Method name prediction from code graphs

The dataset contains 452,741 Python methods extracted from 13,587 GitHub repositories in CodeSearchNet. Each method body is parsed into an abstract syntax tree (AST):

• nodes represent syntax constructs and tokens
• edges represent parent-child relationships in the syntax tree
• the target is the sequence of sub-tokens in the method name

The average graph has 125.2 nodes and 124.2 edges. The name is masked from the input AST using the special token _mask_, including recursive occurrences that could reveal it. This corrects target leakage in the deprecated ogbg-code dataset.

The agent has no internet access, no external data, no pretrained weights. The task runs at two compute budgets, thirty minutes on four CPUs or five hours on sixteen, each with a single GPU.

A good method name is a short semantic summary of what the method does. Predicting it from the body is therefore commonly described as code summarization. It can support developer tools that suggest names during refactoring or flag names that appear inconsistent with an implementation.

It is also a useful representation-learning test. Syntax alone does not determine a name: the model must combine operations, control flow, identifiers, literals, calls, and their structural relationships to infer the developer's intent.

For example, a method that iterates through records, tests a condition, and accumulates matching elements might plausibly be named filter_records. The target is tokenized into a sequence such as:

[filter, records]

Predicting sub-tokens handles naming styles such as filter_records and filterRecords more consistently and avoids treating every full method name as an unrelated class.

The graded names are private. The working data derives only from the dataset's official training split, and no official validation or test labels ship with the task, so nothing can be looked up.

A separate offline verifier checks the schema first: exact keys, exact dtypes, declared lengths in range. A malformed submission scores zero. Valid submissions are graded per function by F1 between the predicted and true subtoken sets, then averaged

where $\hat{T}_i$ and $T_i$ are the predicted and true subtoken sets of function $i$. Order and repeats do not matter, so predicting id, user, get for get_user_id earns full credit. Reported scores are rescaled so a prediction set matching nothing maps to zero and exact naming everywhere maps to one.

The strongest submissions came from runs that scored themselves the way the verifier does, on unique subtoken strings. The top run reached 0.18 on its tuning split within its first few evaluations, spent most of the remaining ones failing to beat that, added two late steps to 0.22, and scored 0.25 on the hidden functions. Runs that measured F1 over vocabulary ids instead, where every out-of-vocabulary name collapses to one unknown token that an unknown prediction matches, read as high as 0.28 on the tuning split and then scored 0.15 to 0.19 on the hidden functions; the gap is the scorer, not overfitting. One run caught this midway, rebuilt its scorer on token strings, watched its best fall from 0.18 to 0.17, and went on to the highest hidden score of its family at 0.19. For every string-scored run the hidden score sits within 0.03 of the run's own best, and always above it.