Biomedical knowledge graph completion

BioKG is a heterogeneous, multi-relational graph. A fact is represented as a directed triplet

The benchmark contains:

• 93,773 entities in total
• 5,088,434 directed triplets
• 5 entity types:
  • 10,687 diseases
  • 17,499 proteins
  • 10,533 drugs
  • 9,969 side effects
  • 45,085 protein functions
• 51 relation types, including 38 drug-drug interaction types and 8 protein-protein interaction types, plus drug-protein, drug-side-effect, and function-function relations.

Relations between entities of the same type, such as protein-protein interactions, are represented symmetrically by including both directions. Other relations remain directed and their direction matters.

The agent has no internet access, no external data, no pretrained weights. The task runs at two compute budgets, thirty minutes on four CPUs or five hours on sixteen, each with a single GPU.

Knowledge graph completion is the standing formulation for inference over data like this: predict the facts nobody has recorded yet, then check the most plausible ones first. Embedding models that place entities and relations in a shared geometric space are the classical approach. Getting one to rank well takes real tuning, since loss choice, negative sampling, and regularization all move the metric.

The graded queries are private. The working data derives only from the dataset's official training split, and no official validation or test labels ship with the task, so nothing can be looked up.

A separate offline verifier checks the schema first: exact keys, exact dtypes, finite scores. A malformed submission scores zero. Valid submissions are graded by mean reciprocal rank over the private queries

where $\mathrm{rank}_q$ is the position of the true candidate among the 1,001 scores of query $q$, with tied scores resolved to the midpoint of their range. The tie rule sets a concrete floor: scoring every candidate identically ranks the true one 501st on average, an MRR near 0.002. Reported scores are rescaled so that trivial floor maps to zero and a perfect ranking maps to one.

Every run trained a knowledge graph embedding from scratch and tuned it against the validation triples, and for most runs the last validation number predicted the hidden score to within a couple of thousandths. Claude's three runs settled on one recipe, ComplEx with N3 regularization trained with a softmax over type-valid candidates and ensembled across checkpoints; each ended between 0.849 and 0.859 on validation, and the verifier scored each within about 0.001 of that. GPT's runs were the exception: its strongest run, a ComplEx checkpoint ensemble with relation-specific degree calibration, plateaued at 0.825 on validation after about twenty evaluations yet scored 0.842 on the hidden queries, and its two sparsest runs also scored higher hidden than on validation. A second GPT run sat at 0.34 for thirty evaluations before jumping to 0.77 and then to 0.833 at its last evaluation. Kimi's TransE runs stalled between 0.61 and 0.70, and only its ComplEx run passed 0.79.