anthology-of-the-sota

LIT-581: Pitfalls of Graph Neural Network Evaluation

2 min read

Properties6
StatusActive
Tagsgraphs-and-networks, analysis-and-evaluation, model-architecture
Compared againstLIT-582 — Semi-Supervised Classification with Graph Convolutional Networks
Cited as “introduced by” bySOTA-350 — Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split
Cited as “paper” byNOTE-318 — Pitfalls of GNN evaluation
Cited as “source” bySOTA-350 — Compare GNN architectures over many random data splits and seeds, with one shared training and hyperparameter-selection procedure for every model, never on a single fixed split

Graph View

Shchur, Mumme, Bojchevski, Günnemann, TU Munich (R2L workshop, NeurIPS 2018) — ARXIV-1811.05868

Key takeaways

  • Two pitfalls: evaluating every model on the same single split of the same three citation graphs, and comparing a new model’s own training procedure against baselines trained differently
  • The protocol: original architectures, with the same optimizer, early stopping, initialization and full-batch training for all. The same grid search, under a matched parameter budget, is selected on average Cora and Citeseer accuracy. 20 labels per class for training and 30 for validation, 100 random splits × 20 initializations, and 8 datasets (4 of them new)
  • Results (Tables 1 and 2a): GNNs beat MLP, logistic regression and label propagation everywhere. Among GNNs there is no clear winner, and on 5 of 8 datasets the second and third are within 1% of the best. By relative accuracy and rank, GCN comes first (99.4%, 2.3), then MoNet (99.0%, 2.7), GraphSAGE-mean (98.3%) and GAT (95.9%, 3.6)
  • Split sensitivity (Table 2b): on the Planetoid split GAT leads Cora (82.8) and Citeseer (71.0). On another random split of the same sizes, GCN leads both (79.0 and 68.6) and MoNet leads Pubmed

Standing in the anthology

Filed from #163 (“graph representation / gnn”). It is compared_against GCN (LIT-582) and sources SOTA-350.

A caveat on its own protocol. Hyperparameters were chosen on Cora and Citeseer and reused on the other six datasets, which can favor whichever model transfers best. GAT’s average is pulled down by rare collapsed runs, while its median is close to the others’. That makes GAT’s result a stability finding, not an accuracy one.

Read — NOTE-318.

Backlinks