Source
Shchur et al. (2018), LIT-581. Read as NOTE-318.
The practice
Shchur et al. made this recommendation in LIT-581, from a re-evaluation of four GNNs under one shared protocol. On the Planetoid split GAT wins Cora and Citeseer. On another split of the same sizes GCN wins both, and over 100 splits × 20 seeds GCN has the best average rank (2.3, against 2.7 for MoNet and GraphSAGE-mean and 3.6 for GAT), with the four non-graph baselines at 7.4–8.8. GAT fell below 40% in 138 of 2,000 runs on Amazon Photo, which is the third bullet.
- Many splits, many seeds. The source used 100 random splits with the same label budget, and 20 initializations each. Report the mean and the distribution, not the best run
- One protocol for everyone: the same optimizer, early stopping, initialization, batching and hyperparameter search, under a matched parameter budget
- Look at the distribution, not just the mean. GAT’s average on Amazon was driven by rare collapsed runs, which is a stability result, not an accuracy one
- Include non-graph baselines (MLP, logistic regression, label propagation), so the value of the graph is measured
Conditions
- Transductive node classification with few labels. Benchmarks with official splits (OGB) fix the split for comparability. Multiple seeds and a shared protocol still apply there
- Select hyperparameters per dataset if you can. The source selected on two datasets and reused the result