Source
Jumper et al. (2021), LIT-583. Read as NOTE-321.
The practice
Jumper et al. introduced this recipe in LIT-583. AlphaFold 2 predicted structures for about 350,000 unlabelled sequences, kept the confident ones, and retrained from scratch on those plus the PDB, under cropping and alignment subsampling. Figure 4a shows the self-distilled network improving on one trained without it, but gives the size only as a plot, against three seeds of the baseline.
- Label the unlabeled pool with the trained model, and keep only predictions the model is confident in. This needs a confidence estimate that tracks accuracy (SOTA-353)
- Retrain from scratch, not by fine-tuning, on a mixture of real labels and distilled ones
- Make the student’s input harder than the teacher’s. AlphaFold crops and subsamples the alignment, so reproducing a prediction requires re-deriving it
- Keep the real labels in the mix. Distilled data supplements the labeled set, it does not replace it
Conditions
- The confidence filter is load-bearing. Without calibrated confidence, errors come back as training labels
- One system, one plotted ablation. The gain’s size is not stated in text