The evidence base — one note per paper, with the takeaways that made it worth keeping and, where it applies, the reason it is no longer.
A note’s status is about the paper’s standing in this anthology, not about whether the paper is correct. That distinction is the point of keeping notes and practices apart: a foundational paper can carry advice that is now wrong, and a paper retired as uninteresting can still be the source of something everybody does. Under the old schema both facts lived in one field on the paper, so neither could be said.
Rejected is the attic. It is a real place in the record with real entries,
and a practice still citing one of them is a finding — luria lint reports it
rather than leaving it to be noticed.
Roughly a third of these notes produced no practice at all. They are here because they are worth reading.
For about two dozen of them — the diffusion and radiance-field corpus, tagged
generative-modeling and vision-and-graphics — the reason used to be a
scope decision: the recommendations were scoped to language-model training by
construction, so such a paper could be read here and not recommended here.
ADR-020 ended that, and ADR-026 — which supersedes it — went further.
The recommendations are scoped by the kind of claim, where the work was
discovered is not the test, and the practice registry now uses the same
thirteen topics this reading list does. So those notes are unread rather than
ineligible, and generative-modeling and vision-and-graphics are no longer
words that exist on only one side of the record.
Several of them carry an instruction one of the thirteen topics holds perfectly well — a preconditioning scheme, a noise-level sampling distribution, a model-in-the-loop annotation bootstrap. Some carry a claim about generative modelling as such, and those now have a word too.
The rest never had even that excuse, and the tag will mislead you if you let
it. adaptation-and-tuning and inference-optimization are also absent
from the practice vocabulary, but they are not ground the recommendations
decline: the record files fine-tuning, preference-training and quantization
practices under training-optimization and systems-optimization, and has
several of each. ADR-003 justified all five extra topics with one sentence,
and that sentence is only true of three of them. A note tagged with one of the
other two needs reading, not classifying — which is how LoRA sat here for a
year sourcing nothing.
By topic
Training optimization (193) — optimizers, learning-rate schedules, batch size, training dynamics, scaling laws and scaling strategies: 001 · 006 · 007 · 009 · 010 · 011 · 012 · 013 · 017 · 018 · 028 · 040 · 042 · 044 · 052 · 058 · 061 · 065 · 068 · 069 · 090 · 099 · 101 · 104 · 105 · 119 · 121 · 122 · 127 · 132 · 139 · 144 · 145 · 146 · 147 · 148 · 149 · 150 · 153 · 155 · 156 · 157 · 158 · 159 · 163 · 186 · 211 · 213 · 221 · 229 · 230 · 231 · 232 · 233 · 234 · 235 · 236 · 237 · 238 · 239 · 240 · 241 · 242 · 245 · 265 · 305 · 360 · 399 · 436 · 437 · 438 · 440 · 441 · 443 · 444 · 445 · 450 · 451 · 452 · 453 · 454 · 455 · 456 · 457 · 459 · 461 · 462 · 470 · 471 · 472 · 473 · 474 · 475 · 476 · 477 · 478 · 483 · 486 · 495 · 501 · 503 · 508 · 514 · 515 · 516 · 519 · 520 · 521 · 522 · 523 · 535 · 537 · 538 · 539 · 540 · 541 · 542 · 544 · 546 · 548 · 556 · 557 · 558 · 562 · 590 · 603 · 605 · 609 · 610 · 616 · 617 · 620 · 626 · 628 · 629 · 634 · 640 · 642 · 646 · 647 · 649 · 651 · 654 · 656 · 657 · 663 · 668 · 671 · 672 · 673 · 675 · 677 · 678 · 680 · 681 · 682 · 683 · 684 · 685 · 686 · 688 · 689 · 690 · 692 · 693 · 700 · 701 · 702 · 706 · 708 · 709 · 710 · 714 · 720 · 721 · 738 · 744 · 746 · 749 · 753 · 758 · 761 · 764 · 765 · 767 · 768 · 770 · 776 · 789 · 791 · 795 · 798 · 799
Systems optimization (26) — hardware utilization, kernels, compilation, memory access patterns — how an operation is executed, not how many bits it is executed in: 004 · 059 · 063 · 066 · 074 · 116 · 139 · 197 · 367 · 378 · 380 · 418 · 512 · 515 · 552 · 567 · 601 · 655 · 706 · 735 · 736 · 737 · 739 · 746 · 750 · 754
Numerics and precision (24) — how many bits, where, and what that costs — number formats, training precision and the failures it causes, post-training quantization, and the interaction between them: 011 · 081 · 102 · 139 · 149 · 186 · 197 · 198 · 200 · 221 · 378 · 380 · 414 · 473 · 512 · 567 · 585 · 586 · 656 · 676 · 750 · 783 · 790 · 800
Model stability (88) — initialization, normalization, gradient handling, regularization, loss-landscape behaviour: 002 · 005 · 014 · 015 · 022 · 023 · 025 · 029 · 032 · 037 · 043 · 047 · 054 · 084 · 088 · 114 · 119 · 139 · 154 · 155 · 198 · 200 · 223 · 251 · 333 · 393 · 394 · 395 · 396 · 472 · 475 · 476 · 478 · 519 · 520 · 521 · 522 · 523 · 525 · 530 · 537 · 538 · 539 · 540 · 541 · 542 · 544 · 548 · 551 · 556 · 557 · 558 · 590 · 591 · 593 · 594 · 595 · 596 · 598 · 599 · 639 · 640 · 651 · 664 · 668 · 673 · 674 · 675 · 680 · 681 · 682 · 684 · 686 · 700 · 704 · 705 · 707 · 708 · 711 · 714 · 733 · 749 · 758 · 764 · 765 · 767 · 786 · 790
Distributed optimization (67) — parallelism and sharding, communication, memory management, checkpointing: 004 · 007 · 016 · 022 · 026 · 027 · 043 · 051 · 056 · 059 · 065 · 083 · 098 · 102 · 158 · 187 · 206 · 212 · 214 · 219 · 220 · 222 · 246 · 252 · 254 · 257 · 273 · 276 · 278 · 279 · 280 · 285 · 288 · 292 · 302 · 303 · 307 · 314 · 315 · 316 · 323 · 325 · 336 · 337 · 344 · 345 · 350 · 354 · 361 · 362 · 366 · 370 · 372 · 373 · 374 · 418 · 605 · 619 · 624 · 735 · 736 · 737 · 739 · 750 · 754 · 762 · 783
Data pipeline (60) — loading, quality assessment and selection, preprocessing, batch preparation: 050 · 053 · 080 · 082 · 087 · 096 · 099 · 117 · 120 · 124 · 125 · 126 · 127 · 166 · 175 · 178 · 184 · 201 · 202 · 203 · 205 · 391 · 392 · 397 · 398 · 399 · 400 · 401 · 403 · 425 · 426 · 427 · 432 · 441 · 442 · 450 · 451 · 484 · 485 · 565 · 583 · 591 · 598 · 599 · 619 · 625 · 648 · 652 · 667 · 671 · 681 · 685 · 691 · 739 · 743 · 744 · 745 · 755 · 757 · 782
Attention techniques (70) — attention variants and alternative mechanisms, implementation optimizations, context length: 008 · 020 · 024 · 033 · 074 · 100 · 106 · 112 · 115 · 133 · 137 · 138 · 139 · 142 · 143 · 161 · 162 · 165 · 174 · 176 · 177 · 190 · 191 · 192 · 193 · 194 · 195 · 204 · 207 · 208 · 209 · 210 · 225 · 387 · 413 · 414 · 428 · 429 · 430 · 431 · 521 · 523 · 527 · 528 · 529 · 530 · 533 · 638 · 640 · 641 · 653 · 655 · 661 · 666 · 708 · 709 · 711 · 723 · 726 · 738 · 747 · 748 · 760 · 763 · 783 · 790 · 799 · 800 · 801 · 802
Model architecture (144) — architecture patterns, component design, structural choices, model families — how the network is shaped, not how many signals it takes in: 008 · 020 · 021 · 023 · 024 · 030 · 035 · 045 · 047 · 048 · 052 · 060 · 061 · 069 · 071 · 080 · 120 · 131 · 134 · 135 · 136 · 137 · 139 · 140 · 141 · 151 · 152 · 160 · 161 · 162 · 165 · 170 · 171 · 173 · 179 · 181 · 182 · 183 · 187 · 188 · 189 · 196 · 199 · 200 · 226 · 227 · 380 · 384 · 406 · 417 · 425 · 428 · 429 · 430 · 431 · 448 · 449 · 464 · 472 · 477 · 479 · 483 · 486 · 493 · 495 · 496 · 497 · 500 · 513 · 525 · 527 · 528 · 530 · 531 · 532 · 534 · 543 · 545 · 546 · 551 · 559 · 560 · 561 · 575 · 579 · 580 · 581 · 582 · 583 · 584 · 587 · 622 · 626 · 639 · 640 · 641 · 651 · 653 · 654 · 657 · 658 · 660 · 662 · 665 · 666 · 667 · 668 · 669 · 670 · 679 · 681 · 683 · 698 · 699 · 708 · 709 · 711 · 714 · 715 · 720 · 723 · 727 · 728 · 729 · 733 · 743 · 747 · 748 · 754 · 758 · 759 · 760 · 763 · 767 · 769 · 771 · 772 · 781 · 785 · 792 · 794 · 799 · 801 · 802
Inference optimization (61) — serving-time decisions — batching, cache layout, compression, sparsity, distillation, sampling algorithms: 024 · 081 · 103 · 110 · 112 · 139 · 185 · 224 · 375 · 376 · 428 · 468 · 479 · 486 · 491 · 498 · 511 · 512 · 516 · 517 · 518 · 529 · 545 · 546 · 547 · 552 · 567 · 585 · 586 · 618 · 629 · 631 · 636 · 637 · 643 · 645 · 646 · 650 · 655 · 676 · 678 · 680 · 683 · 687 · 689 · 701 · 702 · 722 · 748 · 750 · 770 · 775 · 783 · 784 · 787 · 789 · 790 · 791 · 792 · 796 · 800
Adaptation and tuning (88) — taking a trained model somewhere new — fine-tuning and transfer, preference training and safety alignment, parameter-efficient adaptation, context extension: 034 · 035 · 046 · 049 · 055 · 072 · 078 · 079 · 082 · 089 · 091 · 104 · 118 · 119 · 127 · 128 · 129 · 130 · 164 · 167 · 168 · 169 · 172 · 180 · 211 · 230 · 231 · 233 · 234 · 235 · 236 · 237 · 238 · 239 · 240 · 377 · 378 · 379 · 381 · 382 · 383 · 425 · 432 · 433 · 434 · 467 · 473 · 474 · 488 · 491 · 505 · 506 · 516 · 536 · 549 · 574 · 576 · 577 · 578 · 621 · 624 · 633 · 647 · 650 · 670 · 674 · 675 · 687 · 691 · 702 · 715 · 734 · 744 · 755 · 757 · 762 · 769 · 773 · 774 · 776 · 778 · 779 · 782 · 788 · 795 · 797 · 799 · 801
In-context learning (25) — getting behaviour out of a fixed model by what you put in the context — few-shot exemplars, chain of thought, prompting strategy, and what the context can and cannot buy: 035 · 049 · 129 · 379 · 420 · 464 · 467 · 468 · 469 · 470 · 489 · 493 · 502 · 510 · 532 · 533 · 534 · 535 · 536 · 543 · 577 · 733 · 740 · 741 · 757
Representation and encoding (97) — how the signal is encoded before the expensive network sees it — tokenizers and learned latents, positional encoding, and the frequency or basis choices that go with them: 003 · 045 · 048 · 133 · 192 · 193 · 207 · 208 · 209 · 210 · 216 · 341 · 363 · 409 · 410 · 412 · 435 · 458 · 459 · 460 · 485 · 488 · 489 · 494 · 496 · 497 · 498 · 499 · 500 · 505 · 506 · 507 · 508 · 509 · 511 · 526 · 531 · 545 · 547 · 550 · 572 · 587 · 589 · 590 · 591 · 592 · 593 · 594 · 595 · 596 · 597 · 598 · 599 · 600 · 601 · 602 · 603 · 604 · 607 · 608 · 609 · 610 · 612 · 613 · 618 · 619 · 620 · 622 · 623 · 624 · 626 · 634 · 638 · 652 · 657 · 662 · 663 · 664 · 665 · 668 · 670 · 671 · 672 · 679 · 711 · 723 · 724 · 725 · 729 · 730 · 744 · 745 · 747 · 751 · 755 · 757 · 785
Analysis and evaluation (238) — how to find out whether something worked — what to measure, what a measurement cannot tell you, and which comparisons are unsound; theory, interpretability and debugging belong here too: 014 · 019 · 031 · 039 · 041 · 052 · 057 · 060 · 072 · 077 · 085 · 092 · 095 · 123 · 167 · 190 · 228 · 230 · 233 · 235 · 236 · 248 · 261 · 262 · 267 · 271 · 275 · 282 · 285 · 286 · 293 · 296 · 297 · 298 · 301 · 304 · 310 · 311 · 312 · 324 · 326 · 328 · 340 · 346 · 353 · 360 · 364 · 365 · 379 · 382 · 388 · 389 · 390 · 398 · 400 · 401 · 402 · 403 · 404 · 405 · 407 · 408 · 411 · 415 · 416 · 419 · 420 · 421 · 422 · 423 · 424 · 426 · 427 · 433 · 440 · 452 · 455 · 458 · 463 · 464 · 465 · 466 · 470 · 471 · 474 · 476 · 480 · 481 · 482 · 484 · 485 · 486 · 487 · 489 · 490 · 492 · 493 · 495 · 501 · 502 · 503 · 504 · 506 · 507 · 508 · 509 · 510 · 514 · 517 · 518 · 519 · 520 · 524 · 525 · 526 · 528 · 531 · 532 · 533 · 534 · 535 · 536 · 537 · 538 · 539 · 540 · 541 · 542 · 543 · 544 · 548 · 549 · 550 · 553 · 555 · 556 · 557 · 558 · 563 · 565 · 568 · 569 · 570 · 571 · 572 · 574 · 575 · 576 · 577 · 578 · 579 · 580 · 581 · 586 · 588 · 589 · 597 · 600 · 604 · 606 · 607 · 608 · 611 · 614 · 615 · 616 · 617 · 628 · 638 · 639 · 641 · 644 · 645 · 648 · 650 · 653 · 665 · 667 · 669 · 670 · 671 · 672 · 673 · 674 · 675 · 676 · 681 · 684 · 685 · 688 · 690 · 692 · 693 · 694 · 699 · 700 · 703 · 704 · 705 · 708 · 711 · 712 · 713 · 714 · 718 · 719 · 721 · 722 · 724 · 725 · 726 · 727 · 728 · 729 · 730 · 731 · 732 · 734 · 740 · 746 · 752 · 753 · 759 · 761 · 764 · 767 · 768 · 769 · 773 · 782 · 786 · 788 · 793 · 795 · 797 · 799 · 800 · 801
Generative modeling (129) — diffusion, samplers, text-to-image, conditioning and control: 036 · 038 · 062 · 067 · 070 · 073 · 075 · 076 · 093 · 097 · 217 · 381 · 439 · 442 · 446 · 447 · 448 · 449 · 477 · 479 · 483 · 488 · 490 · 491 · 494 · 496 · 497 · 498 · 499 · 500 · 501 · 553 · 554 · 555 · 559 · 560 · 561 · 562 · 563 · 564 · 565 · 566 · 572 · 573 · 584 · 592 · 611 · 614 · 615 · 618 · 619 · 620 · 621 · 622 · 623 · 624 · 625 · 626 · 627 · 629 · 630 · 631 · 632 · 633 · 634 · 635 · 636 · 637 · 642 · 643 · 644 · 645 · 646 · 647 · 648 · 649 · 652 · 658 · 659 · 660 · 661 · 676 · 677 · 678 · 687 · 689 · 691 · 692 · 693 · 699 · 701 · 702 · 703 · 710 · 714 · 720 · 721 · 722 · 743 · 751 · 755 · 770 · 771 · 772 · 773 · 774 · 775 · 776 · 777 · 778 · 779 · 780 · 781 · 782 · 783 · 784 · 785 · 786 · 787 · 788 · 789 · 790 · 791 · 792 · 793 · 794 · 796 · 797 · 798
Vision and graphics (102) — neural rendering, reconstruction, perception, visual foundation models: 064 · 072 · 086 · 094 · 096 · 107 · 108 · 109 · 111 · 113 · 215 · 216 · 218 · 384 · 385 · 386 · 387 · 421 · 422 · 423 · 424 · 435 · 475 · 488 · 494 · 511 · 550 · 551 · 554 · 559 · 560 · 561 · 562 · 564 · 566 · 573 · 587 · 588 · 590 · 591 · 593 · 594 · 595 · 596 · 598 · 599 · 601 · 605 · 611 · 614 · 615 · 618 · 619 · 620 · 621 · 622 · 623 · 624 · 625 · 626 · 627 · 628 · 629 · 631 · 632 · 633 · 634 · 635 · 637 · 643 · 646 · 652 · 657 · 658 · 661 · 662 · 663 · 664 · 687 · 691 · 703 · 706 · 712 · 713 · 715 · 723 · 727 · 728 · 730 · 731 · 732 · 743 · 745 · 747 · 751 · 755 · 770 · 775 · 783 · 784 · 787 · 796
Multimodal learning (23) — what changes when one model has to take in more than one kind of signal — joint architectures and fusion, contrastive image-text training, cross-modal transfer, and what a second modality buys the first: 049 · 071 · 072 · 080 · 135 · 449 · 458 · 483 · 510 · 588 · 605 · 619 · 620 · 632 · 648 · 674 · 706 · 732 · 745 · 747 · 770 · 783 · 796
Deployment and society (14) — what a system does once it is running among people, and how anyone outside it could find out — audits of deployed platforms and recommenders, information ecosystems, moderation and governance, and the access problems and funding pressures of studying any of it from the outside: 480 · 481 · 482 · 487 · 492 · 503 · 524 · 697 · 716 · 717 · 718 · 740 · 741 · 752
Biomolecular modeling (7) — models whose data is molecules — protein and nucleic-acid structure prediction, biological sequence language models, docking and molecular property prediction — what they get right, what their benchmarks can show, and which of their methods travel to other domains: 505 · 506 · 583 · 584 · 679 · 702 · 754
Signal structure (26) — what the data itself is like, independent of any model — the statistics and structure of language, images and other signals that methods exploit or must respect, such as frequency spectra, heavy-tailed and Zipfian distributions, and what counts as a unit: 062 · 205 · 328 · 409 · 410 · 450 · 457 · 484 · 555 · 592 · 601 · 602 · 603 · 607 · 609 · 610 · 612 · 613 · 654 · 659 · 660 · 672 · 689 · 695 · 751 · 767
Concept geometry (14) — how concepts are laid out in a model’s representation space — linear directions and the regions, lattices and spectra built from them, superposition and the dictionaries that pull it apart, reading concepts out with probes and writing them in with steering vectors, and whether different models arrive at the same geometry: 458 · 460 · 526 · 544 · 568 · 571 · 602 · 603 · 604 · 606 · 613 · 696 · 698 · 759
Agents and environments (6) — systems that perceive, act and are changed by the consequences — percept–action loops, world models and the environments agents are trained in, multi-agent and collective behaviour, and what acting rather than only predicting changes about learning, memory and their costs: 502 · 697 · 742 · 756 · 759 · 787
Graphs and networks (6) — graph-structured data and the models that run on it — message passing and graph neural networks, how they are evaluated, graph curvature, rewiring and over-squashing, community structure and partitioning: 386 · 579 · 580 · 581 · 582 · 766
Physical sciences (2) — models whose data is the physical world — learned surrogates for simulation in fluids, PDEs, molecular dynamics and cosmology, foundation models for weather, climate and solar observation — what they are trusted to replace, what their benchmarks can show, and which of their methods travel to other domains: 715 · 763
Human–AI interaction (3) — how people work with a model through an interface — prompting and steering controls, editing and co-writing tools, assistive and co-adaptive interfaces, how such interfaces are scored, and what lab studies of them can and cannot show: 503 · 717 · 741
Tiny models (8) — claims that hold at the small end and not in general — sub-billion-parameter training, where the usual scaling advice inverts: 119 · 123 · 129 · 427 · 484 · 485 · 681 · 719
Capability thresholds (13) — a capability that arrives abruptly rather than smoothly — emergence at scale, grokking after long training, phase transitions in learning — which axis it turns on, and whether the discontinuity is real or an artefact of how it was measured: 085 · 470 · 471 · 537 · 538 · 539 · 540 · 543 · 544 · 617 · 656 · 667 · 769
Flows and transport (30) — generative models as learned transport between distributions — flow matching, rectified flow, stochastic interpolants, normalizing flows and neural ODEs, and the paths, couplings and time schedules they are trained on; Gaussian diffusion is the special case (THEORY-106). A subtopic, worn beside generative-modeling:
447 · 449 · 573 · 619 · 626 · 630 · 636 · 644 · 645 · 678 · 722 · 771 · 772 · 773 · 775 · 777 · 779 · 780 · 781 · 784 · 785 · 787 · 789 · 790 · 791 · 792 · 793 · 794 · 796 · 798
Few-step generation (17) — getting a sample in one or a few network evaluations — progressive, distribution-matching and adversarial distillation, consistency, flow-map, shortcut and mean-flow models, and fast ODE solvers. A subtopic, worn beside generative-modeling or inference-optimization:
067 · 076 · 093 · 629 · 631 · 643 · 646 · 770 · 775 · 783 · 784 · 786 · 787 · 790 · 791 · 793 · 796
By status: In the anthology (710) · Watching (1) · Unreviewed (10) · Superseded (6) · Attic (10)
Chronological
What the status column means in this scheme — the words are luria’s, the meanings are this project’s.
| Status | Means | |
|---|---|---|
Active | In the anthology | Worth reading, whether or not a practice was drawn from it |
Proposed | Watching | Promising and unproven; the note says what would settle it |
Deferred | Unreviewed | In the corpus, not yet read closely enough to place |
Superseded | Superseded | Later work covers this better, and the note names it |
Rejected | Attic | Retired with a reason — too narrow, too theoretical, or simply not interesting enough to carry |
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-001 | Adam: A Method for Stochastic Optimization | Kingma et al. (2014), ARXIV-1412.6980. Introduces adaptive moment estimation for gradient-based optimization. | Active |
| LIT-002 | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift | Ioffe et al. (2015), ARXIV-1502.03167. Introduces normalization of layer inputs during training. | Active |
| LIT-003 v2 | Neural Machine Translation of Rare Words with Subword Units | Sennrich et al. (2015), ARXIV-1508.07909. Introduces byte-pair encoding for tokenization. | Active |
| LIT-004 v4 | Training Deep Nets with Sublinear Memory Cost | Chen et al. (2016), ARXIV-1604.06174. Gradient checkpointing: store activations at O(sqrt(n)) layers and recompute the segments between them in the backward pass, for the cost of one extra forward pass per minibatch. A 1,000-layer ResNet drops from 48G to 7G for 30% more running time. | Active |
| LIT-005 | Layer Normalization | Ba et al. (2016), ARXIV-1607.06450. Introduces normalization across features instead of batch. | Active |
| LIT-006 v2 | SGDR: Stochastic Gradient Descent with Warm Restarts | Loshchilov and Hutter (2016), ARXIV-1608.03983. A duplicate: the migration filed this paper twice, and the other copy carries the reading and the supersession. LIT-042 is the surviving note. | Rejected — by LIT-042; a duplicate of LIT-042, which holds the same paper and the record’s judgement on it |
| LIT-007 | Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour | Goyal et al. (2017), ARXIV-1706.02677. Linear scaling rule for learning rates with batch size. | Active |
| LIT-008 v2 | Attention Is All You Need | Vaswani et al. (2017), ARXIV-1706.03762. Introduces transformer architecture. | Active |
| LIT-009 v2 | Large Batch Training of Convolutional Networks | You et al. (2017), ARXIV-1708.03888. Layer-wise adaptive rate scaling (LARS). | Active |
| LIT-010 | Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates | Smith et al. (2017), ARXIV-1708.07120. One-cycle learning rate policy (1cp). | Active |
| LIT-011 | Mixed Precision Training | Micikevicius et al. (2017), ARXIV-1710.03740. Enables FP16 training with FP32 master weights. | Active |
| LIT-012 v2 | Decoupled Weight Decay Regularization | Loshchilov et al. (2017), ARXIV-1711.05101. Fixes weight decay implementation in Adam. | Active |
| LIT-013 v2 | Population Based Training of Neural Networks | Jaderberg et al. (2017), ARXIV-1711.09846. Automated hyperparameter optimization. | Active |
| LIT-014 v2 | Visualizing the Loss Landscape of Neural Nets | Li et al. (2017), ARXIV-1712.09913. Visualizes loss landscape geometry. | Active |
| LIT-015 v2 | Understanding Batch Normalization | Bjorck et al. (2018), ARXIV-1806.02375. Argues that batch normalization’s main benefit is that it permits substantially larger learning rates, and studies what goes wrong without it at initialisation. | Active |
| LIT-016 v2 | GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism | Huang et al. (2018), ARXIV-1811.06965. Introduces pipeline parallelism for training. | Active |
| LIT-017 v2 | An Empirical Model of Large-Batch Training | McCandlish et al. (2018), ARXIV-1812.06162. Introduces gradient noise scale. | Active |
| LIT-018 v2 | On the Convergence of Adam and Beyond | Reddi et al. (2018), ARXIV-1904.09237. Identifies convergence issues in Adam. | Active |
| LIT-019 v2 | The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks | Frankle and Carbin (2018), ARXIV-1803.03635. A dense network contains a sparse subnetwork that trains to the same accuracy in the same number of steps — but only from its original initialization, which is the part that makes it a claim about initialization rather than about architecture. | Active |
| LIT-020 v4 | Pay Less Attention with Lightweight and Dynamic Convolutions | Wu et al. (2019), ARXIV-1901.10430. Alternative to self-attention. | Active |
| LIT-021 v3 | Mish: A Self Regularized Non-Monotonic Activation Function | Misra et al. (2019), ARXIV-1908.08681. New activation function. | Active |
| LIT-022 v2 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Shoeybi et al. (2019), ARXIV-1909.08053. Model parallel transformers. | Active |
| LIT-023 v3 | Root Mean Square Layer Normalization | Zhang et al. (2019), ARXIV-1910.07467. Simplified layer normalization. | Active |
| LIT-024 v2 | Fast Transformer Decoding: One Write-Head is All You Need | Shazeer (2019), ARXIV-1911.02150. Introduces multi-query attention. | Active |
| LIT-025 v2 | Understanding and Improving Layer Normalization | Xu et al. (2019), ARXIV-1911.07013. LayerNorm’s benefit comes from the derivatives of the mean and variance, not from forward normalization — and its bias and gain increase overfitting risk and do not work in most cases. | Active |
| LIT-026 v3 | Advances and Open Problems in Federated Learning | Kairouz et al. (2019), ARXIV-1912.04977. Comprehensive survey of federated learning. | Active |
| LIT-027 | ZeRO: Memory Optimizations Toward Training Trillion Parameter Models | Rajbhandari et al. (2020), ARXIV-1910.02054. Micro-batch size determined by GPU memory. | Active |
| LIT-028 v5 | Scaling Laws for Neural Language Models | Kaplan et al. (2020), ARXIV-2001.08361. Power-law scaling relationships. | Active |
| LIT-029 v2 | On Layer Normalization in the Transformer Architecture | Xiong et al. (2020), ARXIV-2002.04745. A duplicate: the migration filed this paper twice, and this copy’s one-line reading attributed RMSNorm to it. LIT-114 is the surviving note. | Rejected — by LIT-114; a duplicate of LIT-114, which holds the same paper and reads it correctly |
| LIT-030 v2 | GLU Variants Improve Transformer | Shazeer et al. (2020), ARXIV-2002.05202. Study of gating mechanisms. | Active |
| LIT-031 v2 | Pruned Neural Networks are Surprisingly Modular | Filan et al. (2020), ARXIV-2003.04881. Analysis of neural network modularity. | Superseded — by LIT-041, ARXIV-2110.08058; by later work on clusterability and mechanistic interpretability |
| LIT-032 v2 | PowerNorm: Rethinking Batch Normalization in Transformers | Shen et al. (2020), ARXIV-2003.07845. Alternative to LayerNorm. | Active |
| LIT-033 v4 | Longformer: The Long-Document Transformer | Beltagy et al. (2020), ARXIV-2004.05150. Efficient attention for long sequences. | Active |
| LIT-034 v2 | Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks | Gururangan et al. (2020), ARXIV-2004.10964. Benefits of domain-adaptive pretraining. | Active |
| LIT-035 v2 | Language Models are Few-Shot Learners | Brown et al. (2020), ARXIV-2005.14165. Introduces GPT-3. | Active |
| LIT-036 v2 | Denoising Diffusion Probabilistic Models | Ho et al. (2020), ARXIV-2006.11239. Introduces DDPM framework. | Active |
| LIT-037 v2 | On the difficulty of training Recurrent Neural Networks | Pascanu et al. (2012), ARXIV-1211.5063. The origin of gradient-norm clipping: exploding gradients are a geometric fact about error surfaces with cliffs, and rescaling the gradient when its norm exceeds a threshold is the cheap, correct response. | Active |
| LIT-038 v3 | Denoising Diffusion Implicit Models | Song et al. (2020), ARXIV-2010.02502. Continuous-time formulation of diffusion models. | Active |
| LIT-039 v3 | Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win | Evci et al. (2020), ARXIV-2010.03533. Sparse networks trained from scratch underperform because they have poor gradient flow, and a lottery ticket wins by re-learning the pruned solution it came from rather than by having found a better sparse architecture. | Active |
| LIT-040 v2 | Scaling Laws for Autoregressive Generative Modeling | Henighan et al. (2020), ARXIV-2010.14701. Universal scaling behaviors. | Active |
| LIT-041 | Clusterability in Neural Networks | Filan et al. (2021), ARXIV-2103.03386. Larger models develop more modular features. | Rejected — the result does not look interesting enough to carry |
| LIT-042 v2 | SGDR: Stochastic Gradient Descent with Warm Restarts | Loshchilov and Hutter (2016), ARXIV-1608.03983. Warm restarts: anneal the learning rate on a cosine to a floor, then jump it back up and repeat, with each cycle longer than the last. | Superseded — by LIT-035; single-cycle cosine decay proved better at scale |
| LIT-043 | Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM | Narayanan et al. (2021), ARXIV-2104.04473. 3D parallel training strategy. | Active |
| LIT-044 v2 | Contrastive Learning with Hard Negative Samples | Robinson et al. (2020), ARXIV-2010.04592. Importance of negative sample selection. | Rejected — needs per-case tuning, no consistent cross-domain benefit |
| LIT-045 v2 | RoFormer: Enhanced Transformer with Rotary Position Embedding | Su et al. (2021), ARXIV-2104.09864. Rotation-based positional encoding. | Active |
| LIT-046 v2 | LoRA: Low-Rank Adaptation of Large Language Models | Hu et al. (2021), ARXIV-2106.09685. Freeze the pretrained weights and learn a rank-r product beside each one: the update a fine-tune applies has low intrinsic rank, so representing it in full is wasted memory. | Active |
| LIT-047 v3 | ReZero is All You Need: Fast Convergence at Large Depth | Bachlechner et al. (2020), ARXIV-2003.04887. Gate every residual branch with a single zero-initialized scalar: the network starts as the identity, satisfies dynamical isometry at initialization, and trains at depths where more elaborate schemes are needed otherwise. | Active |
| LIT-048 v4 | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | Press et al. (2021), ARXIV-2108.12409. Linear attention bias. | Active |
| LIT-049 v3 | Learning to Prompt for Vision-Language Models | Zhou et al. (2021), ARXIV-2109.01134. Learnable prompt optimization. | Active |
| LIT-050 v2 | Analyzing and Mitigating Data Stalls in DNN Training | Mohan et al. (2020), ARXIV-2007.06775. Training time is frequently dominated by data stalls — waiting on fetch and preprocessing — and the only way to know is to measure the input pipeline separately from the step. | Active |
| LIT-051 | A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters | Jiang et al. (2020), OSDI ‘20. BytePS: spare CPU and network capacity in a GPU cluster is a resource, and a communication framework that uses it subsumes both all-reduce and parameter-server as special cases. | Active |
| LIT-052 v2 | Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers | Tay et al. (2021), ARXIV-2109.10686. Model shape, not just model size, determines downstream fine-tuning quality; the DeepNarrow strategy reaches T5-Base quality with 50% fewer parameters and 40% faster training. | Active |
| LIT-053 v2 | High Performance I/O For Large Scale Deep Learning | Aizman et al. (2020), ARXIV-2001.01858. Existing distributed filesystems were not built for the access pattern deep learning has; WebDataset stores training data as sharded POSIX tar archives so the pattern becomes sequential reads of large files. | Active |
| LIT-054 v2 | GLM-130B: An Open Bilingual Pre-trained Model | Zeng et al. (2022), ARXIV-2210.02414. A 130B training run reported with its failures intact: the paper’s stated purpose is to unveil how a model at this scale can be successfully pre-trained, and most of what it has to say is about loss spikes and divergence. | Active |
| LIT-055 v3 | P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks | Liu et al. (2021), ARXIV-2110.07602. Prefix tuning applied at every layer rather than only at the input, which is what makes prompt tuning match full fine-tuning at ordinary model sizes instead of only at 10B+. | Active |
| LIT-056 v2 | Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training | Lin et al. (2017), ARXIV-1712.01887. 99.9% of the gradient exchange in distributed SGD is redundant; sending the rest, with four corrections that keep accuracy, compresses gradients 270–600× and puts multi-node training on commodity 1Gbps Ethernet. | Active |
| LIT-057 v2 | What Makes for Good Views for Contrastive Learning? | Tian et al. (2020), ARXIV-2005.10243. Analysis of view generation. | Rejected — theoretical, with no concrete training recommendation |
| LIT-058 v3 | Measuring the Effects of Data Parallelism on Neural Network Training | Shallue et al. (2018), ARXIV-1811.03600. An exhaustive empirical study of batch size: across models, datasets and optimizers, increasing the batch shortens training in a regime of perfect scaling, then hits diminishing returns and finally a wall, and where those transitions fall is workload-dependent. | Active |
| LIT-059 v2 | CheckFreq: Frequent, Fine-Grained DNN Checkpointing | Mohan et al. (2021), FAST ‘21. Checkpoint frequency should be computed, not guessed: online profiling picks an iteration-granularity interval, adaptive tuning holds the overhead under a bound, and two-phase checkpointing pipelines the write against compute. | Active |
| LIT-060 v2 | Improving Language Models by Retrieving from Trillions of Tokens | Borgeaud et al. (2021), ARXIV-2112.04426. Retrieval-augmented language models. | Active |
| LIT-061 v3 | Efficient Large Scale Language Modeling with Mixtures of Experts | Artetxe et al. (2021), ARXIV-2112.10684. “A detailed empirical study of how autoregressive MoE language models scale in comparison with dense models” — across in- and out-of-domain language modelling, zero- and few-shot priming, and full fine-tuning. | Active |
| LIT-062 v3 | High-Resolution Image Synthesis with Latent Diffusion Models | Rombach et al. (2022), ARXIV-2112.10752. Latent space diffusion. | Active |
| LIT-063 v2 | TVM: An Automated End-to-End Optimizing Compiler for Deep Learning | Chen et al. (2018), ARXIV-1802.04799. Graph-level and operator-level optimization in one compiler, with a learned cost model searching the low-level schedule space, so performance is portable across back-ends instead of living in vendor libraries. | Active |
| LIT-064 v3 | Instant Neural Graphics Primitives with a Multiresolution Hash Encoding | Müller et al. (2022), ARXIV-2201.05989. Real-time neural rendering. | Active |
| LIT-065 v2 | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model | Smith et al. (2022), ARXIV-2201.11990. Large scale training techniques. | Active |
| LIT-066 v2 | Data Movement Is All You Need: A Case Study on Optimizing Transformers | Ivanov et al. (2020), ARXIV-2007.00072. Transformer training is bound by data movement, not compute: existing frameworks use suboptimal layouts, and a systematic movement-first optimization cuts data movement 22.91% for a 1.30x speedup on a BERT encoder layer. | Active |
| LIT-067 v3 | Progressive Distillation for Fast Sampling of Diffusion Models | Salimans et al. (2022), ARXIV-2202.00512. Faster sampling through distillation. | Active |
| LIT-068 | Training Compute-Optimal Large Language Models | Hoffmann et al. (2022), ARXIV-2203.15556. Compute-optimal scaling laws. | Active |
| LIT-069 | PaLM: Scaling Language Modeling with Pathways | Chowdhery et al. (2022), ARXIV-2204.02311. Pathways system architecture. | Active |
| LIT-070 v3 | Hierarchical Text-Conditional Image Generation with CLIP Latents | Ramesh et al. (2022), ARXIV-2204.06125. Prior-guided image generation. | Active |
| LIT-071 v3 | CoCa: Contrastive Captioners are Image-Text Foundation Models | Yu et al. (2022), ARXIV-2205.01917. Unified vision-language architecture. | Active |
| LIT-072 v2 | Simple Open-Vocabulary Object Detection with Vision Transformers | Minderer et al. (2022), ARXIV-2205.06230. Open vocabulary detection. | Active |
| LIT-073 v2 | Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding | Saharia et al. (2022), ARXIV-2205.11487. Cascaded diffusion approach. | Active |
| LIT-074 v3 | FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Dao et al. (2022), ARXIV-2205.14135. IO-aware attention implementation. | Active |
| LIT-075 v2 | Elucidating the Design Space of Diffusion-Based Generative Models | Karras et al. (2022), ARXIV-2206.00364. Comprehensive design space analysis. | Active |
| LIT-076 v3 | DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps | Lu et al. (2022), ARXIV-2206.00927. Fast sampling for diffusion models. | Active |
| LIT-077 v2 | Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models | Srivastava et al. (2022), ARXIV-2206.04615. Comprehensive evaluation framework. | Active |
| LIT-078 v3 | An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion | Gal et al. (2022), ARXIV-2208.01618. Efficient personalization method. | Active |
| LIT-079 v2 | DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation | Ruiz et al. (2022), ARXIV-2208.12242. Subject-driven image generation. | Active |
| LIT-080 v2 | PaLI: A Jointly-Scaled Multilingual Language-Image Model | Chen et al. (2022), ARXIV-2209.06794. Joint vision-language scaling. | Active |
| LIT-081 v3 | GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers | Frantar et al. (2022), ARXIV-2210.17323. One-shot weight quantization to 3-4 bits: round column by column and push each rounding error into the columns not yet done, using approximate second-order information from a small calibration set. | Active |
| LIT-082 v3 | Constitutional AI: Harmlessness from AI Feedback | Bai et al. (2022), ARXIV-2212.08073. Train harmlessness without human harm labels: the model critiques and revises its own responses against a written list of principles, and the preference model learns from those comparisons. | Active |
| LIT-083 v2 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | Zhao et al. (2022), ARXIV-2304.11277. Improved memory efficiency over ZeRO. | Active |
| LIT-084 | DeepNet: Scaling Transformers to 1,000 Layers | Wang et al. (2022), ARXIV-2203.00555. Scale-aware normalization. | Active |
| LIT-085 v3 | Progress measures for grokking via mechanistic interpretability | Nanda et al. (2023), ARXIV-2301.05217. Grokking measurement. | Active — measurement rather than actionable practice |
| LIT-086 v4 | K-Planes: Explicit Radiance Fields in Space, Time, and Appearance | Fridovich-Keil et al. (2023), ARXIV-2301.10241. Hybrid representation scheme. | Active |
| LIT-087 v5 | Data Selection for Language Models via Importance Resampling | Xie et al. (2023), ARXIV-2302.03169. Data quality assessment. | Active |
| LIT-088 v3 | Scaling Vision Transformers to 22 Billion Parameters | Zhai et al. (2023), ARXIV-2302.05442. Vision model scaling. | Active |
| LIT-089 v3 | Adding Conditional Control to Text-to-Image Diffusion Models | Zhang et al. (2023), ARXIV-2302.05543. Modular conditioning framework. | Active |
| LIT-090 v2 | Symbolic Discovery of Optimization Algorithms | Chen et al. (2023), ARXIV-2302.06675. Program search over an infinite, sparse space of optimizer programs, which turns up Lion — sign of an interpolated momentum, one state tensor instead of Adam’s two. | Rejected — retired at migration on a description that was not this paper; the reason has not been re-examined |
| LIT-091 v3 | T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models | Mou et al. (2023), ARXIV-2302.08453. Controllable generation. | Active |
| LIT-092 v2 | Understanding Contrastive Learning Requires Incorporating Inductive Biases | Saunshi et al. (2022), ARXIV-2202.14037. Role of inductive biases. | Rejected — theoretical analysis without concrete recommendations |
| LIT-093 v3 | Consistency Models | Song et al. (2023), ARXIV-2303.01469. Single-step generation. | Active |
| LIT-094 v2 | Neural Scene Graphs for Dynamic Scenes | Ost et al. (2020), ARXIV-2011.10379. Dynamic scene representation. | Rejected — too domain-specific for a general training list |
| LIT-095 v3 | The Unreasonable Effectiveness of Deep Features as a Perceptual Metric | Zhang et al. (2018), ARXIV-1801.03924. Deep features from almost any pretrained network, compared layer-wise, match human perceptual judgements far better than L2, SSIM or any hand-designed metric — and the effect is robust to which network and which supervision produced the features. | Active |
| LIT-096 v2 | Segment Anything | Kirillov et al. (2023), ARXIV-2304.02643. Universal segmentation model. | Active |
| LIT-097 v2 | One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale | Bao et al. (2023), ARXIV-2303.06555. Universal sequence modeling. | Active |
| LIT-098 v2 | PipeMare: Asynchronous Pipeline Parallel DNN Training | Yang et al. (2019), ARXIV-1910.05124. Asynchronous pipeline parallelism. | Active |
| LIT-099 v2 | PaLM 2 Technical Report | Anil et al. (2023), ARXIV-2305.10403. Architecture improvements. | Active |
| LIT-100 v2 | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | Ainslie et al. (2023), ARXIV-2305.13245. Efficient attention variant. | Active |
| LIT-101 v3 | AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs | Dubey et al. (2022), ARXIV-2210.06364. Adaptive gradient normalization. | Proposed |
| LIT-102 v3 | ZeRO++: Extremely Efficient Collective Communication for Giant Model Training | Rajbhandari et al. (2023), ARXIV-2306.10209. Communication optimization. | Active |
| LIT-103 v2 | SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills | Agrawal et al. (2023), ARXIV-2308.16369. Prefill saturates the GPU and decode starves it, so split prefills into chunks and fill the rest of each batch with decodes that ride along nearly free. | Active |
| LIT-104 v4 | ReLoRA: High-Rank Training Through Low-Rank Updates | Lialin et al. (2023), ARXIV-2307.05695. Restart a LoRA adapter repeatedly during pretraining so low-rank updates sum to a high-rank one. The restart needs an optimizer reset and a re-warm together (SOTA-426); the paper’s own ablation shows the full-rank warm start carries most of the gain. | Active |
| LIT-105 | Symbolic Discovery of Optimization Algorithms | Chen et al. (2023), ARXIV-2302.06675. A duplicate: the migration filed this paper twice, once under its real title and once under the name of the optimizer it discovers. LIT-090 is the surviving note. | Rejected — by LIT-090; a duplicate of LIT-090, which holds the same paper |
| LIT-106 v2 | FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning | Dao et al. (2023), ARXIV-2307.08691. Improved attention parallelism. | Active |
| LIT-107 v4 | MiDaS v3.1 — A Model Zoo for Robust Monocular Relative Depth Estimation | Birkl et al. (2023), ARXIV-2307.14460. Improved depth estimation. | Active |
| LIT-108 v3 | 3D Gaussian Splatting for Real-Time Radiance Field Rendering | Kerbl et al. (2023), ARXIV-2308.04079. Real-time neural rendering. | Active |
| LIT-109 v3 | NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction | Wang et al. (2022), ARXIV-2212.05231. Accelerated surface reconstruction. | Active |
| LIT-110 v2 | Efficiently Scaling Transformer Inference | Pope et al. (2022), ARXIV-2211.05102. Inference partitioning derived rather than guessed: an analytical model of communication and memory cost picks the best parallel layout for a given model size, batch and latency target, and the answer changes with the target. | Active |
| LIT-111 v3 | OnePose: One-Shot Object Pose Estimation without CAD Models | Sun et al. (2022), ARXIV-2205.12257. Category-level pose estimation. | Active |
| LIT-112 v2 | Efficient Memory Management for Large Language Model Serving with PagedAttention | Kwon et al. (2023), ARXIV-2309.06180. Introduces vLLM. | Active |
| LIT-113 v3 | Gaussian Shell Maps for Efficient 3D Human Generation | Abdal et al. (2023), ARXIV-2311.17857. Shell-based surface representation. | Active |
| LIT-114 v2 | On Layer Normalization in the Transformer Architecture | Xiong et al. (2020), ARXIV-2002.04745. Warmup is not a universal law but a fix for where the normalization sits: Post-LN has large gradients near the output at initialization and needs it, Pre-LN does not. | Active |
| LIT-115 v2 | Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture | Fu et al. (2023), ARXIV-2310.12109. Efficient alternative to attention. | Active |
| LIT-116 v2 | TensorRT-LLM: A Compiler and Runtime for LLMs | A synthetic entry the migration produced: no paper of this title exists, and the identifier it carried belongs to an unrelated physics preprint. Retired rather than deleted. The source field points at the software project itself. | Rejected — software with no published paper behind it, and no reading in the record |
| LIT-117 v2 | Efficient Online Data Mixing For Language Model Pre-Training | Albalak et al. (2023), ARXIV-2312.02406. Online Data Mixing: an Exp3 bandit over data domains, rewarded by per-domain training loss, revising the mixture during training. Reaches the next best method’s final perplexity in 19% fewer iterations at negligible wall-clock cost. | Active |
| LIT-118 v3 | PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding | Li et al. (2023), ARXIV-2312.04461. ID-preserving generation. | Active |
| LIT-119 | Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale | Falcon-LLM Team (2026), technical blogpost. Holds the parameter budget at 90M and pulls the other three levers — data strategy, architecture, optimizer — one at a time: pretraining from scratch on the target SFT or reasoning mix beats pretrain-then-finetune at this scale, Muon with learnable multipliers beats the AdamW recipe, and depth and SSM width beat MLP width. | Active |
| LIT-120 | Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance | Zuo et al. (2025), ARXIV-2507.22448. A parallel hybrid mixer — attention heads and Mamba heads side by side in one block — from 0.5B to 34B, with a µP recipe adapted to it and multi-epoch training on high-quality data; the architecture and the memorization-window measurement LIT-119 builds on. | Active |
| LIT-121 | Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers | Velikanov et al. (2026), ARXIV-2601.04890. Weight decay and gradient noise settle a matrix layer at an equilibrium norm set by the hyperparameters, not the data; a learnable scalar per matrix and per row and column frees it, generalises µP, and improves downstream results with both Adam and Muon. | Active |
| LIT-122 | Muon is Scalable for LLM Training | Liu et al. (2025), ARXIV-2502.16982. Two changes make Muon work at scale without re-tuning: decoupled weight decay, and scaling each update so its RMS matches what AdamW would produce; scaling-law runs show about 2× the compute efficiency of AdamW, demonstrated on a 3B/16B MoE trained for 5.7T tokens. | Active |
| LIT-123 | Wait, Wait, Wait… Why Do Reasoning Models Loop? | Pipis et al. (2025), ARXIV-2512.12895. Reasoning models loop — repeat the same text — at low temperature; larger models loop less and distilled students loop far more than their teachers, which the paper traces to errors in learning and, on a synthetic task, to risk aversion: when the progress-making action is hard to learn and a cyclic one is easy, probability shifts to the cycle. | Active |
| LIT-124 | Efficient Training of Language Models to Fill in the Middle | Bavarian et al. (2022), ARXIV-2207.14255. Moving a span from the middle of a document to its end teaches an autoregressive model to infill, and doing it to a large fraction of the data costs nothing in left-to-right quality — the FIM-for-free result — so the authors recommend training with FIM by default. | Active |
| LIT-125 | Qwen2.5-Coder Technical Report | Hui et al. (2024), ARXIV-2409.12186. Six code models from 0.5B to 32B, continued-pretrained from Qwen2.5 on over 5.5T tokens with cleaned, synthetic and balanced code/text/math data, and trained for fill-in-the-middle at file and repository level; state of the art on more than ten code benchmarks at release. | Active |
| LIT-126 | Structure-Aware Fill-in-the-Middle Pretraining for Code | Gong et al. (2025), ARXIV-2506.00204. Mask complete syntactic structures chosen from the AST instead of random character spans, so FIM examples match how code is actually edited; on 1B and 8B models AST-FIM beats random-character FIM by up to 5 points, and Real-FIM-Eval, built from 30k+ GitHub commits in 12 languages, measures the real task. | Active |
| LIT-127 v2 | DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Shao et al. (2024), ARXIV-2402.03300. A 7B model continued-pretrained on 120B math tokens mined from Common Crawl reaches 51.7% on MATH without tools or voting, and introduces Group Relative Policy Optimization — PPO without a critic, with the baseline taken from a group of sampled outputs — the RL algorithm most later reasoning recipes use. | Active |
| LIT-128 | Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling | Falcon-LLM Team (2026), ARXIV-2601.02346. Falcon-H1-7B taken to a reasoning model by SFT and RL scaling; it matches or beats reasoning models 2× to 7× larger, and its hybrid architecture makes the parallel test-time scaling it relies on (DeepConf) cheap in inference. Its reasoning corpus is the one LIT-119’s tiny reasoning models train on. | Active |
| LIT-129 | Can Tiny Language Models Reason? | Jakimovski (2025), a write-up with code, no paper. SmolLM2-135M-Instruct taken through SFT on plain dialogue, SFT on reasoning traces in think tags, and one epoch of DPO on 50k preference pairs, on a single GPU; the DPO stage converges and the final model gains 3 to 9 points over its base on ARC-C, BBH, MMLU and IFEval. | Active |
| LIT-130 | Olmo 3 | Olmo Team (2025), ARXIV-2512.13961. Fully open 7B and 32B models released as a complete model flow — every stage, checkpoint and dataset — with a Think track built by the standard curriculum (pretrain, SFT on reasoning traces, preference tuning, RL with verifiable rewards) and an RL-Zero track that runs RLVR directly on the base model as an open benchmark for studying RL. | Active |
| LIT-131 | Kimi K3: Open Frontier Intelligence | Kimi Team (2026), ARXIV-2607.24653. A 2.8T-parameter, 104B-active MoE with a 1M-token context, built from three architectural changes the team had published separately — a 69:24 layerwise hybrid of Kimi Delta Attention and gated MLA, Attention Residuals, and a 896-expert LatentMoE with 16 active — for a claimed 2.5× scaling efficiency over Kimi K2, still trained with MuonClip. | Active |
| LIT-132 | Kimi K2: Open Agentic Intelligence | Kimi Team (2025), ARXIV-2507.20534. A 1T-parameter, 32B-active MoE pretrained on 15.5T tokens with zero loss spikes under MuonClip — Muon plus QK-Clip, which rescales query and key projections whenever the largest attention logit crosses a threshold — then post-trained with large-scale agentic data synthesis and a joint RL stage. | Active |
| LIT-133 v2 | Kimi Linear: An Expressive, Efficient Attention Architecture | Kimi Team (2025), ARXIV-2510.26692. Kimi Delta Attention extends Gated DeltaNet with a channel-wise forgetting gate; interleaved with MLA at 3:1 in a 48B-total, 3B-active model trained on 1.4T tokens, the hybrid beats full attention on short-context, long-context and RL evaluations while cutting KV cache by up to 75% and reaching up to 6.3× the decoding throughput at 1M tokens. | Active |
| LIT-134 | Attention Residuals | Kimi Team (2026), ARXIV-2603.15031. Replace the fixed, uniform residual sum with learned, input-dependent attention over preceding layers, computed from one pseudo-query per layer at O(d) parameters; a blocked form keeps it practical at scale, and at 48B parameters on 1.4T tokens it shows a 1.25× compute advantage on scaling laws for under 2% inference overhead. | Active |
| LIT-135 | Qwen3.8-27B | Qwen Team (2026), model card and release blog; no technical report. A 27B dense vision-language model with 48 Gated DeltaNet layers and 16 gated full-attention layers, an MTP draft head and a 262k native context; architecturally identical to Qwen3.6-27B four months earlier, with the generation’s gains coming from post-training on agentic and coding work. | Active |
| LIT-136 | Qwen3-Next: Towards Ultimate Training & Inference Efficiency | Qwen Team (2025), release blog; no paper. An 80B-total, 3B-active MoE that first shipped the 3:1 Gated DeltaNet to gated-attention layout, native multi-token prediction, and zero-centered weight-decayed layernorm; trained on a 15T-token subset for 9.3% of Qwen3-32B’s cost while matching it, with over 10× the throughput past 32k tokens. | Active |
| LIT-137 v2 | Gated Delta Networks: Improving Mamba2 with Delta Rule | Yang et al. (2024), ARXIV-2412.06464. Gating erases memory fast and the delta rule writes to it precisely; combining them in one recurrence, with a chunkwise parallel algorithm for the hardware, beats both Mamba2 and DeltaNet on language modelling, retrieval, length extrapolation and long context at 1.3B/100B tokens, and the paper’s own hybrids with sliding-window attention or Mamba2 layers do better still. | Active |
| LIT-138 | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | Qiu et al. (2025), ARXIV-2505.06708. Thirty gating variants compared on 15B MoE and 1.7B dense models over 3.5T tokens: a head-specific sigmoid gate applied after scaled dot-product attention consistently improves quality, stabilises training, tolerates larger learning rates and scales better, and removes attention sinks and massive activations. | Active |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-140 | mHC: Manifold-Constrained Hyper-Connections | Xie et al. (2025), ARXIV-2512.24880. Hyper-Connections widen the residual stream but break the identity mapping a residual guarantees, which destabilises deep stacks; constraining the residual mixing matrices to be doubly stochastic via Sinkhorn-Knopp restores it while keeping the width, at a reported 6.7% training-time overhead (n = 4, the 27B system result; the scaling study spans 3B, 9B and 27B) — the residual DeepSeek-V4 ships. | Active |
| LIT-141 | Hyper-Connections | Zhu et al. (2024), ARXIV-2409.19606. Replace the residual connection with several parallel streams and learnable mixing between them, at expansion rate 2 or more, to escape the seesaw between vanishing gradients and representation collapse; gains on dense and MoE pretraining — and, per its successor, an identity-mapping property lost along the way. | Superseded — by LIT-140; mHC restores the identity mapping that free hyper-connection mixing gave up, and scales where this did not |
| LIT-142 | DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models | DeepSeek-AI (2025), ARXIV-2512.02556. DeepSeek Sparse Attention: a small FP8 lightning indexer scores every preceding token and only the top-k KV entries enter the attention, fitted onto V3.1’s MLA in multi-query mode by continued pretraining with a 2.1B-token indexer warm-up under frozen dense attention — the sparse mechanism V4 compresses further. | Active |
| LIT-143 | Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention | Yuan et al. (2025), ARXIV-2502.11089. A sparse attention trained from the start rather than applied after: coarse token compression for global context, fine top-k block selection for precision, and a sliding window for locality, with kernels balanced for arithmetic intensity; matches or beats full attention on general, long-context and reasoning evaluations with large speedups at 64k in forward, backward and decoding. | Active |
| LIT-144 | MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies | Hu et al. (2024), ARXIV-2404.06395. Introduces the Warmup-Stable-Decay schedule: hold the learning rate constant for most of training and decay it sharply over roughly the last 10% of tokens, where most of the loss improvement arrives. Because the decay can start at any point, the token budget need not be fixed in advance, intermediate checkpoints stay useful, and new data can be mixed into the decay stage. | Active |
| LIT-145 | Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations | Hägele et al. (2024), ARXIV-2405.18392. A constant learning rate with a cooldown scales as predictably as cosine and matches it in quality, so scaling laws can be fitted from checkpoints along one run rather than from a run per duration; stochastic weight averaging along the trajectory improves further at no training cost. | Active |
| LIT-146 | Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler | Shen et al. (2024), ARXIV-2408.13359. Under a WSD schedule the optimal learning rate follows a power law in the number of training tokens and the batch size; the Power scheduler sets the rate from that law so one value transfers across token budgets and batch sizes, and combined with µP across model sizes too. | Active |
| LIT-147 | Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs | Bergsma et al. (2025), ARXIV-2502.15938. The common recipe decays the learning rate to 10% of its peak; a large empirical study finds that, with the peak tuned, linear decay all the way to zero consistently beats it and other schedules at compute-optimal token budgets, and the paper explains why by reading AdamW as an exponential moving average of updates. | Active |
| LIT-148 | Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer | Yang and Hu (2022), ARXIV-2203.03466. In the Maximal Update Parametrization the optimal learning rate and other hyperparameters stay put as width grows, so they can be tuned on a small proxy and transferred to the full model zero-shot: BERT-large beaten from a 13M-parameter sweep, GPT-3 6.7B beaten at a tuning cost of 7% of one pretraining run. | Active |
| LIT-149 | u-µP: The Unit-Scaled Maximal Update Parametrization | Blake et al. (2024), ARXIV-2407.17465. Combine µP with Unit Scaling so that activations, weights and gradients start at unit scale regardless of size: lower loss than plain µP, FP8 training out of the box, and hyperparameters that can be swept one at a time, with a learning-rate sweep alone reaching near-optimal loss. | Active |
| LIT-150 | Don’t be lazy: CompleteP enables compute-efficient deep transformers | Dey et al. (2025), ARXIV-2505.01618. Some parametrizations, µP among them, fail to transfer the optimal learning rate across depth and leave deep layers learning lazily; CompleteP (the depth exponent α = 1) transfers across depth and keeps every layer non-lazy, saving 11.8% of FLOPs over µP at optimal shapes and 34.4% at 179 layers, and widening the range of compute-efficient width-to-depth ratios. | Active |
| LIT-151 | oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions | Guo et al. (2026), ARXIV-2609.02672. The doubly-stochastic constraint mHC puts on the residual-mixing matrix bounds it from above but not below, so the mixing can only shrink what distinguishes the streams and the widened residual homogenises with depth; constraining the matrix to the rotation group instead bounds it on both sides, and at four streams a pair of unit quaternions gives it in closed form for no extra parameters. | Active |
| LIT-152 v2 | On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability | Qiu et al. (2026), ARXIV-2608.30320. The ablation record behind a Qwen3.8-generation architecture, and the first place the record’s three residual replacements meet in one harness: Gated Residual, mHC and Attention Residuals come out level, separated by inference cost rather than quality. Also the source for judging every architectural change on loss, cost and stability at once. | Active |
| LIT-153 | Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization | Wen et al. (2026), ARXIV-2606.16899. The follow-up that diagnoses why matrix optimizers’ gains shrink with scale: constant decoupled weight decay fixes the equilibrium weight norm and so fixes the angular learning rate. Pinning the Frobenius norms instead recovers 20–30% over the weight-decay baseline. | Active |
| LIT-154 | 2 OLMo 2 Furious | OLMo Team (2025), ARXIV-2501.00656. A fully open 7B–32B recipe whose stated focus is training stability and per-token efficiency, plus the late-stage specialized data mix that the decay phase is for. | Active |
| LIT-155 | Small-scale proxies for large-scale Transformer training instabilities | Wortsman et al. (2023), ARXIV-2309.14322. Instabilities reported only at frontier scale reappear in small models trained at high learning rate, and the large-scale mitigations work there too — which makes instability something a small budget can study. | Active |
| LIT-156 | Fantastic Pretraining Optimizers and Where to Find Them | Wen et al. (2025), ARXIV-2509.02046. Ten optimizers, four scales, tuned fairly and judged at the end of training rather than mid-run: the matrix preconditioners really are fastest, and their advantage over a well-tuned AdamW shrinks with scale — 1.4× at 0.1B to 1.1× at 1.2B. | Active |
| LIT-157 | SOAP: Improving and Stabilizing Shampoo using Adam | Vyas et al. (2024), ARXIV-2409.11321. Shampoo at the 1/2 power is Adafactor run in the eigenbasis of Shampoo’s preconditioner; running Adam there instead gives one extra hyperparameter over Adam and beats both. | Active |
| LIT-158 | A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale | Shi et al. (2023), ARXIV-2309.06497. The engineering that made a second-order optimizer usable at scale: block-diagonal Kronecker preconditioners sharded across GPUs, at most 10% per-step wall-clock over diagonal adaptive methods. | Active |
| LIT-159 | Muon: An optimizer for hidden layers in neural networks | Jordan et al. (2024), blog post. Where Muon comes from: take the momentum matrix for a hidden weight, orthogonalise it with a few Newton–Schulz iterations, and step in that direction — leaving embeddings, the classifier head and the gains and biases to AdamW. | Active |
| LIT-160 | DeepSeek-V3 Technical Report | DeepSeek-AI (2024), ARXIV-2412.19437. 671B total, 37B activated, 14.8T tokens, 2.788M H800 hours — and no irrecoverable loss spike and no rollback in the entire run, which is the claim worth carrying. | Active |
| LIT-161 v2 | Mamba: Linear-Time Sequence Modeling with Selective State Spaces | Gu and Dao (2023), ARXIV-2312.00752. Make the SSM parameters functions of the input and the model can decide what to remember; a hardware-aware recurrent kernel keeps it fast without convolutions. | Active |
| LIT-162 v2 | Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | Dao and Gu (2024), ARXIV-2405.21060. SSMs and attention variants are two decompositions of the same class of structured semiseparable matrices; the duality yields Mamba-2, whose core layer is 2–8× faster. | Active |
| LIT-163 v2 | Better & Faster Large Language Models via Multi-token Prediction | Gloeckle et al. (2024), ARXIV-2404.19737. Predict the next n tokens from n heads on a shared trunk: better sample efficiency at no training cost, growing with model size, and up to 3× faster inference as a side-effect. | Active |
| LIT-164 | DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | DeepSeek-AI (2025), ARXIV-2501.12948. Reasoning incentivised by pure RL on verifiable tasks, with no human-labelled reasoning traces: self- reflection, verification and strategy adaptation emerge rather than being demonstrated. | Active |
| LIT-165 v2 | Mamba-3: Improved Sequence Modeling using State Space Principles | Lahoti et al. (2026), ARXIV-2603.15569. Three changes from an inference-first view — a more expressive discretised recurrence, a complex-valued state update for state tracking, and a MIMO formulation that buys accuracy without decode latency. | Active |
| LIT-166 | Scaling Data-Constrained Language Models | Muennighoff et al. (2023), ARXIV-2305.16264. Up to four epochs of repeated data cost almost nothing against unique data at fixed compute; past that the value of added compute decays to zero. 400 runs, up to 900B tokens and 9B parameters. | Active |
| LIT-167 | Understanding R1-Zero-Like Training: A Critical Perspective | Liu et al. (2025), ARXIV-2503.20783. Two corrections to the R1-Zero story: the base model may already carry what looks like emergent reasoning, and GRPO has a length bias that inflates wrong answers. Dr. GRPO removes the bias. | Active |
| LIT-168 | DAPO: An Open-Source LLM Reinforcement Learning System at Scale | Yu et al. (2025), ARXIV-2503.14476. Decoupled Clip and Dynamic Sampling Policy Optimization, published with the four techniques, the code and the dataset — 50 points on AIME 2024 from a Qwen2.5-32B base. | Active |
| LIT-169 | Direct Preference Optimization: Your Language Model is Secretly a Reward Model | Rafailov et al. (2023), ARXIV-2305.18290. Reparameterise the reward so the optimal policy has a closed form, and RLHF collapses into a single classification loss — no reward model, no sampling during fine-tuning, no RL loop. | Active |
| LIT-170 | DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models | Dai et al. (2024), ARXIV-2401.06066. Two changes to top-K routing that make experts specialise: cut them finer so the activated combination is more expressive, and isolate a few shared experts to absorb the common knowledge the routed ones were duplicating. | Active |
| LIT-171 | Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts | Wang et al. (2024), ARXIV-2408.15664. Balance expert load with a bias added to the routing scores and updated from recent load, instead of with an auxiliary loss whose gradients interfere with the objective you care about. | Active |
| LIT-172 | Tulu 3: Pushing Frontiers in Open Language Model Post-Training | Lambert et al. (2024), ARXIV-2411.15124. The post-training recipe published in full — data, code, decontamination and the negative results — and the origin of Reinforcement Learning with Verifiable Rewards as a named stage. | Active |
| LIT-173 | RWKV-7 “Goose” with Expressive Dynamic State Evolution | Peng et al. (2025), ARXIV-2503.14456. A generalised delta rule with vector-valued gating and in-context learning rates — provably able to track state and recognise all regular languages, which Transformers under standard conjectures cannot. | Active |
| LIT-174 | DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model | DeepSeek-AI (2024), ARXIV-2405.04434. Where Multi-head Latent Attention comes from: compress the KV cache into a latent vector, cutting it 93.3% and raising maximum generation throughput 5.76×. | Active |
| LIT-175 | Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining | Chen et al. (2026), ARXIV-2606.16246. Autoregressive pretraining overfits badly under heavy repetition; three orthogonal families of augmentation — token noise, sequence permutation, offset targets — delay it and make hundreds of epochs on a fixed corpus productive. | Active |
| LIT-176 | Native Hybrid Attention for Efficient Sequence Modeling | Du et al. (2025), ARXIV-2510.07019. Hybridise within the layer instead of between layers: a linear RNN maintains long-term KV slots, a sliding window supplies short-term tokens, and one softmax attends over both. | Active |
| LIT-177 | Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention | Li et al. (2026), ARXIV-2606.26560. The delta rule corrects only at the address it is writing to, so stale content elsewhere can never be removed; EDA adds an erase step at an independently chosen address. | Active |
| LIT-178 | Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset | Su et al. (2024), ARXIV-2412.02595. Aggressive model-based filtering throws away 90% of the web, which is fine at 1T tokens and ruinous at 15T; classifier ensembling plus synthetic rephrasing keeps four times the unique tokens at the same quality. | Active |
| LIT-179 | The Llama 3 Herd of Models | Grattafiori et al. (2024), ARXIV-2407.21783. The period’s most-cited dense recipe, published at length: a 405B dense Transformer at 128K context, with the empirical evaluation and the compositional route to image, video and speech. | Active |
| LIT-180 v2 | Group Sequence Policy Optimization | Zheng et al. (2025), ARXIV-2507.18071. Define the importance ratio on sequence likelihood rather than per token, and clip, reward and optimise at sequence level — which stabilises MoE reinforcement learning in particular. | Active |
| LIT-181 v2 | Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections | Liu et al. (2026), ARXIV-2603.20896. Three named defects in mHC’s doubly-stochastic constraint — identity degeneration, an expressivity bottleneck from non-negativity, and parameterization cost — answered by moving the feasible set to a spectral-norm sphere. | Active |
| LIT-182 | Qwen3 Technical Report | Yang et al. (2025), ARXIV-2505.09388. Thinking and non-thinking modes unified in one model with a thinking budget the caller sets, plus flagship-to-small distillation; 0.6B to 235B, dense and MoE, 119 languages. | Active |
| LIT-183 | Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning | NVIDIA (2025), ARXIV-2512.20848. A 30B-A3B MoE hybrid Mamba-Transformer on 25T tokens: more accurate than its predecessor at under half the activated parameters, and up to 3.3× the inference throughput of comparable open models. | Active |
| LIT-184 | The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale | Penedo et al. (2024), ARXIV-2406.17557. 15T tokens from 96 Common Crawl snapshots, with every design choice documented and ablated — and FineWeb-Edu, the 1.3T educational filter that moves MMLU and ARC sharply. | Active |
| LIT-185 v3 | EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test | Li et al. (2025), ARXIV-2503.01840. Drop feature prediction for direct token prediction and fuse multi-layer features, so the draft model finally benefits from more training data: up to 6.5× speedup. | Active |
| LIT-186 | Scaling Laws for Precision | Kumar et al. (2024), ARXIV-2411.04330. Training in low precision reduces a model’s effective parameter count, and post-training quantization degrades more the longer a model was trained — so past some point extra pretraining data is actively harmful. | Active |
| LIT-187 v2 | GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding | Lepikhin et al. (2020), ARXIV-2006.16668. Puts the mixture-of-experts layer into a transformer and shards it across 2048 TPUs with annotations rather than a rewrite: a 600B multilingual translation model trained in four days, and the top-2 routing that later work names as the thing to fix. | Active |
| LIT-188 | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Shazeer et al. (2017), ARXIV-1701.06538. The sparsely-gated mixture-of-experts layer: thousands of feed-forward experts with a trainable gate selecting a few per example, so model capacity can grow by 1000× at small increases in compute. The paper conditional computation had been promising in theory and not delivering in practice until. | Active |
| LIT-189 | Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | Fedus et al. (2021), ARXIV-2101.03961. Route each token to exactly one expert instead of two: simpler, cheaper per token, and stable in bfloat16 with selective casting — a 1.6T-parameter model and a 7× pretraining speedup over T5-Base at matched compute. | Active |
| LIT-190 | Massive Activations in Large Language Models | Sun et al. (2024), ARXIV-2402.17762. A handful of activations in an LLM run up to 100,000× larger than the rest, at fixed feature dimensions and a few token positions, and they act as fixed biases rather than input-dependent features: setting them to their own mean leaves the model intact, setting them to zero destroys it. They are how the model smuggles an attention bias into a mechanism that has none, and giving attention an explicit learnable bias makes them disappear at no cost in quality. | Active |
| LIT-191 | Efficient Streaming Language Models with Attention Sinks | Xiao et al. (2023), ARXIV-2309.17453. Window attention collapses when the first few tokens leave the cache — Llama-2-13B goes from 5.4 to 5158 perplexity — because a softmax must put its mass somewhere and models learn to dump the surplus on whatever is globally visible, which is the initial tokens. Keeping four of their KV entries alongside a rolling window restores it at no finetuning cost, and a single learnable sink token trained in from the start replaces all four. | Active |
| LIT-192 | Extending Context Window of Large Language Models via Positional Interpolation | Chen et al. (2023), ARXIV-2306.15595. Down-scale the position indices to fit the window the model was trained on, instead of extrapolating past it. LLaMA 7B-65B reach 32768 tokens — 16× — after fine-tuning for under 1000 steps, where naive fine-tuning at the longer length moved the effective window from 2048 to 2560 in more than 10000 batches. The argument is about smoothness: interpolating a learned attention-score function between points it already fits is bounded, extrapolating it is not. | Active |
| LIT-193 | YaRN: Efficient Context Window Extension of Large Language Models | Peng et al. (2023), ARXIV-2309.00071. Position interpolation stretches every RoPE dimension equally, which destroys the high-frequency components and stalls around a scale factor of 8. YaRN scales by wavelength instead — leaving dimensions that never complete a rotation alone, interpolating the ones that do — and adds a temperature on the attention logits implemented for free inside the rotary embedding. Llama 2 to 128k with 10× fewer tokens and 2.5× fewer steps than PI. | Active |
| LIT-194 | Gated Linear Attention Transformers with Hardware-Efficient Training | Yang et al. (2023), ARXIV-2312.06635. Linear attention was slower in practice than optimised softmax attention despite better asymptotics, because nobody had made it I/O-aware. FlashLinearAttention fixes that — faster than FlashAttention-2 as a standalone layer even at 1K — and the algorithm generalises to a data-dependent forget gate. The resulting GLA Transformer matches LLaMA-architecture baselines, beats Mamba on throughput at matched size, and generalises from 2K training to over 20K. | Active |
| LIT-195 | Parallelizing Linear Transformers with the Delta Rule over Sequence Length | Yang et al. (2024), ARXIV-2406.06484. The delta rule fixes what linear attention is worst at — associative recall — but its original algorithm did not parallelize over sequence length, so it never left small scale. A reparameterization as products of Householder matrices makes it chunkwise, and a 1.3B DeltaNet on 100B tokens beats Mamba and GLA. It also reports the hybrid result: interleaving with sliding-window or two global attention layers beats a strong Transformer baseline. | Active |
| LIT-196 | LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts | Elango et al., NVIDIA (2026), ARXIV-2601.18089. Existing MoE designs are argued from sparsity and tuned for offline throughput, which is the wrong objective for interactive serving where expert computation is bandwidth- bound, not compute-bound. LatentMoE down-projects tokens into a narrow latent space before routing and keeps the routed experts’ weights there, so both the dispatch traffic and the weight-loading bandwidth fall by the width ratio. Explored to 95B parameters over 1T tokens; adopted by Nemotron-3 Super and Ultra, and by Kimi K3. | Active |
| LIT-197 | Microscaling Data Formats for Deep Learning | Rouhani et al. (2023), ARXIV-2310.10537. The MX formats: a block of 32 elements sharing one 8-bit power-of-two scale, with the elements themselves in FP8, FP6, FP4 or INT8. Over two dozen benchmarks — 8-bit MX runs inference on FP32 checkpoints with no calibration, 6-bit matches FP32 for training weights, activations and gradients with no recipe change, and 4-bit weights cost only a minor drop. The first sub-8-bit training result of that kind. | Active |
| LIT-198 v2 | Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention | Qiu et al. (2025), ARXIV-2510.04212. The first mechanistic account of why training with flash attention in low precision explodes. Two things have to coincide: attention representations that are low-rank and similar across steps and tokens, and rounding error in BF16 addition that is biased rather than symmetric. The bias acts as a coefficient on the repeated low-rank update, so the errors accumulate instead of cancelling and the spectral norm of weights runs away. A minimal change to flash attention that de-biases the rounding stops it. | Active |
| LIT-199 | Language Modeling with Gated Convolutional Networks | Dauphin et al. (2016), ARXIV-1612.08083. Where the gated linear unit comes from. Its argument is about gradients, not expressivity: an LSTM-style tanh gate multiplies two downscaling factors into the gradient, which vanishes as layers stack, while gating a linear unit leaves a path with no downscaling — a multiplicative skip connection. State of the art on WikiText-103, and the first non-recurrent model competitive with strong recurrent ones at that scale. | Active |
| LIT-200 v3 | PowLU: An Activation Function for Stable Pre-Training of LLMs | Jiang et al. (2026), ARXIV-2605.25704. For large positive inputs SwiGLU approximates x², and that quadratic amplification is what enlarges the output range and produces outliers, particularly in low-precision training. PowLU replaces it with a rational power function giving adaptive nonlinearity and bounded growth. Scaling-law experiments plus 7.9B and 124B Ling models against SwiGLU, and against SwiGLU-Clip at 7.9B only. | Active |
| LIT-201 | Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance | Ye et al. (2024), ARXIV-2403.16952. Mixture proportions are tuned by heuristic because nobody had shown they were predictable. They are: loss is a fittable function of the proportions, so training a handful of small mixtures predicts performance on mixtures never run. Nested with the scaling laws for steps and model size, a 1B model over 100B RedPajama tokens matched the default mixture trained 48% longer. | Active |
| LIT-202 | Deduplicating Training Data Makes Language Models Better | Lee et al. (2021), ARXIV-2107.06499. Standard corpora are full of near-duplicates — C4 contains one 61-word English sentence repeated thousands of times. Deduplicating makes models emit memorised text ten times less often and reach the same or better accuracy in fewer steps, and it fixes evaluation too: train-test overlap affects a large fraction of standard validation sets. | Active |
| LIT-203 | Fewer Truncations Improve Language Modeling | Ding et al. (2024), ARXIV-2404.10830. Everyone concatenates documents and splits at a fixed length to avoid padding, which shreds documents into incomplete pieces. Best-fit Packing treats the grouping as bin packing instead and eliminates unnecessary truncation at the same training efficiency — no padding added. Across 22 tasks at 7B–13B: +4.7% reading comprehension, +16.8% context following, +9.2% program synthesis, and up to 58.3% less closed-domain hallucination. | Active |
| LIT-204 | Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models | De et al., Google DeepMind (2024), ARXIV-2402.19427. Two models: Hawk, a pure RNN with gated linear recurrences that exceeds Mamba’s reported downstream numbers; and Griffin, which mixes those recurrences with local attention and matches Llama-2 on over 6× fewer tokens. Scaled to 14B, with the sharding recipe published, matching Transformer hardware efficiency in training and beating it on inference latency and throughput. | Active |
| LIT-205 v2 | How to Synthesize Text Data without Model Collapse? | Zhu et al. (2024), ARXIV-2412.14689. Pre-training across proportions of synthetic data gives a negative correlation between that proportion and performance, and the statistical signature is distributional shift plus over-concentration of n-gram features. The proposed answer is not better generation but token-level editing of human text into semi-synthetic data, with a proof that the test error is then bounded. | Active |
| LIT-206 | Ring Attention with Blockwise Transformers for Near-Infinite Context | Liu et al. (2023), ARXIV-2310.01889. Compute attention and the feedforward blockwise, distribute the blocks across devices in a ring, and overlap the communication of key/value blocks with the computation of the next block. Sequence length scales with device count — no approximation, no extra communication or computation cost — which is what makes million-token contexts a sharding question rather than a memory ceiling. | Active |
| LIT-207 v2 | The Impact of Positional Encoding on Length Generalization in Transformers | Kazemnejad et al. (2023), ARXIV-2305.19466. Five positional schemes trained from scratch under identical hyperparameters — absolute, T5’s relative bias, ALiBi, rotary, and none at all — and the one that generalizes to unseen lengths is none at all. It also reports that such a model represents both absolute and relative position anyway, which is why removing the encoding removes a constraint rather than information — though the absolute half of that is Haviv et al. 2022 (LIT-665), a year earlier and with the mechanism. | Active |
| LIT-208 | Rope to Nope and Back Again: A New Hybrid Attention Strategy | Yang et al. (2025), ARXIV-2501.18795. The paper that says what NoPE and RoPE layers are each for, by measuring where their attention mass goes: NoPE layers retrieve, RoPE layers attend locally, and the two roles are cleaner when the RoPE layers are confined to a sliding window. The resulting architecture — one full-attention NoPE layer per three windowed RoPE layers, 8B parameters, 5T tokens — beats a full-attention RoPE model on long and short context, at about a quarter of the KV cache. | Active |
| LIT-209 | SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling | Puvvada et al. (2025), ARXIV-2504.08719. The same layout as LIT-208, reached independently and three months later — NoPE global layers interleaved 1:3 with sliding-window RoPE layers — plus the two things that paper does not have: a dynamic scaling of attention scores at inference, which is what keeps the NoPE layers working far past the training length, and a demonstration that an existing 8B RoPE model pretrained on 15T tokens can be converted to this architecture by continued training without losing its short-context scores. | Active |
| LIT-210 | Round and Round We Go! What makes Rotary Positional Encodings useful? | Barbero et al. (2024), ARXIV-2410.06205. Opens up a trained Gemma 7B to ask what RoPE is actually doing, and the answer is not the one RoPE’s own paper gave: attention does not reliably decay with distance, and the model instead splits the frequency band in two — the highest frequencies build robust positional attention heads, the lowest carry semantic content in channels that are provably not robust over long context. Removing the lowest frequencies (p-RoPE) fixes that and improves perplexity at 2B, and p is a dial whose ends are exactly RoPE and NoPE. | Active |
| LIT-211 | Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning | Qiu et al. (2025), ARXIV-2509.24372. Evolution strategies applied to the full parameter space of an LLM, at billion scale, without dimensionality reduction — which the field had assumed impossible. Against PPO and GRPO on Countdown across Qwen2.5 (0.5B–7B) and LLaMA3 (1B–8B), ES improves over the base model by 36.4% on average against 21.3% for GRPO and 17.9% for PPO, and does it with one fixed hyperparameter set while RL got a sweep per experiment. | Active |
| LIT-212 | DiLoCo: Distributed Low-Communication Training of Language Models | Douillard et al. (2023), ARXIV-2311.08105. Training on islands of badly connected devices instead of one tightly coupled cluster: many inner AdamW steps per worker, Nesterov momentum as the outer optimizer over the resulting deltas, synchronised rarely. On C4 with 8 workers it matches fully synchronous training while communicating 500× less, and tolerates workers appearing and disappearing mid-run. | Active |
| LIT-213 | The Road Less Scheduled | Defazio et al. (2024), ARXIV-2405.15682. A third answer to the question the schedule chain is about: do not schedule. Iterate averaging and scheduling turn out to be the same mechanism, and the averaged form needs no stopping time T, no schedule shape and no hyperparameter beyond what AdamW already has — while matching schedules that do know T. Won the MLCommons 2024 AlgoPerf Self-Tuning track. | Active |
| LIT-214 | Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch | Douillard et al. (2025), ARXIV-2501.18512. DiLoCo made synchronisation rare and left its size alone, so peak bandwidth was unchanged. Three changes fix that: synchronise subsets of parameters in sequence rather than all at once, let workers keep training while they synchronise, and quantise what crosses the link. Billion-scale training at matched quality with two orders of magnitude less bandwidth. | Active |
| LIT-215 v2 | V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning | Assran et al. (2025), ARXIV-2506.09985. The JEPA objective at internet scale: over 1M hours of video, action-free, then post-trained into an action-conditioned world model on under 62 hours of unlabelled robot video. 77.3 on Something-Something v2, 39.7 recall-at-5 on Epic-Kitchens-100, 84.0 on PerceptionTest once aligned to an 8B LLM — and zero-shot pick-and-place on Franka arms in labs that contributed no training data. | Active |
| LIT-216 v3 | Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture | Assran et al. (2023), ARXIV-2301.08243. I-JEPA: predict the representations of target blocks from a context block in the same image, rather than the pixels or an augmentation-invariant embedding. No hand-crafted augmentations, and the masking strategy is the design choice that does the work — large semantic targets, a spatially distributed context. A ViT-Huge/14 trains on ImageNet in under 72 hours on 16 A100s. | Active |
| LIT-217 | Large Language Diffusion Models | Nie et al. (2025), ARXIV-2502.09992. LLaDA: an 8B language model trained from scratch under the ordinary pretrain-then-SFT paradigm, with the autoregressive factorization replaced by masked diffusion — a forward masking process and a reverse process that predicts masked tokens, optimizing a likelihood lower bound. Competitive with LLaMA3 8B on in-context learning, and beats GPT-4o on reversal poem completion. | Active |
| LIT-218 v2 | Video models are zero-shot learners and reasoners | Wiedemer et al. (2025), ARXIV-2509.20328. The argument is an analogy and it is stated as one: large generative models on web-scale data made language models generalist, the same primitives now describe video models, so vision may be about to follow. Veo 3 is shown solving segmentation, edge detection, image editing, affordance recognition, tool-use simulation and early visual reasoning (mazes, symmetry) that it was never trained on. | Active |
| LIT-219 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | Li et al. (2020), ARXIV-2006.15704. The design and evaluation of PyTorch’s DistributedDataParallel, whose three acceleration techniques are “bucketing gradients, overlapping computation with communication, and skipping gradient synchronization” — near-linear scalability on 256 GPUs. | Active |
| LIT-220 | Horovod: fast and easy distributed deep learning in TensorFlow | Sergeev and Del Balso (2018), ARXIV-1802.05799. Ring-allreduce for TensorFlow, and the origin of Tensor Fusion — batching small tensors before the collective, because ring-allreduce “utilizes the network in an optimal way if the tensors are large enough, but does not work as efficiently or quickly if they are very small.” | Active |
| LIT-221 | Train With Mixed Precision — NVIDIA deep learning performance guide | NVIDIA, the Train With Mixed Precision user guide. Where the dynamic loss scaling constants everyone runs actually come from: “we successfully trained networks with N = 2000, increasing scaling factor by 2, decreasing scaling factor by 0.5” — reported as one tested configuration, with “many other settings are valid as well.” | Active |
| LIT-222 | CoreWeave Training Benchmarks Whitepaper | CoreWeave (2025). Fits a right-censored exponential survival model to real job durations and interruptions, giving a per-GPU failure rate of 3,748.25 days/failure and an MTTF of 3.66 days at 1,024 GPUs — the quantity a checkpoint interval should be chosen against, and one the record had no source for. | Active |
| LIT-223 v2 | How Does Batch Normalization Help Optimization? | Santurkar et al. (2018), ARXIV-1805.11604. “Distributional stability of layer inputs has little to do with the success of BatchNorm” — what it does instead is make the optimization landscape significantly smoother. | Active |
| LIT-224 | Orca: A Distributed Serving System for Transformer-Based Generative Models | Yu et al. (2022), OSDI ‘22. Introduces iteration-level scheduling — schedule at the granularity of one model iteration rather than one request — and selective batching. 36.9× throughput over FasterTransformer at equal latency on GPT-3 175B. The origin of what the field calls continuous batching. | Active |
| LIT-225 | Generating Long Sequences with Sparse Transformers | Child et al. (2019), ARXIV-1904.10509. The paper the record’s sliding window comes from. Factorizes causal attention into two steps - a local window of the previous l positions and a second head that reaches across windows - giving O(n*sqrt(n)) and, on two of three datasets, a LOWER loss than the dense attention it replaces. | Active |
| LIT-226 | MoEfication: Transformer Feed-forward Layers are Mixtures of Experts | Zhang et al. (2021), ARXIV-2110.01786. Most inputs activate only a tiny fraction of an FFN’s neurons, and the neurons that fire together can be partitioned into experts with a router bolted on afterwards — the same parameters, conditionally used. 10–30% of FFN parameters per input retains over 95% of performance. | Active |
| LIT-227 | Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints | Komatsuzaki et al. (2022), ARXIV-2212.05055. Initialize a mixture-of-experts model from a dense checkpoint instead of from scratch. Upcycled T5 and ViT models beat their dense counterparts using ~50% of the dense pretraining sunk cost, and beat MoE models trained from scratch on 100% of it. | Active |
| LIT-228 | Emergent Modularity in Pre-trained Transformers | Zhang et al. (2023), ARXIV-2305.18390. Neurons in a pre-trained transformer are functionally specialized, the specialization clusters into mixture-of-experts-shaped groups, and perturbing a functional expert damages the corresponding function. The modular structure stabilizes early in pre-training — earlier than the neurons themselves do. | Active |
| LIT-229 | Evolution Strategies at the Hyperscale | Sarkar et al. (2025), ARXIV-2511.16652. EGGROLL — Evolution Guided GeneRal Optimisation via Low-rank Learning. Replaces the full-rank Gaussian perturbation each ES worker draws with a rank-r outer product, cutting auxiliary memory per layer from O(mn) to O(r(m+n)) and raising arithmetic intensity enough for a hundredfold throughput gain at billion scale, at up to 91% of pure batch-inference throughput. The population average is still high-rank, and the low-rank update converges to the full-rank one at O(1/r). | Active |
| LIT-230 v3 | Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO | Ba et al. (2026), ARXIV-2608.27351. The first study of what ES post-training DOES rather than whether it wins. Three findings: ES raises pass@1 and pass@k together while GRPO raises pass@1 and falls below the base model on pass@16 and pass@32 in 15 of 18 comparisons; ES’s 40x larger parameter drift is functionally sparse, concentrated in LayerNorm and attention projections, and does not produce broad forgetting; and the population size needed for stable ES falls as the model grows. | Active |
| LIT-231 v2 | Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging | Gu et al. (2026), ARXIV-2608.05541. Argues that full-parameter ES on billion-parameter LLMs is ineffective because almost all random perturbations are near-orthogonal to a useful descent direction, and replaces the search space rather than the optimizer: a handful of cheap GRPO runs supply LoRA descent directions, and CMA-ES then searches layer-wise DARE-TIES merging coefficients over their span. Beats GRPO-LoRA by about 1% with 10% fewer gradient updates — on models at 1.5B and below. | Active |
| LIT-232 | EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale | Kaya and Hashemi (2026), ARXIV-2609.10980. Works out what EGGROLL’s low-rank update actually converges to and finds it is a resolvent-filtered gradient, which can be nonconservative and can turn a local optimum into a repelling one — while being exact on every quadratic objective at every rank. The practical yield is LOO-ROLL, which replaces the two antithetic evaluations per direction with one leave-one-out evaluation, halving estimator MSE in transformer blocks at equal cost. | Active |
| LIT-233 v3 | Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights | Gan and Isola (2026), ARXIV-2603.12228. Measures the neighbourhood of pretrained weights and finds it dense with task-improving Gaussian perturbations, with the density rising monotonically with model scale — 0% of perturbations match base accuracy on GSM8K at 0.5B against 64% at 32B — and finds those perturbations to be specialists rather than generalists. The method built on it, RandOpt, post-trains in one parallel step and is competitive with PPO, GRPO and ES at equal training FLOPs, at the cost of K forward passes at inference. | Active |
| LIT-234 v2 | Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies | Hayes et al. (2026), ARXIV-2608.12679. Takes evolution strategies up to 32B and asks one question: what does post-training do to the breadth of the output distribution? RL raises pass@1 and lowers pass@k, eventually below the base model; ES raises both, and never falls below base at any k or scale tested. The mechanism is visible in the accuracy histogram — RL increases the number of prompts solved in none of k samples, putting a hard ceiling on pass@k that no extra sampling can lift. | Active |
| LIT-235 v2 | Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training | Hoy et al. (2026), ARXIV-2604.01499. Resolves the drift dispute by decomposing the ES update: an on-manifold component that changes the loss, and an off-manifold component that is loss-invariant and therefore performs a random walk whose squared norm grows as sigma^2 d T / N. In a landscape with many flat directions the second dominates — which is why ES moves two orders of magnitude further than GRPO, why the two update directions are nearly orthogonal, and why the two solutions are nevertheless linearly connected with no loss barrier. | Active |
| LIT-236 v2 | The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective | Liang et al. (2026), ARXIV-2602.00170. One geometric property accounts for two puzzles at once: fine-tuning landscapes are low-dimensional in curvature, so a handful of stiff directions carry the improvement while a near-zero bulk absorbs variance. That gives rise-then-decay dynamics under fixed hyperparameters — in GRPO as well as ES — and it gives degeneracy, where many ambient perturbations share the same projection onto the stiff subspace, so a population of about 30 keeps finding improvements from 0.5B to 7B with no rightward shift. | Active |
| LIT-237 v2 | Evolutionary Strategies lead to Catastrophic Forgetting in LLMs | Abdi et al. (2026), ARXIV-2601.20861. The first independent attempt to reproduce ES-beats-GRPO, and it does not: ES lands close but GRPO stays ahead on every task but one, at 1B and 1.5B. Its larger contribution is the forgetting curve — held-out HellaSwag falls steadily as Countdown training continues past the point where Countdown itself has converged, tracing a convex Pareto front, with ES parameter drift roughly 1000x GRPO’s and ES updates dense where GRPO’s are about 95% sparse. | Active |
| LIT-238 v2 | Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies | Schweighofer et al. (2026), ARXIV-2605.30148. Answers the forgetting result with three findings: the degradation is transient drift rather than irreversible forgetting — HellaSwag falls 8% over 300 iterations and returns to baseline by the end — it is not specific to ES, since GRPO forgets too on ProofWriter, and it is driven by the random walk Hoy et al. predicted, whose size falls with population as their equation says. Anchored Weight Decay, an L1 or L2 pull toward the initial weights applied in the update rule, buys a population-128 reduction in drift at population 30 for 1-2% runtime. | Active |
| LIT-239 v2 | ESSA: Evolutionary Strategies for Scalable Alignment | Korotyshova et al. (2025), ARXIV-2507.04453. The earliest paper in this record’s ES line, and it takes the opposite route to LIT-211: shrink the search space rather than search the full one. SFT-train LoRA adapters, take their SVD, and let CMA-ES optimize only the top singular values. On GSM8K with Qwen2.5-Math-7B it passes 0.80 accuracy in under 25 minutes against GRPO’s 50+, on 100 training examples against GRPO’s 5,978 — and it aligns a 32B model at INT4 and INT8, where backpropagation cannot run at all. | Active |
| LIT-240 v2 | ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning | Sun et al. (2026), ARXIV-2602.01003. Adds a sharpness-aware step to ES: move the parameters against the reward-weighted direction first, recompute the update at that neighbouring point, and apply it at the original — steering the solution toward flat regions. On GSM8K across Qwen2.5 at 0.5B-7B it averages 78.27% against standard ES’s 75.97%, PPO’s 77.72% and GRPO’s 78.34%, at inference-level GPU memory. | Active |
| LIT-241 | Proximal Policy Optimization Algorithms | Schulman, Wolski, Dhariwal, Radford and Klimov (2017), ARXIV-1707.06347. The baseline every paper in this record’s evolution-strategies line measures against, and the algorithm GRPO is a modification of. Its contribution is one objective: clip the probability ratio to [1-eps, 1+eps], take the minimum of clipped and unclipped so the result is a pessimistic lower bound, and you get trust-region behaviour from first-order optimization and a few lines of change. | Active |
| LIT-242 | Evolution Strategies as a Scalable Alternative to Reinforcement Learning | Salimans, Ho, Chen, Sidor and Sutskever (2017), ARXIV-1703.03864. The paper ten of the twelve papers in this record’s evolution-strategies line cite, and the source of the mechanism they all inherit: synchronize random seeds across workers and each one can reconstruct every other’s perturbation, so a parameter update costs one scalar per worker instead of a gradient. 3D humanoid walking in 10 minutes on 1,440 cores, linear speedup, and competitive Atari results in an hour. | Active |
| LIT-245 | Bayesian Learning via Stochastic Gradient Langevin Dynamics | Welling and Teh (2011), Bayesian Learning via Stochastic Gradient Langevin Dynamics. Adding correctly scaled Gaussian noise to a decaying-step-size SGD update turns the optimizer into a posterior sampler, with no accept-reject step. | Active |
| LIT-246 | Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts | Ryabinin and Gusev (2020), ARXIV-2002.04013. Learning@home: train a mixture of experts spread over unreliable volunteer hardware, routed through a DHT, so no worker holds the whole model. | Active |
| LIT-248 | Topics in propagation of chaos | Sznitman (1991), DOI:10.1007/BFb0085169. The standard treatment of propagation of chaos: N interacting particles become asymptotically independent as N grows, with the empirical measure converging at rate 1/sqrt(N). | Active |
| LIT-251 v2 | The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks | Entezari et al. (2021), ARXIV-2110.06296. Conjectures that most of the loss barrier between two independently trained networks is permutation, not genuine disagreement: account for the hidden-unit symmetry and there would likely be no barrier. The evidence is indirect, and the paper’s own permutation search did not remove the barrier for deep networks. | Active |
| LIT-252 | OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training | Jaghouar et al. (2024), ARXIV-2407.07852. OpenDiLoCo: a reproduction and open implementation of DiLoCo, trained across continents at 90-95% compute utilization. | Active |
| LIT-254 | Stochastic Gradient Push for Distributed Deep Learning | Assran et al. (2018), ARXIV-1811.10792. Stochastic gradient push over directed, time-varying graphs, with the communication overlapped against the next local step. | Active |
| LIT-257 | Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates | Yin et al. (2018), ARXIV-1803.01498. Coordinate-wise trimmed mean and coordinate-wise median as Byzantine-robust aggregators, with statistical rates that are order-optimal, and the median needs no knowledge of the corruption fraction. | Active |
| LIT-261 | The mean field analysis of the Kuramoto model on graphs II. Asymptotic stability of the incoherent state, center manifold reduction, and bifurcations | Chiba and Medvedev (2017), ARXIV-1709.08305. The companion stability analysis: the incoherent state of the graph Kuramoto model, its centre- manifold reduction, and the bifurcations at the transition. | Active |
| LIT-262 | Consensus Based Sampling | Carrillo et al. (2021), ARXIV-2106.02519. Consensus-based sampling: an interacting-particle method that produces Gaussian posterior approximations without derivatives, tuned by an interpolation parameter between optimization and sampling. | Active |
| LIT-265 | Large Batch Optimization for Deep Learning: Training BERT in 76 minutes | You et al. (2019), ARXIV-1904.00962. LAMB: normalize each layer’s update by the ratio of weight norm to update norm, which lets BERT train at batch size 32k without per-batch-size retuning. | Active |
| LIT-267 | Mean Field Analysis of Neural Networks: A Law of Large Numbers | Sirignano and Spiliopoulos (2018), ARXIV-1805.01053. The law-of-large- numbers half of the mean-field limit for single-hidden-layer networks: the empirical measure of parameters converges to a deterministic limit. | Active |
| LIT-271 | A Mean Field View of the Landscape of Two-Layers Neural Networks | Mei et al. (2018), ARXIV-1804.06561. The distribution of neurons in a wide two-layer network evolves by a Wasserstein gradient flow, so SGD on the finite network is a noisy discretization of a deterministic PDE. | Active |
| LIT-273 | Distributed Optimization with Gradient Tracking over Heterogeneous Delay-Prone Directed Networks | Makridis et al. (2025), ARXIV-2504.10964. Gradient tracking over directed networks with heterogeneous delays, with an a-priori step-size bound computed from the delay bound. | Active |
| LIT-275 | The mean field analysis for the Kuramoto model on graphs I. The mean field equation and transition point formulas | Chiba and Medvedev (2016), ARXIV-1612.06493. The mean-field limit of the Kuramoto model on a graph is an equation in the graphon, and the synchronization threshold is set by the extreme eigenvalues of that operator rather than by average degree. | Active |
| LIT-276 | Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs | Defazio et al. (2025), ARXIV-2512.17131. Replace DiLoCo’s outer optimizer with primal averaging, making the smoothing strength a continuous hyperparameter independent of the inner step count. | Active |
| LIT-278 | 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed | Tang et al. (2021), ARXIV-2102.02888. Adam’s variance term stabilizes early in training; freeze it after a warmup and the update becomes linear enough for error-compensated 1-bit compression to work. | Active |
| LIT-279 | HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent | Niu et al. (2011), ARXIV-1106.5730. Lock-free parallel SGD on shared memory: let cores overwrite each other without locks, and when the problem is sparse enough the collisions are rare and the convergence rate is nearly the same. | Active |
| LIT-280 | signSGD: Compressed Optimisation for Non-Convex Problems | Bernstein et al. (2018), ARXIV-1802.04434. Sending only the sign of each gradient coordinate is a 32x compression that still converges, and with majority-vote aggregation the reduction also applies to the return path. | Active |
| LIT-282 | Convex Analysis of the Mean Field Langevin Dynamics | Nitanda et al. (2022), ARXIV-2201.10469. Mean-field Langevin dynamics read as convex optimization in measure space, with a duality gap that can be monitored and a log-Sobolev inequality giving linear convergence. | Active |
| LIT-285 v2 | Coordination of groups of mobile autonomous agents using nearest neighbor rules | Jadbabaie, Lin and Morse (2003), DOI:10.1109/TAC.2003.812781. Nearest-neighbour heading averaging on undirected graphs converges to a common value for any switching sequence whose graphs are jointly connected across contiguous, bounded time intervals. The proof is via Wolfowitz’s ergodic-product theorem. It is asymptotic only, with no rate, and its priority is contested by Bertsekas and Tsitsiklis against their 1980s agreement results (LIT-325). The system is shown stable although no common quadratic Lyapunov function exists. | Active |
| LIT-286 | The Kuramoto model: A simple paradigm for synchronization phenomena | Acebron et al. (2005), DOI:10.1103/RevModPhys.77.137. The standard review of the Kuramoto model: the synchronization transition, the order parameter, and what noise does to the critical coupling. | Active |
| LIT-288 | Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling | Zhu et al. (2025), ARXIV-2504.11645. Tighter finite-time rates for heterogeneous federated stochastic approximation under Markovian sampling, where naive local averaging converges to the wrong fixed point. | Active |
| LIT-292 | Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms | Wang and Joshi (2018), ARXIV-1808.07576. Cooperative SGD: one framework whose special cases are local SGD, elastic averaging and decentralized SGD, so their error floors can be compared in the same terms. | Active |
| LIT-293 | Emergent Behavior in Flocks | Cucker and Smale (2007), DOI:10.1109/TAC.2007.895842. Flocking emerges unconditionally when the interaction between agents decays slowly enough with distance, and only conditionally when it decays faster. | Active |
| LIT-296 | The Variational Formulation of the Fokker-Planck Equation | Jordan et al. (1998), DOI:10.1137/S0036141096303359. The Fokker-Planck equation is the gradient flow of free energy in the Wasserstein metric, realized by an implicit variational scheme. | Active |
| LIT-297 | Reconciling modern machine learning practice and the bias-variance trade-off | Belkin et al. (2018), ARXIV-1812.11118. The double-descent curve: test error rises to a peak at the interpolation threshold and then falls again past it, so more parameters past the point of fitting the training set help rather than hurt. | Active |
| LIT-298 | On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport | Chizat and Bach (2018), ARXIV-1805.09545. Gradient descent on the neurons of an overparameterized two-layer network converges to a global optimum of the measure problem, given a separation condition on the initialization. | Active |
| LIT-301 | Products of indecomposable, aperiodic, stochastic matrices | Wolfowitz (1963), DOI:10.1090/S0002-9939-1963-0154756-3. Products of indecomposable aperiodic stochastic matrices converge to a rank-one limit — the theorem every gossip and consensus convergence proof reduces to. | Active |
| LIT-302 | Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent | Lian et al. (2017), ARXIV-1705.09056. Decentralized parallel SGD, in which workers average only with neighbours, matches centralized SGD in rate while removing the server bottleneck, and wins outright on low-bandwidth or high- latency networks. | Active |
| LIT-303 | SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient | Ryabinin et al. (2023), ARXIV-2301.11913. SWARM parallelism: pipeline stages over unreliable heterogeneous workers connected by slow links, rebalanced as workers come and go. | Active |
| LIT-304 | Consensus Problems in Networks of Agents With Switching Topology and Time-Delays | Olfati-Saber and Murray (2004), DOI:10.1109/TAC.2004.834113. Consensus on directed graphs: average consensus iff the graph is balanced, convergence rate the Fiedler value of the mirror graph, switching among balanced strongly connected graphs, and an exact bound on tolerable uniform delay. | Active |
| LIT-305 | A Bayesian Perspective on Generalization and Stochastic Gradient Descent | Smith and Le (2017), ARXIV-1710.06451. Generalization is set by a noise scale g = eps*N/B rather than by batch size alone, which is why the learning rate must rise with the batch, and the same quantity predicts an optimal batch size. | Active |
| LIT-307 | Communication-Efficient Learning of Deep Networks from Decentralized Data | McMahan et al. (2016), ARXIV-1602.05629. FedAvg: average the weights of clients that each ran several local epochs, rather than averaging one gradient per round, and the round count falls by one to two orders of magnitude. | Active |
| LIT-310 | Phase transitions, logarithmic Sobolev inequalities, and uniform-in-time propagation of chaos for weakly interacting diffusions | Delgadino et al. (2021), ARXIV-2112.06304. Uniform-in-N log-Sobolev inequalities for weakly interacting diffusions, which is what makes propagation of chaos hold uniformly in time rather than only on finite horizons. | Active |
| LIT-311 | An effective theory of collective deep learning | Arola-Fernández and Lacasa (2023), ARXIV-2310.12802. An ensemble of networks coupled through their weights undergoes a phase transition into a collective-learning regime, in which each learns classes it never saw. | Active |
| LIT-312 | Exact Solution for On-Line Learning in Multilayer Neural Networks | Saad and Solla (1995), DOI:10.1103/PhysRevLett.74.4337. The companion Letter: a closed set of order-parameter equations for online learning in a two-layer network, solved exactly. | Active |
| LIT-314 | Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction | Tyurin (2025), ARXIV-2506.23836. A lower-bound construction showing that unbiased compression cannot improve both the uplink and downlink terms at once, which bounds what centralized compression schemes can buy. | Active |
| LIT-315 | Epidemic Learning: Boosting Decentralized Learning with Randomized Communication | Vos et al. (2023), ARXIV-2310.01972. Epidemic Learning: sample a fresh random set of s neighbours each round instead of using a fixed topology, which converges faster than any static graph. | Active |
| LIT-316 | Distributed Deep Learning in Open Collaborations | Diskin et al. (2021), ARXIV-2106.10207. DeDLOC: collaborative training across volunteered hardware with adaptive averaging, keeping the run equivalent to standard large-batch SGD at a fixed global batch size. | Active |
| LIT-323 | Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD | Wang et al. (2020), ARXIV-2002.09539. Overlap Local-SGD hides the averaging latency behind local computation using an anchor model, so small H becomes affordable. | Active |
| LIT-324 | Broken Neural Scaling Laws | Caballero et al. (2022), ARXIV-2210.14891. Broken neural scaling laws: a functional form with breaks that fits and extrapolates the scaling behaviours a single power law cannot, including non-monotone and inflected curves. | Active |
| LIT-325 | Distributed asynchronous deterministic and stochastic gradient optimization algorithms | Tsitsiklis et al. (1986), DOI:10.1109/TAC.1986.1104412. The founding analysis of distributed asynchronous gradient methods: convergence holds for any finite delay bound B, and the step size must shrink with B. | Active |
| LIT-326 | A Necessary and Sufficient Condition for Consensus Over Random Networks | Tahbaz-Salehi and Jadbabaie (2008), DOI:10.1109/TAC.2008.917743. Consensus over random networks holds if and only if the expected topology has a spanning tree — the condition is on the average graph, not on each round. | Active |
| LIT-328 v2 | Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks | Bordelon et al. (2020), ARXIV-2002.02561. Learning curves for kernel regression follow a power law whose exponent is set by the decay of the kernel-target spectrum, which connects spectral structure to observed scaling exponents. | Active |
| LIT-333 v2 | Git Re-Basin: Merging Models modulo Permutation Symmetries | Ainsworth et al. (2022), ARXIV-2209.04836. Git Re-Basin: three algorithms for finding the permutation that aligns one trained network to another. After alignment their linear interpolation has zero barrier on MNIST and on sufficiently wide ResNets on CIFAR-10. It does not at standard width, and not on ImageNet. | Active |
| LIT-336 | Async-HFL: Efficient and Robust Asynchronous Federated Learning in Hierarchical IoT Networks | Yu et al. (2023), ARXIV-2301.06646. Async-HFL: hierarchical federated learning over long-tailed wireless delays, selecting devices by gradient diversity rather than by latency alone. | Active |
| LIT-337 | PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization | Vogels et al. (2019), ARXIV-1905.13727. PowerSGD compresses the gradient to a rank-k factorization found by one power-iteration step, which is all- reducible, and error feedback is what makes it converge. | Active |
| LIT-340 | Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup | Goldt et al. (2019), ARXIV-1906.08632. Exact ODEs for the order parameters of a two-layer network trained by online SGD in the teacher-student setup, extending the Saad-Solla analysis to the modern setting. | Active |
| LIT-341 | A Growing Neural Gas Network Learns Topologies | Fritzke (1994), [A Growing Neural Gas Network Learns Topologies](https://pro ceedings.neurips.cc/paper_files/paper/1994/hash/d56b9fc4b0f1be8871f5e1c40c00 67e7-Abstract.html). Growing neural gas: insert units where the accumulated error is largest, so the network finds its own size and topology without an annealing schedule. | Active |
| LIT-344 | Byzantine-Tolerant Machine Learning | Blanchard et al. (2017), ARXIV-1703.02757. Averaging gradients is not Byzantine-tolerant at any cluster size — one worker can steer the average anywhere. Krum instead selects the gradient closest to its own nearest neighbours. | Active |
| LIT-345 | Generalized EXTRA stochastic gradient Langevin dynamics | Gurbuzbalaban et al. (2024), ARXIV-2412.01993. A generalized EXTRA correction removes the network-induced bias that decentralized SGLD otherwise leaves in the sampled posterior. | Active |
| LIT-346 | On-line learning in soft committee machines | Saad and Solla (1995), DOI:10.1103/PhysRevE.52.4225. The exact learning dynamics of a soft committee machine under online gradient descent, including the symmetric plateau and the specialization that ends it. | Active |
| LIT-350 | Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism | Ramasinghe et al. (2025), ARXIV-2506.01260. Project the model-parallel communication onto a low-rank subspace updated every few hundred steps, reaching about 100x compression for pipeline-parallel decentralized training. | Active |
| LIT-353 | Emergence of stochastic flocking for the discrete Cucker-Smale model with randomly switching topologies | Dong et al. (2019), ARXIV-1912.11949. Cucker-Smale flocking under randomly switching topologies: almost-sure velocity consensus given a union-spanning- tree condition on the switching sequence. | Active |
| LIT-354 | Asynchronous Stochastic Gradient Descent with Delay Compensation | Zheng et al. (2016), ARXIV-1609.08326. Asynchronous SGD applies a gradient computed at stale parameters; a first-order Taylor correction using a cheap Hessian approximation recovers most of the accuracy the delay costs. | Active |
| LIT-360 v2 | Neural Tangent Kernel: Convergence and Generalization in Neural Networks | Jacot et al. (2018), ARXIV-1806.07572. In the infinite-width limit under standard parametrization, training follows a fixed kernel — the network is linear in its parameters around initialization, and no features are learned. | Active |
| LIT-361 | Local SGD Converges Fast and Communicates Little | Stich (2018), ARXIV-1805.09767. Local SGD keeps the linear speedup of minibatch SGD while communicating only every H = O(sqrt(T/(Kb))) steps. | Active |
| LIT-362 | Don’t Use Large Mini-Batches, Use Local SGD | Lin et al. (2018), ARXIV-1808.07217. Post-local SGD: run minibatch SGD until the first learning-rate decay and switch to local SGD after it, recovering the generalization that large-batch training loses. | Active |
| LIT-363 | ’Neural-gas’ network for vector quantization and its application to time-series prediction | Martinetz et al. (1993), DOI:10.1109/72.238311. Neural gas: adapt units by the rank of their distance to the input rather than by a fixed lattice neighbourhood, so no topology need be assumed. | Active |
| LIT-364 | Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed | Refinetti et al. (2021), ARXIV-2102.11742. A two-layer network with a few hidden units beats kernel methods on a Gaussian-mixture classification task in the high-dimensional limit, which separates feature learning from lazy training. | Deferred |
| LIT-365 | Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach | Rotskoff and Vanden-Eijnden (2018), ARXIV-1805.00915. The same mean-field limit reached as an interacting particle system, with approximation error scaling as 1/n in width rather than 1/sqrt(n). | Active |
| LIT-366 | Asynchronous Local Computations in Distributed Bayesian Learning | Bhar et al. (2023), ARXIV-2311.03496. Asynchronous decentralized Langevin sampling with several local unadjusted-Langevin steps per communication round. | Active |
| LIT-367 | ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning | Rajbhandari et al. (2021), ARXIV-2104.07857. ZeRO-Infinity offloads parameters, gradients and optimizer state to CPU and NVMe with bandwidth- centric partitioning, so model size stops being bounded by aggregate GPU memory. | Active |
| LIT-370 | Deep learning with Elastic Averaging SGD | Zhang et al. (2014), ARXIV-1412.6651. Let each worker keep its own parameters and pull it toward a centre variable with an elastic force, so workers may explore apart and communication can be infrequent. | Active |
| LIT-372 | Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices | Ryabinin et al. (2021), ARXIV-2103.03239. Moshpit SGD: average over a random grid of small groups rather than by gossip, reaching exponentially fast averaging on unreliable heterogeneous workers. | Active |
| LIT-373 | SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum | Wang et al. (2019), ARXIV-1910.00643. SlowMo: wrap any local-update or decentralized base optimizer in an outer momentum step over the averaged iterates. | Active |
| LIT-374 | LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates | Fokam et al. (2024), ARXIV-2410.05985. LAYUP: decouple the forward and backward passes into separate threads with layer-wise updates, so straggler devices stop setting the step time. | Active |
| LIT-375 | Accelerating Large Language Model Decoding with Speculative Sampling | Chen et al. (2023), ARXIV-2302.01318. The same algorithm as LIT-376, arrived at independently two months later and demonstrated at a scale the first paper did not reach: Chinchilla, 70B, in a distributed setup, 2-2.5x decoding speedup with the target distribution preserved “within hardware numerics”. Its sharpest observation is that the resulting tokens per second “often exceeds the idealised ceiling on auto-regressive sampling speed imposed by the memory bandwidth” — which is the premise of the whole family, measured. | Active |
| LIT-376 | Fast Inference from Transformers via Speculative Decoding | Leviathan et al. (2022), ARXIV-2211.17192. Decoding is serial and memory-bandwidth-bound, so the arithmetic units are idle. Speculative decoding spends them: a cheap draft model guesses gamma tokens, the target model scores all gamma + 1 positions in one parallel pass, and an accept-reject rule keeps a prefix of the guesses. The output distribution is exactly the target’s — not approximately — and the serial count can never exceed plain autoregressive decoding. 2x-3x on T5-XXL with identical outputs. | Active |
| LIT-377 v3 | Training language models to follow instructions with human feedback | Ouyang et al. (2022), ARXIV-2203.02155. InstructGPT: the paper that turned reinforcement learning from human feedback from a summarization technique into the default shape of post-training. Three steps — supervised fine-tuning on labeler demonstrations, a reward model trained on ranked outputs, then PPO against that reward with a per-token KL penalty. A 1.3B model tuned this way is preferred to 175B GPT-3, which is the result the field acted on. | Active |
| LIT-378 | QLoRA: Efficient Finetuning of Quantized LLMs | Dettmers et al. (2023), ARXIV-2305.14314. Back-propagate through a frozen 4-bit base model into 16-bit LoRA adapters: 65B fine-tuning drops from >780GB to <48GB with no loss against a 16-bit fully fine-tuned baseline. Carries a second finding the quantization headline tends to bury — LoRA on the query and value projections alone does not reach full fine-tuning at scale, and the adapter COUNT, not the rank, is what closes the gap. | Active |
| LIT-379 v2 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Akyürek et al. (2024), ARXIV-2411.07279. Temporarily update the model’s parameters at inference, using a loss built from the test instance’s own in-context examples. On ARC-AGI this is up to 6x a fine-tuned baseline — 53.0% with an 8B model, 61.9% ensembled with program synthesis, which the paper puts at average human performance. Read as evidence about in-context learning’s limits as much as about the technique. | Active |
| LIT-380 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ma et al. (2024), ARXIV-2402.17764. Every weight is ternary, {-1, 0, +1}, and the model is trained that way from scratch rather than quantized afterwards. From 3B upward it matches an FP16 LLaMA of the same size and token budget on perplexity and end-task accuracy, while the matrix multiply reduces to integer addition — a compute paradigm rather than a compression ratio. | Active |
| LIT-381 | Scaling Diffusion Language Models via Adaptation from Autoregressive Models | Gong et al. (2024), ARXIV-2410.17891. Diffusion language models are hard to train from scratch at scale and there are a great many trained autoregressive ones, so convert instead: continual pretraining on under 200B tokens turns GPT2 and LLaMA at 127M–7B into DiffuGPT and DiffuLLaMA. The conversion rests on a stated connection between the AR and diffusion objectives rather than on the two being retrained independently. | Active |
| LIT-382 | Large Language Models Cannot Self-Correct Reasoning Yet | Huang et al. (2023), ARXIV-2310.01798. The negative result the self-correction literature is answering: asked to revise its own reasoning with no external feedback, a model does not reliably improve, and often gets worse. Names the thing being tested — INTRINSIC self-correction — and that distinction is most of the paper’s lasting value. | Active |
| LIT-383 | Training Language Models to Self-Correct via Reinforcement Learning | Kumar et al. (2024), ARXIV-2409.12917. SCoRe: multi-turn online RL on entirely self-generated correction traces, which teaches the self-correction that prompting could not produce. +15.6% on MATH and +9.1% on HumanEval for Gemini 1.0 Pro and 1.5 Flash. Its more transferable half is the diagnosis of why supervised fine-tuning on correction traces fails. | Active |
| LIT-384 | VGGT: Visual Geometry Grounded Transformer | Wang et al. (2025), ARXIV-2503.11651. One feed-forward pass infers camera parameters, point maps, depth maps and 3D point tracks together, from one view or hundreds, in under a second — beating methods that post-process with geometry optimisation. The claim that matters is that the specialisation into separate tasks was never necessary. | Active |
| LIT-385 | DUSt3R: Geometric 3D Vision Made Easy | Wang et al. (2023), ARXIV-2312.14132. Reconstruct from uncalibrated, unposed images by regressing pointmaps, with no camera intrinsics or extrinsics supplied. The inversion is the content: pixel matches and relative and absolute camera come back OUT of the 3D output, where the classical pipeline demands them as inputs before it can triangulate anything. | Active |
| LIT-386 | SuperGlue: Learning Feature Matching with Graph Neural Networks | Sarlin et al. (2019), ARXIV-1911.11763. Match two sets of local features by learning the matcher rather than hand-designing it: a graph neural network predicts assignment costs and a differentiable optimal transport step solves them, finding correspondences and rejecting unmatchable points in one operation. Replaces “hand-designed heuristics” with priors over geometry learned end to end — and still consumes keypoints somebody else detected. | Active |
| LIT-387 | LoFTR: Detector-Free Local Feature Matching with Transformers | Sun et al. (2021), ARXIV-2104.00680. Drop the keypoint detector: establish dense matches coarsely and refine them, with self and cross attention producing descriptors conditioned on BOTH images. The argument is that the detector is the binding constraint — it cannot produce repeatable interest points in low-texture regions, which is exactly where matching is hard. | Active |
| LIT-388 v2 | Evaluating Large Language Models Trained on Code | Chen et al. (2021), ARXIV-2107.03374. Introduces HumanEval, which measures FUNCTIONAL correctness of generated programs against unit tests rather than textual similarity — and defines pass@k along with the unbiased estimator the record already recommends without citing. The benchmark is the better-known half; the metric is the half this anthology was leaning on. | Active |
| LIT-389 | Training Verifiers to Solve Math Word Problems | Cobbe et al. (2021), ARXIV-2110.14168. Introduces GSM8K — 8.5K grade-school math word problems, chosen because they are conceptually simple and models still failed them, which isolates multi-step reasoning from mathematical difficulty. Also introduces verifier-ranked best-of-n sampling, the ancestor of the reranking the record recommends elsewhere. | Active |
| LIT-390 | Measuring Massive Multitask Language Understanding | Hendrycks et al. (2020), ARXIV-2009.03300. MMLU: 57 subjects from elementary mathematics to law, measuring breadth of acquired knowledge rather than reasoning. Its own abstract names the insensitivity that matters most — models “frequently do not know when they are wrong”, so a high MMLU score is not evidence of calibration. | Active |
| LIT-391 | DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining | Xie et al. (2023), ARXIV-2305.10429. Train a 280M proxy under group DRO to produce domain weights with no knowledge of downstream tasks, then resample and train an 8B model with them: +6.5 points average few-shot accuracy and the baseline reached in 2.6x fewer steps. The design point is the objective — worst-case EXCESS loss against a reference model, because worst-case raw loss would chase the noisiest domain. | Active |
| LIT-392 | Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models | Chen et al. (2023), ARXIV-2307.14430. Training data has a prerequisite structure: skills are learned in an order, and training on a prerequisite makes the dependent skill reachable with less data. Formalises a skill and an ordered skill set in terms of the data that induces them, and samples over skill mixtures online. | Active |
| LIT-393 | Understanding Dropout | Baldi and Sadowski (2013), NIPS 26:2814-2822. The mathematics behind the weight-scaling rule. For a single logistic unit the normalized weighted geometric mean over all dropout configurations is exactly the unit evaluated at the expected input, so one forward pass with scaled weights computes it — and logistic and constant functions are the only functions with that property. In deep networks the same argument holds through three recursive equations of which only one is an approximation, good to second order and exact when activities are consistent across subnetworks. The same paper derives the adaptive weight-decay term, which is the other account. | Active |
| LIT-394 | Improving neural networks by preventing co-adaptation of feature detectors | Hinton et al. (2012), ARXIV-1207.0580. The three-page note that first states dropout: randomly omit half the feature detectors on each training case, and the overfitting a large network shows on a small training set is greatly reduced. The mechanism it names is co-adaptation — a detector that is only useful in the company of several specific others — and the fix is to make that company unreliable. Everything the record holds about dropout descends from this; the evidence is in LIT-395 two years later. | Active |
| LIT-395 | Dropout: A Simple Way to Prevent Neural Networks from Overfitting | Srivastava et al. (2014), JMLR 15:1929-1958. The definitive dropout paper, and the one that carries the conditions rather than just the technique: a “sweet spot” in dataset size outside which dropout buys nothing in either direction, a 2-3x training-time cost, the weight-scaling rule that makes a single test-time pass approximate the ensemble, and a practical guide (p = 0.5-0.8 hidden, 0.8 input, 10-100x the learning rate, momentum 0.95-0.99, max-norm c = 3-4) whose coupling between those knobs is the part most citations drop. | Active |
| LIT-396 | Dropout Training as Adaptive Regularization | Wager et al. (2013), ARXIV-1307.1493. For generalized linear models, dropout is first-order equivalent to L2 applied after scaling the features by an estimate of the inverse diagonal Fisher information — so the penalty it imposes is data-dependent rather than isotropic, which is what distinguishes it from weight decay. The paper also connects the result to AdaGrad, and turns it into a semi-supervised method that builds the regularizer from unlabeled data. | Active |
| LIT-397 | Deep Learning on a Data Diet: Finding Important Examples Early in Training | Paul et al. (2021), ARXIV-2107.07075. Two scores — GraNd and EL2N — that rank training examples by importance a few epochs into training rather than at the end of it, and transfer across architectures. Also the finding that the very highest-scoring examples should be excluded, more so under label noise. | Active |
| LIT-398 | DataComp-LM: In search of the next generation of training sets for language models | Li et al. (2024), ARXIV-2406.11794. A controlled testbed for data curation — a fixed 240T-token corpus, fixed training recipes at 412M–7B, and 53 evaluations — whose baseline finding is that model-based filtering is what makes a high-quality training set. The position two documents in this record already argue with. | Active |
| LIT-399 | Beyond neural scaling laws: beating power law scaling via data pruning | Sorscher et al. (2022), ARXIV-2206.14486. Power-law scaling in dataset size can be beaten — reduced toward exponential — if you have a good ranking of which examples to discard and you prune harder as the corpus grows. Which end of the ranking to discard inverts with data abundance. | Active |
| LIT-400 | Estimating Training Data Influence by Tracing Gradient Descent | Pruthi et al. (2020), ARXIV-2002.08484. Attributes a prediction to training data by tracing the actual optimization: how much did the loss on the test point move on the steps where this training example was used? Needs only gradients, saved checkpoints and a loss — no Hessian, no convexity. | Active |
| LIT-401 | Studying Large Language Model Generalization with Influence Functions | Grosse et al. (2023), ARXIV-2308.03296. Scales influence functions to language models up to 52B parameters with an EK-FAC approximation to the inverse-Hessian-vector product, then uses them to ask what training data a behaviour came from. Generalization gets more abstract with scale — and stays startlingly sensitive to word order. | Active |
| LIT-402 | If Influence Functions are the Answer, Then What is the Question? | Bae et al. (2022), ARXIV-2209.05364. Decomposes the gap between influence estimates and leave-one-out retraining into five terms, finds that three of them are not errors at all, and identifies what influence functions on neural networks do estimate: the proximal Bregman response function. | Active |
| LIT-403 | Understanding Black-box Predictions via Influence Functions | Koh and Liang (2017), ARXIV-1703.04730. Traces a prediction back through the learning algorithm to the training points responsible for it, using a classical technique from robust statistics made tractable by needing only gradients and Hessian-vector products. The origin of training-data attribution in deep learning. | Active |
| LIT-404 | HellaSwag: Can a Machine Really Finish Your Sentence? | Zellers et al. (2019), ARXIV-1905.07830. Commonsense sentence completion whose wrong answers are generated and then filtered against the models of the day until those models fail — so the benchmark’s difficulty is defined relative to a specific generation of systems, which is the caveat to carry. | Active |
| LIT-405 | On the Measure of Intelligence | Chollet (2019), ARXIV-1911.01547. Argues that measuring skill measures the wrong thing, and introduces ARC — a puzzle benchmark built so that every task is structurally novel, which is what makes it resistant to the memorization that saturates ordinary benchmarks. | Active |
| LIT-406 | Qwen2.5 Technical Report | Qwen Team (2024), ARXIV-2412.15115. The open-weight family this record measures against more than any other — forty-five documents here name it, and until now none of them could point at it. | Active |
| LIT-407 | Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge | Clark et al. (2018), ARXIV-1803.05457. Grade-school science questions split into an Easy set and a Challenge set, the latter selected because retrieval and word-co-occurrence baselines both fail on it. The “ARC” that appears in this record’s evaluation suites — and not the one SOTA-235 reports on. | Active |
| LIT-408 | Program Synthesis with Large Language Models | Austin et al. (2021), ARXIV-2108.07732. Introduces MBPP — Mostly Basic Python Problems — short crowd-sourced tasks each with three test cases, the entry-level counterpart to HumanEval and the other half of the pair this record’s code-generation results are reported on. | Active |
| LIT-409 v2 | Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs | Feucht et al. (2024), ARXIV-2406.20086, EMNLP 2024. The last token of a multi-token word or named entity shows a pronounced “erasure”: probes that recover neighbouring and current token identity elsewhere fail there, in early layers specifically. The paper reads that as the footprint of a mechanism forming lexical items, scores sequences for “lexicality” and reads out an LLM’s implicit vocabulary — the first attempt to do so. Its framing is the sharp part for this record: lexical items are units of meaning that are non-compositional, and the example given is a multi-word expression. | Active |
| LIT-410 v2 | The English phrase-as-lemma construction: When a phrase masquerades as a word, people play along | Goldberg and Shirtz (2025), Language 101(2):291-320. English has a construction — a don't-mess-with-me driver — in which a unit with the internal syntax of a phrase, up to a whole sentence, occupies a slot reserved for a word, and speakers treat it as a word: four preregistered surveys (N = 685 plus 70) find such sentences judged to presuppose more shared common knowledge, and to be wittier and more sarcastic, than normed near-paraphrases. The effect survives restricting to high-frequency phrases and shows no frequency dependence. No model was run, and the finding is about the signal rather than anything trained on it: it sources THEORY-022, the claim that a lexical unit is delimited by construction rather than by frequency — which is what SOTA-007’s vocabulary is fit to. See ADR-045 for the scope call it forced. | Active |
| LIT-411 | Measuring Mathematical Problem Solving With the MATH Dataset | Hendrycks et al. (2021), ARXIV-2103.03874. 12,500 competition mathematics problems, each with a full step-by-step solution — which is why the dataset became a training signal as well as a benchmark, and why a result on it needs to say which it was used as. | Active |
| LIT-412 | From Tokens to Words: On the Inner Lexicon of LLMs | Kaplan et al. (2024), ARXIV-2410.05864, ICLR 2025. LLMs recombine sub-word sequences into whole-word representations at the word’s last token, mostly in early and middle layers, and the effect is robust to arbitrary splits, typos and words the tokenizer never had. The control is what makes it: a probe on the last token reaches 89% word-vs-nonword accuracy, the same probe on the penultimate token only 61%, so it is the completed word rather than token co-occurrence. Turned into a vocabulary-expansion method whose core parameters stay frozen. | Active |
| LIT-413 | Attention Is Off By One | Miller (2023), a blog post. Proposes softmax₁(x)ᵢ = exp(xᵢ)/(1 + Σⱼ exp(xⱼ)) — one added to the denominator — so a head can decline to attend at all, and calls the result QuietAttention. It reports no experiments and says so: “I’m 99.44% sure that it will resolve the outlier feedback loop” is the entirety of the evidence. Filed because it is the clearest statement of the mechanism the record holds in THEORY-019 and because it is what most people have actually read, not because it established anything. | Active |
| LIT-414 | Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing | Bondarenko et al. (2023), ARXIV-2306.12929, NeurIPS 2023. The diagnosis the attention-sink literature rests on, and two architectural fixes with the experiment attached. A head trying to learn a no-op must drive its softmax input ever larger to approximate exact zeros, which is what creates the outliers that break quantization. Clipped softmax and gated attention each remove them: BERT-base goes from 1294 W8A8 perplexity to 4.55, with max infinity-norm from 735 to 20. Its gated attention is a per-head sigmoid on the attention output — the recommendation SOTA-134 makes, two years earlier, as Qiu et al. themselves say. | Active |
| LIT-415 | Stream of Search (SoS): Learning to Search in Language | Gandhi et al. (2024), ARXIV-2404.03683. Filed for its task, not its method. Countdown — combine input numbers with arithmetic operations to reach a target — is the verifiable puzzle twenty-three documents here report results on, and this is the paper that put it in front of this literature. | Active |
| LIT-416 | GPQA: A Graduate-Level Google-Proof Q&A Benchmark | Rein et al. (2023), ARXIV-2311.12022. 448 multiple-choice questions written by domain PhDs, validated so that skilled non-experts with unrestricted web access reach 34% against experts’ 65%. The gap is the instrument — and it is small enough that the benchmark’s own noise floor is worth knowing. | Active |
| LIT-417 | The Remarkable Robustness of LLMs: Stages of Inference? | Lad et al. (2024), ARXIV-2406.19384. Deleting a layer outright, or swapping two adjacent ones, leaves 72-95% of top-1 predictions unchanged with no fine-tuning — but only in the middle. Interventions on the first and last layers degrade badly, and swapping hurts less than dropping. The paper reads that localized sensitivity as four stages of inference, with the question mark in its own title: detokenization, feature engineering, prediction ensembling, residual sharpening. The record files the measurement and the framework as separate claims because their evidence is not the same strength. | Active |
| LIT-418 | HybridFlow: A Flexible and Efficient RLHF Framework | Sheng et al. (2024), ARXIV-2409.19256. The paper behind verl — a hybrid single/multi-controller execution model for the RLHF dataflow, with a 3D resharding engine between the training and generation phases. Named in ADR-032 as one of the four references at the top of the backlog it opened. | Active |
| LIT-419 | OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems | He et al. (2024), ARXIV-2402.14008. 8,476 olympiad-level maths and physics problems, bilingual and partly multimodal, on which the best model at publication scored 17.97% — a floor-effect instrument that the record uses to separate sub-8B models, which is the caveat to carry. | Active |
| LIT-420 | Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them | Suzgun et al. (2022), ARXIV-2210.09261. The 23 BIG-bench tasks where models had not yet beaten the average human rater, split out as BBH — a suite selected by a model generation’s shortfall, and one whose headline finding is that the shortfall was partly a prompting artifact. | Active |
| LIT-421 | Gradient-based learning applied to document recognition | LeCun et al. (1998), Proceedings of the IEEE. The paper MNIST comes from. In this record it is cited for the dataset alone, and almost always in one role: the small convex-ish problem where a claim about influence, attribution or retraining can be checked against ground truth. | Active |
| LIT-422 | ImageNet: A large-scale hierarchical image database | Deng et al. (2009), CVPR. The database itself — millions of images organised on the WordNet noun hierarchy. Almost nothing in this record is measured on it: an “ImageNet” number here is nearly always ILSVRC (LIT-423), the thousand-class challenge built from a slice of it. | Active |
| LIT-423 | ImageNet Large Scale Visual Recognition Challenge | Russakovsky et al. (2014), ARXIV-1409.0575. The thousand-class challenge built on a slice of LIT-422’s database, and the instrument almost every “ImageNet top-1” number in this record is actually measured on — including the human-error figure those numbers are read against. | Active |
| LIT-424 | Learning Multiple Layers of Features from Tiny Images | Krizhevsky (2009), University of Toronto technical report. The source of CIFAR-10 and CIFAR-100 — 32×32 labelled images drawn from the Tiny Images collection. Sixty-nine documents here measure something on it, which makes it the record’s most-used instrument after ILSVRC. | Active |
| LIT-425 v2 | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | Raffel et al. (2019), ARXIV-1910.10683. T5’s paper, and the origin of C4 — the heuristically filtered Common Crawl slice that forty-six documents here train on. Filed for the corpus: the record cites C4 constantly and had no document behind it. | Active |
| LIT-426 v2 | The Pile: An 800GB Dataset of Diverse Text for Language Modeling | Gao et al. (2020), ARXIV-2101.00027. 800GB assembled from 22 named sub-corpora with declared weights. The named domains and the default mixture are why the data-mixing line in this record is measured on it — and every “over The Pile’s default weights” number is relative to a choice its authors made, not to an optimum. | Active |
| LIT-427 | SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model | Ben Allal et al. (2025), ARXIV-2502.02737. The model report behind SmolLM2-135M-Instruct, which is the base LIT-129’s whole experiment runs on — and the source of the overtraining ratio that makes a 135M model worth fine-tuning at all. | Active |
| LIT-428 | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | Katharopoulos et al. (2020), ARXIV-2006.16236. Where linear attention comes from, and the identity the record argues about in twelve documents without being able to cite it: drop the softmax for a kernel feature map, associativity reorders the product to O(N), and the autoregressive case is then literally an RNN. | Active |
| LIT-429 | Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence | Peng et al. (2024), ARXIV-2404.05892. RWKV-5 and RWKV-6, and the two changes the line is still built on: the state becomes matrix-valued rather than vector-valued, and the decay becomes a function of the input rather than a learned constant. | Active |
| LIT-430 | RWKV: Reinventing RNNs for the Transformer Era | Peng et al. (2023), ARXIV-2305.13048. The paper the RWKV line starts from, and the first to take the architecture to 14B: AFT’s position bias becomes an exponential decay over relative position, which makes the layer trainable in parallel and runnable as a constant-memory recurrence. | Active |
| LIT-431 | An Attention Free Transformer | Zhai et al. (2021), ARXIV-2105.14103. Drops the query-key product entirely: keys and values are combined with a learned position bias and the query enters element-wise. The record holds it because it is where RWKV’s letters come from — R, W, K, V is this construction renamed. | Active |
| LIT-432 | Finetuned Language Models Are Zero-Shot Learners | Wei et al. (2021), ARXIV-2109.01652. FLAN, and the name “instruction tuning”: fine-tune on 60+ existing NLP tasks rewritten as natural-language instructions and zero-shot performance on unseen task types improves. The rival post-training answer to LIT-377’s, and the one InstructGPT measured itself against. | Active |
| LIT-433 | Learning to summarize from human feedback | Stiennon et al. (2020), ARXIV-2009.01325. The step where the preference loop crosses into language: human comparisons between summaries, a reward model fit to them, PPO against that reward. The summaries beat the human reference summaries, and beat much larger models fine-tuned supervised — which is the result that made the loop look like a general answer rather than a control technique. | Active |
| LIT-434 | Deep reinforcement learning from human preferences | Christiano et al. (2017), ARXIV-1706.03741. The construction the record’s whole post-training spine assumes and could not cite: when the reward function is the thing you cannot write down, fit one to human comparisons instead and optimize against the fit. Atari and simulated locomotion, on feedback covering under 1% of the agent’s interactions. | Active |
| LIT-435 | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis | Mildenhall et al. (2020), ARXIV-2003.08934. The coordinate network the record’s one scene-representation practice exists to replace. An MLP maps a 5D coordinate — position and viewing direction — to density and view-dependent colour; differentiable volume rendering means posed images are the only supervision needed. | Active |
| LIT-436 | Scalable Optimization in the Modular Norm | Large et al. (2024), ARXIV-2405.14813. Generalises the per-layer natural norm to the whole weight space: the modular norm is defined recursively alongside the architecture. Normalising any base optimizer’s updates in it makes the learning rate transferable across width and depth, with no optimizer-specific scale factors. | Active |
| LIT-437 | A Spectral Condition for Feature Learning | Yang et al. (2023), ARXIV-2310.17813. Feature learning at large width follows from scaling the spectral norm of weights and their updates like sqrt(fan-out/fan-in) — not from the Frobenius-norm and entry-size heuristics in common use. It also yields an elementary derivation of muP. | Active |
| LIT-438 | Modular Duality in Deep Learning | Bernstein and Newhouse (2024), ARXIV-2410.21265. The gradient is a dual vector and should not be subtracted from the weights without a duality map. Modular dualization builds that map recursively from per-layer operator norms; the Linear case is a rectangular Newton-Schulz iteration, and a variant of it set NanoGPT speed records — which is where Muon comes from. | Active |
| LIT-439 | Deep Unsupervised Learning using Nonequilibrium Thermodynamics | Sohl-Dickstein et al. (2015), ARXIV-1503.03585. Where diffusion models come from: destroy structure with a slow forward process, learn the reverse, and get a generative model that is flexible and tractable — a trade the paper frames as the central problem it is solving. | Active |
| LIT-440 | How much do language models memorize? | Morris et al. (2025), ARXIV-2505.24832. Separates what a model stores about a specific dataset from what it stores about the data-generating process, and measures the first by training on uniform random bitstrings where the second is impossible. GPT-style transformers hold about 3.6 bits per parameter, near-independent of precision, and the capacity is linear in parameter count across hundreds of models from 500K to 1.5B. | Active |
| LIT-441 | Pre-training under infinite compute | Kim et al. (2025), ARXIV-2509.14786. Under a fixed corpus and no compute limit, the standard recipe of more epochs and more parameters overfits — and the fix is regularization, at roughly 30x the customary weight decay. Ensembling independently trained models then beats parameter scaling outright, and distilling the ensemble keeps most of the gain. The paper also proposes judging such recipes by their scaling law’s asymptote rather than by loss at a budget. | Active |
| LIT-442 | Diffusion Beats Autoregressive in Data-Constrained Settings | Prabhudesai et al. (2025), ARXIV-2507.15857. Refits Muennighoff’s data-constrained scaling law for masked diffusion and gets a data-reuse half-life of 512 epochs against autoregressive training’s 32. Under a fixed corpus, AR wins at Chinchilla-optimal compute and then overfits; diffusion starts far worse and keeps improving to a lower final loss. A closed-form critical-compute threshold says which side of the crossover you are on. | Active |
| LIT-443 | Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training | Bergsma et al. (2025), ARXIV-2505.13738. Hundreds of muP-trained models say the AdamW timescale — not the weight decay itself — is the quantity that scales, and it follows a power law in tokens-per-parameter with exponent about -0.52. Optimal and critical batch size both scale as power laws in the token budget D, independently of model size, which is not what Kaplan’s compute exponent says. | Active |
| LIT-444 | Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful | Marek et al. (2025), ARXIV-2507.07101. Batch size one trains stably, and the reason small batches looked unstable is that beta_2 was held fixed while the batch changed. Hold the second moment’s half-life fixed in tokens instead and small batches match or beat large ones per FLOP, are far more robust to hyperparameter choice, and let plain SGD without momentum match AdamW at 1.3B. | Active |
| LIT-445 | How Does Critical Batch Size Scale in Pre-training? | Zhang et al. (2024), ARXIV-2410.21676. Decouples model size from data size, which prior batch-size studies had scaled together, and finds critical batch size tracks the token budget and is almost flat in model size. Backed by an argument from muP — past a width, more width does not raise it — and by a least-squares analysis giving the same shape. 85M to 1.2B on C4. | Active |
| LIT-446 | Variational Diffusion Models | Kingma et al. (2021), ARXIV-2107.00630. Rewrites the diffusion VLB as an integral over signal-to-noise ratio, and the expression collapses: in continuous time the bound depends on the noise schedule only through its two endpoints. Variance-preserving and variance-exploding specifications are therefore equivalent up to a rescaling, and the schedule’s shape is free — spent here on minimizing the loss estimator’s variance. | Active |
| LIT-447 v2 | SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers | Ma et al. (2024), ARXIV-2401.08740. Holds DiT’s architecture, parameter count and GFLOPs fixed and varies four transport choices one at a time: discrete vs continuous time, score vs velocity prediction, the interpolant connecting the distributions, and the sampler. Velocity prediction and a linear interpolant each help substantially; the sampler’s diffusion coefficient turns out to be choosable after training. | Active |
| LIT-448 | Scalable Diffusion Models with Transformers | Peebles and Xie (2022), ARXIV-2212.09748. Replaces the U-Net backbone every diffusion model had inherited from DDPM with a transformer over latent patches, and shows FID falls monotonically with forward-pass GFLOPs however the GFLOPs are bought — depth, width or token count. DiT-XL/2 reaches 2.27 FID on class-conditional ImageNet 256. | Active |
| LIT-449 v2 | Scaling Rectified Flow Transformers for High-Resolution Image Synthesis | Esser et al. (2024), ARXIV-2403.03206. A 61-way comparison of diffusion and rectified-flow formulations settles on a straight-line path with timesteps drawn from a logit-normal, then scales it to 8B for text-to-image. Adds a resolution-dependent shift of the timestep schedule, and MMDiT, which gives each modality its own weights while letting the two token streams attend jointly. | Active |
| LIT-450 v3 | How do language models learn facts? Dynamics, curricula and hallucinations | Zucchet et al. (2025), ARXIV-2503.21676. Factual recall is learned in three phases with a long plateau between generic statistics and individual knowledge, and the plateau is the attention recall circuit forming — shown by patching a reference model’s attention patterns in, at which point the plateau disappears. Imbalanced data shortens the plateau, up to a point, and the paper argues it slows acquisition afterwards; a warm-up on a subset of individuals, then all of them, beat the fixed distributions tried (single seed). | Active |
| LIT-451 | Data Mixing Can Induce Phase Transitions in Knowledge Acquisition | Gu et al. (2025), ARXIV-2505.18091. Knowledge acquired from a knowledge-dense dataset scales linearly with model size when that dataset is trained on alone — and stops doing so once it is mixed into web text. Below a critical mixing ratio the model memorises almost nothing however long it trains; below a critical model size, likewise. The critical ratio follows a power law in model size, so the transitions are predictable. | Active |
| LIT-452 | Scaling Laws for Fact Memorization of Large Language Models | Lu et al. (2024), ARXIV-2406.15720. Fact capacity scales linearly with model size and saturates in training epochs along a negative exponential. Extrapolated, memorising all 15B Wikidata triples wants about 1000B non-embedding parameters trained for 100 epochs. Redundant facts cost full price unless they share direction and structure, so parameters are an expensive place to keep a knowledge base. | Active |
| LIT-453 v2 | Understanding Optimization in Deep Learning with Central Flows | Cohen et al. (2024), ARXIV-2410.24206. The trajectory an optimizer actually takes oscillates at the edge of stability; its time-average is a differential equation — the central flow — that predicts the real trajectory numerically. Reading it: at the edge of stability the learning rate does not set the step size, and adaptive optimizers do not merely adapt to curvature, they steer away from it so they can take larger steps. | Active |
| LIT-454 | The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion | Kunin et al. (2021), ARXIV-2107.09133. Long after the loss has converged the weights keep moving, with distance travelled growing as a power law with a non-trivial exponent. Derives SGD at finite learning rate and batch size as an underdamped Langevin equation and shows what drives the motion is not the training loss but a modified loss plus probability currents that make the trajectory oscillate rather than diffuse randomly. | Active |
| LIT-455 | Hidden Breakthroughs in Language Model Training | Kangaslahti et al. (2025), ARXIV-2506.15872. A smooth loss curve is not evidence of smooth training. POLCA decomposes the change in loss along a low-rank basis of the training subspace and per example, and the breakthroughs the aggregate hides come back — on synthetic arithmetic it recovers the skill of carrying a digit, which clustering the exact loss curves does not. | Active |
| LIT-456 | Muon is Not That Special: Random or Inverted Spectra Work Just as Well | Shumaylov et al. (2026), ARXIV-2605.11181. Replace the gradient’s singular values with chaotic noise and the optimizer still matches Muon. The geometric account of why spectral optimizers work does not survive its own control; what survives is that Muon’s optimal step size is constant. | Active |
| LIT-457 v2 | When do spectral gradient updates help in deep learning? | Davis and Drusvyatskiy (2025), ARXIV-2512.04299. A spectral update beats a Euclidean one on a block exactly when the gradient’s squared nuclear-to-Frobenius ratio exceeds the stable rank of that block’s incoming activations — a measurable condition, and they show it is satisfied throughout transformer training. | Active |
| LIT-458 v2 | The Platonic Representation Hypothesis | Huh et al. (2024), ARXIV-2405.07987. Representations are converging: across architectures, objectives and modalities, and more so as models get larger. The conjecture is that they are converging on a statistical model of whatever generated the data. The mathematical argument holds only for bijective observations, which the authors say plainly and test at the boundary. | Active |
| LIT-459 | Contextures: The Mechanism of Representation Learning | Zhai (2025), ARXIV-2504.19792 — a CMU dissertation. A representation is learned from the association between the input and a context variable; an encoder capturing the maximum information of that association is optimal on tasks compatible with the context. A context is most useful when the association is neither too strong nor too weak, and mixing contexts is a cheap way to get there. | Active |
| LIT-460 v2 | The Lattice Representation Hypothesis of Large Language Models | Xiong (2026), ARXIV-2603.01227. If concepts are linear directions with thresholds, each concept is a half-space, and intersecting half-spaces gives a concept lattice — so conceptual inclusion, intersection and union become geometric meet and join. Tested on WordNet sub-hierarchies. The Linear Representation Hypothesis gives binary concepts; this is the set-theoretic structure over them. | Active |
| LIT-461 | Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability | Cohen et al. (2021), ARXIV-2103.00065. Full-batch gradient descent drives the maximum Hessian eigenvalue up until it reaches 2/eta, then trains there: the loss goes non-monotone over short timescales and keeps falling over long ones. The paper the record has been treating as established premise without holding it. | Active |
| LIT-462 v2 | Spectral Condition for μP under Width-Depth Scaling | Zheng et al. (2026), ARXIV-2603.00541. Extends the spectral condition for muP from width to width-and-depth, and finds the rule is set by how many transformations a residual branch contains: one gives Depth-muP, two or more gives CompleteP. Transformers are the second case. | Active |
| LIT-463 v2 | Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity | Lawsen (2025), ARXIV-2506.09250. Argues the reported accuracy collapse is the evaluation: output-token limits, a scorer that cannot tell refusal from failure, and River Crossing instances that have no solution yet count as failures. One claim is a fact; the rest is argued. | Active |
| LIT-464 | Theoretical limitations of multi-layer Transformer | Chen et al. (2024), ARXIV-2412.02975. The first unconditional lower bound for a decoder-only transformer with more than one layer: any constant-depth decoder needs polynomially many parameters to compose functions sequentially. Depth, an encoder, or chain of thought each buy what width cannot. | Active |
| LIT-465 | Auxiliary task demands mask the capabilities of smaller language models | Hu and Frank (2024), ARXIV-2404.02418. Across four capacities, the higher-demand way of asking scores lower than the lower-demand way of asking the same question — and the gap is widest for the smallest and least-trained models, so demands are mistaken for missing capability. COLM 2024. | Active |
| LIT-466 | The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity | Shojaee et al. (2025), ARXIV-2506.06941. Controllable puzzles instead of contaminated benchmarks, and reasoning models collapse to zero accuracy past a complexity threshold while reducing their thinking tokens. NeurIPS 2025; the collapse reading is publicly contested and the three-regime finding is not. | Active |
| LIT-467 | Chain-of-Thought Prompting Elicits Reasoning in Large Language Models | Wei et al. (2022), ARXIV-2201.11903. Put worked reasoning steps in the few-shot exemplars and a 540B model more than doubles on GSM8K. Three ablations rule out the obvious alternative explanations, and the effect is absent — sometimes negative — below about 100B parameters. | Active |
| LIT-468 | Self-Consistency Improves Chain of Thought Reasoning in Language Models | Wang et al. (2022), ARXIV-2203.11171. Replace greedy decoding with sample-and-marginalize: draw several reasoning paths, take the most common final answer. GSM8K +17.9, SVAMP +11.0, AQuA +12.2. The gain saturates by about five or ten samples. | Active |
| LIT-469 | Large Language Models are Zero-Shot Reasoners | Kojima et al. (2022), ARXIV-2205.11916. One fixed sentence — “Let’s think step by step” — with no exemplars at all takes MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. It beats eight-shot standard prompting and still loses to hand-written few-shot chains. | Active |
| LIT-470 | Emergent Abilities of Large Language Models | Wei et al. (2022), ARXIV-2206.07682. Names and defines emergent abilities — present in larger models, absent in smaller ones, and therefore not predictable by extrapolating a scaling law. A survey of existing curves rather than new experiments: eight few-shot examples across five model families, plus prompting strategies that only help past a threshold. Section 5.1 raises the metric explanation itself and argues against it on two specific grounds. | Active |
| LIT-471 | Are Emergent Abilities of Large Language Models a Mirage? | Schaeffer et al. (2023), ARXIV-2304.15004. A sharp capability curve can be produced from a smooth one by the metric alone. Rescoring fixed GPT-3 outputs under Token Edit Distance removes the emergence; >92% of hand-annotated BIG-Bench emergent abilities sit under two metrics, one nonlinear and one discontinuous; and emergence is induced on demand in vision models that had never shown it. | Active |
| LIT-472 | nGPT: Normalized Transformer with Representation Learning on the Hypersphere | Loshchilov et al. (2024), ARXIV-2410.01131. Normalize every embedding-dimension vector in every matrix to the unit hypersphere after each step, delete all normalization layers, weight decay and LR warmup, and make each block’s residual contribution a learnable per-dimension step size. 4× / 10× / 20× fewer tokens to the same loss at 1k / 4k / 8k context, at 0.5B and 1B on OpenWebText — and 60–80% more time per step. | Active |
| LIT-473 | Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost | Xu, Miikkulainen and Qiu (2026), ARXIV-2602.03120. Fine-tune a quantized model in its own integer space: accumulate the part of each evolution-strategies update that is smaller than the lattice spacing until it crosses a grid point, and rematerialize that accumulator from stored seeds so it costs kilobytes instead of an FP16 copy of the weights. Countdown, Qwen2.5 at 1.5B and 3B, INT4/INT8/W8A8. | Active |
| LIT-474 | The Coverage Principle: How Pre-Training Enables Post-Training | Chen, Huang, Golowich, Malladi, Block, Ash, Krishnamurthy and Foster (2025), ARXIV-2510.15020. Names the quantity cross-entropy is a bad proxy for: the coverage profile, the probability mass a model puts on rare high-quality responses. Proves it is necessary and sufficient for Best-of-N to succeed, that next-token prediction optimizes it faster than it optimizes cross-entropy, and that sequence-level KL carries a linear dependence on sequence length that coverage does not. | Active |
| LIT-475 | Dropout Reduces Underfitting | Liu, Xu, Jin, Shen and Darrell (2023), ARXIV-2303.01500. Dropout applied only for the first stretch of training and then switched off lowers training loss and raises test accuracy on models too small to overfit — the regime where standard dropout costs up to 6 points. The mechanism is measured: early dropout reduces the angle between mini-batch gradients and the whole-dataset gradient. | Active |
| LIT-476 | Estimating the Probability of Sampling a Trained Neural Network at Random | Scherlis and Belrose (2025), ARXIV-2501.18812. An estimator for the measure, under the initialization distribution, of the region around a trained network whose behaviour matches it. Negative log of that measure is a description length. A ConvNeXt deliberately trained to generalize badly has a smaller such region — detectable on clean held-out data where its behaviour is indistinguishable. | Active |
| LIT-477 | NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation | Li, Teh and Pascanu (2025), ARXIV-2503.24322. Train each block independently to denoise a noisy label embedding given the raw input — diffusion machinery pointed at a classifier. No forward or backward pass across the network at training time, about half the GPU memory, and test accuracy matching backprop on MNIST and CIFAR-10. Every block sees the input, which is what makes independence possible and what makes the comparison awkward. | Active |
| LIT-478 v3 | Neural networks with late-phase weights | von Oswald, Kobayashi, Meulemans, Henning, Grewe and Sacramento (2020), ARXIV-2007.12927. Late in training, replicate a small subset of the weights — BatchNorm’s scale and shift will do — train K copies that share everything else, and average them into one model at the end. Inference cost unchanged. WRN 28-10 on CIFAR-100: 81.35 → 82.87. Starting the same procedure at initialization instead fails to beat the baseline. | Active |
| LIT-479 v2 | The Diffusion Duality | Sahoo, Deschenaux, Gokaslan, Wang, Chiu and Kuleshov (2025), ARXIV-2506.10892. Uniform-state discrete diffusion is the argmax of a Gaussian diffusion, which lets Gaussian techniques transfer: a variance-reducing curriculum doubles training speed, and consistency distillation cuts sampling from 1024 steps to 8. Masked diffusion still wins on likelihood; this wins in the few-step regime, where masked models cannot revise what they have already emitted. | Active |
| LIT-480 | Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X During the 2024 U.S. Presidential Election | Ye, Luceri and Ferrara (2024), ARXIV-2411.01852. 120 sock-puppet accounts, six weeks, 9.79M tweets from Twitter/X’s “For You” timeline. The method is the transferable part: puppets that never interact, follow sets controlled to the subject, and exposure weighted by rank rather than counted. Accounts following nobody get the most diverse recommendations, so personalization concentrates rather than broadens. | Active |
| LIT-481 | Industry Influence in High-Profile Social Media Research | Bak-Coleman, West, O’Connor and Bergstrom (2026), ARXIV-2601.11507. 49% of social media papers in Science, Nature and PNAS have a disclosable industry tie and most go undisclosed — from 21% of authors, so a concentrated group rather than broad engagement. Ties are over-represented in research on what users share and sparse in research on platform dynamics. | Active |
| LIT-482 | Moving towards informative and actionable social media research | Bak-Coleman, Lewandowsky, Lorenz-Spreen, Narayanan, Orben and Oswald (2025), ARXIV-2505.09254. Individual-level randomized trials do not estimate collective effects, for four named reasons — non-linearity across scale, hysteresis, feedback in time, and SUTVA violation through the network. A null from such a trial is not evidence of no effect, and the positive recommendation is to evaluate specific interventions instead of net effects. | Active |
| LIT-483 | Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models | Liang et al. (2024), ARXIV-2411.04996. Untie every non-embedding parameter by modality — feed-forward, attention projections, layer norms — and keep global self-attention over the whole sequence. Matches a dense baseline at 55.8% of the FLOPs in the Chameleon setting and 37.2% with speech added. The component ablation is the useful part: the feed-forward carries most of it, attention adds less, layer norms add nothing. | Active |
| LIT-484 v2 | TinyStories: How Small Can Language Models Be and Still Speak Coherent English? | Eldan and Li (2023), ARXIV-2305.07759. Restrict the training corpus to what a three-year-old’s vocabulary can say and a model below 10M parameters — or one with a single transformer block — writes fluent, grammatical, multi-paragraph stories. The finding is about the corpus, not the architecture: what stops small models speaking is the breadth of what they are asked to learn, not the complexity of language. | Active |
| LIT-485 | Parameterized Synthetic Text Generation with SimpleStories | Finke et al. (2025), ARXIV-2504.09184. Sampling three words from a child’s vocabulary was not enough: 59.38% of TinyStories begins “once upon a time”. Parameterize the generating prompt by theme, topic, style, narrative feature, grammar feature and author persona, and constrain the opening part of speech and letter, and every diversity metric moves. The one clean ablation in the paper is about something else — a 4,096-token domain tokenizer beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality at fixed data and architecture. | Active |
| LIT-486 | Parallel Scaling Law for Language Models | Chen et al. (2025), ARXIV-2505.10475. Run the same weights over P learnably-prefixed copies of the input and aggregate the outputs: the loss falls as if the parameter count had risen by O(log P). Fitted across 0.5B–4.4B and P = 1…8 on two corpora. At batch size 1, matching a parameter-scaled model costs 22× less added memory and 6× less added latency — and P× the training FLOPs, which is the real price. | Active |
| LIT-487 | Introducing AI to an Online Petition Platform Changed Outputs but not Outcomes | Corpus et al. (2025), ARXIV-2511.13949. Change.org’s “write with AI” tool reached the US, GB and Canada eleven weeks before Australia. Across 1.5M petitions, access made petitions 49% longer, more lexically varied, 1.5 grades harder to read, and rated higher on quality and persuasiveness — moving them toward the profile of successful pre-AI petitions — while raising inter-petition similarity 23% and leaving outcomes flat or worse. The features that used to predict success stopped predicting it. | Active |
| LIT-488 | gen2seg: Generative Models Enable Generalizable Instance Segmentation | Khangaonkar and Pirsiavash (2025), ARXIV-2505.15263. Finetune Stable Diffusion or MAE end-to-end — encoder and decoder — for category-agnostic instance segmentation on indoor furnishings and cars only, and the model segments people, animals, x-rays and impressionist paintings. It approaches SAM across five domains on 0.3% of the annotated masks, and beats it ~3× on fine structures. The controls are the argument: the same backbone under a standard segmentation head scores near zero. | Active |
| LIT-489 | Measuring In-Context Computation Complexity via Hidden State Prediction | Herrmann, Csordás and Schmidhuber (2025), ARXIV-2503.13431. Next-token loss cannot tell hard from interesting: random noise scores high and requires no computation, memorized recitation scores low. Insert a variational bottleneck with a learned autoregressive prior mid-network and measure the KL between posterior and prior — the nats of hidden-state information the past did not predict. That quantity separates in-context learning from memorization and noise, and tracks formal-language description length and maths difficulty after controlling for next-token loss. | Active |
| LIT-490 | Denoising as Projection: Constrained Optimization with Gradient-Guided Diffusion | Zhang et al. (2026), ARXIV-2608.29507. Standard gradient guidance adds the objective gradient after the denoising step, which can push the sample off the learned data geometry. Swap the order — gradient first, then one pretrained-denoiser call — and the denoiser acts as an approximate projection, making the sampler an inexact projected-gradient method with convergence guarantees on linear subspaces, compact convex sets and compact Riemannian submanifolds. One gradient and one denoiser call per step; no score Jacobians, no fine-tuning, no extra samples. | Active |
| LIT-491 | Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models | Eyring et al. (2025), ARXIV-2508.09968. Reward-guided test-time noise optimization costs 20–40 seconds a sample. Train a LoRA hypernetwork once to predict the optimized noise instead, and recover roughly half the gain at 0.1 s of added latency — 33–300× faster. The comparison that carries it: direct reward fine-tuning of the same generator makes it worse (GenEval 0.73 → 0.62 at four steps), because the KL term that would stop reward-hacking is intractable in data space and tractable in noise space. | Active |
| LIT-492 | The unintended consequences of large language models as a labor-augmenting technology in science | Duede, Gross, Crockett and Bergstrom (2026), ARXIV-2607.17397. A marginal-value-theorem model of how a time-saving tool changes what its users produce, assuming the tool is perfect — faster with no added error and no cost. Removing time raises the long-run rate of return, which is the opportunity cost of time, so developed work is developed less thoroughly. Which work gets developed at all moves in opposite directions depending on the phase accelerated, and thoroughness reverses sign in the third phase — a split the paper’s own closing sentence does not carry. | Active |
| LIT-493 | Language Modeling by Language Models | Cheng, Clark and Richardson (2025), ARXIV-2506.20249. An LLM agent system that proposed, implemented and pretrained 1,062 verified language-model block designs at 14M–350M parameters. The result that survives its own controls is the code generation: decomposing a long program into units and freezing each one that passes an execution-based checker takes valid-design rate from 6% to 92%. The architecture-discovery headline does not survive as well — the five best designs average 0.6 points below the five human seeds they were bred from. | Active |
| LIT-494 v2 | GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting | Dong, Wang, Zheng, Chen, Lu and Tang (2025), ARXIV-2501.15619. Replaces the fixed grid of a VQ image tokenizer with K 2D Gaussians whose feature coefficient is quantized while position, rotation and scale stay continuous. Reports rFID 1.61 on ImageNet-1K against LlamaGen’s 2.19 with a 16× smaller codebook. The paper’s own word for the result is semi-discrete, and the comparison is at equal token count rather than equal rate — each token carries five floats the baselines’ tokens do not. | Active |
| LIT-495 | On the Statistical Query Complexity of Learning Semiautomata: a Random Walk Approach | Giapitzakis, Fountoulakis, Nichani and Lee (2025), ARXIV-2510.04115. The first Statistical Query hardness result for semiautomata, and the first where the hardness comes from the transition structure alone rather than from a hard language embedded in the machine. Two distinct machines in the constructed family agree at chance after O(N² ln N) random symbols, so the signal separating them vanishes exponentially in sequence length. The regime it needs is unusual and belongs beside the result: an alphabet of size Ω(N³ ln N). | Active |
| LIT-496 | Taming Transformers for High-Resolution Image Synthesis | Esser, Rombach and Ommer (2020), ARXIV-2012.09841. Adds a perceptual loss and a patch-based discriminator to the VQ autoencoder, with an adaptive weight λ = ∇[L_rec] / (∇[L_GAN] + δ) computed from the decoder’s last layer. The result is that f = 16 compression stops destroying perceptual quality, so a 256×256 image becomes 16×16 tokens a transformer can model. Every tokenizer in this line since is this architecture with the codebook retuned. | Active |
| LIT-497 | Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation | Sun et al. (2024), ARXIV-2406.06525. Keeps VQGAN’s encoder-quantizer-decoder and retunes the codebook: ℓ₂-normalized codes, vector dimension 8 instead of 256, size 16384. Utilization goes from 0.29% to 97% and rFID from 9.21 to 2.19. Its other measurement is the one the record needed — the same tokenizer spans rFID 2.19 → 0.70 purely by changing how many tokens the image gets. | Active |
| LIT-498 | MaskGIT: Masked Generative Image Transformer | Chang, Zhang, Jiang, Liu and Freeman (2022), ARXIV-2202.04200. Keeps VQGAN’s tokenizer and replaces its raster-order autoregressive prior with a bidirectional masked transformer decoded in parallel: predict every token, keep the most confident, re-predict the rest. 8 steps instead of 256, up to 64× faster, and better — ImageNet 256×256 FID 6.18 vs 15.78, IS 182.1 vs 78.3. The mask schedule matters and the concave family wins; cosine is best in every experiment. | Active |
| LIT-499 | Neural Discrete Representation Learning | van den Oord, Vinyals and Kavukcuoglu (2017), ARXIV-1711.00937. The paper that made discrete latents trainable: nearest-neighbour lookup into a learned codebook, gradients copied past the non-differentiable step by a straight-through estimator, and two L2 terms — one moving codes toward the encoder, one committing the encoder to its code. ImageNet at 128×128×3 compressed to 32×32 indices over K = 512, a 42.6× bit reduction, with a PixelCNN learned over the indices as the prior. | Active |
| LIT-500 | Vector-quantized Image Modeling with Improved VQGAN | Yu et al. (2021), ARXIV-2110.04627. The origin of the codebook advice SOTA-306 rests on, and it says something sharper than its descendants repeat: factorize the code, so lookup happens in a low-dimensional space while the embedding stays high-dimensional, and ℓ₂-normalize both sides. Its ablation puts codebook usage at 4% at lookup dimension 256, 95% at 16 — and 2% with normalization removed, which is the largest single effect in the table. | Active |
| LIT-501 | The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation | Dufour, Efros and Pérez (2026), ARXIV-2606.20536. Several hundred SiT networks trained on class-conditional ImageNet 256×256, with FID treated as a random variable over a panel of training seeds × sampling seeds. Retraining moves FID 3.2× more than resampling does, the coefficient of variation holds at 1–2% across four model sizes and ten compute budgets, and a lucky training seed reaches the same FID with up to 2× less compute. | Active |
| LIT-502 | MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation | Wu, Xiang, Tang, Chen, Zhang and Su (2026), ARXIV-2606.00610. A multi-agent GraphRAG system, filed for its pilot study rather than its system: across three published GraphRAG pipelines, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops relevance (38.5% against 62.9%), and end-task accuracy falls with it. Their own filtering probe finds 40% of extracted triples can be deleted without cost. | Active |
| LIT-503 v2 | First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization | Reddy, Levine and Dragan (2022), ARXIV-2205.12381. Proposes scoring a control interface by the mutual information between the operator’s command and the state transition it induces — no labels, no reward, no knowledge of the task. Validated on 540K examples (Spearman ρ = 0.43, in 4 of 5 domains) and used to learn an interface from scratch in under 30 minutes. The fifth domain correlated negatively, and fixing it needs a horizon hyperparameter whose setting requires the task knowledge the method assumes away. | Active |
| LIT-504 | Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? | Méloux, Maniu, Portet and Peyrard (2025), ARXIV-2502.20914. Asks whether mechanistic interpretability’s own criteria pick out a unique explanation, and answers no at every stage by exhaustive enumeration on small MLPs. No network had exactly one circuit interpretation, under 2% had exactly one valid minimal mapping, and the median count rises from 38 to 910,000 as width goes from 2 to 5 — with both figures explicitly lower bounds. | Active |
| LIT-505 v2 | Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences | Rives et al. (2021), PNAS 118(15) — the protein language model the biological-sequence line runs from. A transformer trained by masked-token prediction on 250 million protein sequences, with no structural or functional labels, produces representations in which secondary structure, tertiary contacts and remote homology are linearly recoverable. Held here as the ancestor of LIT-506, which reimplements the recipe for RNA and copies its downstream head. | Active |
| LIT-506 v2 | Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions | Chen et al. (2022), ARXIV-2204.00300 — a 12-layer BERT encoder trained by masked-token prediction on 23.7 million unannotated ncRNA sequences from RNAcentral, then used as a frozen feature extractor for secondary structure, 3D contact, protein-RNA binding and UTR regulation. Beats twelve secondary-structure methods (F1 0.941 on ArchiveII600, 0.704 on bpRNA TS0). Read as NOTE-252, which records that the paper’s one clean input-swap control moves R² by 0.002 on the generalization set. | Active |
| LIT-507 | Adaptive Estimators Show Information Compression in Deep Neural Networks | Chelombiev, Houghton and O’Donnell (2019), ARXIV-1902.09037 — the third voice, and the one whose body is weaker than its abstract. Argues that LIT-509’s ReLU result is an artifact of non-adaptive binning and that with adaptive estimators saturation is not required for compression. Its own Figure 6 shows that averaged over 50 initializations a ReLU network has no distinct phase at all, and it agrees with the rebuttal that hidden-layer compression does not correlate with generalization. Read as NOTE-254. | Active |
| LIT-508 | Opening the Black Box of Deep Neural Networks via Information | Shwartz-Ziv and Tishby (2017), ARXIV-1703.00810 — the claim the information-bottleneck theory of deep learning rests on. Training is said to have two phases, a short fitting phase and a long compression phase in which I(X;T) falls; compression is said to explain generalization; and the compression is attributed to the diffusion-like behaviour of SGD. Measured by binning arctan activations into 30 equal intervals between -1 and 1 over 4096 input patterns. Held as the trunk of a three-way dispute — see LIT-509 and LIT-507. | Active |
| LIT-509 v2 | On the Information Bottleneck Theory of Deep Learning | Saxe et al. (2018), ICLR — the rebuttal. Takes the three claims of LIT-508 and shows none holds in general. The compression phase tracks the nonlinearity: tanh compresses, ReLU does not, replicated with the original authors’ own code, and consistent across binning, a kernel density estimator and — for layer entropy, on the original task only — the Kraskov estimator. Compression and generalization dissociate in all four combinations. Full-batch gradient descent compresses as much as SGD. Read as NOTE-253. | Active |
| LIT-510 | Gemini: A Family of Highly Capable Multimodal Models | Gemini Team, Google (2023), ARXIV-2312.11805 — held for its evaluation methodology rather than its models. Its appendices publish three things the record wanted: an MMLU sweep in which the ordering against GPT-4 flips depending on the inference procedure; a measurement of how cheaply a benchmark can be inflated (100 fine-tuning steps on HellaSwag-adjacent web extracts take Ultra to 96.0% at 1-shot); and a FLEURS ablation retraining without the benchmark’s own training set. Read as NOTE-255. | Active |
| LIT-511 | Image-GS: Content-Adaptive Image Representation via 2D Gaussians | Zhang et al. (2024), ARXIV-2407.01866 — the 2D descendant of LIT-108, and the paper that does the rate accounting LIT-494 skipped. Anisotropic colored 2D Gaussians, spawned by image-gradient magnitude and grown where error persists, decoded at 0.3K MACs per pixel against C3’s 3K at comparable bitrate. Beats six neural image representations at matched model size across the bitrate range, and beats JPEG below 0.244 bpp. Read as NOTE-256. | Active |
| LIT-512 | SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models | Li et al. (2024), ARXIV-2411.05007 — 4-bit weights and activations for diffusion transformers. Shift outliers from activations into the weights by smoothing, peel the dominant singular values into a 16-bit rank-32 branch, quantize only the residual. Two propositions bound the output error by the magnitudes rather than only the rounding errors, which is why shrinking the residual works. Naïvely the extra branch costs 57% latency; fused, the system reaches 3.0× over a W4A16 baseline and 3.5× memory reduction on 12B FLUX.1. Read as NOTE-257. | Active |
| LIT-513 | mHC-lite: You Don’t Need 20 Sinkhorn-Knopp Iterations | Yang (2026), ARXIV-2601.05732 — not a fourth constraint but a repair of the third. Keeps LIT-140’s doubly-stochastic constraint and makes it exact, building the matrix as a convex combination of permutation matrices (Birkhoff-von Neumann) instead of approximating the projection with 20 Sinkhorn-Knopp iterations. Measures the approximation it replaces: the column sums of the layer-wise product deviate from 1 by up to 220% in a 24-layer network. Read as NOTE-258. | Active |
| LIT-514 | Rethinking Early Stopping: Refine, Then Calibrate | Berta, Holzmüller, Jordan and Bach (2025), ARXIV-2501.19195 — the record’s first document on probability calibration. A proper loss splits into calibration error and refinement error; the two are not minimized at the same epoch, so stopping on validation loss lands at a compromise optimal for neither. Stop on validation loss after temperature scaling instead, then calibrate. 196 tabular datasets across three model families, plus a vision benchmark of ten runs per dataset. Read as NOTE-259. | Active |
| LIT-515 | Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training | Liu et al. (2025), ARXIV-2506.12037 — full-parameter training one block at a time, with blocks cut on layer boundaries so the optimized kernels still apply. Reaches comparable or better perplexity at 2B on about three times the iterations, using one A800 where full-parameter needs two. Read as NOTE-260. | Active |
| LIT-516 | From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications | Jaiswal et al. (2024), ARXIV-2407.11239 — how nearly low-rank a transformer weight matrix is depends on which matrix and how deep: Query/Key and MLP Gate converge to low rank, MLP Up/Down and Value do not, and middle blocks resist while the first and last few give way. Picking the rank per matrix instead of globally is ~6.4× better in perplexity than uniform reduction at 30% on LLaMA-2 7B and ~47× better at 40% on 13B. Read as NOTE-264. | Active |
| LIT-517 | Small Singular Values Matter: A Random Matrix Analysis of Transformer Models | Staats, Thamm and Rosenow (2024), ARXIV-2410.17770 — the singular spectrum of a trained transformer weight matrix is a Marchenko-Pastur bulk plus outliers, and in non-square matrices the outliers appear at both ends. Removing the smallest decile of the Down-Projection drops Llama-3 8B on GSM8K from 43.2% to 2.0%, while the smallest decile of the square Attention-Output matrix costs almost nothing. Magnitude order is not importance order. Read as NOTE-263. | Active |
| LIT-518 | The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction | Sharma, Ash and Misra (2023), ARXIV-2312.13558 — replacing one weight matrix by a low-rank approximation, especially an MLP matrix in a later layer, can improve accuracy: GPT-J on CounterFact goes from 13.3% to 24.1% off a single layer, with no retraining. The gains land on facts rare in the training corpus. Filed for the debate it anchors rather than read in full. | Active |
| LIT-519 | Complexity-Guided Component-wise Initialization for Language Model Pretraining | Garbers and Oh (2026), ARXIV-2607.09204 — eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile: effective-rank entropy falls in the last few blocks, most sharply in the residual-writing matrices W_O and W_down, whose Frobenius norms rise almost linearly with depth. Initializing to imitate that profile changes the spectra and does not improve results. Read as NOTE-262. | Active |
| LIT-520 | The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training | Zhang, Zhou, Jia, Chen and Cheng (2026), ARXIV-2605.26489 — the trace-normalized singular spectrum stops moving long before the weight matrices do. Variation peaks near 10⁻² in the first ~1000 steps, while validation loss falls from ~10 to ~4.0, then settles at a 10⁻⁴ floor that coincides with the slow-descent phase. Holds on GPT-2 and LLaMA, across schedules, weight decays, AdamW and Muon. Read as NOTE-261. | Active |
| LIT-521 | Taming Transformer Without Using Learning Rate Warmup | Qi et al. (2025), ARXIV-2505.21910 — attention entropy collapse comes in two modes, and only one crashes the model: sparse and low-rank is fatal, sparse alone is benign. The driver is spectral energy concentration in W_q^T W_k. Suppressing it by bounding the learning rate with Weyl’s inequality trains ViT, Swin and GPT without warmup, matching warmed-up AdamW on all five configurations. Read as NOTE-266. | Active |
| LIT-522 | Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation | Shi et al. (2026), ARXIV-2605.12492 — instead of bounding or renormalizing a weight matrix’s spectrum, never change it: update W by left and right orthogonal transformations so the singular values are invariant and only the singular vectors rotate. On a 60M LLaMA with every normalization layer removed, AdamW and Muon produce NaNs and Pion trains 9.6B tokens. Read as NOTE-267. | Active |
| LIT-523 | Stabilizing Transformer Training by Preventing Attention Entropy Collapse | Zhai et al. (2023), ARXIV-2303.06296 — proves a tight lower bound on attention entropy that falls like Ω(Tσe^{-σ}) in the spectral norm of W_K W_Q^T, then removes the growth by reparameterizing every linear layer as γ·W/σ(W). A ViT trained this way reaches 82.2% on ImageNet without pre-LN, warmup, weight decay or an adaptive optimizer. Read as NOTE-265. | Active |
| LIT-524 | Ghost in the Transformer: Detecting Model Reuse with Invariant Spectral Signatures | Wang, Ma, Xinyi and Li (2025), ARXIV-2511.06390 — take the singular spectra of W_q W_k^T and W_v W_o per layer. Those products are invariant to the permutation and scaling transforms that hide derivation while preserving function, so the spectra fingerprint the model. F1 0.9867 over 55 model pairs, data-free, against 0.9610 for the best prior method. Read as NOTE-268. | Active |
| LIT-525 | Transformers Learn Low Sensitivity Functions: Investigations and Implications | Vasudeva et al. (2024), ARXIV-2403.06925 — sensitivity to token-wise random input perturbations is one number that separates transformers from MLPs, CNNs, ConvMixers and LSTMs across vision and language. It also predicts robustness, tracks flatness, can be regularized for, and keeps falling through a grokking plateau while the loss does not move. Read as NOTE-271. | Active |
| LIT-526 | Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations | Acharya, Rimal and Dhakal (2026), ARXIV-2605.01609 — contextual concept directions sit in the low-variance tail of the unembedding covariance in 17 of 17 models, while static vocabulary contrasts sit at the top. The paper also reports a robust null against the hypothesis it set out to test, and labels each of its own claims robust, partial or suggestive. Read as NOTE-270. | Active |
| LIT-527 | Leaner Transformers: More Heads, Less Depth | Saratchandran, Teney and Lucey (2025), ARXIV-2505.20802 — a theorem that more attention heads lower the condition number of the attention block, and twelve redesigned architectures trading heads for layers. In the five configurations where the MLP width is held fixed, accuracy holds or improves at 29–53% fewer parameters. Read as NOTE-269. | Active |
| LIT-528 | The emergence of clusters in self-attention dynamics | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), ARXIV-2305.05465 — treat tokens as interacting particles and self-attention as their dynamics. Tokens cluster, and the spectrum of the value matrix decides the limiting geometry: V = I gives the vertices of a convex polytope, a simple positive leading eigenvalue gives at most three parallel hyperplanes, V = −I gives one cluster at the origin. In one dimension the attention matrix provably converges to a low-rank Boolean matrix. Read as NOTE-272. | Active |
| LIT-529 | Linformer: Self-Attention with Linear Complexity | Wang, Li, Khabsa, Fang and Ma (2020), ARXIV-2006.04768 — the measurement the record needed, attached to a method it has already moved past. SVD of the attention matrix in RoBERTa-base and RoBERTa-large over 10k sentences shows a long-tail spectrum at every layer, head and task, skewing further in higher layers; Theorem 1 gives a rank Θ(log n) approximation. Read as NOTE-273. | Active |
| LIT-530 | Attention is not all you need: pure attention loses rank doubly exponentially with depth | Dong, Cordonnier and Loukas (2021), ARXIV-2103.03404 — strip the skip connections and MLPs out of a transformer and its output converges doubly exponentially to a rank-1 matrix with identical rows. Skip connections stop it, MLPs slow it, and layer normalization provably does nothing — it is a right-multiplication, and right-multiplication cannot raise rank. Read as NOTE-274. | Active |
| LIT-531 | Deep Learning and the Information Bottleneck Principle | Tishby and Zaslavsky (2015), ARXIV-1503.02406 — the five-page position paper the whole information-bottleneck line descends from. It proposes quantifying every layer by I(X;h) and I(Y;h), draws the information plane as an explicitly qualitative figure with a hypothesized layer path, and states that approaching the IB limit requires stochastic mapping between the layers — the condition its successors were later refuted for dropping. Read as NOTE-275. | Active |
| LIT-532 | What learning algorithm is in-context learning? Investigations with linear models | Akyürek, Schuurmans, Andreas, Ma and Zhou (2022), ARXIV-2211.15661, ICLR 2023 — the independent construction, and the finding the title undersells. A transformer needs constant depth and O(d) hidden size for a gradient step and O(d²) for a Sherman–Morrison ridge update; but the trained model’s behaviour moves through three algorithmic phases as depth grows — gradient descent, then ridge, then ordinary least squares — and at every noise level the ridge parameter that best fits it is the one that minimizes Bayes risk. Read as NOTE-277. | Active |
| LIT-533 | Transformers learn in-context by gradient descent | von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov and Vladymyrov (2022), ARXIV-2212.07677, ICML 2023 — the paper that named the mesa-optimizer. Proposition 1 constructs key, query, value and projection matrices for which one linear self-attention layer is exactly one gradient-descent step on an implicit linear model, and a single trained layer is shown to find approximately those weights. Deeper trained models do not match plain gradient descent; they match GD++, gradient descent with an iterative curvature correction — a result of this paper’s own. Read as NOTE-280. | Active |
| LIT-534 | What Can Transformers Learn In-Context? A Case Study of Simple Function Classes | Garg, Tsipras, Liang and Valiant (2022), ARXIV-2208.01066, NeurIPS 2022 — the setup that turned “how does in-context learning work” into a question with a checkable answer. A 9.5M-parameter GPT-2 trained from scratch on sequences of (x, f(x)) pairs in-context-learns unseen linear functions at the accuracy of the optimal least-squares estimator, and sparse linear functions, two-layer ReLU networks and depth-4 decision trees at or above their task-specific algorithms. Read as NOTE-279. | Active |
| LIT-535 | Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression | Fu, Chen, Jia and Sharan (2023), ARXIV-2310.17086, NeurIPS 2024 — the challenge from inside the same setup. In Garg et al.’s exact training regime, successive transformer layers track Iterative Newton linearly (roughly 3 iterations per middle layer) and gradient descent only exponentially; on data with condition number 100 the transformer is unaffected while GD needs 2,000 steps, which twelve layers cannot hold. Theorem 5.1 builds k Newton iterations in k + 8 layers with O(d) width. Read as NOTE-278. | Active |
| LIT-536 | Do pretrained Transformers Learn In-Context by Gradient Descent? | Shen, Mishra and Khashabi (2023), ARXIV-2310.08540, ICML 2024 — the challenge from outside. It separates two hypotheses the field had been conflating: that ICL in a naturally pretrained model is gradient descent, and that transformer weights exist that simulate it. Theorem 1 says an algorithm equivalent to ICL must share its order sensitivity, and gradient descent is order-stable while ICL is not; on LLaMA-7B, ICL and fine-tuning disagree on accuracy, top-10 token overlap and overlap cosine similarity across four datasets. Read as NOTE-276. | Active |
| LIT-537 | Grokking as the Transition from Lazy to Rich Training Dynamics | Kumar, Bordelon, Gershman and Pehlevan (2023), ARXIV-2310.06110, ICLR 2024 — the counterexample. A two-layer MLP groks on modular arithmetic with no weight decay at all, and its parameter norm rises through the transition, which the weight-norm accounts cannot produce. The proposed replacement: grokking is the network leaving the lazy (linearised) regime and starting to learn features, controlled by an output-scale parameter α and by the alignment between the initial neural tangent kernel and the target. Read as NOTE-284. | Active |
| LIT-538 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power, Burda, Edwards, Babuschkin and Misra (2022), ARXIV-2201.02177 — the paper that named grokking. A small transformer on division mod 97 at 50% data reaches near-perfect training accuracy in under 10³ steps and near-perfect validation accuracy around 10⁶, with almost no generalization before 10⁵. The regime dependence is measured here, not later: near 25–30% data on S₅, removing 1% of the training set raises median time to generalization by 40–50%, while time to fit stays at 10³–10⁴ throughout. Weight decay more than halves the data needed. Read as NOTE-281. | Active |
| LIT-539 | Explaining grokking through circuit efficiency | Varma, Shah, Kenton, Kramár and Kumar (2023), ARXIV-2309.02390, DeepMind — the account that earns its place by predicting things nobody had seen. Memorising and generalising circuits both fit the training set; weight decay prefers whichever produces more logit per unit parameter norm; memorisation gets less efficient as the dataset grows and generalisation does not, so there is a critical dataset size D_crit where the preference flips. From that the paper predicts ungrokking and semi-grokking, and then observes both. Read as NOTE-283. | Active |
| LIT-540 | Omnigrok: Grokking Beyond Algorithmic Data | Liu, Michaud and Tegmark (2022), ARXIV-2210.01117, ICLR 2023 — the paper that answers whether grokking is specific to algorithmic data. No: it is induced on MNIST, IMDb and QM9 by shrinking the training set and scaling the initialization, and eliminated on algorithmic data by constraining the weight norm. The proposed mechanism is the “LU” mismatch — reduced training loss is L-shaped in weight norm and test loss U-shaped, so above the critical norm the two disagree. Read as NOTE-282. | Active |
| LIT-541 | Deep Learning is Singular, and That’s Good | Murfet, Wei, Gong, Li, Gell-Redman and Quella (2020), ARXIV-2010.11560 — the invitation, and the trunk under a school the record has been citing without holding. Neural networks are singular statistical models: the set of optimal parameters is a real analytic variety with singularities, so the Laplace approximation and anything that divides by the determinant of the Hessian do not apply. The correct effective parameter count is the real log canonical threshold λ, which counts how many directions change the model, not how sharply the loss rises in them. Read as NOTE-286. | Active |
| LIT-542 | The Local Learning Coefficient: A Singularity-Aware Complexity Measure | Lau, Furman, Wang, Murfet and Wei (2023), ARXIV-2308.12108 — the instrument. It localizes Watanabe’s learning coefficient to a single minimum, defines it as the exponent in V(ε) = c ε^λ (−log ε)^{m−1} + …, and gives an SGLD-based estimator λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] with β* = 1/log n. Validated against known theoretical values on deep linear networks up to 100M parameters. On ResNet18/CIFAR10 it separates training configurations whose training losses have all collapsed to zero. Read as NOTE-285. | Active |
| LIT-543 | Loss Landscape Degeneracy and Stagewise Development in Transformers | Hoogland, Wang, Farrugia-Roberts, Carroll, Wei and Murfet (2024), ARXIV-2402.02364, TMLR 2025 — the instrument applied. Tracking the LLC through training divides two transformers into five stages each, and the boundaries coincide with structural changes the loss curve does not show: bigrams, then n-grams, then previous-token heads, then the induction circuit. In the in-context regression transformer, in-context learning is acquired in LR2 and then deteriorates across LR3 and LR4 as the model specializes to its pre-training distribution. Read as NOTE-287. | Active |
| LIT-544 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei and Murfet (2023), ARXIV-2310.06301 — the paper that separates two things everyone calls a phase transition. In the Toy Model of Superposition at two hidden dimensions, regular k-gons are critical points with derivable local learning coefficients; the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α] then predicts a Bayesian 5 → 6 transition at n_cr ≈ 600, observed at 600 ≤ n ≤ 700. The same critical points govern SGD plateaus — and the 5 → 6 transition is not one of the dynamical ones. Read as NOTE-288. | Active |
| LIT-545 | Gemma 3n model overview | Google DeepMind (2025), model documentation with no accompanying paper. The origin of Per-Layer Embeddings (PLE): each token id looks up a small vector for every layer, and each layer gates it into the residual stream. The table is about 2.35B parameters in E2B and is kept off the accelerator. A design disclosure and not evidence: Google published no ablation. Cited by the Qwen3.8-Flash-Next report (LIT-152) as the precedent for embedding tables held in host memory. | Active |
| LIT-546 | MatFormer: Nested Transformer for Elastic Inference | Devvrit, Kudugunta, Kusupati et al. (2023), ARXIV-2310.07707. Nest four FFN widths inside one Transformer, so each smaller FFN is the first neurons of the larger. Train one width per step, sampled uniformly, and pick a different width per layer at inference to extract sizes nobody trained. Up to 850M parameters it matches four separately trained sizes at the same total compute. The small submodels beat theirs partly because shared weights see up to 4× the tokens. | Active |
| LIT-547 | Matryoshka Representation Learning | Kusupati, Bhatt, Rege et al. (2022), ARXIV-2205.13147. Add the task loss at about log(d) prefix lengths of one embedding (8, 16, …, 2048), and every prefix works as a representation. ResNet50 on ImageNet: each prefix is at least as accurate as a separately trained model of that width. Shortlisting with 16 dimensions and re-ranking with 2048 matches full-width retrieval at 128× fewer FLOPs, 14× faster in wall-clock time. At full width it costs a few tenths of a point on the web-scale models. | Active |
| LIT-548 | Feature Learning in Infinite-Width Neural Networks | Yang and Hu (2020), ARXIV-2011.14522 — Tensor Programs IV, where µP is introduced. Among parametrizations that scale initialization, multipliers and learning rate with width, every stable, nontrivial one either learns features in the wide limit or trains as a kernel, never both. The standard parametrization is stable only at an O(1/width) learning rate, which puts it in the kernel regime. µP is the one that updates every layer maximally, and its maximum stable learning rate does not move with width. | Active |
| LIT-549 | ReFT: Representation Finetuning for Language Models | Wu, Arora et al. (2024), ARXIV-2404.03592. Freeze the model and train a low-rank edit to the hidden state at a few prompt positions: h + Rᵀ(Wh + b − Rh), with R having orthonormal rows. It uses 15–65× fewer parameters than LoRA. It leads on commonsense QA (LLaMA-7B, 80.2 against LoRA’s 74.7), leads on GPT-4-judged instruction following, and is level on GLUE. It loses on arithmetic chain of thought (GSM8K 26.0 against 37.5). Baseline numbers are copied from earlier papers. | Active |
| LIT-550 | Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains | Tancik, Srinivasan, Mildenhall et al. (2020), ARXIV-2006.10739. A plain MLP on low-dimensional coordinates learns high frequencies so slowly that in practice it never does. In the NTK picture this is because the kernel’s eigenvalues fall off fast with frequency. Mapping the input through sinusoids makes the kernel stationary, with a bandwidth set by the frequencies. Random Gaussian frequencies beat NeRF’s positional encoding on all seven tasks, and only their scale matters. | Active |
| LIT-551 | Implicit Neural Representations with Periodic Activation Functions | Sitzmann, Martel et al. (2020), ARXIV-2006.09661. An MLP with sin activations fits images, video, audio and signed distance functions, and, unlike ReLU networks, their derivatives, because a SIREN’s derivative is itself a SIREN. It trains only with a specific initialization: weights U(±√(6/n)) keep every pre-activation near N(0,1), and the first layer is scaled by ω₀ = 30. The initialization’s activation statistics are verified. Its role in earlier periodic networks’ failures is asserted, not ablated. | Active |
| LIT-552 | XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models | Dong et al. (2024), ARXIV-2411.15100. Constrained decoding sets the logits of grammar-violating tokens to −∞. Doing that for a context-free grammar means interpreting a pushdown automaton over the whole vocabulary every step. XGrammar precomputes, per automaton node, the verdict for tokens that do not depend on the rest of the stack (over 99% for JSON on Llama-3.1), and builds the mask on the CPU during the GPU forward pass. Per-token mask time is 36 µs against milliseconds for earlier engines, with end-to-end overhead of about 0.1 ms per token. | Active |
| LIT-553 | Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise | Bansal et al. (2022), ARXIV-2208.09392. Train a network to invert blur, masking, downsampling or snow at every severity, then sample by alternating restoration and re-degradation, and something generative happens without Gaussian noise. The durable result is the sampler, x_{s−1} = x_s − D(x̂₀, s) + D(x̂₀, s−1), which is exact for degradations linear in severity whatever the restorer’s errors. The headline, that noise is not needed, is contradicted by its own Table 5: noiseless blur generation is FID 97.00 on CelebA against 23.11 with noise, and a σ = 0.002 noise sprinkle brings it to 49.45. | Active |
| LIT-554 | Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion | Chen, Marti Monso, Du, Simchowitz, Tedrake, Sitzmann (2024), ARXIV-2407.01392. Train a causal sequence model to denoise tokens that each carry an independent noise level. One model then samples autoregressively, like next-token prediction, and supports guidance over a whole horizon, like full-sequence diffusion. Video rollouts stay stable past 1,000 frames where teacher forcing diverges, shown qualitatively only. On D4RL mazes it beats Diffuser using its own generated actions. A convolutional RNN, not a transformer, at small scale. | Active |
| LIT-555 v2 | Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models | Hsueh et al. (2025), ARXIV-2511.16904. Degrade with blur and noise together, and predict the blurry image and the missing high frequencies with two heads. Blur lets the model use the dependency of fine detail on coarse structure. Noise keeps intermediate states on the data manifold. Sweeping the blur-to-noise ratio on CIFAR-10 at 35 steps gives FID 1.97 at BNR = 0 (EDM), 1.85 at BNR = 0.5 and 11.97 at BNR = 10. A small gain, and a large loss as the process goes cold. | Active |
| LIT-556 | Tensor Programs III: Neural Matrix Laws | Yang (2020), ARXIV-2009.10685. At random initialization and infinite width, a network’s pre-activations become asymptotically free (in the random-matrix sense) of its weights, for any architecture: the Free Independence Principle. This rigorously justifies the Jacobian singular-value calculations behind dynamical isometry, which had assumed it, and gives a second justification of the gradient independence assumption. The general Master Theorem is also what TP-IV uses. | Active |
| LIT-557 | Tensor Programs II: Neural Tangent Kernel for Any Architecture | Yang (2020), ARXIV-2006.14548. The neural tangent kernel of a randomly initialized network of any standard architecture converges to a deterministic limit as width grows, and the paper shows how to compute it. It also gives a checkable condition (Simple GIA Check) under which the common heuristic that backprop’s Wᵀ is independent of forward W is correct, and cases where that heuristic gives wrong answers. | Active |
| LIT-558 | Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes | Yang (2019), ARXIV-1910.12478. At random initialization, any network built from MLPs, RNNs, convolutions, pooling, skip connections, attention, batchnorm or layernorm converges to a Gaussian process as width grows. This is the NNGP correspondence extended from MLPs and CNNs to all standard architectures, proved by expressing the forward pass in the NETSOR language. The first paper of the series and the tutorial for its machinery. A statement about initialization, not training. | Active |
| LIT-559 | Alias-Free Generative Adversarial Networks | Karras et al. (2021), ARXIV-2106.12423. StyleGAN3: fine detail in GAN images sticks to pixel coordinates (“texture sticking”) because the generator aliases. Treat every feature map as a bandlimited continuous signal. Low-pass filter around each nonlinearity at 2× temporary upsampling, feed Fourier features instead of a constant, drop noise inputs. The generator becomes translation-equivariant (EQ-T 63 dB), or also rotation-equivariant (StyleGAN3-R, EQ-R 40 dB), at StyleGAN2’s FID. | Active |
| LIT-560 | Analyzing and Improving the Image Quality of StyleGAN | Karras et al. (2019), ARXIV-1912.04958. StyleGAN2 traces StyleGAN’s droplet artifacts to instance normalization and replaces it with weight demodulation (FID unchanged, artifacts gone). It traces its phase artifacts to progressive growing and replaces growing with a fixed output-skip generator and residual discriminator (FFHQ FID 4.34 → 3.31). It runs R1 only every 16 minibatches at no cost, and adds path-length regularization, which trades FID against smoothness on less structured data. | Active |
| LIT-561 | A Style-Based Generator Architecture for Generative Adversarial Networks | Karras, Laine, Aila (2018), ARXIV-1812.04948. StyleGAN: a mapping network turns the latent z into an intermediate w, and w modulates every synthesis layer through AdaIN. Per-layer noise inputs supply stochastic detail, and mixing two w’s across layers regularizes the model toward scale-specific control. Built on progressive growing. FFHQ FID goes from 8.04 (Progressive GAN) to 4.40, most of it from tuning (5.25) and the mapping network and styles (4.85). Also introduced the FFHQ dataset and the perceptual path length metric. | Active |
| LIT-562 | Projected GANs Converge Faster | Sauer, Chitta, Müller, Geiger (2021), ARXIV-2111.01007. Feed real and generated images through a frozen ImageNet-pretrained EfficientNet, mix its multi-scale features with fixed random 1×1 and 3×3 convolutions, and put a small discriminator on each scale. FID reaches StyleGAN2’s best on LSUN-Church after 1.1M images instead of 88M, and beats it on 22 datasets. A later analysis (LIT-563) shows part of the gain is in FID’s perceptual null space: at equal FID, humans and a CLIP-feature distance prefer StyleGAN2. | Active |
| LIT-563 | The Role of ImageNet Classes in Fréchet Inception Distance | Kynkäänniemi, Karras, Aittala, Aila, Lehtinen (2022), ARXIV-2203.06026. FID’s feature space sits one affine map from ImageNet class logits, so it attends to whatever ImageNet class an image evokes. That is often not the subject: seat belts and bow ties on FFHQ faces. Resampling a fixed generator’s outputs to match those statistics cuts FFHQ FID from 5.30 to 1.78 (−66%) while a CLIP-space distance moves 2.76 → 2.64. A Projected FastGAN with an ImageNet-pretrained discriminator matches StyleGAN2’s FID and loses to it in CLIP space and with humans. | Active |
| LIT-564 | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks | Zhu, Park, Isola, Efros (2017), ARXIV-1703.10593. Learn G: X → Y and F: Y → X from unpaired collections, with adversarial losses in each domain and a cycle-consistency loss that makes F(G(x)) ≈ x and G(F(y)) ≈ y. The cycle term turns an under-constrained mapping into a usable one for colour and texture changes (horse → zebra, photo ↔ painting, summer ↔ winter). The authors report that it fails on geometric changes. Its own ablation has a one-directional cycle beating the full model on labels → photo. | Active |
| LIT-565 | FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark | Fang et al. (2025), ARXIV-2509.09680. 6 million FLUX-generated images with 20 million bilingual captions, organized by six characteristics (imagination, entity, text rendering, style, affection, composition), and with “Generation Chain-of-Thought” breakdowns. Curation cost 15,000 A100 GPU-days. Released with PRISM-Bench, seven tracks scored by vision-language models, used to evaluate 19 text-to-image models. A dataset and a benchmark, with no training ablation read here. | Active |
| LIT-566 | SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis | Podell et al. (2023), ARXIV-2307.01952. SDXL: a 2.6B-parameter UNet latent diffusion model, 3× Stable Diffusion’s, with two text encoders (CLIP ViT-L and OpenCLIP ViT-bigG) and pooled-text conditioning. It is conditioned on each training image’s original size and crop, trained on multiple aspect ratios, and followed by an optional refiner applied SDEdit-style. The one quantitative ablation is size conditioning on class-conditional ImageNet. The headline comparisons are user studies. | Active |
| LIT-567 | MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models | Frantar, Castro, Chen, Hoefler, Alistarh (2024), ARXIV-2408.11743. An FP16×INT4 matrix-multiply kernel, the “Marlin format” being its quantization-specific weight layout. It keeps weight-only quantization’s memory-bandwidth speed-up through batched serving: about 3.9× over FP16 (near the 3.87× ideal for 4-bit, group 128) up to batch 16–32, falling toward 1.5× at 128 as the problem turns compute-bound. In vLLM, about 2.8× lower time per output token on an A6000, and 2.3–3.2× single-GPU end-to-end at batch 16 or below, falling to 1.1–1.2× at 128. | Active |
| LIT-568 | Are Sparse Autoencoders Useful? A Case Study in Sparse Probing | Kantamneni, Engels et al. (2025), ARXIV-2502.16681. On 113 probing datasets over Gemma-2-9B, and replicated on Llama-3.1-8B, logistic-regression probes are compared with probes on SAE latents. Selection is done the way a practitioner would, by validation AUC over a “quiver” of methods. Adding SAE probes does not help in standard conditions (−0.003 AUC) or under data scarcity, class imbalance, label noise or covariate shift. Earlier positive results for SAE probes disappear against stronger baselines, including some of the authors’ own. | Active |
| LIT-569 | Eliciting Latent Predictions from Transformers with the Tuned Lens | Belrose et al. (2023), ARXIV-2303.08112. The logit lens is unreliable: it fails on BLOOM, OPT-125M and GPT-Neo, and even where it works it is biased, 4–5 bits of KL from the final marginal on GPT-Neo-2.7B. The tuned lens trains one affine translator per layer by distillation toward the final logits. It has lower perplexity and bias on every model family tested up to 20B, and features it relies on also matter to the model (Spearman 0.89). Version 6 (2025) adds that every lens in the paper was undertrained, and that Muon trains them much better. | Active |
| LIT-570 | interpreting GPT: the logit lens | nostalgebraist (2020), a LessWrong post. Apply GPT-2’s final layer norm and unembedding to each intermediate layer’s residual stream and read the result as a next-token distribution. On GPT-2 those distributions make sense and converge on the final prediction well before the last layer, and they almost never resemble the input tokens. The post calls it a partial view. | Active |
| LIT-571 | Scaling and evaluating sparse autoencoders | Gao et al. (2024), ARXIV-2406.04093. TopK sparse autoencoders keep the k largest latents, which sets L0 directly and replaces the L1 penalty. They beat ReLU SAEs on the sparsity-reconstruction frontier, and the gap widens with scale. Encoder-as-decoder-transpose initialization plus an auxiliary dead-latent loss leave 7% dead latents in a 16M-latent SAE on GPT-4, against up to 90% without them. Clean scaling laws in latents and k. That SAE scores 98.2% “loss recovered”, yet substituted into GPT-4 it matches a model trained on 10% of GPT-4’s compute. | Active |
| LIT-572 | FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison | Black Forest Labs (2025), technical report. DiT-XL flow models on ImageNet 256² in four latent spaces (SD-VAE, FLUX.1-VAE, FLUX.2-VAE, RAE), 30 configurations each: training timestep distribution, training shift and sampling shift, with and without REPA. The training shift moves FID by 61–86%, and without tuning it per space the rankings flip. FLUX.1’s VAE reconstructs far better than SD’s and is less learnable (gFID 10.1 against 7.7). FLUX.2’s VAE, with 8× SD’s dimensionality and semantic regularization, gets both (LPIPS 0.27, gFID 3.7). FLUX.2 is that latent plus Mistral-3 24B and a rectified-flow transformer. | Active |
| LIT-573 | FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space | Black Forest Labs (2025), ARXIV-2506.15742. The only paper describing FLUX.1: a 12B rectified-flow transformer in a 16-channel VAE latent, with double-stream then 38 single-stream blocks, fused feed-forward layers and 3D RoPE. Kontext fine-tunes it for editing by appending context-image latents to the sequence, offset by a “virtual time step” in RoPE. The evidence is human preference against other systems and a reconstruction table. It has no ablations. | Active |
| LIT-574 | Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models | Hase, Bansal, Kim, Ghandeharioun (NeurIPS 2023), ARXIV-2301.04213. On 652 facts known to GPT-J, causal tracing often peaks outside the layers ROME and MEMIT edit. Yet editing at those layers works regardless. In a regression of ROME’s rewrite score, the edit layer explains 94.7% of the variance, and adding the fact’s tracing effect raises it to 94.8%. Robust across editing methods (ROME, MEMIT, fine-tuning), metrics, models and datasets. Localization, as causal tracing measures it, does not tell you where to edit. | Active |
| LIT-575 | Transformer Feed-Forward Layers Are Key-Value Memories | Geva, Schuster, Berant, Levy (2021), ARXIV-2012.14913. Read each FFN hidden unit as a memory. Its input weight is a key that fires on human-recognizable prefix patterns: shallow n-grams in lower layers, semantic topics in upper ones. Its output weight is a value that, projected through the output embedding, favors next tokens. The evidence is 160 sampled keys in one 16-layer WikiText-103 model. Top value predictions agree with the keys’ triggers at most 3.5% of the time, in upper layers only. Layer outputs are mostly compositions, matching no single memory’s prediction. | Active |
| LIT-576 | Mass-Editing Memory in a Transformer | Meng, Sen Sharma, Andonian, Belinkov, Bau (2023), ARXIV-2210.07229. MEMIT writes thousands of facts at once by spreading each update over a range of mid-layer MLPs (layers 3–8 in GPT-J) in one batched least-squares solve. At 10,000 CounterFact edits it scores 85.8 on GPT-J (ROME 50.3, fine-tuning 67.6 with generation collapse). Neighborhood success falls from 83.5 to 73.7. | Active |
| LIT-577 | Evaluating the Ripple Effects of Knowledge Editing in Language Models | Cohen, Biran, Yoran, Globerson, Geva (2023), ARXIV-2307.12976. An edit implies other facts: logical consequences, multi-hop compositions, aliases. RippleEdits tests 5K edits on six such criteria. ROME, MEMIT and MEND average 38–66 across GPT-2, GPT-J, GPT-NeoX and LLaMA. Logical generalization can be as low as 5.5–7.0. Stating the new fact in the prompt (in-context editing) scores best, beating ROME by more than 10 points on GPT-NeoX and by 29 on LLaMA. | Active |
| LIT-578 | Locating and Editing Factual Associations in GPT | Meng, Bau, Andonian, Belinkov (2022), ARXIV-2202.05262. Causal tracing: corrupt the subject tokens, then restore single hidden states to see which carry the fact. The strongest effects are at mid-layer MLPs on the last subject token. ROME writes a new fact as a rank-one update to one such MLP, treated as a linear associative memory. On CounterFact it scores 89.2 (GPT-2 XL) and 91.5 (GPT-J) against at most 68.7 for the baselines. It edits one fact at a time and degrades after about 10. | Active |
| LIT-579 | Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark | Tönshoff, Ritzert, Rosenbluth, Grohe (2023), ARXIV-2309.00367. On the Long-Range Graph Benchmark, graph transformers were reported to beat message-passing GNNs. After a basic hyperparameter sweep within the same 500k-parameter budget, GCN, GINE and GatedGCN all beat GPS on Peptides-func and Peptides-struct. Most of the gain comes from a 2-layer prediction head instead of a linear one. On the superpixel datasets, input feature normalization accounts for at least half the gain of most models, and GPS stays ahead there. The link-prediction metric’s code computed raw MRR where the paper specified filtered MRR. | Active |
| LIT-580 | How Powerful are Graph Neural Networks? | Xu, Hu, Leskovec, Jegelka (ICLR 2019), ARXIV-1810.00826. Message-passing GNNs are at most as discriminative as the 1-WL isomorphism test, and they reach that bound when aggregation and readout are injective on multisets. Sum aggregation with an MLP update (GIN) is; mean (GCN) and max (GraphSAGE) are not. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and GIN scores 92.4 and 57.5. On the other seven benchmarks, most differences are within a standard deviation. | Active |
| LIT-581 | Pitfalls of Graph Neural Network Evaluation | Shchur, Mumme, Bojchevski, Günnemann (R2L workshop, NeurIPS 2018), ARXIV-1811.05868. GCN, GAT, MoNet and GraphSAGE were run with one shared training and tuning procedure, over 100 random splits × 20 seeds on 8 datasets. GCN has the best average rank (2.3) and relative accuracy (99.4%). On Planetoid’s single split GAT wins Cora and Citeseer; on another random split of the same sizes GCN wins them. GAT’s lower average on Amazon comes from 138 of 2,000 runs collapsing. | Active |
| LIT-582 | Semi-Supervised Classification with Graph Convolutional Networks | Kipf and Welling (ICLR 2017), ARXIV-1609.02907. The GCN layer is H′ = σ(D̃^{-1/2}ÃD̃^{-1/2} H W), with self-loops added and the adjacency symmetrically normalized. It is motivated as a first-order approximation of spectral graph convolution. Two layers reach 81.5% on Cora, 70.3% on Citeseer and 79.0% on Pubmed, all on the single Planetoid split. Models deeper than about 7 layers are hard to train without residual connections. | Active |
| LIT-583 | Highly accurate protein structure prediction with AlphaFold | Jumper et al. (Nature 2021), DOI 10.1038/s41586-021-03819-2. AlphaFold 2 predicts protein structure from sequence and a multiple sequence alignment. At CASP14 its median backbone error was 0.96 Å r.m.s.d.95, against 2.8 Å for the next best method. Its per-residue confidence, pLDDT, tracks true accuracy (r = 0.76 over 10,795 chains). Self-distillation on its own confident predictions for 350,000 unlabeled sequences improves it. Accuracy falls sharply when the alignment has fewer than about 30 sequences. | Active |
| LIT-584 | Accurate structure prediction of biomolecular interactions with AlphaFold 3 | Abramson et al. (Nature 2024), DOI 10.1038/s41586-024-07487-w. AlphaFold 3 predicts complexes of proteins, nucleic acids, ligands, ions and modified residues. A pairformer with lighter MSA processing replaces the Evoformer, and a diffusion module over raw atom coordinates replaces the structure module, with no built-in rotational equivariance. It beats Vina on PoseBusters without the pocket information Vina gets (P = 2.27×10⁻¹³). The generative head hallucinates structure in disordered regions, which cross-distillation from AlphaFold-Multimer reduces. Chirality is violated in 4.4% of PoseBusters predictions even after ranking penalties. | Active |
| LIT-585 | AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration | Lin et al. (2023), ARXIV-2306.00978. Weight-only low-bit quantization that protects the ~1% of channels whose activations are largest, by scaling them up before rounding rather than keeping them in higher precision. No backpropagation and no reconstruction, so it does not overfit its calibration set. | Active |
| LIT-586 v2 | LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale | Dettmers et al. (2022), ARXIV-2208.07339. Int8 inference for the feed-forward and attention projections, at no measured cost in quality up to 175B, by isolating a small set of systematically emergent outlier feature dimensions into a 16-bit multiplication and quantizing the other 99.9% vector-wise. | Active |
| LIT-587 | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale | Dosovitskiy et al. (2020), ARXIV-2010.11929. Cut the image into fixed patches, linearly embed them, and feed the sequence to an unmodified transformer encoder. The result is not that the architecture is better — it is that at enough pre-training data the convolutional prior stops paying for itself, and below that threshold it still does. | Active |
| LIT-588 v2 | Learning Transferable Visual Models From Natural Language Supervision | Radford et al. (2021), ARXIV-2103.00020. Train an image encoder and a text encoder to match 400M (image, text) pairs contrastively, then build a classifier for any label set by embedding its class names. The most durable result is not the zero-shot accuracy — it is that supervised adaptation which raises ImageNet by 9.2% buys no average robustness. | Active |
| LIT-589 | Representation Learning with Contrastive Predictive Coding | Oord et al. (2018), ARXIV-1807.03748. Predict the future in latent space and score it as a density ratio against sampled negatives, rather than reconstructing the observation. The loss is named InfoNCE here, and it is the objective CLIP, SimCLR and MoCo all run; the bound it maximizes is capped at log N, which the folklore about negatives leaves out. | Active |
| LIT-590 | Momentum Contrast for Unsupervised Visual Representation Learning | He et al. (2019), ARXIV-1911.05722. Reframes contrastive learning as dictionary look-up and asks for a dictionary that is both large and consistent: a queue makes it large independently of the batch, and an EMA-updated key encoder makes it consistent. The momentum ablation is the result that propagated — at m=0 it does not train at all. | Active |
| LIT-591 | A Simple Framework for Contrastive Learning of Visual Representations | Chen et al. (2020), ARXIV-2002.05709. Strips contrastive learning to augment, encode, project, NT-Xent — no memory bank, no special architecture — and then ablates each piece. The two findings that outlived the method: the augmentation pair is the pretext task, and the projection head is a sacrificial layer you throw away. | Active |
| LIT-592 | Latent Dirichlet Allocation | Blei, Ng and Jordan (2003), JMLR 3:993–1022. If the words of a document are exchangeable (bag of words), de Finetti’s theorem says the document is a mixture over a latent parameter. LDA makes that parameter a Dirichlet-distributed vector of topic proportions, one per document, with each topic a distribution over words. Fitted by variational EM, it has lower held-out perplexity than a unigram, a mixture-of-unigrams and a pLSI model on two corpora, and the two baselines overfit badly as the number of topics grows. Its 50 topic proportions stand in for 15,818 word features in Reuters classification. | Active |
| LIT-593 | Exploring Simple Siamese Representation Learning | Chen & He (2020), ARXIV-2011.10566. Strips a joint-embedding method to nothing: no negatives, no large batch, no momentum encoder — shared weights, a predictor on one side, a stop-gradient on the other. It works, which makes stop-gradient the candidate for the load-bearing piece, and contradicts BYOL’s account of its own method. | Active |
| LIT-594 | Bootstrap your own latent: A new approach to self-supervised Learning | Grill et al. (2020), ARXIV-2006.07733. Learns a representation with no negatives at all: an online network predicts a slow EMA copy of itself under a different augmentation. 74.3% ImageNet linear. Why it does not collapse is a hypothesis in the paper, not a result, and LIT-593 disputes which part is doing the work. | Active |
| LIT-595 | VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning | Bardes et al. (2021), ARXIV-2105.04906. Three explicit terms — a hinge keeping each embedding dimension’s standard deviation above a threshold, an invariance term, and a decorrelation term — applied to each branch separately. Needs no weight sharing, batch norm, stop-gradient or memory bank, and the variance term transplants into other methods and improves them. | Active |
| LIT-596 | Barlow Twins: Self-Supervised Learning via Redundancy Reduction | Zbontar et al. (2021), ARXIV-2103.03230. Removes every asymmetry — no predictor, no stop-gradient, no moving average — and prevents collapse in the objective instead: drive the cross-correlation matrix between the two branches’ outputs toward the identity. 73.2% ImageNet linear, and unlike its neighbours it wants a very high-dimensional output. | Active |
| LIT-597 | Reading Tea Leaves: How Humans Interpret Topic Models | Chang, Boyd-Graber, Gerrish, Wang and Blei (NIPS 2009). Two Mechanical Turk tasks measure whether a topic model’s latent space means anything to people. In word intrusion, subjects find the odd word out among a topic’s top five. In topic intrusion, they find the topic that does not belong to a document. On pLSI, LDA and CTM at 50, 100 and 150 topics over NYT and Wikipedia, CTM has the best held-out likelihood and the worst human scores. LDA is usually best. The paper concludes that likelihood does not track interpretability, and it reads the trend as a negative correlation. | Active |
| LIT-598 | Unsupervised Learning of Visual Features by Contrasting Cluster Assignments | Caron et al. (2020), ARXIV-2006.09882. Predict one view’s cluster code from another view’s representation instead of comparing features pairwise, with an equipartition constraint doing the anti-collapse work. 75.3% ImageNet linear — and multi-crop, which is the part that transplants into other methods for 2-4 points. | Active |
| LIT-599 v3 | DINOv2: Learning Robust Visual Features without Supervision | Oquab et al. (2023), ARXIV-2304.07193. The thesis is that existing self-supervised objectives already produce general-purpose features if the data is curated enough, so the contribution is a retrieval-based curation pipeline and the engineering to scale it — not a new objective. Frozen features, no finetuning, competitive with the best open weakly-supervised models. | Active |
| LIT-600 | Automatic Evaluation of Topic Coherence | Newman, Lau, Grieser and Baldwin (NAACL 2010). Humans rated 237 LDA topics from news and books for coherence. The paper then tested automated scores built from WordNet, Wikipedia and Google. The simplest measure, the average pointwise mutual information of a topic’s top-word pairs counted in 10-word windows over Wikipedia, reaches a Spearman correlation of about 0.77 with the human mean. Inter-annotator agreement, measured differently, is 0.73–0.82. WordNet measures are erratic, and Google drops from 0.80 on news to 0.51 on books. | Active |
| LIT-601 | Masked Autoencoders Are Scalable Vision Learners | He et al. (2021), ARXIV-2111.06377. Mask 75% of the patches, run the encoder on the visible quarter only, and reconstruct pixels with a small decoder you then throw away. 3x faster pretraining, 87.8% with ViT-Huge on ImageNet-1K alone. The 75% ratio, five times BERT’s 15%, is found by a sweep, and the paper explains it with an argument about the signal’s redundancy. | Active |
| LIT-602 v2 | GloVe: Global Vectors for Word Representation | Pennington, Socher and Manning (EMNLP 2014), DOI 10.3115/v1/D14-1162. Meaning shows in ratios of co-occurrence probabilities, not in the probabilities themselves, so word vectors should make differences encode log ratios. That leads to a weighted least-squares fit of wᵢ·w̃ⱼ + bᵢ + b̃ⱼ to log Xᵢⱼ over nonzero counts, with a weight capped at x_max = 100 and exponent 3/4. It scores 75.0% on the word analogy test with 42B tokens, and 71.7% against word2vec skip-gram’s 69.1% on the same 6B tokens. The word2vec side of that comparison ran at default settings. | Active |
| LIT-603 v3 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov, Sutskever, Chen, Corrado and Dean (NIPS 2013), ARXIV-1310.4546. Makes skip-gram fast and better. Negative sampling replaces the softmax with logistic regression against k noise words drawn from the unigram distribution raised to the 3/4 power. Frequent words are subsampled, with discard probability 1 − √(t/f) at t ≈ 10⁻⁵, which runs 2–10× faster and helps rare words. High-PMI bigrams become single tokens. Vector sums compose meanings, for example Russian + river ≈ Volga River. | Active |
| LIT-604 | Efficient Estimation of Word Representations in Vector Space | Mikolov, Chen, Corrado and Dean (2013), ARXIV-1301.3781. Two log-linear architectures drop the hidden layer of a neural language model. CBOW predicts a word from its averaged context, and skip-gram predicts the context from the word. The paper also introduces the Semantic-Syntactic Word Relationship test (8,869 semantic and 10,675 syntactic analogy questions), answered by vector arithmetic. On it, skip-gram at 300 dimensions reaches 53.3% on 783M words in about 3 days on one CPU, far ahead of earlier public embeddings. The comparisons are not controlled for data size or dimension. | Active |
| LIT-605 v2 | Sigmoid Loss for Language Image Pre-Training | Zhai et al. (2023), ARXIV-2303.15343. Replace CLIP’s softmax with a pairwise sigmoid, so the loss never needs a global view of the similarity matrix — no all-gathers, only a per-device block in memory, and better results below 16k batch. Then trains at batch size one million and finds the benefit saturated at 32k. | Active |
| LIT-606 | The Linear Representation Hypothesis and the Geometry of Large Language Models | Park, Choe and Veitch (ICML 2024), ARXIV-2311.03658. “Concepts are directions” has three readings: a shared difference between word pairs, a linear probe, and a steering vector. The paper formalizes the first with counterfactual pairs in the unembedding and embedding spaces, and proves the first is a probe and the second a steering vector. Training identifies the representation only up to an invertible linear map, so Euclidean cosine has no special meaning. An inner product under which causally separable concepts are orthogonal unifies the two spaces, and one such product is the inverse covariance of the unembedding rows. Evidence is 27 concepts on LLaMA-2 7B, with a Gemma-2B comparison in the appendix. | Active |
| LIT-607 v3 | Improving Distributional Similarity with Lessons Learned from Word Embeddings | Levy, Goldberg and Dagan (TACL 2015). Takes the design choices bundled into word2vec and GloVe (dynamic windows, subsampling, shifted PMI, context-distribution smoothing, adding context vectors, eigenvalue weighting), transfers them to count-based PPMI and SVD, and compares all four methods under the same tuning: 672 representations, 8 similarity and analogy datasets, and cross-validated selection. Once tuned alike, no method wins consistently. A single hyperparameter often matters more than the choice of method. SGNS beats GloVe on every task, reversing GloVe’s headline comparison. Smoothing the context distribution by 0.75 is the one setting that helps everywhere. | Active |
| LIT-608 | Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors | Baroni, Dinu and Kruszewski (ACL 2014). The first broad comparison of count-based distributional vectors (36 PPMI/LMI models, full, SVD and NMF) against word2vec CBOW vectors (48 models), on 14 lexical-semantics benchmarks and a 2.8B-token corpus. Predict wins on almost every task, often by 10 points or more, and is far more robust to its worst settings. The count models were not given word2vec’s context smoothing or negative shift, and SVD kept its singular-value weighting. LIT-607 traces the result to those differences and finds no consistent winner when both are tuned alike. | Active |
| LIT-609 v2 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov et al. (2013), ARXIV-1310.4546. Where negative sampling comes from — and it is NCE with the consistency deliberately dropped, kept only because the vectors stay good. Also: the number of negatives that helps falls as the dataset grows, which is the opposite of what the vision contrastive line later assumed. | Superseded — by LIT-603; a duplicate of LIT-603, filed concurrently by |
| LIT-610 v2 | Efficient Estimation of Word Representations in Vector Space | Mikolov et al. (2013), ARXIV-1301.3781. CBOW and Skip-gram: drop the non-linear hidden layer that made neural language models expensive, and word vectors become cheap enough to train on 1.6 billion words in under a day. The linear-analogy finding is here, and so is the test set built to stop it being an anecdote. | Superseded — by LIT-604; a duplicate of LIT-604, filed concurrently by |
| LIT-611 v2 | GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium | Heusel et al. (2017), ARXIV-1706.08500. Where FID comes from — and it comes from a subsection of a paper about two time-scale update rules and Nash equilibria. Fit a Gaussian to Inception activations for each distribution and take the Wasserstein-2 distance between them. Validated by monotonicity under six synthetic corruptions. | Active |
| LIT-612 | Neural Word Embedding as Implicit Matrix Factorization | Levy and Goldberg (NeurIPS 2014). With enough dimensions, skip-gram with negative sampling reaches its optimum when w·c = PMI(w, c) − log k, so it implicitly factorizes the word-context PMI matrix shifted by log of the number of negatives. NCE factorizes log P(w|c) − log k in the same way. The factorization is weighted: frequent pairs cost more to get wrong. Shifted PPMI and SVD over it match SGNS on word similarity. SGNS stays ahead on syntactic analogies, which the authors attribute to the weighting. | Active |
| LIT-613 | A Latent Variable Model Approach to PMI-based Word Embeddings | Arora, Li, Liang, Ma and Risteski (TACL 2016). A generative model of text: a discourse vector drifts slowly on a random walk, and each word is emitted with probability ∝ exp(⟨c, v_w⟩). If word vectors are isotropic, the partition function is nearly constant, and integrating out the discourse gives log p(w,w’) ≈ ‖v_w + v_w’‖²/2d − 2 log Z, so PMI ≈ ⟨v_w, v_w’⟩/d in low dimensions. Isotropy also explains why analogies survive large fitting error: solving for v_a − v_b is a regression that denoises by √(d/n). The squared norms of the fitted vectors correlate 0.75 with log frequency, and the partition function concentrates within about ±10%. | Active |
| LIT-614 | Improved Techniques for Training GANs | Salimans et al. (2016), ARXIV-1606.03498. Where the Inception Score comes from — exp(E_x KL(p(y|x) ‖ p(y))), proposed as a cheap stand-in for a Mechanical Turk study it correlates with, on 50k samples because part of what it measures is diversity. The incumbent LIT-611 displaced, and the paper that first tied generative evaluation to ImageNet class structure on purpose. | Active |
| LIT-615 | Demystifying MMD GANs | Bińkowski et al. (2018), ARXIV-1801.01401. Where KID comes from, and where the case against comparing FID numbers comes from. The plug-in FID estimator is biased, no unbiased estimator of FID exists, and at d=2048 with 50,000 samples it reliably returns the wrong ordering between two models with a standard deviation small enough to hide it. KID is the squared MMD in the same Inception features, with a cubic kernel and a simple unbiased estimator. | Active |
| LIT-616 | On Calibration of Modern Neural Networks | Guo et al. (2017), ARXIV-1706.04599. Modern networks are badly calibrated where their 1998-era ancestors were not, and depth, width, BatchNorm and less weight decay each make it worse while improving accuracy. The mechanism is that NLL overfits while 0/1 loss keeps improving. The remedy is one scalar: temperature scaling beats every richer method tried, including the ones that contain it. | Active |
| LIT-617 | Scaling Language Models: Methods, Analysis & Insights from Training Gopher | Rae et al. (2021), ARXIV-2112.11446. Six models on one dataset at one token budget, 44M to 280B, evaluated on 152 tasks — a controlled study of parameter scale. The gains are nonuniform: 51.2% of tasks improve by over 25%, 10.5% not at all, and Gopher is worse than its own smaller models on Abstract Algebra, Temporal Sequences and High School Mathematics. Also the allocation LIT-068 corrects: 280B parameters on 300B tokens is 1.07 tokens per parameter against Chinchilla’s 20. | Active |
| LIT-618 v2 | LTX-Video: Realtime Video Latent Diffusion | HaCohen et al., Lightricks (2024), ARXIV-2501.00103. It trades latent size for speed: a 32×32×8 VAE with 128 channels (1:192) moves patchification into the autoencoder, and the decoder does the last denoising step. It generates 5s of 768×512 video in about 2s on an H100 at 20 steps. The denoising decoder, the central idea, is supported by one internal preference study with no reported numbers. | Active |
| LIT-619 v2 | Wan: Open and Advanced Large-Scale Video Generative Models | Wan Team, Alibaba (2025), ARXIV-2503.20314. The open Wan2.1 video models, 1.3B and 14B, and the full recipe behind them. The recipe covers data curation, a 127M causal 3D VAE compressing 4×8×8, a cross-attention DiT trained by rectified flow on an image-then-video resolution curriculum, the 2D context parallelism it needs, and a mask-conditioned image-to-video variant. The abstract claims video scaling laws in data and model size, but the report plots no scaling curve. | Active |
| LIT-620 v2 | HunyuanVideo: A Systematic Framework For Large Video Generative Models | Kong et al., Tencent Hunyuan (2024), ARXIV-2412.03603. A 13B open video model: FLUX-style dual-then-single-stream transformer, a multimodal LLM as text encoder, a from-scratch causal 3D VAE. It fits the line’s only explicit video compute-optimal scaling law, on a proxy model that differs from the shipped one in objective, text encoder and conditioning. It reports no controlled ablations. | Active |
| LIT-621 v2 | Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models | Blattmann et al., LMU Munich and NVIDIA (2023), ARXIV-2304.08818. Video LDM turns a pretrained image latent diffusion model into a video model by inserting temporal layers and fine-tuning the decoder on video. Starting from the image model halves driving-scene FVD against end-to-end training (534 against 1155). The paper counts 2.2B of the text-to-video model’s 3.1B parameters as trained, and that count leaves out the spatial layers and decoder it also fine-tuned. The abstract describes training only temporal layers. | Active |
| LIT-622 v2 | CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer | Yang, Teng et al., Tsinghua and Zhipu AI (2024), ARXIV-2408.06072. The first widely used open DiT for text-to-video. It has a MAGVIT-style causal 3D VAE, one sequence over text and video with modality-specific adaLN, and full 3D attention. Full 3D attention beats factorized attention at 1.1× to 2.3× the forward time, rising with resolution. The architecture ablations are absent from the first arXiv version and present in the third, and the 2B model ships the position encoding they rejected. | Active |
| LIT-623 v2 | Language Model Beats Diffusion — Tokenizer is Key to Visual Generation | Yu et al., Google and CMU (2023), ARXIV-2310.05737. MAGVIT-v2 is a causal 3D-CNN tokenizer with lookup-free quantization. The causal design lets images and video share one tokenizer, though no experiment uses the video tokenizer for images, and LFQ lets generation keep improving as the vocabulary grows, where VQ turns worse. The title’s “beats diffusion” holds only with guidance, and the “language model” is a masked, MaskGIT-style one. | Active |
| LIT-624 v3 | Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model | Step-Video Team, StepFun (2025), ARXIV-2502.10248. A 30B text-to-video DiT on a 16×16×8 VAE that reconstructs as well as HunyuanVideo’s 8×8×4 on its own benchmark, though not on Open-Sora 2.0’s, so the model covers 204 frames. It also applies DPO to video (a 55/45 preference score, ties split) and runs at 32% MFU. It reaches parity with Movie Gen and loses to the strongest commercial model it names, despite the abstract’s “state-of-the-art”. | Active |
| LIT-625 v2 | Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets | Blattmann, Dockhorn, Kulal et al., Stability AI (2023), ARXIV-2311.15127. Video LDM’s architecture is held fixed, and the paper studies what the data does. Curating 4× fewer clips beats the uncurated set, and the advantage survives high-quality fine-tuning. It is the only controlled video-curation study in this line. One of its own filters lost the ablation and was kept anyway. | Active |
| LIT-626 v2 | Movie Gen: A Cast of Media Foundation Models | The Movie Gen team, Meta (2024), ARXIV-2410.13720. A 30B video model built from a Llama 3-style transformer, flow matching and an 8×8×8 temporal autoencoder, with the most controlled ablations in the video line. At 5B, flow matching beats diffusion and video captions beat frame captions. The “LLaMa3 scaling laws” claim rests on image-only runs at four sizes. | Active |
| LIT-627 v2 | Video Diffusion Models | Ho, Salimans et al., Google (2022), ARXIV-2204.03458. The first diffusion model for video: a U-Net factorized into space and time and trained jointly on videos and still frames. Adding eight still frames per video cuts FVD from 202 to 58 at fixed model, batch and steps, though not at fixed frames per step. The abstract’s explanation, lower gradient variance, is never measured. | Active |
| LIT-628 | Better plain ViT baselines for ImageNet-1k | Beyer, Zhai and Kolesnikov (2022), ARXIV-2205.01580. Four pages that move a plain ViT-S/16 on ImageNet-1k from 66.8% to 80.0% with five changes, none of them novel and none of them a regulariser. The original recipe scores 66.8 / 67.2 / 67.1 at 90 / 150 / 300 epochs — it cannot use more training at all. The belief the abstract names is that ViT needs sophisticated regularisation at this scale; the answer is that standard augmentation is sufficient. | Active |
| LIT-629 v3 | Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion | Huang et al., Adobe Research and UT Austin (2025), ARXIV-2506.08009. Self Forcing trains an autoregressive video model on its own rollouts, conditioning on self-generated context and scoring the whole clip with a distribution-level loss. On Wan2.1-1.3B it beats teacher-forced and diffusion-forced training under every objective tried, and holds quality where they degrade frame by frame. It reaches 17 fps at 0.69s latency on one H100. The comparison against slower models is confounded by a 14B teacher. | Active |
| LIT-630 v3 | Flow Matching for Generative Modeling | Lipman et al., Meta AI and Weizmann (2022), ARXIV-2210.02747. A continuous normalizing flow can be trained without simulation by regressing onto a per-sample conditional vector field. Diffusion paths are special cases. The straight “OT” path beats the diffusion path under the same objective, architecture and epochs, and it needs about 60% of the function evaluations. It used uniform timesteps. | Active |
| LIT-631 v3 | From Slow Bidirectional to Fast Autoregressive Video Diffusion Models | Yin et al., MIT and Adobe (2024), ARXIV-2412.07772. CausVid distills a bidirectional video DiT into a 4-step block-causal student with distribution matching, cutting first-frame latency from 219s to 1.3s at 9.4 fps. The controlled result is that a bidirectional teacher makes a better causal student than a causal one. The authors attribute this to error accumulation, but the causal teacher is also the weaker model, and the ablation does not separate the two. The paper does not claim real time, and it reports that the student flickers more and is less diverse. | Active |
| LIT-632 v2 | Make-A-Video: Text-to-Video Generation without Text-Video Data | Singer et al., Meta AI (2022), ARXIV-2209.14792. An unCLIP text-to-image model is extended into video with identity-initialized temporal layers trained on video whose captions are discarded. Text alignment comes entirely from images. That limits it: it “can not learn associations between text and phenomenon that can only be inferred in videos”. The paper has no ablations, and its title describes the training signal, not the data. | Active |
| LIT-633 v2 | AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning | Guo et al., CUHK, Shanghai AI Lab and Stanford (2023), ARXIV-2307.04725. Stable Diffusion 1.5 is frozen and only zero-initialized temporal “motion modules” are trained. The modules then plug into community checkpoints fine-tuned from the same base, with no tuning. Transfer is measured by one table against two baselines not built for the task. The first version’s limitations section, which reports failure on stylized checkpoints, was dropped from the second. | Active |
| LIT-634 v2 | Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k | The Open-Sora team, HPC-AI Tech (2025), ARXIV-2503.09642. An 11B video model initialized from FLUX, with most of its compute at 256px and a 768px text/image-to-video stage that is short in iterations but takes 37% of the compute. The “$200k” is one final diffusion-model run at an assumed H200 rental rate. It excludes data, captioning, failed runs, and the FLUX and HunyuanVideo-VAE weights it starts from. A 4×32×32 autoencoder is a further efficiency experiment, and the model adapted to it is unconverged and weaker. | Active |
| LIT-635 v2 | Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning | Girdhar et al., Meta GenAI (2023), ARXIV-2311.10709. Text-to-video is factored into generating an image and then a video conditioned on it, on a frozen text-to-image model. The report runs controlled human-preference ablations of the factorization, zero terminal SNR, low-then-high resolution training and a fine-tune on 1.6K clips filtered on motion, aesthetics and CLIP score together. Movie Gen builds on this recipe. Freezing the image layers beats fine-tuning them only narrowly (55/58). | Active |
| LIT-636 v4 | Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow | Liu, Gong and Liu, UT Austin (2022), ARXIV-2209.03003. It names rectified flow: regress a velocity onto x1 − x0 along the straight interpolation, which beats VP and sub-VP probability-flow ODEs on CIFAR-10 at the same architecture (FID 2.58 against 3.93). The “fast” in the title needs reflow or distillation. Without reflow or distillation one-step FID is 378. Distillation alone gives 6.18. One reflow gives 12.21 undistilled and 4.85 distilled, and full-simulation quality gets worse. | Active |
| LIT-637 v2 | Imagen Video: High Definition Video Generation with Diffusion Models | Ho et al., Google (2022), ARXIV-2210.02303. VDM scaled into a seven-model, 11.6B cascade producing 128 frames at 1280×768. It is the first report to add a separate image-text corpus (60M internal pairs plus LAION-400M) to joint training, and it asserts the benefit without an ablation. Its tested results are that v-prediction beats ε-prediction for video super-resolution, and that distillation to 8 steps is 18× faster at a small metric cost under constant guidance and none under oscillating guidance. | Active |
| LIT-638 | A Length-Extrapolatable Transformer | Sun et al. (2022), ARXIV-2212.10554. Defines attention resolution — a scalar for how well an attention pattern distinguishes token distance, estimable before training — and measures it across the extrapolation boundary. RoPE collapses 0.91 → 0.08 at twice its training length; ALiBi holds 0.81 → 0.88 and is the only incumbent that does. The record holds both papers and four practices about RoPE, and had no measure of this. | Active |
| LIT-639 | ResiDual: Transformer with Dual Residual Connections | Xie et al. (2023), ARXIV-2304.14802. Gives the representation-collapse argument a rate: in a Pre-LN transformer the per-layer change in the normalised hidden state decays as O(1/√k), and adding a layer to an N−1 layer model moves the output by O(1/√N). Also measures what Pre-LN’s no-warmup claim costs — 32.28 against 35.12 BLEU at E6D6, and 31.82 against 35.18 at E12D12. | Active |
| LIT-640 v3 | Query-Key Normalization for Transformers | Henry, Dachapally, Pawar and Chen (2020), ARXIV-2010.04245. The origin of QK-norm, which thirty documents in this record name and SOTA-192 recommends while sourcing an adopter. ℓ₂-normalize each query and key along the head dimension before the dot product, then scale by a learnable parameter instead of dividing by √d — so the temperature becomes a trained quantity rather than a constant. +0.928 BLEU averaged over five low-resource pairs. | Active |
| LIT-641 | Attention Approximates Sparse Distributed Memory | Bricken and Pehlevan (2021), ARXIV-2111.05498. Attention’s update rule is Kanerva’s sparse distributed memory read, given two conditions: L² normalized vectors and a fitted softmax temperature β. The retrodiction is the interesting part — those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], the range that interpolates between SDM’s optimality criteria. | Active |
| LIT-642 | Professor Forcing: A New Algorithm for Training Recurrent Networks | Lamb, Goyal et al., MILA (2016), ARXIV-1610.09038. A discriminator is trained to tell a recurrent network’s hidden-state dynamics under teacher forcing from its dynamics when free-running, and the generator is trained to fool it alongside the usual likelihood loss. Human raters prefer its samples for handwriting generated 20 times past the training length and for raw audio. The likelihood gains are small, the sequential-MNIST claim has no teacher-forcing baseline, and word-level language modeling showed no difference. | Active |
| LIT-643 | One-step Diffusion with Distribution Matching Distillation | Yin et al., MIT and Adobe (2023), ARXIV-2311.18828. DMD distills a diffusion model into a one-step generator by descending an approximate KL(p_fake ‖ p_real), whose gradient is the difference of a frozen “real” score and an online “fake” score. It adds an LPIPS regression loss onto precomputed teacher ODE outputs. It reaches 2.62 FID on ImageNet-64 and 11.49 on zero-shot COCO-30k. The ablation that makes the regression loss look necessary ran its no-regression arm at a different learning rate, and its mode-collapse evidence is qualitative. | Active |
| LIT-644 v2 | Building Normalizing Flows with Stochastic Interpolants | Albergo and Vanden-Eijnden (2022), ARXIV-2209.15571. A continuous normalizing flow can be trained without ODE solves by regressing a velocity onto the time derivative of any interpolant between two densities. It is one of three concurrent origins of this simulation-free objective, with Flow Matching and Rectified Flow. Its path is not the straight line: every experiment uses the trigonometric cos/sin interpolant, which SiT later calls GVP. The image results arrived only in v3, and they are quoted against diffusion baselines, not matched against them. | Active |
| LIT-645 | Stochastic Interpolants: A Unifying Framework for Flows and Diffusions | Albergo, Boffi and Vanden-Eijnden (2023), ARXIV-2303.08797. The earlier interpolant objective is extended with a latent Gaussian term. One learned velocity and score then give an ODE and a family of SDEs, all sharing the interpolant’s marginals, and the SDE’s diffusion coefficient can be chosen after training. For SDEs the regression losses bound the KL, and for the ODE they do not. This is the framework SiT instantiates. The evidence that SDE sampling beats ODE sampling is a 2-D checkerboard and a 128-D Gaussian mixture. The paper reports no image metric and no diffusion baseline. | Active |
| LIT-646 | Improved Distribution Matching Distillation for Fast Image Synthesis | Yin et al., MIT and Adobe (2024), ARXIV-2405.14867. DMD2 drops DMD’s regression loss. It stabilizes training by updating the fake score five times per generator step, adds a GAN loss on real images, and trains multi-step students on their own simulated inputs. The ImageNet ablation is clean: removing the loss costs 2.62 → 3.48 FID, and the update rule restores 2.61. The student is slightly less diverse than its teacher (LPIPS diversity 0.61 against 0.64). The paper does not attribute this to the reverse KL. | Active |
| LIT-647 | Sequence Level Training with Recurrent Neural Networks | Ranzato, Chopra, Auli and Zaremba, Facebook AI Research (2015), ARXIV-1511.06732. The paper that named exposure bias. MIXER anneals a cross-entropy-trained RNN toward REINFORCE on the test metric over its own samples, and beats cross-entropy by 1 to 3 points on three tasks with greedy decoding. Its baselines that avoid exposure bias without a sequence-level reward do not reliably help, so the gain is not shown to come from fixing exposure bias. | Active |
| LIT-648 | Improving Image Generation with Better Captions | Betker et al., OpenAI and Microsoft (2023), no arXiv id. Identical T5-conditioned latent diffusion models are trained on the same images with alt-text, short synthetic or descriptive synthetic captions. The descriptive ones give higher CLIP score, and more synthetic in the blend is better up to the 95% tested. The controlled evidence is CLIP score alone, at 256px, and on ground-truth prompts the gain is inside the checkpoint noise. The paper does not describe the DALL-E 3 model, and it says its DALL-E 3 comparisons are not a measurement of recaptioning. | Active |
| LIT-649 | Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks | Bengio, Vinyals, Jaitly and Shazeer, Google (2015), ARXIV-1506.03099. During training, each previous-token input is replaced by the model’s own prediction with a probability that grows on a schedule. On captioning, parsing and frame-level speech recognition with small LSTMs this beats training on ground truth only, and training only on the model’s own output does badly. The gains are single runs with no variance reported. Feeding uniform random tokens recovers part of the captioning gain, and the paper never measures how errors grow with sequence length. | Active |
| LIT-650 | Competition-Level Code Generation with AlphaCode | Li et al. (2022), ARXIV-2203.07814. Top 54.3% on ten Codeforces contests, by sampling up to a million programs per problem and filtering 99% of them away on the example tests. Two things the record needs are incidental to that: HumanEval’s false-positive rate, hand-measured at 30%, and the fact that pass@k assumes every sample can be submitted and is therefore an upper bound on what a system with a submission budget achieves. | Active |
| LIT-651 | Transformers without Tears: Improving the Normalization of Self-Attention | Nguyen and Salazar (2019), ARXIV-1910.05895. The first systematic evaluation of pre-norm in the base Transformer regime — and it reports a result the record’s universal pre-norm practice does not carry: on high-resource WMT’14 English-German, post-norm wins, 27.58 to 26.83. Also introduces ScaleNorm, the ℓ₂-normalization-with-one-learned-scalar that LIT-640’s QK-norm is an application of. | Active |
| LIT-652 v2 | Video generation models as world simulators | Brooks, Peebles et al., OpenAI (2024), no arXiv id. Sora is a latent diffusion transformer over spacetime patches of a video autoencoder’s latents. It is trained jointly on images and videos at their native durations, resolutions and aspect ratios, with captions from a DALL-E 3-style recaptioner. The report withholds model and implementation details, and it has no table, number, metric or baseline. Every claim, including the scaling comparison and the square-crop ablation, is shown by example videos only. The video reports filed after it cite it and benchmark against it, and it supplies no measurement of its own. | Active |
| LIT-653 | Softmax is not Enough (for Sharp Size Generalisation) | Veličković, Perivolaropoulos, Barbero and Pascanu (2024), ARXIV-2410.01104. Twenty-three practices in this record name the softmax and no note held it. This one proves a limit on it: in a Transformer over a finite vocabulary, attention coefficients must decay towards zero as the number of items grows, so a head that is sharp in distribution cannot stay sharp on larger inputs. Max retrieval falls from 98.6% at 16 items to 12.4% at 16,384. | Active |
| LIT-654 | Muon Outperforms Adam in Tail-End Associative Memory Learning | Wang et al. (2025), ARXIV-2509.26030. Ninety-four documents in this record name Muon and seventeen practices recommend or condition on it; five theories mention it and none says why it works. This one ablates Muon block by block: applying it to the value-output weights and the FFN alone almost recovers the full-Muon trajectory, and applying it to query-key contributes little. The account is that Muon’s update matches the outer-product structure of associative memory, giving isotropic spectra and better learning of rare classes. | Active |
| LIT-655 | Self-attention Does Not Need O(n²) Memory | Rabe and Staats (2021), ARXIV-2112.05682. Six months before FlashAttention: attention in O(1) memory per query, self-attention in O(log n), a practical O(√n) implementation, exact rather than approximate, 59× less memory overhead at sequence length 16,384. And the number the record needed and did not have: on TPU the same algorithm is “within a few percent of the runtime of the standard implementation” — the memory saving is algorithmic and the speedup is not. | Active |
| LIT-656 | Intriguing Properties of Quantization at Scale | Ahmadian et al. (2023), ARXIV-2305.19268. A controlled study from 410M to 52B, same architecture, varying weight decay, dropout, gradient clipping and training dtype: outlier dimensions are not an inherent product of scale but of the optimization conditions during pre-training. Their 52B model gains 0.08% under plain INT8 (77.31 → 77.39 average) where OPT-66B loses about 42%. The record’s largest cluster had the emergence framing in an Active practice’s title. | Active |
| LIT-657 | Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution | Dehghani, Mustafa et al., Google DeepMind (2023), ARXIV-2307.06304. Packs patches from several images into each fixed-length sequence, with per-example attention masks and pooling. A ViT can then take any aspect ratio and a sampled resolution per image. It leads ViT at each of 12 compute-matched budgets, and matches the best ViT with 4x less compute. The paper attributes the gain chiefly to seeing about 5x more images, not to aspect ratio. The one test that isolates aspect ratio is a linear probe on fairness attributes. | Active |
| LIT-658 | FiT: Flexible Vision Transformer for Diffusion Model | Lu et al. (2024), ARXIV-2402.12376. A DiT variant trained on images without cropping or upsampling. Large images are only resized down to at most 256² pixels at native aspect ratio, and sequences are padded to 256 tokens. Table 3 isolates this pipeline change on DiT-B/2 at 400K steps. At a square 256² output, FID moves 44.83 → 43.34 and sFID worsens 8.49 → 11.11. At 160×320 and 224×448, FID falls from 91.32 and 109.1 to 50.51 and 52.55, but the fixed model never trained on those shapes. | Active |
| LIT-659 | On the Importance of Noise Scheduling for Diffusion Models | Chen, Google Brain (2023), ARXIV-2301.10972. Higher-resolution images are less corrupted by the same per-pixel noise, so the best schedule moves with resolution. It sweeps schedule functions (Table 3) and an input scaling factor b, with x_t = sqrt(γ)·b·x0 + sqrt(1−γ)·ε (Table 4), on ImageNet 64, 128 and 256 with RIN. The best b falls as resolution rises. At 256 with a 1−t schedule, b = 1 gives FID 7.21 and b = 0.4 gives 3.52. This is concurrent with simple diffusion and is the second origin of the resolution-dependent shift. | Active |
| LIT-660 | simple diffusion: End-to-end diffusion for high resolution images | Hoogeboom, Heek and Salimans, Google Brain (2023), ARXIV-2301.11093. Trains single-stage pixel-space diffusion up to 512x512 with four changes. The first is to define the noise schedule at a reference resolution and shift its log-SNR by 2·log(ref/d) at resolution d. Table 2 is the controlled measurement. On ImageNet 256 the unshifted cosine schedule gives FID 7.65 train / 6.87 eval, and the schedule shifted to 32 gives 3.76 / 3.71. This is the origin, with the concurrent Chen (2023), of the resolution-dependent schedule shift SD3 later derives again. | Active |
| LIT-661 | Photorealistic Video Generation with Diffusion Models | Gupta, Yu et al., Stanford and Google (2023), ARXIV-2312.06662. A latent video diffusion transformer over a causal image-and-video encoder. It alternates per-frame spatial attention with local spatiotemporal windows. Two single-run ablations bear on the record’s video practices. Adding a separate corpus of ~970M image-text pairs to ~89M text-video pairs takes zero-shot UCF-101 FVD from 598.8 to 344.5 at 419M (Table 5), with compute not stated. And windowed attention matches full self-attention on UCF-101 (FVD 55.3–59.6 against 59.9) while training 1.7x faster (Table 3b). | Active |
| LIT-662 | Vision Transformers Need Registers | Darcet, Oquab, Mairal and Bojanowski (2023), ARXIV-2309.16588. 107 documents in this record say “ViT” and none held this. Large trained ViTs put high-norm outlier tokens on background patches — 2.37% of tokens in DINOv2 ViT-g — which hold little local information and a lot of global information. The model is using redundant patches as scratch space. Append a few tokens whose only job is to be scratch space and the artifacts vanish. | Active |
| LIT-663 | DINOv3 | Siméoni et al. (2025), ARXIV-2508.10104. The finding the record needs is not the model: train a self-supervised ViT long enough and its dense features rot while its global metrics keep improving. Segmentation mIoU declines after ~200k iterations and for the 7B falls below its early levels, while ImageNet classification rises monotonically throughout. Gram anchoring — pull the student’s patch Gram matrix towards an early teacher’s — repairs it, and it is a different defect from the register artifacts. | Active |
| LIT-664 | Emerging Properties in Self-Supervised Vision Transformers | Caron et al. (2021), ARXIV-2104.14294. The origin of the DINO line, which 31 documents here reference and 12 practices touch. Self-supervised ViT features carry explicit semantic segmentation that supervised ViTs and convnets do not show as clearly, and reach 78.3% ImageNet top-1 with a plain k-NN — no finetuning, no linear probe, no augmentation. Collapse is avoided by centering and sharpening the teacher output alone, with no predictor. | Active |
| LIT-665 | Transformer Language Models without Positional Encodings Still Learn Positional Information | Haviv, Ram, Press, Izsak and Levy (2022), ARXIV-2203.16634. The result the record’s position-encoding cluster rests on, credited here to a paper a year younger. A decoder-only LM with no positional encoding at all is competitive; probing shows it acquires absolute position within four layers. The mechanism is the causal mask, and the control that shows it is a bidirectional MLM, where NoPos perplexity is 147.18 against 4.00. | Active |
| LIT-666 v2 | Sparse Distributed Memory | Kanerva (1988), MIT Press. An associative memory in a high-dimensional binary space: a pattern is written to every fixed-address neuron within a Hamming radius of it, and read back as the thresholded sum over the neurons within that radius of the query. Held because LIT-641 shows attention’s softmax approximates this read — the weighting it induces, the intersection of two hyperspheres, decays near-exponentially in distance — and the record carried that account (THEORY-097) without its source. Filed from secondary accounts; the book has not been read here. | Active |
| LIT-667 | Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization | Wang, Yue, Su and Sun (2024), ARXIV-2405.15071. Grokking outside algorithmic data, on reasoning over stored facts — and a named correction to the critical-data-size hypothesis the record holds as THEORY-071. With the inferred/atomic ratio φ held at 9.0, varying the training-set size changes nothing; varying φ at fixed size moves grokking speed monotonically and at φ = 18.0 all but removes it. Also traces the generalizing circuit for two tasks and explains from its layout why composition never generalizes out of distribution while comparison does. | Active |
| LIT-668 v2 | ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | Lan, Chen, Goodman, Gimpel, Sharma and Soricut (2019), ARXIV-1909.11942. The measurement behind cross-layer parameter sharing, which this record started recommending — via the fix THEORY-103 endorses — without holding a single paper on it. The sharp result is an asymmetry: sharing the attention parameters across every layer costs +0.1 average at E = 128, and sharing the feed-forward parameters costs −1.4. Also carries a priority claim the record’s dropout practice should have: the first report that dropout hurts a large transformer. | Active |
| LIT-669 | Sparse Distributed Memory and Related Models | Kanerva (1992), RIACS TR 92.10, published in 1993 in Hassoun (ed.), Associative Neural Memories (Oxford). SDM restated as a matrix memory: a fixed random address matrix A, a wide layer of hard locations, and a modifiable contents matrix C trained by the outer-product rule. Stored words are weighted by the overlaps of activation sets. The activation probability that maximizes signal-to-noise is p = (2MT)^(-1/3), and capacity is about 10% of the number of locations at one bit error in a thousand. This is the source of two of the three optimal radii LIT-641 fits β against. The third, critical distance, is not in it. | Active |
| LIT-670 | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Devlin, Chang, Lee and Toutanova (2018), ARXIV-1810.04805. The model 89 files in this record referred to and none held. Two things it is cited for here turn out not to be in it: the 15% masking rate was never swept — “in all of our experiments, we mask 15%”, and the one appendix table headed “Masking Rates” varies the 80/10/10 replacement split at a fixed 15% — and its NSP ablation removes the loss while keeping the paired-segment input format, which LIT-671 later separates and reverses. | Active |
| LIT-671 | RoBERTa: A Robustly Optimized BERT Pretraining Approach | Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov (2019), ARXIV-1907.11692. A replication study that finds BERT was significantly undertrained and, more usefully here, that BERT’s own NSP ablation could not have shown what it claimed: it removed the loss while keeping the paired-segment input format. Separate the two and removing NSP matches or slightly improves. The paper is careful about its own margins — dynamic masking is “comparable or slightly better”, and byte-level BPE is adopted despite being slightly worse. | Active |
| LIT-672 | Should You Mask 15% in Masked Language Modeling? | Wettig, Gao, Zhong and Chen (2022), ARXIV-2202.08005. The sweep BERT never ran, and the answer the record has been openly missing: 15% is not universally optimal, and the optimum tracks model size — 40% at 354M, 20% at 124M, 15% at 51M. It also decomposes the masking rate into a corruption rate and a prediction rate whose effects are antagonistic, and uses that to find BERT’s 80-10-10 substitution rule worse than plain [MASK]. | Active |
| LIT-673 | Averaging Weights Leads to Wider Optima and Better Generalization | Izmailov, Podoprikhin, Garipov, Vetrov and Wilson (2018), ARXIV-1803.05407. Average the weights visited along the tail of an SGD trajectory under a cyclical or high constant learning rate. The paper LIT-478 named as the first thing missing from this record’s weight-space ensembling coverage, and the baseline SOTA-288’s promotion condition asks for. From pretrained torchvision models, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at the inference cost of one model. One gotcha, stated in the paper: batch normalization statistics have to be recomputed with a pass over the data. | Active |
| LIT-674 | Robust fine-tuning of zero-shot models | Wortsman, Ilharco, Kim, Li, Kornblith, Roelofs, Gontijo-Lopes, Hajishirzi, Farhadi, Namkoong and Schmidt (2021), ARXIV-2109.01903. Fine-tuning a zero-shot model buys target accuracy and costs robustness. Linearly interpolating the zero-shot and fine-tuned weights recovers both: at α = 0.5, +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model while reference accuracy falls by at most 0.3 pp and usually rises. No extra cost at training or inference. And it names the precondition: averaging all the layers of two unrelated networks “typically fails, achieving no better accuracy than a randomly initialized neural network” — this works because the trajectory is shared. | Active |
| LIT-675 | Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time | Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith and Schmidt (2022), ARXIV-2203.05482. The dependency SOTA-407’s consensus note named as unheld. A hyperparameter sweep throws away every model but one; average them instead. The greedy recipe — sort by held-out accuracy, add each model only if held-out accuracy improves — is “no worse than the best individual model on the held-out validation set” by construction, at O(1) inference cost against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member. It does not move that practice’s consensus, because it is the same first author. | Active |
| LIT-676 v4 | DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models | Lu, Zhou, Bao, Chen, Li and Zhu (2022), ARXIV-2211.01095. The regime SOTA-203 does not mention. At a classifier guidance scale of 8.0 on ImageNet 256×256, FID at 10 function evaluations is 13.04 for first-order DDIM, 114.62 for DPM-Solver-2 and 164.74 for DPM-Solver-3 — higher order is monotonically catastrophic, and the practice’s own framing of DDIM as “the least accurate member of the family” inverts. Two causes, one of which the record already holds as SOTA-202. The fix is order 2, multistep, on the data-prediction parameterization. | Active |
| LIT-677 | Diffusion Models With Learned Adaptive Noise | Sahoo, Gokaslan, De Sa and Kuleshov (NeurIPS 2024), ARXIV-2312.13236. MuLAN replaces VDM’s scalar learned schedule with a per-pixel schedule conditioned on a learned auxiliary latent. At equal steps and architecture the bound moves 2.65 → 2.60 bits/dim on CIFAR-10, and it reaches VDM’s 2.65 in 2M steps instead of 10M. Its own ablations confirm VDM’s invariance where VDM claimed it. A scalar schedule conditioned on the input buys nothing, and a multivariate schedule conditioned on time only is “comparable to that of VDM”. The headline 2.55 is a different estimator. | Active |
| LIT-678 v2 | Diffusion Meets Flow Matching: Two Sides of the Same Coin | Gao, Hoogeboom, Heek, De Bortoli, Murphy and Salimans (2024), a blog post at diffusionflow.github.io. With a Gaussian source, flow matching is a diffusion model with the schedule α_t = 1−t, σ_t = t, and the flow-matching Euler sampler is DDIM. What separates the named recipes is three choices: the loss weighting, the network output and the sampling schedule. “Straight paths” holds only for a single-point target, and a variance-preserving schedule can be straighter for wide data. The content is derivation and toy illustration only: no trained models and no benchmarks. | Active |
| LIT-679 | Dynamic Chunking for End-to-End Hierarchical Sequence Modeling | Hwang, Wang and Gu (2025; ICLR 2026), ARXIV-2507.07955. H-Net is a byte-level hierarchical model that learns where to cut chunks. Compared with a GPT-2 BPE Transformer at matched bytes per batch and FLOPs per byte, a chain of baselines separates each ingredient. One stage of learned chunking only ties BPE and whitespace chunking (0.728 against 0.730 against 0.726 bits/byte at XL). The gain is the second stage (0.715, +2.7 downstream average). The largest gains are on Chinese. It is limited to 1.3B-equivalent FLOPs and trains about 2× slower in wall-clock. | Active |
| LIT-680 | Distilling the Knowledge in a Neural Network | Hinton, Vinyals and Dean (NIPS 2014 Deep Learning Workshop), ARXIV-1503.02531. This is where the word “distillation” in 112 files of this record comes from. It trains a student on the teacher’s softmax at a raised temperature, adds a down-weighted hard-label term, and scales the soft term by T². On speech, a 10-model ensemble distils into one model of the same size as each member, keeping 86% of the frame-accuracy gain and all of the 0.2-point WER gain. Three things usually quoted from it are not in it: the phrase “dark knowledge”, a high temperature on the realistic task (the best was T = 2), and an ablation of T². | Active |
| LIT-681 | Cramming: Training a Language Model on a Single GPU in One Day | Geiping and Goldstein (ICML 2023), ARXIV-2212.14034. A BERT-style MLM trained from scratch for 24 hours on one consumer GPU, which reaches 79.2–80.4 GLUE against BERT-base’s 80.9. The transferable finding is negative: model shape does not beat the scaling law at this budget. Every reshape it tried lands near the same final loss, because a smaller model’s extra throughput cancels its slower learning. So the only architectural lever is to cut step time at constant parameter count. Its data result is the cheapest filter in the record: drop text the tokenizer compresses badly. | Active |
| LIT-682 | ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models | Defazio (2026), ARXIV-2605.19095, a single-author tech report from the author of LIT-213. Plain Schedule-Free fails at large batch (“a scaling ‘cliff’ at 2M tokens per batch”). Its small weight decay was “compensating for gradient norm drift, in a somewhat hacky way”. ScheduleFree+ fixes both: it restores inner momentum and adds Polyak step sizes with AdamC-style weight decay. From 120M to 2B it beats WSD at every size, and it beats tuned linear decay except at the largest short runs. The ladder changes six things at once, and the one quantified margin, 31%, is for the 120M model only. | Active |
| LIT-683 | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach | Geiping et al. (NeurIPS 2025), ARXIV-2502.05171. Huginn has a 2-layer prelude, a 4-layer core iterated r times with the input re-injected every iteration, and a 2-layer coda: 3.5B parameters trained on 800B tokens with r sampled per step (mean 32) and backpropagation through only the last 8 iterations. Accuracy rises with test-time iterations and saturates per task. Recurrence is shown to buy accuracy per parameter, not per FLOP. The only controlled baseline has the same parameters and about 1/16 of the compute. “Equivalent to 50B parameters” is a FLOP count, not a comparison. | Active |
| LIT-684 | Model Merging in Pre-training of Large Language Models | Li et al. (ByteDance Seed, 2025), ARXIV-2505.12082. A uniform average of about ten recent checkpoints on a constant-learning-rate (WSD stable-phase) run beats the latest checkpoint at every size tested, up to 20B/200B MoE and 70B dense. It roughly tracks where an anneal would land. The travelling claim, that constant LR plus merging “can effectively match the performance of an annealed model at any point”, rests on one fork of one 1.3B/13B model. At the end of that fork’s 250B-token decay, the annealed model is about 1.5 MMLU ahead. The baselines in the big tables are unannealed checkpoints. | Active |
| LIT-685 | Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs | Chen, Wang, Xiao, Wang, Chen, Cai, He and Wang (ACL 2025), ARXIV-2507.10613, published at ACL as “Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies”. It claims that gains decelerate faster than a power law when data is “dense” or when tokens per parameter run high, and fits a modified Chinchilla law over more than 400 models from 20M to 7B. None of it should be leaned on. The density metric’s embedding, dimension and cluster count are never stated. The fitted law has no density term, only two logistic multipliers on D/N bounded in [1.5, 2). And the paper’s own Table 4 contradicts its “across all model sizes”. | Active |
| LIT-686 | On the distance between two neural networks and the stability of learning | Bernstein, Vahdat, Yue and Liu (NeurIPS 2020), ARXIV-2002.03432. A deep network’s trust region is not quadratic. The change in its function and Jacobian is bounded by a product of per-layer relative perturbations (“deep relative trust”), and Figure 1 shows loss and gradient breaking down quasi-exponentially in that quantity at depth 16. The optimizer it derives, Fromage, is LARS plus a norm-correcting prefactor. It is the Frobenius antecedent of the modular-norm line. LIT-437, which shares an author, names its conditioning assumption as the flaw and replaces Frobenius with spectral. | Active |
| LIT-687 v2 | Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think | Martin Garcia, Knaebel, Schmidt, de Geus, Hermans and Leibe (WACV 2025), ARXIV-2409.11355. Marigold’s “dismal” few-step depth came from the DDIM scheduler’s leading timestep spacing. The spacing hands a pure-noise input a timestep that claims an almost clean one. Switching to trailing spacing makes one step usable (NYUv2 AbsRel 5.7). The fixed single step is still behind the 50-step, 10-member ensemble on all five depth sets. What beats that ensemble everywhere is end-to-end fine-tuning as a one-step model with a task loss, and it works nearly as well from plain Stable Diffusion. | Active |
| LIT-688 v2 | Reconciling Kaplan and Chinchilla Scaling Laws | Pearce and Song (TMLR 2024), ARXIV-2406.12907. Kaplan’s N_opt ∝ C^0.73 against Chinchilla’s C^0.50 is mostly a counting artifact. Kaplan counted non-embedding parameters and fit at small scale, where embeddings are much of the model. Simulating the Chinchilla study under those conditions gives 0.74–0.78. The result is a simulation from Chinchilla’s fitted surface, extrapolated four or more orders of magnitude below its range. The experiments are five models of 0.8–4.6M parameters at context length 16. They also find that the learning-rate schedule, which Chinchilla blamed, is not the main cause. | Active |
| LIT-689 | Common Diffusion Noise Schedules and Sample Steps are Flawed | Lin, Liu, Li and Yang (WACV 2024), ARXIV-2305.08891. Common schedules leave signal at the last timestep. Stable Diffusion’s x_T is 0.068·x₀ + 0.998·ε. The model learns to keep the leaked per-channel mean, and pure-noise inference then pins samples to “medium brightness”. Many samplers also never start at t = T. It proposes four fixes: zero terminal SNR, v-prediction, sampling from the last timestep, and a guidance rescale. The only numeric result applies all four at once (COCO FID 22.96 → 21.66). Every single-fix ablation is qualitative, on one to three prompts. It credits the trailing discretization to DPM-Solver, and it never calls the spacing fix “slight”. | Active |
| LIT-690 | Resolving Discrepancies in Compute-Optimal Scaling of Language Models | Porian, Wortsman, Jitsev, Schmidt and Carmon (NeurIPS 2024), ARXIV-2406.19146. It reproduces Kaplan’s allocation exponent (0.835 on RefinedWeb, 0.864 on OpenWebText2) over 900+ runs from 5M to 901M, then removes it in steps. Counting the head’s FLOPs takes 0.129 off, a warmup short enough for small models takes 0.104, and tuning learning rate, batch and β₂ per size, with a constant learning rate, takes 0.105, landing on 0.497. Chinchilla’s proposed cause, matching the cosine decay to each run, moves it 0.031 on a side branch. “careful learning rate decay is not essential for the validity of their scaling law.” | Active |
| LIT-691 | Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation | Ke, Obukhov, Huang, Metzger, Daudt and Schindler (CVPR 2024), ARXIV-2312.02145. Marigold fine-tunes Stable Diffusion v2’s U-Net on 74K synthetic images to denoise a depth latent conditioned on the image latent, with the VAE frozen. It samples 50 DDIM steps and ensembles 10 runs, and ranks first on average across five zero-shot depth benchmarks. Its step-count curve, one step at ≈34% NYUv2 AbsRel, was measured under diffusers’ default leading spacing, which the paper never names. That is the scheduler bug LIT-687 corrects. The claim that a generative prior is what helps is not tested. | Active |
| LIT-692 | Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation | Kingma and Gao (NeurIPS 2023), ARXIV-2303.00848. Every standard diffusion loss is ½∫ w(λ) E‖ε̂ − ε‖² dλ for an implied weighting w. Given w, the loss is invariant to the schedule’s interior shape, and the schedule is only an importance sampler. If w falls monotonically in log-SNR, the loss is the expected ELBO of Gaussian-noise-augmented data. App. D.3 derives that flow matching’s weighting equals v-MSE’s under a cosine schedule. In its 64×64 ablations the training schedule makes no difference, and the weighting and the sampler each move FID by 0.1–0.3. Monotone weightings tie EDM. They do not beat it. | Active |
| LIT-693 v3 | Classifier-Free Diffusion Guidance | Ho and Salimans (2022), ARXIV-2207.12598. The technique SOTA-202, SOTA-203 and SOTA-410 all turn on, which this record had never held. Drop the conditioning on a fraction of training examples and one network learns both scores; at sampling, extrapolate away from the unconditional one. The trade it buys is severe and measured: on ImageNet 64×64, w from 0.1 to 4.0 moves FID from 1.55 to 26.22 while IS goes 66.11 to 260.2. Any FID or IS comparison across guidance scales is therefore meaningless, which is the missing source for SOTA-307’s instruction to search the scale per cell. | Active |
| LIT-694 | Rigor with Machine Learning from Field Theory to the Poincaré Conjecture | Gukov et al. (2024), ARXIV-2402.13321. Stochastic, black-box ML can still yield rigorous results in mathematics and physics along two routes. The first is conjecture generation followed by human proof, or RL searches whose outputs can be checked exactly; this is how an RL/Bayesian-optimized ribbon verifier ruled out over 800 candidate counterexamples to the smooth 4D Poincaré conjecture. The second is importing ML theory, such as NNGP/NN-field-theory and gradient-descent metric flows that include Perelman’s Ricci flow. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-695 | Language Design as Information Renormalization | Gallego et al. (2022), ARXIV-1708.01525. The paper models Chomsky’s MERGE as a coarse-graining tensor, so that sentence probabilities form mostly loop-free stochastic tensor networks. It argues from this that tree-like syntax gives polynomially decaying (long-range) correlations. It then concludes that tree-structured networks (it names deep CNNs, as Tree Tensor Networks) fit language, while MPS-like models (RNNs, HMMs) with exponentially decaying correlations do not. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-696 v2 | Rarely categorical, highly separable representations along the cortical hierarchy | Posani et al. In 14,000+ IBL Neuropixels units (4,617 passing a ΔR² ≥ 0.015 selectivity threshold), the clustering of 8-D encoding-model selectivity vectors beats a covariance-matched Gaussian null in only a handful of the ~20 testable cortical regions (VISp, AUDp, SSp-ul at Bonferroni p < 0.05; clustering z falls with hierarchy position, ρ = −0.62). Pooled modules and the whole cortex do cluster (whole-cortex z = 8.0). Among the 16 regions with enough trials, once the 16 task conditions are merged into pairwise-decodable “independent” ones (M_IC from 5 to 16, rising with hierarchy, ρ = 0.77), ≥ 95% of random balanced dichotomies are linearly decodable above a shuffle null in every region except gustatory cortex (≈0.82). | Active — read in full 2026-09-25 (NOTE-353); a careful, well-controlled, openly reviewed large-scale study worth reading for its method (Gaussian-null categoricality test, independent-conditions merging, dichotomy separability) and for its explicit warning that above-chance linear decodability of one variable is weak evidence in a high-dimensional code. The headline “all regions maximally separable” rests on a lenient above-chance criterion and on 16 regions, not 43. |
| LIT-697 | AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society | Piao et al. (2025), ARXIV-2502.08691. The paper presents a simulator of about 10k LLM-driven agents in a realistic urban, social and economic environment. It claims that the simulator’s outcomes on polarization, inflammatory-message spread, UBI, hurricane mobility shocks and urban sustainability qualitatively match real-world empirical findings, which would make it usable as a testbed for social science. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-698 | Topos and Stacks of Deep Neural Networks | Belfiore et al. (2021), ARXIV-2106.14587. The authors claim that every DNN architecture defines a Grothendieck topos of sheaves over a site built from its graph, that backpropagation is a flow of morphisms in that topos, and that layer invariances (CNN translations, LSTM structure) are stacks, and they conjecture that these stacks explain generalization and support a homological notion of “semantic information”. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-699 v2 | Diffusion Models Beat GANs on Image Synthesis | Dhariwal and Nichol (2021), ARXIV-2105.05233. The classifier-guidance paper, the ablated U-Net the diffusion literature then treated as the default, and the source of the guidance-scale trade. Its title claim is metric-dependent and the paper says so: ADM-G beats BigGAN-deep on FID and recall at 128, 256 and 512, and loses on precision at all three — 0.78 against 0.86, 0.82 against 0.87, 0.87 against 0.88. It also asserts, once, in the introduction, that the gradient scale can be raised “by an order of magnitude without obtaining adversarial examples”, and runs no experiment anywhere in the paper that tests it. | Active |
| LIT-700 | Non-Euclidean Gradient Descent Operates at the Edge of Stability | Islamov, Crawshaw, Cohen and Gower (ICML 2026), ARXIV-2603.05002. Steepest descent under any norm — Spectral GD, ℓ∞-descent, Block CD, and the normalized forms SignGD and normalized Spectral GD — also sharpens to 2/η and trains there, if sharpness is measured in the update’s own norm: max_{‖d‖=1} dᵀ∇²L d. The ordinary top Hessian eigenvalue stays well below 2/η for Spectral GD and ℓ∞-descent, so the Euclidean diagnostic misses the regime entirely. Full-batch, CIFAR-10 and Tiny Shakespeare scale; the non-Euclidean sharpness is a multi-restart Frank-Wolfe estimate, not an exact value. | Active |
| LIT-701 | Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution | Lou, Meng and Ermon (2023), ARXIV-2310.16834. SEDD: learn the ratios p_t(y)/p_t(x) of a discrete diffusion with score entropy, a Bregman loss that keeps them positive and yields an ELBO. The first discrete diffusion model to reach GPT-2-level zero-shot perplexity, and the absorbing (mask) process beats the uniform one on every table. The GPT-2 comparison is against the released model, trained on different data and re-evaluated without a sliding window. | Active |
| LIT-702 | Simple and Effective Masked Diffusion Language Models | Sahoo et al. (2024), ARXIV-2406.07524. MDLM: the masked-diffusion training objective reduces to a weighted average of BERT-style masked-LM losses, a continuous-time ELBO whose value is invariant to the noise schedule. It is the strongest discrete diffusion language model at its scale — beating SEDD at matched tokens — and still trails a retrained autoregressive baseline in-domain (LM1B 27.04 vs 22.32; OWT 23.21 vs 17.54), which the authors state as the limitation. | Active |
| LIT-703 | Improved Precision and Recall Metric for Assessing Generative Models | Kynkäänniemi, Karras, Laine, Lehtinen and Aila (2019), ARXIV-1904.06991. The definitions SOTA-425 rests on: build a k-NN hypersphere approximation of each distribution’s manifold in a feature space, then precision is the fraction of generated samples inside the real manifold and recall the fraction of real samples inside the generated one. The features are VGG-16 fc2, not Inception — the record had this wrong. Its Figure 4 is the demonstration: two StyleGAN setups at FID 16.9 and 16.7 where one produces good samples at low variety and the other is “practically all images broken”, and the FID-optimised configuration has visible facial distortion. | Active |
| LIT-704 | Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning | Gal and Ghahramani (2015; ICML 2016), ARXIV-1506.02142. With dropout before every weight layer and L2 weight decay, the training objective is, for a suitable precision τ and length-scale l, a variational objective whose approximate posterior is Bernoulli-masked weight matrices under a deep Gaussian process. Averaging T stochastic forward passes (“MC dropout”) then gives a predictive mean, and their spread plus τ⁻¹ gives a predictive variance. The equivalence is exact only up to an approximated KL term, and the derivation is in a separate appendix paper. The benchmark comparisons use one-hidden-layer, 50-unit networks on UCI regression. | Active |
| LIT-705 | Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift | Li, Chen, Hu and Yang (2018; CVPR 2019), ARXIV-1801.05134. Dropout changes a unit’s variance between train and test mode, and a batch-norm layer downstream keeps normalizing by the train-mode variance. Dropout 0.5 in every DenseNet bottleneck costs 77.42% → 68.55% on CIFAR-100, and the model then misclassifies its own training data in eval mode. The damage is measured and large. The remedy’s gain is not: one dropout layer after the last BN gives about 0.2 top-1 on ImageNet and mixed, mostly sub-0.3 changes on CIFAR, with no spread reported. | Active |
| LIT-706 | Scaling Language-Image Pre-training via Masking | Li et al. (2022), ARXIV-2212.00794 — FLIP. Train CLIP with 50–75% of image patches removed and the image encoder run on the rest: the saved compute buys more pairs per hour and a 2–4× larger batch at the same memory, which outweighs the information lost per image. ViT-L/16 on LAION-400M reaches its CLIP reproduction’s accuracy more than 3× faster. The accuracy gain at a fixed schedule is 1–2 points at L and nothing at B/16, and at equal batch size masking 50% buys nothing — the batch is doing the work. | Active |
| LIT-707 | Analysis of dropout learning regarded as ensemble learning | Hara, Saitoh and Shouno (2017), ARXIV-1706.06859. In a teacher-student soft-committee machine (erf units, N = 1000, 100 student units, one fixed set of N inputs reused), dropout at p = 0.5 reaches lower test MSE than an ensemble of two independently trained 50-unit halves. Its residual error is about that of SGD with a tuned L2 penalty. The title says analysis, but no analytic result is derived. Everything is simulation curves averaged over 10 trials, with no numbers or spread reported. | Active |
| LIT-708 | A Controlled Study of Attention-Only Transformers | Ndubuaku et al. (2026), ARXIV-2607.18363. Removing every feed-forward layer from a decoder costs 0.47 nats at matched depth and 0.26 at matched training FLOPs, but only 0.0055 nats (0.27% of loss) at matched parameters once the freed budget goes into attention depth: 20 attention-only layers against 4 standard blocks, 24M parameters, 105B tokens of a synthetic reasoning corpus, with a separate learning-rate sweep for each arm. The remaining gap sits on low-context tokens, where the answer must come from the weights. A pre-registered test on fineweb-edu predicted 0.02–0.05 nats and measured 0.040. Removing QK-norm made the attention-only stack diverge; residual gates and ReZero changed nothing. Everything is at or below 87M parameters. | Active |
| LIT-709 | Primer: Searching for Efficient Transformers for Language Modeling | So et al. (NeurIPS 2021), ARXIV-2109.08668. An evolutionary search over TensorFlow primitives, at a fixed 24-hour training budget, finds two changes that carry most of the gain. One squares the feed-forward ReLU. The other adds a causal width-3 depthwise convolution after each per-head Q, K and V projection (MDHA). The only direct activation comparison has squared ReLU ahead of SwiGLU and ReGLU, in one 110M C4 figure. “4.2×” is against the original ReLU T5 at 537M. Against the SwiGLU + RMSNorm baseline the saving is 2× at equal final quality. On encoder–decoder masked LM, Primer-EZ does no better than that baseline. | Active |
| LIT-710 | Efficient Diffusion Training via Min-SNR Weighting Strategy | Hang et al. (ICCV 2023), ARXIV-2303.09556. Weight each timestep’s x0-space loss by min(SNR(t), γ), γ = 5, which is min(γ/SNR, 1) on an ε loss and min(SNR, γ)/(SNR+1) on a v loss. The headline “3.4× faster” is against an x0-prediction baseline with constant weight, on one ViT-B run. Against the ε-prediction default most people train, the UNet gain is 8.55 → 7.32 FID at 200K iterations and 4.21 → 4.14 at 1M. It speeds early training much more than it moves the end point. The “conflicting gradients between timesteps” explanation rests on one fine-tuning probe and a proxy objective. | Active |
| LIT-711 | Do Transformer Modifications Transfer Across Implementations and Applications? | Narang et al. (2021), ARXIV-2102.11972. About fifty published transformer modifications reimplemented in one codebase (T5 / Mesh TensorFlow, 223M encoder-decoder) with hyperparameters held fixed, scored on C4 pre-training loss, SuperGLUE, XSum, WebQuestions and WMT’14 En-De. Most do not help. The ones that do are small (GLU-variant activations, RMSNorm, untied embeddings), cost parameters or time (Switch/MoE, product-key memory, mixture of softmaxes), or were invented in the same codebase. The Universal Transformer, ReZero, Fixup, cross-layer sharing, the Evolved Transformer and lightweight/dynamic convolutions all come out worse than the vanilla baseline. The caveat that travels least: the positive results are also mostly same-codebase — GLU Variants (LIT-030) was run in this codebase at this exact configuration — so the paper is a strong negative test and a weak positive one. | Active |
| LIT-712 | Axiomatic Attribution for Deep Networks | Sundararajan, Taly and Yan (ICML 2017), ARXIV-1703.01365. Integrated Gradients: attribute a prediction to input features by integrating the gradient along the straight line from a baseline x′ to the input, times (x − x′). Its case is axiomatic, not empirical. It satisfies Sensitivity and Implementation Invariance, which gradients, DeconvNets and Guided Backprop (Sensitivity) and DeepLift and LRP (Implementation Invariance) each break, and its attributions sum to F(x) − F(x′). It is the unique symmetry-preserving path method. No experiment measures whether its maps are faithful. The only comparison with another method is visual (Figure 2). The baseline is a free choice that changes the answer. | Active |
| LIT-713 v2 | Sanity Checks for Saliency Maps | Adebayo, Gilmer, Muelly, Goodfellow, Hardt and Kim (NeurIPS 2018), ARXIV-1810.03292. Two randomization tests for any explanation method. Compare its maps on the trained model against the same architecture with weights re-initialized (top-down, or one layer at a time), and against a model trained on permuted labels. A method whose maps survive either cannot support model debugging or explain what the model learned from data. Gradients and GradCAM pass. Guided Backprop and Guided GradCAM are invariant to the higher layers’ weights. Integrated Gradients and gradient⊙input get no verdict. Their maps keep the input’s structure (rank correlation with absolute values and SSIM stay high), while their sign decorrelates at once. The lesson that travels is the negative one: judging a saliency map by how it looks is not a test. | Active |
| LIT-714 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras et al. (2023), ARXIV-2312.02696 — EDM2. Two contributions that do not depend on each other. Rebuilding the ADM U-Net so every layer preserves activation, weight and update magnitude takes ImageNet-512 FID from 8.00 to 2.56 at equal compute, the largest single step (6.96 → 3.75) being weight normalization on the learned layers. And post-hoc EMA: store two power-function averages at snapshots and reconstruct any EMA length after training, which shows the optimal length moving with architecture, training time, learning rate, guidance weight and — by a factor of seven under guidance — with which metric you score it. | Active |
| LIT-715 | Surya: Foundation Model for Heliophysics | Roy et al. (2025), ARXIV-2508.14112. A 366M-parameter transformer pretrained on full-resolution (4096²) multi-instrument SDO imagery to forecast the next frame, then rollout-tuned, forecasts solar dynamics zero-shot. With LoRA fine-tuning it beats task-specific baselines on four downstream heliophysics tasks. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-716 | The Impact of Generative AI on Social Media: An Experimental Study | Møller et al. (2025), ARXIV-2506.14295. In a randomized experiment with 680 US participants, generative-AI writing aids raised engagement and output on a simulated social platform but lowered perceived quality and authenticity, with negative spill-over into threads, and no single tool improved both producer and consumer experience. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-717 | Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task | Kosmyna et al. (2025), ARXIV-2506.08872. In a small four-month EEG study, participants who wrote essays with ChatGPT showed the weakest brain connectivity, the lowest sense of ownership and the worst recall of their own essays. Those switched from ChatGPT to no tools stayed under-engaged, which the authors call accumulated “cognitive debt”. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-718 | Automated decision-making as domination | Burrell (2024), DOI-10.5210/fm.v29i4.13630. Defining algorithmic injustice as allocative unfairness is a product of computer science’s status and funding, not of the harms themselves. Redefining it as domination, following Iris Marion Young, widens the remedies from code fixes to law, participation, resistance and bans. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-719 | Discovering cognitive strategies with tiny recurrent neural networks | Ji-An et al. (2023), DOI-10.1038/s41586-025-09142-4. GRUs with only 1–4 units predict individual animals’ and humans’ choices in six reward-learning tasks better than 30+ classical cognitive models of the same dimensionality. Because they are so small, they can be read as dynamical systems, exposing strategies (state-dependent learning rates, perseveration, reward-induced indifference) that no classical model contains. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-720 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras, Aittala, Lehtinen, Hellsten, Aila and Laine (2023), ARXIV-2312.02696. Post-hoc EMA: keep two power-function weight averages during training, snapshot them periodically, and afterwards reconstruct the average for any decay length by least squares — reconstruction error falling as O(1/n⁴) in the snapshot count. That makes the averaging length a sweep instead of a pre-training guess, and the consequence is the headline number here: with the length fixed at 13%, varying the learning-rate decay over t_ref ∈ [30k, 160k] moves FID by up to 72%; sweeping the length post hoc brings the whole bracket within 10% of the optimum. Also EDM2’s magnitude-preserving architecture, reaching FID 1.91 on ImageNet-512 without guidance. | Superseded — by LIT-714; A duplicate. 2312.02696 was already held as LIT-714, filed hours earlier in #375 and merged while this note was being written; LIT-714 is the fuller reading — the whole ablation ladder, the metric-disagreement finding, and NOTE-370 with the appendices — and ten documents cite it against this note’s four, all of which were its own. Retired rather than deleted so the mistake stays visible and the formerly: alias keeps any stray citation resolving. Nothing here is lost: everything this note said, LIT-714 or SOTA-428 says already, including the learning-rate bracket. |
| LIT-721 | Guiding a Diffusion Model with a Bad Version of Itself | Karras, Aittala, Kynkäänniemi, Lehtinen, Aila and Laine (2024), ARXIV-2406.02507. CFG’s quality gain is not the class emphasis. It comes from the unconditional reference model being worse — a harder task on a smaller training slice — so the density ratio p₁/p₀ falls off away from the manifold and guidance points inward. Replace the unconditional model with a smaller, less-trained copy of the same conditional model and the quality gain survives without the diversity loss: ImageNet-512 FID 2.56 → 1.34 on EDM2-S, 1.25 on XXL, 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The mechanism is tested with a negative control: degrade the two models differently and guidance stops helping entirely. | Active |
| LIT-722 | Score-Based Generative Modeling through Stochastic Differential Equations | Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole (2020), ARXIV-2011.13456. The framework this record has been standing on without holding. Classifier guidance is Eq. 14 here, six months before ADM: the conditional reverse-time SDE adds ∇ₓ log pₜ(y|x) to the unconditional score, and §5 gives the recipe for the noise-conditioned classifier that supplies it — sample (x(0), y), noise it through the tractable forward SDE, train with a mixture of cross-entropy losses over timesteps. Also the probability flow ODE with exact likelihoods, predictor-corrector sampling, CIFAR-10 IS 9.89 / FID 2.20 / 2.99 bits/dim, and the first 1024×1024 samples from a score model. Its §I.2 imputation approximation is the “replacement” SOTA-397 exists to reject. | Active |
| LIT-723 | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows | Liu, Lin, Cao, Hu et al. (2021), ARXIV-2103.14030. A vision transformer with a hierarchical feature pyramid and self-attention confined to shifted local windows, linear in image size rather than quadratic. Nine documents here run experiments on it and none said what it is. Its ablation is the reason to hold it: every design element is measured on three tasks, and the absolute position embedding this record recommends elsewhere helps ImageNet classification (+0.4 top-1) while hurting COCO detection (−0.2 box AP) and ADE20K segmentation (−0.6 mIoU). | Active |
| LIT-724 v2 | Investigating Sanity Checks for Saliency Maps with Image and Text Classification | Kokhlikyan, Miglani, Alsallakh, Martin and Reblitz-Richardson (2021), ARXIV-2106.07475. The follow-up SOTA-430 and NOTE-365 both named as the first thing to read against the sanity checks, unread until now. Three findings, all narrowing what a failed randomization test means: the failure is attributable to the input multiplier rather than to the attribution; the verdict does not transfer from image to text, where global Integrated Gradients becomes parameter-sensitive; and SSIM and Spearman rank correlation return opposite verdicts on the same experiment. | Active |
| LIT-725 | On the (In)fidelity and Sensitivity for Explanations | Yeh, Hsieh, Suggala, Inouye and Ravikumar (2019), ARXIV-1901.09392, NeurIPS 2019. The definitions behind the two numbers LIT-724 reports and this record had started quoting: infidelity and max-sensitivity. Two properties a reader needs and neither citing document carried — max-sensitivity is minimised by a constant explanation, and infidelity is defined relative to a chosen perturbation distribution for which several existing methods are each the optimum. It also reproduces SOTA-430’s metric split on a third model family, and builds the planted-feature setting SOTA-430’s promote_when asks for while pointing it at a different question. | Active |
| LIT-726 | Talagrand Meets Talagrand: Upper and Lower Bounds on Expected Soft Maxima of Gaussian Processes with Finite Index Sets | Chu et al. (2025), ARXIV-2502.06709. Take a centered Gaussian process on a finite set T with max variance σ² and minimum separation a. The expected softmax-weighted average g(β) = E[Σ_t X_t e^{βX_t}/Z] satisfies g(β) ≤ √(2σ²·E D(ν_β‖ν₀)) for every β (Thm 1, D = KL from uniform). For β ≥ β*, g(β) ≥ c·a·√(E D(ν_β‖ν₀)), with c the Sudakov constant (1/17 via Marcus–Rosen) (Thm 2). For i.i.d. N(0,σ²) logits there is an exact identity, g(β) = βσ²(1 − E‖ν_β‖²₂) (Lemma 2). The quenched free energy is bounded above by √(2σ²·E D_{1/2}(ν_β‖ν₀)), with an i.i.d. lower bound of (cσ/2)√(E D_{1/2}) (Thms 4–5). | Active |
| LIT-727 v4 | Striving for Simplicity: The All Convolutional Net | Springenberg, Dosovitskiy, Brox and Riedmiller (2014), ARXIV-1412.6806, ICLR 2015 workshop. Guided backpropagation’s defining paper, and so the referent for SOTA-430’s headline failure case — the method whose maps stay “visually and quantitatively similar” after the weights above the lowest layers are destroyed. The masking rule is stated here in one sentence: zero the gradient wherever either the top gradient or the bottom activation is negative. The paper is about an architecture; guided backprop arrives in §4.2 as a tool for analysing it, and the property later found to be its defect is presented here as its advantage. | Active |
| LIT-728 v2 | Visualizing and Understanding Convolutional Networks | Zeiler and Fergus (2013), ARXIV-1311.2901. Two attribution families start here and the record depends on both. The deconvnet is the approach guided backprop (LIT-727) is explicitly “a new variant of”, and so is the antecedent of SOTA-430’s headline failure case. Occlusion sensitivity — slide a grey square, watch the class probability — is the perturbation-based family LIT-725 names as its own origin, and LIT-725’s infidelity is its quantitative descendant. Also an early depth-against-parameters ablation: removing the fully connected layers costs little, removing them and two middle conv layers is “dramatically worse”. | Active |
| LIT-729 v2 | When Explanations Lie: Why Many Modified BP Attributions Fail | Sixt, Granz and Landgraf (2019), ARXIV-1912.09818, ICML 2020. The account SOTA-430’s failing row was missing — for one family, and not the family this record said it was for. Theorem 1: a backpropagation rule that keeps only non-negative relevance gives a product of non-negative matrices, which converges to a rank-1 matrix, and C v = c γᵀ v = λc — once converged the later layers can only change the map’s sign. That covers DTD, LRP-α1β0, Excitation BP and PatternAttribution. It explicitly does not cover Guided Backprop or Deconv, whose ReLU on the gradient makes the backward pass non-linear and which it defers to Nie et al. (2018), now LIT-730. | Active |
| LIT-730 | A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations | Nie, Zhang and Patel (2018), ARXIV-1805.07039, ICML 2018. The third mechanism, and the one SOTA-430’s headline failure row needed. Theorem 1: in a random three-layer CNN, guided backpropagation recovers the input, s_k^GBP(x) ≈ x, regardless of the class label. Theorem 2: saliency map and DeconvNet in the same network are N(0, I) — noise. So GBP and DeconvNet are doing (partial) image recovery, “unrelated to the decision-making of neural networks”, and the two causes are the backward ReLU and a CNN’s local connections. Four arms test it, including removing local connections and an adversarial attack that changes the class while barely changing the input. | Active |
| LIT-731 | SmoothGrad: removing noise by adding noise | Smilkov, Thorat, Kim, Viégas and Wattenberg (2017), ARXIV-1706.03825. The smoothing eight held documents name — averaging the gradient over n noisy copies of the input, with 10–20% noise and n ≈ 50, past which “there was little apparent change”. Its own evidence is entirely qualitative and it says so: “since quantitative evaluation of a map remains an unsolved problem, we again focus on qualitative evaluation.” And its §3.1 visualization lessons contain informal statements of two things this record spent two days building accounts for: the signed-versus-absolute ambiguity, and the input multiplier putting the input’s own edges into the map. | Active |
| LIT-732 | Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization | Selvaraju, Cogswell, Das, Vedantam, Parikh and Batra (2016; ICCV 2017, IJCV 2019), ARXIV-1610.02391. The referent behind both halves of SOTA-430’s verdict table — GradCAM, which passes the randomization test, and Guided GradCAM, which fails it, are both defined here, and the second is defined as the pointwise product of the first with guided backpropagation. Grad-CAM reads one layer: gradients of the class logit into the last conv feature maps, global-average-pooled into per-channel weights. Also carried: a 2016 human study in which guided backprop, the best-looking method, comes last of four — 44.44% against Deconvolution’s 53.33% — a number its authors did not gloss until v4 in December 2019, fourteen months after LIT-713 made the same point another way. | Active |
| LIT-733 | Language Models are Unsupervised Multitask Learners | Radford et al. (2019) — GPT-2. Language models trained on WebText begin to perform tasks zero-shot when conditioned on a suitable context, and the performance rises log-linearly with capacity up to the 1.5B model, which still underfits. Its §2.3 carries two architecture changes the field kept: layer norm moved to the input of each sub-block, and residual-layer weights scaled at initialization by 1/sqrt(N), N the number of residual layers, to account for accumulation on the residual path with depth. | Active |
| LIT-734 | Fine-Tuning Language Models from Human Preferences | Ziegler et al. (2019), ARXIV-1909.08593. The first time the human-preference loop is run on a pretrained language model: a 774M GPT-2 fine-tuned with RL against a reward model fit to human choices, with a KL penalty holding it near the pretrained model. On stylistic continuation, 5,000 comparisons give a policy humans prefer 86% of the time over zero-shot. On TL;DR and CNN/DM summarization, 60,000 comparisons produce “smart copiers” that labelers like but that may be exploiting the labelers’ heuristics, the problem LIT-433 set out to fix. | Active |
| LIT-735 | Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads | Jangda et al. (2021), ARXIV-2105.05720 — CoCoNet (ASPLOS 2022). A DSL and compiler that treat computation and collectives as one program, so they can be fused and overlapped. Overlapping a tensor-parallel MatMul with its AllReduce in fine-grained chunks hides over 80% of the MatMul and runs 1.36x faster than issuing them in sequence; in Megatron-LM, the overlapped schedules speed up model-parallel inference 1.48-1.51x and pipeline-parallel inference further, and the data-parallel gain on BERT training comes from fusing the optimizer into the collective rather than from overlap. | Active |
| LIT-736 | Reducing Activation Recomputation in Large Transformer Models | Korthikanti et al. (2022), ARXIV-2205.05198. Introduces sequence parallelism: the layer norms and dropouts that tensor parallelism leaves replicated are split along the sequence dimension, and the tensor-parallel all-reduces become an all-gather and a reduce-scatter that move the same bytes, so activation memory per layer falls by the tensor-parallel degree at no extra communication. With selective recomputation of the attention core it cuts activation memory about 5x, and over full recomputation lifts end-to-end throughput 29.0-32.1% on models from 22B to 1T parameters. | Active |
| LIT-737 | 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training | Afroz et al. (2025), ARXIV-2511.14124 (SoCC ‘25). A tensor caching and migration system for offloaded LLM training across GPU, CPU and NVMe, reporting up to 2× faster training than prior offloading methods. Its motivating measurement is the one the record uses: on an L40S over PCIe 4.0 ×16, a 16 MB FP32 host-to-device copy takes 1.65 ms (10.16 GB/s) from pageable memory and 0.68 ms (24.74 GB/s) from pinned memory. | Active |
| LIT-738 | Shortformer: Better Language Modeling Using Shorter Inputs | Press, Smith and Lewis (2020), ARXIV-2012.15832. Training a transformer language model on short subsequences first and then at the full length is faster and better than training at the full length throughout: on WikiText-103 (247M parameters, final length 3,072), 128 tokens for the first 50 epochs gives 17.52 dev perplexity against the baseline’s 18.65. The paper credits the two-stage routine to BERT, which used it only for speed. Also introduces position-infused attention. | Active |
| LIT-739 | TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems | Abadi et al. (2016), ARXIV-1603.04467 — the TensorFlow white paper (dated November 2015). Describes TensorFlow’s dataflow-graph interface and its execution from a single device to clusters; among its graph features are input operations that read from storage directly and queues that let input be prefetched from disk while the previous batch is still being computed. | Active |
| LIT-740 | Role-Play with Large Language Models | Shanahan, McDonell and Reynolds (2023), ARXIV-2305.16367 — a conceptual paper, with no experiments. A base LLM behind a dialogue prompt is best described as role-playing the character the context implies. More precisely, it keeps a superposition of characters consistent with the conversation so far and commits to none of them, and every reply narrows the set. The framing lets folk-psychological words be used without ascribing beliefs or goals to the model. It also gives a behavioural test for kinds of falsehood: invented answers vary under regeneration, “good faith” errors do not, and “deliberate” deception shows when the same question is asked from different contexts. | Active |
| LIT-741 | Textoshop: Interactions Inspired by Drawing Software to Facilitate Text Editing | Masson, Kim and Chevalier (2024), ARXIV-2409.17088 (CHI 2025). A GPT-4o-backed text editor that replaces typed prompts with drawing-software operations, each of which sends an engineered prompt. Resizing a selection shortens or expands it, rotating it reorders the words, a colour picker sets tone, boolean operations merge passages, and layers hold versions. In a within-subject study with 12 participants, against a text editor plus a ChatGPT replica on the same model, participants rated themselves more successful (4.6 vs 3.9), gave a SUS of 90 against 69, and finished tasks 55 s faster. The quality of the resulting text was not measured. | Active |
| LIT-742 | The Computational Foundations of Collective Intelligence | Pilgrim et al. (2025), ARXIV-2509.07999. A conceptual framework with no new data or model. A collective differs from an individual in its computational resources, which are sensing, state (memory), processes and actions, and in the constraints that distribution imposes on them: coordination and cooperation. Wisdom of the crowd, collective sensing, division of labour and cultural learning are each read as one of those resource advantages being put to use. It is illustrated with golden shiners, ant nest-site choice and pigeon homing. | Active |
| LIT-743 | Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation | Bahmani et al. (2025), ARXIV-2509.19296 — Lyra. Train a 3D Gaussian Splatting decoder on the latents of a frozen camera-controlled video diffusion model (GEN3C), supervised only by that model’s own RGB decodings along six camera trajectories per image, with no real multi-view data. Decoding latents rather than pixels is what makes 726 views per scene fit in memory. Single image to 3D beats the reported numbers of ZeroNVS, ViewCrafter, Wonderland and Bolt3D on RealEstate10K, DL3DV and Tanks-and-Temples, and adding real multi-view data to the synthetic supervision does not help. | Active |
| LIT-744 | Text Embeddings by Weakly-Supervised Contrastive Pre-training | Wang et al. (2022), ARXIV-2212.03533 — E5. A BERT-initialised bi-encoder trained with InfoNCE and in-batch negatives (batch 32,768) on ~270M web text pairs, cut from ~1.3B by a consistency filter that keeps a pair only if a model trained on the noisy set ranks its passage in the top 2 of 1M. With no labelled data the base model averages 42.9 nDCG@10 on 15 BEIR datasets against BM25’s 41.7, the first unsupervised model reported above BM25; after fine-tuning on MS-MARCO, NQ and NLI the 330M large model averages 61.4 on 56 MTEB English datasets, above the 4.8B GTR-xxl and Sentence-T5-xxl. | Active |
| LIT-745 | SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features | Tschannen et al. (2025), ARXIV-2502.14786. Keep SigLIP’s architecture and sigmoid loss, and add a captioning-and-grounding decoder loss (LocCa) throughout, self-distillation and masked prediction for the last 20% of training, a 90/10 English/multilingual mix with de-biasing filters, and distillation through active data selection for the smallest models. It beats SigLIP at every size and resolution compared on zero-shot classification, retrieval, VLM transfer, dense probing and localization (RefCOCO val, L/256: 67.3 → 86.0). None of the ingredients is ablated, so the gains cannot be credited to any one of them. | Active |
| LIT-746 | Position: Curvature Matrices Should Be Democratized via Linear Operators | Dangel et al. (2025), ARXIV-2501.19183 — a position paper with a library behind it: expose the Hessian, GGN, Fisher flavours and (E)KFAC as linear operators, so applications ask only for matrix-vector products and the batching, scaling and determinism pitfalls live in one tested place. Its measurements are costs in units of a gradient, on ResNet-50 and nanoGPT: computing KFAC’s factors takes 1.5–2.5 gradients and applying them 0.1–0.2, while a Hessian-vector product takes 4.5–5.5 gradients and 3–4.5× the memory in PyTorch. | Active |
| LIT-747 | Perceiver: General Perception with Iterative Attention | Jaegle et al. (2021), ARXIV-2103.03206 — the Perceiver. A learned latent array of N=512 queries cross-attends to the raw input (50,176 ImageNet pixels), so attention costs O(MN) rather than O(M²), and a deep latent transformer then runs at O(N²) per layer, independent of input size. With 2D Fourier-feature positions and no convolution it reaches 78.0% top-1 on ImageNet without pretraining, unchanged when the pixels are permuted, while ViT-B-16 and ResNet-50 given the same features fall to 61.7 and 39.4. The same architecture runs on audio, video and point clouds. | Active |
| LIT-748 | SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator | Chen et al. (2024), ARXIV-2412.12094 — SepLLM. Let each token attend only to a few initial tokens, the previous n tokens, and every earlier separator token (punctuation and whitespace), on the hypothesis that a segment’s content is condensed into the separator that ends it. Training-free on Llama-3-8B-Instruct it keeps GSM8K-CoT at 77.18 against 77.79 with 47% of the KV cache, where StreamingLLM at the same budget gets 70.89. Trained that way from scratch at 160M it reaches lower loss than full attention at equal FLOPs. | Active |
| LIT-749 | bert2BERT: Towards Reusable Pretrained Language Models | Chen et al. (2021), ARXIV-2110.07143 — bert2BERT. Initialize a larger Transformer language model from a smaller pretrained one by Net2Net-style width expansion, then stack layers for depth. Growing a 54M BERT (12 layers, width 512) into BERT-base (width 768) with function-preserving initialization saves 30.4% of pre-training FLOPs at equal GLUE and SQuAD scores. Its recommended variant breaks function preservation by borrowing the next layer’s weights and adds a two-stage schedule, which saves 45.2% on BERT-base and 47% on a GPT-base decoder. | Active |
| LIT-750 | ZipNN: Lossless Compression for AI Models | Hershcovitch et al. (2024), ARXIV-2411.05239. Losslessly compressing a trained model’s file works almost entirely through the float exponent: about 40 of 256 values occur, and the exponent stream codes to ~33% while sign and mantissa stay near 100%. Regular BF16 models therefore shrink to ~66%, FP32 to ~83%, and models rounded after training (“clean”) to as little as 34–48%. Splitting the exponent into its own stream and coding it with Huffman alone, without LZ, beats Zstd by 17% in size and 62% in single-thread speed on Llama-3.1 BF16. | Active |
| LIT-751 | ”Principal Components” Enable A New Language of Images | Wen, Zhao et al. (2025), ARXIV-2503.08685 — Semanticist. A ViT encoder emits a causal 1D sequence of continuous tokens that conditions a DiT diffusion decoder. During training, every token after a uniformly sampled index is replaced by a learned null token (“nested CFG”), so earlier tokens carry more. The result is coarse-to-fine and PCA-like, and any prefix decodes. ImageNet-256 rFID 0.78 with 256×16 floats against ImageFolder’s 0.80 at 286×32. An autoregressive model trained on only the first 32 tokens reaches gFID 2.57. The diffusion decoder is argued to keep image spectrum out of the tokens, but that is shown against other tokenizers, not ablated. | Active |
| LIT-752 | Position: AI/ML Influencers Have a Place in the Academic Process | Weissburg, Arora, Wang, Pan and Wang (2024), ICML 2024 position paper. Papers tweeted by the two ML paper-sharing accounts @_akhaliq and @arankomatsuzaki have median citations of 24 and 31 against 14 and 12 for controls matched on venue, year, topic embedding and author prominence, while OpenReview scores of tweeted papers and their controls are indistinguishable. A negative-outcome-control estimate keeps a positive causal effect on being highly cited at every threshold tried, largest for less-cited papers. | Active |
| LIT-753 | Rigorous dynamical mean field theory for stochastic gradient descent methods | Gerbelot et al. (2022), ARXIV-2210.06591 — a proof that the discrete-time dynamical mean-field equations of statistical physics are exact for a family of first-order methods (multi-pass SGD, heavy-ball and Nesterov momentum, Langevin noise, learning-rate schedules) learning a shallow estimator on Gaussian data, in the limit where samples and dimension grow together and the mini-batch is a fixed fraction of the data. On a teacher-student perceptron the solved equations match simulation at d = 1000 across learning rates and batch fractions. | Active |
| LIT-754 | Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism | Wang et al. (2022), ARXIV-2211.00235. Move the Evoformer’s outer product mean to the end of the block, so the MSA and pair stacks no longer depend on each other within a block; this matches the original block’s accuracy and speed. Then put the two stacks on two GPUs. On a batch of one protein per device, where data parallelism cannot scale further, this cuts end-to-end AlphaFold2 training from 5.8 to 4.2 days on UniFold and from 6.7 to 4.9 days on HelixFold, using twice the GPUs. | Active |
| LIT-755 | Wonderland: Navigating 3D Scenes from a Single Image | Liang et al. (2024), ARXIV-2412.12091 — Wonderland (CVPR 2025). Fine-tune CogVideoX-5B-I2V for camera control with a ControlNet branch plus LoRA, then train a reconstructor (LaLRM) that regresses 3D Gaussians directly from the video latents rather than decoded frames. Reading latents beat reading RGB at matched cost (RealEstate10K PSNR 27.10 against 25.06). The reconstructor is trained on captured RealEstate10K, ACID and DL3DV first, then fine-tuned on those plus 20K videos the video model generated, supervised by its own decoded frames; that addition moved benchmark PSNR by at most 0.09. | Active |
| LIT-756 | The Work Capacity of Channels with Memory: Maximum Extractable Work in Percept-Action Loops | Fiderer et al. (2025), ARXIV-2504.06209. A theory paper about agents modelled as finite-state hidden Markov channels interacting with an environment. It defines a channel’s work capacity, the best rate at which any agent can extract work from it under the second law. When the agent’s actions do not affect its percepts, the efficient agent is the maximally predictive one, as earlier results showed. Once actions feed back into percepts, that is no longer true. Theorem 3 gives an environment where, for every memory size, the work-efficient agents, the predictive agents and the agents that randomize their actions and forget them are disjoint sets. | Active |
| LIT-757 | Promptagator: Few-shot Dense Retrieval From 8 Examples | Dai et al. (2022), ARXIV-2209.11755 — Promptagator. Prompt a 137B FLAN with at most 8 examples from the target task to write queries for its corpus, keep a generated pair only if a retriever trained on the generated pairs ranks its source passage top-1, and train a 110M dual encoder on what is left. On 11 BEIR datasets it averages 47.8 nDCG@10 against 46.6 for SPLADE v2 and 46.2 for ColBERT v2, both trained on MS MARCO. The filter is worth 2.5 points on average and lowers three of the eleven datasets. | Active |
| LIT-758 | Net2Net: Accelerating Learning via Knowledge Transfer | Chen, Goodfellow and Shlens (2015), ARXIV-1511.05641 — Net2Net, ICLR 2016. Initialize a larger network so it computes exactly the function of a trained smaller one, then train it normally. Net2WiderNet replicates randomly chosen units and divides their outgoing weights by the replication count, and Net2DeeperNet inserts identity layers. On ImageNet Inception-BN, a widened student reaches the randomly initialized network’s final validation accuracy roughly 2×10⁶ minibatches sooner, with no loss of final accuracy. | Active |
| LIT-759 | From monoliths to modules: Decomposing transducers for efficient world modelling | Boyd et al. (2025), ARXIV-2512.02193 — a formal paper with no experiments. A world model is a transducer (a stochastic input-output machine with latent memory, generalizing POMDPs), and composing transducers is a Kronecker product of their operators. The paper inverts this. Two information measures, Intransducibility (with latents) and Acausality (observables only), are zero exactly when one block of variables can be a causal module driven by the rest. Peeling such modules off factors a monolithic model into a DAG of sub-transducers that can be learned and inferred separately. Minimal predictive models (ε-transducers) are closed under composition. | Active |
| LIT-760 | Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks | Lee et al. (2018), ARXIV-1810.00825 — the Set Transformer (ICML 2019). An attention encoder–decoder for set inputs. Its induced set attention block (ISAB) routes a set of n elements through m learned “inducing points”, so each block costs O(nm) rather than O(n²), and its pooling by multihead attention (PMA) aggregates the set with k learned seed vectors in place of mean or max pooling. Attention-based pooling is the larger win on tasks whose outputs interact (amortized clustering), and 16 inducing points matched or beat full self-attention there; each block still maps back to the input size, so a stack of them scales with n at every layer. | Active |
| LIT-761 | Synaptic Field Theory for Neural Networks | Lee, Lee and Yi (2025), ARXIV-2503.08827 — a hep-th proposal with no experiments. Gradient descent with momentum, Ẅ + γẆ + ∂C/∂W = 0, follows from an action weighted by e^{γt}, the volume factor of de Sitter space with Hubble rate H = γ/d. Treat the weight indices as spatial coordinates and take the continuum limit, and training becomes a field theory with the data as external sources. Only toy networks give a local one: periodic, nearest-neighbour wiring with linear or quadratic activations. | Active |
| LIT-762 | Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models | Kandpal et al. (2023), ARXIV-2306.04529 — Git-Theta, a Git extension that versions a checkpoint per parameter group rather than as one blob. Unchanged groups (detected with a locality-sensitive hash) are stored as references; changed groups are stored as typed updates (dense, sparse, low-rank, (IA)³) that are reconstructed by walking back through history. In one T0-3B workflow it stored 41.5 GB against Git LFS’s 57.0 GB, the saving coming almost entirely from a LoRA commit (0.27 GB against 11.4); dense fine-tunes were stored nearly in full, and every add was slower than Git LFS. | Active |
| LIT-763 | Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems | Zhdanov et al. (2025), ARXIV-2502.17019 — Erwin. Build a ball tree over an irregular point set so that every ball at a level holds the same number of points in contiguous memory, then run full self-attention inside each ball, coarsen and refine through the tree in a U-Net, and alternate with a tree built on a rotated copy of the cloud so balls exchange information. Runtime fits n^1.054. Best on cosmology at large training sets, on EAGLE turbulence (3× faster and 8× less memory than EAGLE’s own model), and on ShapeNet-Car; a clear failure on the Airfoil PDE mesh. | Active |
| LIT-764 | A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima | Xie et al. (2020), ARXIV-2002.03495 (ICLR 2021). Near a minimum, SGD’s gradient-noise covariance is approximately the Hessian divided by the batch size; feed that into a Kramers escape-time calculation and the mean time to leave a valley is exponential in B/η times the inverse Hessian eigenvalue along the escape direction. So SGD favours flat minima exponentially, where gradient descent with injected white noise favours them only polynomially. Validated as escape rates on test functions and networks of ten hidden units; generalization is not measured. | Active |
| LIT-765 | Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization | Samragh et al. (2024), ARXIV-2409.12903 — HyperCloning. Initialize a wider language model from a smaller pretrained one of the same depth by tiling each weight matrix into n×n blocks scaled by 1/n, so every hidden vector is n stacked copies of the small model’s and the logits match exactly. Against random initialization with every other setting identical, the cloned models reach the random baseline’s final 10-task accuracy 2.2–4× sooner and finish higher, on OPT, Pythia and OLMo targets of 1.3–2.9B. The small model’s own pretraining is not charged to the comparison. | Active |
| LIT-766 | Community Detection on Networks with Ricci Flow | Ni et al. (2019), ARXIV-1907.03993 (Scientific Reports). Ollivier–Ricci curvature on a graph comes from optimal transport between the neighbourhoods of an edge’s endpoints. Edges between communities come out negatively curved and edges inside them positively curved. A discrete Ricci flow stretches negative edges and shrinks positive ones, and thresholding the resulting weights (“surgery”) separates the communities. On a 500-node, two-block SBM, ARI goes from nearly 100% at p_inter/p_intra = 0.5 to nearly 0% at 0.55. On LFR graphs it is the most stable of the methods compared. | Active |
| LIT-767 | Understanding Deep Learning via Notions of Rank | Razin (2024), ARXIV-2408.02111 — PhD thesis. Gradient descent’s implicit regularization is better read as rank minimization than norm minimization: on a 2×2 matrix completion problem, a depth L ≥ 2 linear network trained by gradient flow drives every norm and quasi-norm to infinity, with probability at least 0.5 over arbitrarily small initialization, while rank falls to its minimum. The same bias toward low (tensor, hierarchical-tensor) rank is proved for polynomial-activation networks equivalent to tensor factorizations; it predicts convolutional networks’ locality, which a regularizer derived from it partly undoes on long-range tasks. | Active |
| LIT-768 | Kronecker-factored Approximate Curvature (KFAC) From Scratch | Dangel et al. (2025), ARXIV-2507.05127 — a tutorial, math and PyTorch side by side, for the original KFAC of Martens and Grosse: fully-connected layers, no weight sharing. One scaffold covers the GGN, the Monte-Carlo Fisher and the empirical Fisher, and the only thing that changes between them is which vectors are backpropagated. Its practical content is the tests: KFAC is exact on a single data point, and on a deep linear network with square loss for the GGN and MC Fisher but not for the empirical Fisher, which is how to catch the scaling and factor-order bugs it says are easy to ship. | Active |
| LIT-769 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li (2025), ARXIV-2506.20699. A theory paper with no experiments. It defines a problem’s width, the minimum number of locally feasible contexts needed to cover it, and states that width is incomparable with VC dimension and that a learner with fewer contexts than the width has an error floor no amount of within-context training removes. From the additive split of complexity into a routing term and a within-context term it argues that the parts of a model that keep context structure should not be trained by the gradients that fit predictions inside a context, and it reads hallucination, reward hacking and deceptive alignment as failures of that structure. The current version (v2, 2026) replaced a v1 about a “Context-Content Uncertainty Principle” for brains and machines. | Active |
| LIT-770 v2 | DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation | Yu et al., Texas A&M and ByteDance (2026), ARXIV-2610.02188. DMAD replaces DMD’s fake-score critic with a discriminator and trains the student on a linear loss on its logit. At the unrestricted discriminator optimum that gradient equals DMD’s idealized one (App. A, checked); nothing bounds it off the optimum, where training runs with one discriminator step per student step. One-step ImageNet-64 FID is 1.24 against DMD2’s 1.28, at 570K iterations, and 1.04 with an ImageNet-pretrained discriminator scored only by FID. Four-step SDXL reaches 14.47 FID at lower CLIP. Wan2.1 VBench margins are 0.1–0.3 points. No diversity metric and no seeds are reported. | Active |
| LIT-771 | Glow: Generative Flow with Invertible 1x1 Convolutions | Kingma and Dhariwal, OpenAI (2018), ARXIV-1807.03039. A flow step is actnorm (a per-channel affine layer initialized from one batch), a learned invertible 1×1 convolution in place of Real NVP’s fixed channel permutation, and an affine coupling layer. The 1×1 convolution is the one controlled result: lower CIFAR-10 NLL than reversal or random permutation with either coupling, three seeds, about 7% more wall-clock (Fig. 3). Against Real NVP’s published numbers it is lower on all six datasets (CIFAR-10 3.35 against 3.49 bits/dim), with much larger models. The 256×256 faces are trained on 5-bit images and sampled at temperature 0.7. | Active |
| LIT-772 | Neural Ordinary Differential Equations | Chen, Rubanova, Bettencourt and Duvenaud, Toronto and Vector (2018), ARXIV-1806.07366. Parameterize a hidden state’s derivative with a network, integrate it with a black-box solver, and backpropagate by solving the adjoint ODE backwards, at memory constant in depth (§2, Alg. 1). For densities, the log-density of a state moving under dz/dt = f obeys d log p/dt = −tr(∂f/∂z) (Theorem 1), so a continuous normalizing flow needs a trace, not a determinant. The CNF evidence is 2-D toys. An MNIST ODE-Net matches a ResNet at 0.42% against 0.41% error. | Active |
| LIT-773 | Improving Video Generation with Human Feedback | Liu, Liu, Liang et al., CUHK, Tsinghua and Kuaishou (2025), ARXIV-2501.13918. Three parts. A 182k-pair human preference set over videos from 12 text-to-video systems, rated separately for visual quality, motion and text alignment. VideoReward, a Qwen2-VL-2B reward model trained with a Bradley–Terry loss that models ties. Flow-DPO, Diffusion-DPO rewritten for rectified flow. Carried over exactly, the loss weights the KL by β(1 − t)², which goes to zero at the noise end; that variant raises the reward’s quality scores and drops its text alignment below the base model’s. A constant β fixes it. On an internal 1B video model with LoRA, constant-β Flow-DPO beats SFT and Flow-RWR, and humans prefer it to the base 44% to 27% with 29% ties. Training labels and automatic win rates come from the same reward model, and VBench barely moves. | Active |
| LIT-774 | Diffusion Model Alignment Using Direct Preference Optimization | Wallace, Dang, Rafailov et al., Salesforce AI and Stanford (2023), ARXIV-2311.12908. DPO for diffusion models. The intractable image likelihood is replaced by a bound on the path likelihood, and the reverse chain is approximated by the forward one. The loss compares how much the model beats a frozen reference at denoising the preferred image against how much it beats it on the rejected one, at the same noise and timestep. Trained offline on 851,293 Pick-a-Pic pairs, the tuned SDXL-base is preferred to the untuned base 70.0% of the time on PartiPrompts, and to SDXL base+refiner 69%, by majority vote of five crowdworkers. β is 2000–5000 with the learning rate scaled by 1/β. Fine-tuning on the preferred images alone makes SDXL worse. Checkpoints are chosen by PickScore, which was trained on the same source. There is one run per setting and no intervals. | Active |
| LIT-775 | Inductive Moment Matching | Zhou, Ermon and Song, Luma AI and Stanford (2025), ARXIV-2503.07565. A one-network, from-scratch few-step model that maps the interpolant’s marginal at t to the one at s < t. It is trained by matching, with MMD over groups of samples, the distributions reached from t and from a nearby r. Consistency models are its one-sample, first-moment case, and that case collapses on ImageNet. ImageNet-256 FID is 8.05 at one step and 1.99 at eight, with guidance at sampling time, so 16 evaluations. Training is 1.2M steps at batch 4,096. CIFAR-10 2-step 1.98. One run per row. | Active |
| LIT-776 | Training Diffusion Models with Reinforcement Learning | Black, Janner, Du, Kostrikov and Levine, UC Berkeley and MIT (2023), ARXIV-2305.13301. Treat each denoising step of a stochastic sampler as an action. Each step is then an isotropic Gaussian with an exact likelihood, so policy gradient (DDPO) can fine-tune a diffusion model on any black-box reward of the final image. On Stable Diffusion v1.4, with JPEG compressibility, LAION aesthetic score and a LLaVA + BERTScore alignment reward, both DDPO estimators beat reward-weighted regression per reward query, and the PPO-clipped one is slightly ahead. There is no KL term. Over-optimization is shown and handled by picking checkpoints by hand. Prompt sets are narrow (45–398 animals), there is no human evaluation, and the run count is not stated. | Active |
| LIT-777 | Flow Matching Guide and Code | Lipman et al., FAIR, MIT and Weizmann (2024), ARXIV-2412.06264. A tutorial and PyTorch library, with no new experiments. It states flow matching as “choose a path, regress its generator” across Euclidean, Riemannian, discrete and general Markov-process state spaces. Its claims useful to practice are proved or attributed. The straight conditional path minimizes a bound on kinetic energy and is optimal transport only for a one-point target. Every affine scheduler gives the same exact-ODE endpoint, so a schedule matters only through training and discretization. Sampling t beats weighting by t, citing SD3. It does not cover few-step generation. | Active |
| LIT-778 | DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models | Fan, Watkins, Du, Liu, Ryu, Boutilier, Abbeel, Ghavamzadeh, K. Lee and K. Lee, Google Research, UW–Madison, UC Berkeley, Amazon and KAIST (2023), ARXIV-2305.16381. Policy gradient over the denoising steps, as in DDPO, with a KL to the pre-trained model. The KL on the final image is intractable, so each step’s KL is summed, which bounds it from above. Stable Diffusion v1.5 with LoRA is trained on ImageReward, one model per prompt, against the same reward-weighted objective trained offline on 20K samples. Online RL scores higher on ImageReward and on the held-out aesthetic predictor, and eight raters prefer it. The KL ablation is one prompt. The training reward is also the main metric, and no human compares against the base model. | Active |
| LIT-779 v2 | Flow-GRPO: Training Flow Matching Models via Online RL | Liu, Liu et al., CUHK, Tsinghua and Kuaishou (2025), ARXIV-2505.05470. GRPO on a flow-matching image model. The deterministic sampler is swapped for an SDE with the same marginals, built from the model’s own velocity, which makes each step a Gaussian policy with a closed-form KL. Rollouts use 10 steps and evaluation 40. SD3.5-M with LoRA: GenEval 0.63 → 0.95, OCR accuracy 0.59 → 0.92, PickScore 21.72 → 23.31. Without the KL penalty, quality metrics fall on two tasks and diversity collapses on the third. GenEval is trained on its own templates and scored by its own detector. Quality is judged only by reward models, and there is one run per setting. | Active |
| LIT-780 | FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models | Grathwohl, Chen, Bettencourt, Sutskever and Duvenaud, Toronto, Vector and OpenAI (2018), ARXIV-1810.01367. Replaces the CNF’s exact Jacobian trace with Hutchinson’s estimator, one noise vector fixed per solve, so the log-density costs O(D) per evaluation and the dynamics network is unrestricted (§3, Eq. 8). Best reversible model on all five tabular sets, behind the autoregressive MAF-DDSF on four. MNIST 0.99 bits/dim against Glow’s 1.05; CIFAR-10 3.40 against Glow’s 3.35, with under 2% of Glow’s parameters. Training took about five days on six GPUs, and NFE grows during training until it can become “prohibitive” (§6). | Active |
| LIT-781 | Density estimation using Real NVP | Dinh, Sohl-Dickstein and Bengio, Google Brain (2016), ARXIV-1605.08803. The affine coupling layer: half the dimensions pass through unchanged and parameterize a scale and shift of the other half, so the Jacobian is triangular, its log-determinant is a sum, and the inverse costs the same as the forward pass whatever networks compute the scale and shift (§3.2–3.3). Stacked with squeeze and factor-out in a multi-scale architecture, it gives exact likelihood, exact latent inference and one-pass sampling. It does not beat PixelRNN’s likelihood on any dataset it shares with it (CIFAR-10 3.49 against 3.00 bits/dim, Table 1), and sample quality is shown, not measured. | Active |
| LIT-782 | Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation | Kirstain, Polyak, Singer, Matiana, Penna and Levy, Tel Aviv University and Stability AI (2023), ARXIV-2305.01569. A web app that generates two images per prompt and logs which one the user keeps, or a tie. The paper’s version holds 583,747 training pairs over 37,523 prompts from 4,375 users. PickScore is CLIP-H fine-tuned on those pairs with a reward-model loss. It predicts held-out user choices at 70.5% under a tie-aware accuracy where chance is 56.8%, against 68.0% for the authors’ expert annotators. Its ranking of 45 model configurations correlates 0.790 with real users’ Elo. On MS-COCO captions FID ranks nine configurations at −0.900 against human experts, because people prefer higher guidance. Every validation is in-distribution or uses colleague annotators. Diffusion-DPO trains on this data and Flow-GRPO optimizes PickScore as a reward. | Active |
| LIT-783 | Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency | Zheng et al., Tsinghua and NVIDIA (2025), ARXIV-2510.08431. sCM made to run on 14B text-to-image and text-to-video models, with a FlashAttention-2 JVP kernel under FSDP and context parallelism. Pure sCM loses fine detail and temporal coherence there. Adding a DMD loss at weight 0.01 (rCM) fixes it. Four-step Wan2.1 VBench is 84.43 at 1.3B against a re-implemented DMD2’s 84.56, and 84.92 at 14B; GenEval tracks the teacher at 1–4 steps. The diversity advantage over DMD2 is shown in samples only. One run per row. | Active |
| LIT-784 | Mean Flows for One-step Generative Modeling | Geng, Deng, Bai, Kolter and He, CMU and MIT (2025), ARXIV-2505.13447. Train a network for the average velocity u(z, r, t) over an interval, using the identity u = v − (t − r)·du/dt with du/dt from a Jacobian-vector product and the target under stop-gradient. With r = t it is flow matching. From scratch on ImageNet-256 with DiT-XL/2 and guidance folded into the target, one-step FID is 3.43 at 240 epochs and two-step 2.20 at 1,000. The best one-step baselines it beats, Shortcut 10.60 and IMM 7.77, are copied rows at other budgets. Behind iCT on CIFAR-10. One run per row. | Active |
| LIT-785 | STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis | Gu et al., Apple (2025), ARXIV-2506.06276. TarFlow moved into a VAE latent with a deep-shallow split: one deep Transformer flow block and five 2-layer ones. It adds a decoder fine-tuned on noisy latents in place of score denoising, and a guidance rule derived from the Gaussian score. ImageNet 256 FID 2.40 at 1.4B against a retrained pixel TarFlow’s 5.56, with SiT’s quoted 2.06 and MAR’s 1.55 still ahead. ImageNet 512 3.00 against EDM2’s 1.25; COCO zero-shot 9.1; GenEval 0.56. The only head-to-head against diffusion is a plot against an in-house 2.1B DiT trained on 200M images (STARFlow’s schedule is 400M) with a stock decoder. Sampling is about 2.2 s per 256×256 image with guidance. | Active |
| LIT-786 | Improved Techniques for Training Consistency Models | Song and Dhariwal, OpenAI (2023), ARXIV-2310.14189. Consistency training from random weights, without a teacher or LPIPS. The teacher network takes no EMA, because with one the limiting objective carries no information about the data. A Pseudo-Huber loss replaces LPIPS, whose ImageNet features can leak into FID. The step count doubles on a schedule, and noise levels are drawn log-normally. One-step FID is 2.51 on CIFAR-10 and 3.25 on ImageNet-64, two-step 2.24 and 2.77, beating consistency distillation. Ablations are CIFAR-10 curves, one run each. | Active |
| LIT-787 | One Step Diffusion via Shortcut Models | Frans, Hafner, Levine and Abbeel, UC Berkeley (2024), ARXIV-2410.12557. A flow-matching network also conditioned on step size d, trained in one run: the flow-matching loss at d = 0, and a target for step 2d built from two of its own d-steps. On DiT-B at matched compute, one model gives 6.9 / 13.8 / 20.5 FID at 128 / 4 / 1 steps on CelebA-HQ-256 and 15.5 / 28.3 / 40.3 on ImageNet-256. Progressive distillation is better at one step (14.8 and 35.6) and cannot sample at many. The re-run consistency baselines are far below their published numbers. DiT-XL reaches 10.6 in one step. One run per cell. | Active |
| LIT-788 | ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation | Xu, Liu, Wu, Tong, Li, Ding, Tang and Dong, Tsinghua, Zhipu AI and BUPT (2023), ARXIV-2304.05977. A reward model for text-to-image preference: a BLIP backbone trained on 136,892 ranked pairs over 8,878 DiffusionDB prompts. It picks the rater’s choice 65.14% of the time, against 54.82% for CLIP similarity, about as often as one annotator agrees with another (65.3%). ReFL, added in v3, backpropagates the reward through one denoising step taken late in the chain. On Stable Diffusion v1.4 it is ranked over the base model 58.79% of the time, ahead of filtering, reward weighting and RAFT. Best-of-64 under the same reward reaches 73.33%. Its premise, that a diffusion model has no likelihood for RLHF to use, is true of the whole chain and not of a step. | Active |
| LIT-789 | Multisample Flow Matching: Straightening Flows with Minibatch Couplings | Pooladian et al., Meta AI, NYU and Weizmann (2023), ARXIV-2304.14772. Flow matching with noise and data paired within each minibatch by OT (BatchOT) or a cheaper stable matching keeps the marginals exact. As batch size grows the paths straighten toward the OT map. On ImageNet-64 the FID-20 point drops from 29 Euler steps to 12, and adaptive FID from 13.93 to 12.37, at equal likelihood and 4% more time per iteration. Low-NFE numbers stay far from one-step quality: 38.86 FID at 4 Euler steps on ImageNet-32. One run per row. | Active |
| LIT-790 | Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models | Lu and Song, OpenAI (2024), ARXIV-2410.11081. Continuous-time consistency models had been unstable. A trigonometric flow parameterization, identity time input with positional embeddings, normalized tangents, learned loss weights and a Flash Attention JVP make them train to 1.5B parameters. Continuous time beats every discrete step count tried. Two-step FID is 2.06 on CIFAR-10, 1.48 on ImageNet-64 and 1.88 on ImageNet-512, against teachers at 2.01, 1.33 and 1.73. Every model, “consistency training” included, starts from the pretrained diffusion model. | Active |
| LIT-791 | Improving the Training of Rectified Flows | Lee, Lin and Fanti, CMU and Microsoft Research (2024), ARXIV-2405.20320. One reflow round is enough, argued informally, and the 2-rectified flow then trains better with a U-shaped timestep distribution and an LPIPS-Huber loss. Started from EDM, with no separate distillation stage, one-step FID is 3.07 on CIFAR-10 and 4.31 on ImageNet-64. Its own table lists DMD at 2.62 on ImageNet-64. There is no distillation-alone arm from the same teacher, and many-step quality is not compared with the pre-reflow model. | Active |
| LIT-792 | Discrete Flow Matching | Gat et al., FAIR and Weizmann (2024), ARXIV-2407.15595. Flow matching on discrete tokens as a continuous-time Markov chain, with velocities of the same form as the continuous case, any coupling, a family of schedulers, and corrector sampling that can re-mask emitted tokens. At 150M its generative perplexity beats SEDD’s in float32 (GPT-2 29.3 against 40.5 at 1,024 NFE). In double precision every discrete model’s perplexity rises 2–3.5 times and the gap narrows (97.5 against 109.5). At 1.7B on code it reaches HumanEval pass@1 6.7 against a same-data autoregressive model’s 14.3. | Active |
| LIT-793 | Flow map matching with stochastic interpolants: A mathematical framework for consistency models | Boffi, Albergo and Vanden-Eijnden, Courant Institute (2024), ARXIV-2406.07507. Learn the two-time flow map X_{s,t} of a probability-flow ODE. Distillation from a velocity field can use a Lagrangian loss (new) or an Eulerian one, which is the continuous-time limit of consistency distillation; both bound a Wasserstein distance. Consistency training’s update is shown not to descend any loss. On CIFAR-10 the Lagrangian loss distils far better than the Eulerian: FID 7.13 against 48.32 at two steps, teacher 5.53. Direct training on ImageNet-32 gives 16.90 FID at four steps. Small models; no one-step image table. | Active |
| LIT-794 | Normalizing Flows are Capable Generative Models | Zhai et al., Apple (2024), ARXIV-2412.06329. TarFlow stacks causal Transformers over image patches as affine autoregressive flows, reversing the order between blocks, trained by exact maximum likelihood. A likelihood model reaches 2.99 bits/dim on unconditional ImageNet 64×64, below every diffusion and autoregressive number it quotes (Table 2). Sampling needs three additions: Gaussian input noise about 25× the dequantization noise, a Tweedie denoising step using the flow’s own score, and classifier-free guidance on the affine parameters. Best FIDs are 2.66 at ImageNet 64 and 5.03 at 128, against quoted 1.55 (EDM) and 1.94. Sampling is sequential: 32 samples at 64×64 take about two minutes on one A100. | Active |
| LIT-795 | Scaling Laws for Reward Model Overoptimization | Gao, Schulman and Hilton, OpenAI (2022), ARXIV-2210.10760. InstructGPT’s 6B reward model stands in for people and labels comparisons for proxy reward models of 3M to 3B parameters. Optimizing a 1.2B policy against a proxy, the gold reward rises and then falls as a function of d = √KL from the initial policy: d(α − βd) for best-of-n and d(α − β log d) for PPO. The coefficients move smoothly with the proxy’s size, so the peak is predictable. Larger proxies and more comparison data push the peak out, and a 6B policy peaks at the same KL as a 1.2B one. A KL penalty in the reward did not improve gold reward at a given KL; its effect was like early stopping, a result the authors flag as possibly hyperparameter-sensitive. RL spends far more KL than best-of-n, so KL does not compare optimization across methods. Synthetic labels, one environment. | Active |
| LIT-796 | InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation | Liu, Zhang, Ma, Peng and Liu, UT Austin and Helixon (2023), ARXIV-2309.06380. Reflow Stable Diffusion once, then distil the reflowed model to one step with an LPIPS regression. Against distilling SD 1.4 directly, at the same 100K steps and 3.2M generated pairs, reflow first gives one-step FID 31.0 against 40.9 on COCO-2017-5k and 20.0 against 34.6 on COCO-2014-30k. The scaled model, InstaFlow-0.9B, scores 23.4 and 13.10. The teacher is a diffusion model, not a rectified flow, and the distiller is a regression, not distribution matching. One run per row. | Active |
| LIT-797 | Aligning Text-to-Image Models using Human Feedback | Lee, Liu, Ryu, Watkins, Du, Boutilier, Abbeel, Ghavamzadeh and Gu, Google Research and UC Berkeley (2023), ARXIV-2302.12192. RLHF’s recipe carried to a text-to-image model, without the RL. Two labellers mark 27,528 Stable Diffusion v1.5 images good or bad on templated count, colour and background prompts. A small head on frozen CLIP embeddings learns that label. One offline round of reward-weighted likelihood plus a pre-training loss then fine-tunes the model. Alignment wins 50% of 120 prompts by a two-thirds vote against 3% for the base. Fidelity loses: FID 13.97 to 16.76, and 3% against 20% when compared with best-of-16 sampling from the base under the same reward. One run, no diversity measure. | Active |
| LIT-798 | Improving and generalizing flow-based generative models with minibatch optimal transport | Tong et al., Mila (2023), ARXIV-2302.00482. Flow matching trains on any coupling of source and target samples. Pairing each minibatch by exact optimal transport (OT-CFM) gives near-OT paths in 2-D (normalized path energy 0.02–0.09 against 0.2–2.7 for the independent coupling, five seeds) at under 1% overhead. On CIFAR-10 its gain is small: adaptive-solver FID 3.58 against 3.66 at 134 against 146 NFE, and the independent coupling is better at 1,000 Euler steps. Its rerun of the path comparison, at a recipe that beats Flow Matching’s own, has the straight path ahead of VP at 100 Euler steps (4.64 against 7.77). | Active |
| LIT-799 | Olmo Hybrid: From Theory to Practice and Back | Merrill et al., Ai2 and Lambda (2026), ARXIV-2604.03444. Olmo 3 7B with its sliding-window layers replaced by Gated DeltaNet (negative eigenvalues), at 3:1 with full attention, trained to 6T tokens. It reaches Olmo 3’s MMLU in 49% fewer tokens and beats it in every domain after mid-training, though data mix, schedule and mid-training batch also changed. Scaling fits from 60M to 1B give the hybrid a lower data coefficient (83.7 against 94.9, non-overlapping CIs); the projected 1.3–1.9× token savings have CIs that include 1 at every size but 3B. A theorem shows hybrids express a task neither parent can. Removing negative eigenvalues barely changes the scaling, which weakens the expressivity explanation. | Active |
| LIT-800 | Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM | Kozyrev and Maiboroda, Minima (2026), ARXIV-2609.04098. Public 4-bit builds of Qwen3.8-27B kept its 48 Gated DeltaNet layers in 8 or 16 bits for fear that errors in a recurrence compound. Quantizing all 496 linear layers to NVFP4 W4A4 by calibration alone stays within seed noise of BF16 on five task suites (average −0.52) at 17.5 GiB. The perplexity cost is the largest of the recipes compared, +0.72 at 4K, but it shrinks along a 32K window. In replayed layers the gate projections everyone protected are the least sensitive. State error holds at a plateau of about 12.6% over 32K tokens, and a state impulse is erased within hundreds to a few thousand steps. One model, one format. | Active |
| LIT-801 | Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension | Bertsch, Soldaini, Gormley, Neubig, Hajishirzi, Lo and Groeneveld, Ai2 and CMU (2026), ARXIV-2608.10296. Twenty-six 7–8B models, 140B tokens each on identical data, extended identically to 64K. QK norm, GQA, sliding-window layers and a 4K pretraining length each cost long-context ability, and together they compound: HELMET at 32K runs from 56.4 down to 29.9. Counting how many of the four a model has predicts that score with in-sample R² 0.67 (0.61 leave-one-out). Pretraining loss predicts it at R² 0.29. QK norm is the largest single cost, about 4 to 6 points. GQA plus sliding windows costs about 9, against 1.1 for windows alone. One seed per configuration, and initialization alone moves scores by 7.7% of the range on average. | Active |
| LIT-802 | Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | Hatamizadeh, Choi and Kautz, NVIDIA (2026), ARXIV-2605.22791. The delta rule’s scalar beta decides two things at once: how much to erase along the key and how much of the value to write. Splitting it into a channel-wise erase gate on the key and a channel-wise write gate on the value, on top of KDA’s channel-wise decay, gives the best average of six recurrent mixers at 1.3B/100B tokens and matched state size (53.11 against 52.39 for Mamba-3 MIMO and 52.28 for KDA; 53.97 hybrid). One run each. The erase gate carries most of the gain. Its multi-key retrieval lead is real, but Gated DeltaNet beats it on the word-needle task at 4K, 60.6 against 31.8. | Active |