in-context-learning.
18 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-049 | Learning to Prompt for Vision-Language Models | Replaces hand-written CLIP prompts with continuous context vectors learned end-to-end while every pretrained parameter stays frozen. One or two shots beat prompt engineering; sixteen shots average about 15% gain, over 45% at the best. The paper also reports where it overfits and names the cause as noisy labels. | Read |
| NOTE-167 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Turn the in-context examples into a loss and take gradient steps on them at inference, then discard the update. Up to 6x a fine-tuned baseline on ARC and +7.3 points on BIG-Bench Hard at 10-shot. The finding underneath is about in-context learning: the same examples are worth several times more as gradient than as context. | Read |
| NOTE-214 | Limitations of multi-layer Transformers | First unconditional lower bound for a decoder-only transformer of more than one layer. Reading it: the bound comes from causal masking rather than from circuit complexity, and it buys an exponential depth-width tradeoff, an encoder/decoder separation, and a provable benefit of chain of thought. | Read |
| NOTE-216 | Chain-of-Thought Prompting | Worked reasoning steps in the few-shot exemplars, nothing else. Reading it: the three ablations are the contribution — it is not the equation, not the extra tokens, and not knowledge activation — and the effect is absent or negative below about 100B parameters. | Read |
| NOTE-217 | Self-Consistency | Sample several chains, take the majority answer. Reading it: it is a decoding change rather than a prompt change, so it composes with everything else in the topic — and it counts answers, not reasoning, so it does not make the traces trustworthy. | Read |
| NOTE-218 | Zero-Shot CoT | One fixed sentence, no exemplars, twelve datasets. Reading it: the ordering is the practical content — it loses to hand-written few-shot chains and beats eight-shot standard prompting, so it is the baseline to try first rather than the method to settle on. | Read |
| NOTE-219 | Emergent Abilities | The paper that named the phenomenon, read for what it actually argues rather than what it is cited for. It raises the metric explanation itself, in §5.1, and declines it on two grounds — one of which was later answered and one of which still stands. | Read |
| NOTE-238 | PHi: prediction of hidden states | Read for the instrument. The diagnosis of next-token loss is clean and the boring/interesting separation is convincing on two architectures. The reasoning-selection result is weaker than the abstract implies, and the paper says so itself in a sentence most readers will pass over: picking by next-token loss alone already scores 71%. | Read |
| NOTE-242 | Genesys and what survives its own controls | Read from the #180 worklist for the architecture-discovery claim and filed for the code-generation one. Unit-by-unit generation with an execution-based checker takes valid output from 6% to 92% on the same proposals, with a stated mechanism that has nothing to do with architectures. The discovered architectures are a wash against their own seeds, by the paper’s own table. | Read |
| NOTE-249 | MemGraphRAG: the pilot study is the contribution | Read from the #180 worklist in rank order. The system is a multi-agent GraphRAG pipeline with single-run numbers; the durable part is §3, which measures other people’s systems and finds graph expansion trading relevance for recall at a net loss on the end task. That is the filing. | Read |
| NOTE-255 | Gemini 1.0: complete disclosure, and an abstract that picks the flattering row | Read from the #180 worklist for the model and filed for the appendices. The headline — first model to exceed human-expert MMLU — holds under one inference procedure out of three the report itself publishes, and under the other two GPT-4 is ahead. Nothing is concealed. The disambiguating numbers are in the same document, which is what makes this the corpus’s cleanest instance of DP-010. | Read |
| NOTE-276 | Order-sensitivity refutes it, and the models everyone measured were trained for the task | Read as the challenge from outside the setup. It separates the hypothesis that pretrained models do in-context learning by gradient descent from the hypothesis that transformer weights exist which simulate it, and shows the literature had been testing only the second. Theorem 1 then gives a contradiction from order sensitivity alone, and LLaMA-7B disagrees with fine-tuning on three metrics across four datasets. | Read |
| NOTE-277 | The algorithm changes with depth, and the one it lands on is Bayes-optimal ridge | Read as the second, independent arrival at the mesa-optimization construction. Its title asks which algorithm, and its answer is that there is no single one: shallow learners sit nearest gradient descent, deeper ones nearest ridge, deepest nearest ordinary least squares, and under label noise the best-fitting ridge parameter is the Bayes-optimal σ²/τ² at every setting. The gradient-descent phase is the small-model phase. | Read |
| NOTE-278 | Linear in Newton steps, exponential in gradient steps: a rate is what separates the rivals | Read as the challenge from inside the gradient-descent line’s own setup. Matching each transformer layer to the best number of steps of a candidate algorithm gives a linear trend against Iterative Newton — about 3 iterations per middle layer — and an exponential one against gradient descent; on data with condition number 100 the transformer is unchanged while GD needs 2,000 steps a 12-layer model cannot hold. The method is the contribution: fits cannot separate these rivals and rates can. | Read |
| NOTE-279 | The setup that made the mechanism question answerable | Read because every later claim about what in-context learning is measures in this setup. A 9.5M-parameter GPT-2 trained from scratch on (x, f(x)) sequences matches ordinary least squares on unseen linear functions and Lasso on sparse ones, in a single forward pass. The training objective is the setup’s power and, later, the ground of the main objection to everything built on it. | Read |
| NOTE-280 | The construction is exact, and the trained model it describes is one layer deep | Read as the trunk of the mesa-optimization line. Proposition 1’s weight construction is exact and a single trained linear-self-attention layer approximately finds it. The result that matters most for what came after is four pages later and rarely quoted: deeper trained models do not match gradient descent, they match GD++ — gradient descent on data transformed by I − γXXᵀ — which is the paper’s own evidence that plain GD is the wrong description beyond one layer. | Read |
| NOTE-287 | Stage boundaries found without knowing in advance what to look for | Read as the application that makes the LLC matter to this record. Critical points of the LLC curve divide two transformers into five stages each, and the boundaries land on bigrams, n-grams, previous-token heads and the induction circuit — found without a mechanistic hypothesis in hand, which is what LIT-085’s progress measures require. Its in-context regression model acquires in-context learning and then loses it. | Read |
| NOTE-314 | RippleEdits | Knowledge-editing benchmarks test the edited fact and leave its consequences untested. On six ripple criteria, weight editors average 38–66, and prompting with the new fact beats them. Main text read. | Read |