Brown et al. (2020) — ARXIV-2005.14165
Key takeaways
- Introduces GPT-3
- Scale enables few-shot learning
- Task-agnostic language model
- Emergence of in-context learning
1 min read
| Status | Active |
| Version | 2 |
| Tags | model-architecture, adaptation-and-tuning, in-context-learning |
| Extended by | LIT-467 — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models |
| Supersedes | LIT-042 — SGDR: Stochastic Gradient Descent with Warm Restarts |
| Cited as “introduced by” by | SOTA-036 — Train a decoder-only transformer on a broad web corpus with a fixed context and a single next-token objective, SOTA-037 — LM in-context learning emerges at scale, SOTA-038 — ICL permits few-shot task adaptability |
| Cited as “source” by | SOTA-036 — Train a decoder-only transformer on a broad web corpus with a fixed context and a single next-token objective, SOTA-037 — LM in-context learning emerges at scale, SOTA-038 — ICL permits few-shot task adaptability, SOTA-039 — single cycle of cosine decay is sufficient lr schedule |
Brown et al. (2020) — ARXIV-2005.14165