Build an LLM Playground
LLM Overview and Foundations
- Data collection (manual crawling, Common Crawl)
- Data cleaning (RefinedWeb, Dolma, FineWeb)
- Tokenization (e.g., BPE)
- Architecture (neural networks, Transformers, GPT family, Llama family)
- Text generation (greedy and beam search, top-k, top-p)
- SFT
- RL and RLHF (verifiable tasks, reward models, PPO, etc.)
- Traditional metrics
- Task-specific benchmarks
- Human evaluation and leaderboards