the kl divergence term appears frequently in formulations of machine learning, often as a term in the loss function. in relation to language models it typically appears in the objective at the reinforcement learning stage rlfh $\max_{\pi_\theta} \mathbb{E}_{x \sim D,y \sim \pi_\theta(y \mid x)} \left[ r_\phi(x, y) \right] - \beta \mathrm{D}_{\mathrm{KL}} \left[ \pi_\theta(y \mid x) \mid\mid \pi^{\text{SFT}}(y \mid x) \right].$ grpo $$ J_{GRPO} (\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min \left( r \hat{A}{i,t}, \text{clip} \left(r, 1 - \epsilon, 1 + \epsilon \right) \hat{A}{i,t} \right)...

Supervised fine-tuning (SFT) has long been the go-to method for refining large-scale language models. While effective, it suffers from well-known bottlenecks: high dependency on curated annotations, scalability constraints, and an inherent ceiling on generalization. DeepSeek-R1-Zero turns this paradigm on its head, proving that reinforcement learning (RL), when applied correctly, can independently drive substantial improvements in reasoning and logical inference. This article dissects DeepSeek-R1-Zero—an approach that ditches SFT entirely in favor of pure reinforcement learning—and DeepSeek-R1, which integrates RL into a more conventional pipeline with an initial fine-tuning phase. The...

Late on the ball again... let's see what all the r1 hype is about. DeepSeekMath presents a large-scale mathematical pretraining dataset and model, structured around dataset curation, supervised fine-tuning (SFT), and reinforcement learning (RL). The paper provides a meticulous breakdown of how the dataset was constructed, why certain architectural choices were made, and how the resulting model performs across key benchmarks. The dataset is constructed from Common Crawl and refined through a fastText-based classifier, ultimately yielding 120 billion tokens. This is an order of magnitude larger than existing math...

lately, i've been feeling confused and anxious about what i want to focus on next. i feel like i can never escape the generalist schtick because there are too many areas i want to understand. at the same time, i want to nosedive into something like vllm and really get into the weeds—it’s been too long since i’ve contributed to an open project, and i want to do some proper coding. but it's a hefty time commitment before you can contribute something meaningful. i've also been reaaallyy into reasoning...

DeepSeek V3 is an astonashing feat of engineering, model performance aside (the topic seems contentious, i've heard mixed opinions so far, apparently it keeps getting stuck in a reasoning spiral), being able to train a model of this capacity on a 2048 H800 cluster.... in just 4 GPU days? Crazy. They only spent $5M dollars training this thing. Also, from what I hear, the DeepSeek team is barely >100 people, and it's all in-house Chinese talent. I mean if you were worried about China's development before... yeah I don't...