DeepSeek V3 is an astonashing feat of engineering, model performance aside (the topic seems contentious, i've heard mixed opinions so far, apparently it keeps getting stuck in a reasoning spiral), being able to train a model of this capacity on a 2048 H800 cluster.... in just 4 GPU days? Crazy. They only spent $5M dollars training this thing. Also, from what I hear, the DeepSeek team is barely >100 people, and it's all in-house Chinese talent. I mean if you were worried about China's development before... yeah I don't...

i don't typically cover the cv domain but this paper won 'best paper' at neurips and i heard its a really good paper for people unfamiliar with the field. you may also have heard of the controversy surrounding this paper, the paper is a collaboration between peking university and bytedance, and bytedance is suing the main author because apparently he was sabotaging other projects, slowing down colleagues, everything in attempt to hog more compute for his project. this project. which now won best paper and outperformed previous autoregressive image...

another paper from FAIR that has garnered significant attention this december. chain-of-thought (cot) reasoning has long been known (relative to the existence of llms) to enhance the reasoning performance of language models. not only does it improve reasoning, but it also allows users to verify the steps taken to reach an answer. in 2024, the trend has been to use cot reasoning traces as a training signal. o1 was the first notable release in a class of models explicitly trained to produce such reasoning traces. generally, the sentiment is...

it's been a while. happy to be back reading some papers, i've got some that i want to get through over the christmas break. BLT, byte latent transformer, is an attempted takedown of the bpe tokenizer monopoly by allowing a transformer to train on raw bytes without a fixed vocabulary. the papers proposes dynamic segmentation of raw bytes into patches based on the entropy of the next-byte prediction, creating contextualized groupings of bytes with uniform information density. while tokenization-based models are "stuck" allocating the same amount of compute to...