How to Build a Small Code LLM from Scratch: Meet theOG-50M

How to Build a Small Code LLM from Scratch: Meet theOG-50M

Follow The Progress   FollowFollowFollow How to Build a Small Code LLM from Scratch: Meet theOG-50M Everybody wants a bigger model. I went the other way. theOG-50M is a 50-million-parameter model that writes Python, and I built it from nothing: no pretrained weights,...
Chat Format, Labels and Sequence Packing for LLM Training

Chat Format, Labels and Sequence Packing for LLM Training

Follow The Progress   FollowFollowFollow Chat Format, Labels and Sequence Packing for LLM Training Strip away the marketing and a language model learns exactly one thing: given the tokens so far, guess the next one. Writing a whole function is that guess repeated,...
Robot art toys with is pilot

Get microModels updates

New theOG versions, benchmark results and what custom nodes can do. A short email when there is something worth knowing.

You have Successfully Subscribed!