[P] OpenLanguageModel (OLM): A modular, readable PyTorch LLM library — feedback & contributors welcome

Hey all,
We’re building OpenLanguageModel (OLM): an open-source PyTorch library for training and experimenting with language models, with a focus on being simple, hackable, and performance-aware.

Repo: https://github.com/openlanguagemodel/openlanguagemodel
Website/docs: https://openlanguagemodel.github.io/openlanguagemodel/

The main idea:
OLM is trying to hit three goals at the same time (which most repos only hit one of):

1. Starter-friendly: You can train a small LM in very few lines, and the code is written to be read. Removing giant abstractions and the “magic” training loops you can’t follow. It’s meant for people who want to learn how LLMs are built by actually touching the code, without hitting the large learning curve of pytorch and HuggingFace.
2. Researcher-friendly: Everything is built from modular blocks (attention, FFN, norms, activations, losses, etc.). You can swap components, implement new ideas, or rebuild GPT/LLaMA-style architectures without rewriting the whole training stack. Useful for prototyping quickly
3. Compute-aware: We’re not ignoring performance: the design is aimed at good GPU utilization and modern training setups, with things like FlashAttention / torch.compile, distributed training, and MoE in mind. It is built ENTIRELY on pytorch, and we achieve SOTA on GPU optimisation

Why:
A lot of LLM repos today are either huge black boxes or research code that’s painful to extend. OLM tries to stay small, readable, and flexible, while still scaling toward serious training.

Status:

• We’ve trained a few ~150M models using OLM
• v2.1 is out, and we’re now moving toward multi-node training and RLHF

We’d really love:

• People trying it and giving honest feedback
• API/design critiques
• Contributions

If you care about clean ML code and experimenting with LLMs, check it out!

Thanks