Qwen3 4b Instruct from 1% to 99% on wordle in 11 hours with a single 5090 with PPO.

Hello everyone! I’ve previously worked on transformers and ppo for traditional rl environments but I wanted to see if my training would scale up to llm fine tuning (it does at least for wordle).

I modified/rewrote my gridworld agents from mapox-trainer to load Qwen3 weights and created a new value approximation architecture to better take advantage of latent state from pretraining.

The end result is custom llm inference and training infrastructure in jax that can fine tune with Qwen with online rl in a reasonable amount of time on a single consumer gpu at least for this narrow task.

I’d like to expand the framework to explore new training methods, models and environments but I would appreciate any feedback on the project in its current state.

The code for the repo: valm
Writeup and preliminary training results: https://gabrielkeith.dev/posts/valm