Tpo-torch: Stable RLHF alignment in PyTorch using Target Policy Optimization

Hey everyone,

RLHF alignment using standard Proximal Policy Optimization (PPO) can be notoriously tricky to stabilize during LLM post-training due to policy collapse and high sensitivity to hyperparameters.

I built Tpo-torch to explore Target Policy Optimization (TPO) as a cleaner, more stable alternative for preference alignment directly in PyTorch.

Key Focus Areas:

• Mitigating policy collapse without requiring aggressive KL-divergence penalties.

• Modular, lightweight, and readable implementation designed for research and custom fine-tuning pipelines.

• Integrated stability benchmarks comparing policy drift against standard PPO.

I’ll drop the GitHub repository link in the comments below! I’d love to hear feedback from anyone experimenting with alignment, preference optimization, or RLHF.

Repo link : [https://github.com/Griffith-7/Tpo-torch.git](https://github.com/Griffith-7/Tpo-torch.git)