Limitations of RLHF as a static preference optimization paradigm for LLMs — towards interactive / multi-agent formulations?

Following up on some thoughts around RLHF and LLM training.

Most current RLHF pipelines can be framed as optimizing a policy πₜ (the LLM) against a learned reward model r_φ that approximates human preference distributions over outputs. In practice, this is often implemented with PPO-style updates under KL constraints relative to a reference policy.

This setup works well for alignment and helpfulness, but it has a few structural properties that seem limiting:

1. Static reward modeling
The reward model is trained on pairwise (or ranked) human feedback over isolated outputs.
This implicitly assumes:

• i.i.d. samples
• short-horizon evaluation
• no evolving environment dynamics

There’s no notion of reward emerging from interaction tracjectories.

2. Lack of temporal credit assignment
Most RLHF setups optimize over very short horizons (often single responses or short chains).
This avoids hard credit assignment problems, but also means:

• no delayed rewards
• no long-term policy consequences
• minimal pressure for consistent reasoning across turns

3. No persistent environment / state
LLMs operate in effectively stateless or shallow-context environments:

• no persistent world model
• no environment transitions
• no endogenous dynamics driven by agent actions

This contrasts with standard RL settings where policies must adapt to environment evolution.

4. Absence of adversarial or multi-agent pressure
In many domains, capability emerges from:

• competition (self-play)
• adversarial dynamics
• equilibrium-seeking behavior

RLHF largely removes this by collapsing feedback into a single scalar reward signal approximating human preference.

Given these constraints, RLHF seems closer to:

than to full RL in the sense of learning under environment dynamics.

This raises a few questions:

• Can we frame LLM post-training as a multi-agent RL problem, where models interact (e.g., debate, critique, collaboration) and rewards emerge from outcomes over trajectories rather than static labels?
• Would self-play or population-based training (analogous to AlphaZero-style setups) be meaningful in language domains, especially for reasoning tasks?
• How would we handle long-horizon credit assignment for reasoning quality, where correctness or usefulness only becomes clear after extended interaction?
• Is there a viable way to construct environments for language models where:
  • state evolves
  • actions have persistent effects
  • reward is delayed and context-dependent

Intuitively, RLHF captures alignment to human preference distributions, but may underutilize RL’s strengths in:

• learning under interaction
• adapting to dynamic systems
• improving through adversarial pressure

Curious if people here are working on:

• multi-agent LLM training setups
• debate/self-play frameworks
• trajectory-level reward modeling for reasoning

Would appreciate pointers to papers or ongoing work in this direction.