How RL fit into tool-using LLM agent (MCP, hybrid-policies)

Hey!

I’ve been thinking about how RL fits into modern LLM agents that use tools (like MCP-style setups), and I’m a bit stuck conceptually.

I understand how to frame a classic RL setup with Gymnasium, define the environment, actions, reward function, do reward shaping, etc.

But in current agent paradigms, the LLM is already doing a lot of implicit reasoning and exploration when deciding which tools to call and how.

So I’m not sure how RL cleanly applies here.

If you try to train a policy over tool usage, do you lose the natural exploration and flexibility of the LLM?

Or is RL more about shaping high-level decisions (like tool selection sequences) rather than low-level token generation?

I’ve been thinking about hybrid approaches where:

sometimes the agent follows a learned policy

sometimes it falls back to LLM-driven exploration

but I don’t have a clear mental model of how to structure that efficiently.

Has anyone worked on or seen solid approaches for combining RL with tool-using LLM agents in a practical way? (After Finetunning without touching any llm weigths!!)

Especially in setups where the agent interacts with multiple tools dynamically.

thanks for your insights!