Training an Agent to walk using Procedural Animation in my Game Engine

I am creating my own game/simulation engine since the last year. Currently i am working on procedural animation and i am having some trouble with it.

The agent can learn to balance itself easily but when i try to teach it to walk, it just can’t do it. It moves only about 0.5 on x-axis and then falls down or the episode ends(due to maximum time limit). I am kind of new to this procedural animation stuff but i’ve seen some videos of it.

Can anyone tell me what’s the problem with my agent here? The max reward won’t rise after a few episodes. I am using Box2D for physics and LibTorch to train the network. The renderer is made by me using OpenGL and i am trying to train it to walk from scratch.

I don’t think that the problem is in physics or other parts of my engine. Because i’ve already did pendulum and double pendulum balancing and training the agent to stand without falling down. But i can’t get it to walk. I’ve tried different reward functions but those did not work so i added a very simple reward that can tell the agent to always move forward. Here is my current reward related code : –

// Forward velocity reward float reward = vel.x * 0.1f; // Penalize falling - if root body angle is too large float angle = rootBody->GetAngle(); if (std::abs(angle) > 1.2f) // ~70 degrees { reward -= 1.0f; brainComponent.done = true; // end episode on fall } 

Btw i am using the PPO algorithm here. If this much info is not enough, feel free to ask me. It would be nice to hear your suggestions if you’ve worked on this kind of problem before.

https://reddit.com/link/1umh562/video/jsw7otnv31bh1/player