Strategy to efficiently debug and do reward shaping for Reinforcement Learning

Hi everyone,

I’am a student in France working on a drone guidance project using reinforcement learning. The goal is to make a drone reach a sequence of checkpoints, or track a moving checkpoint using vision-based guidance as I implement this on my FPV drone, and this drone has the PX4 controller so the pipeline for the loop is : Guidance –> give accels –> PX4.

So far, I have first built everything in Python, I implemented a proportional guidance law and it worked quite well in simulation, but it did not perform very well once I used the camera-based observations.

Then, I move to an RL-based pipeline with RL policy –> accels –> PX4. I implemented the full pipeline in simulation and it technically works, but I’m seeing a lot of strange behaviours : oscillations, bang-bang commande law abusements,…. My suspicion is that the issue may be due to the reward function. I have tried tuning and cooking the reward many times but each version seems to produce a new unexpected problems or strange behaviours rather than the one I actually want. I have tried to plot many metrics to understand what is happening but debugging this RL guidance law has become frustrating.

Does anyone have suggestions or advice for debugging this kind of RL guidance or RL related problem please ? In particular, I would like to have some advice on reward shaping and how to efficiently debug trained RL policy,…

Any advice, refs, or practical debuggings, tips or discussions would be really helpful for me !

Thanks a lot and I wish you a good day !