Actor-Critic for Car Racing can’t get past the first corner

Hi! I am trying to explore and learn some RL algorithms and implement them in Gym’s Car Racing environment ( https://gymnasium.farama.org/environments/box2d/car_racing/ ).

Instead of using the image on the screen for my state, I measure the distance from the car to the edge of the track at 5 points (90º left, 45º left, forwards, 45º right, 90º right), along with the car’s current speed, and pass that as my state. I also give a fixed -1 reward if the car goes off-track (all distance readings are ≈ 0)

DQN worked well, however as I’ve tried training this now (roughly 1000 races), the car accelerates along the first straight, and brakes to a halt just before it reaches the end of the first straight. At that point, there is little that can be done to salvage the situation, as the apex of the corner has been missed, and any acceleration will cause it to go off track.

Can anyone suggest how to get over this issue? I’ve attached the code at the link below.

https://hastebin.com/share/xukuxihudi.python

Thank you!