PPO vs SAC on real robot

I’m working on an RL project using two algorithms: SAC and PPO. The project consists of a robot made up of 3 arms (controlled by 3 servos) attached to a plate on which a ball will roll. Through infrared sensors placed under the plate, I can detect the ball’s position (observations) and move the plate (pitch and roll actions) to bring the ball to the center and stabilize it.

The reward is defined based on the distance from the center of the plate, the ball’s speed which must be kept low, and a penalty for overly jerky robot movements.

When training with PPO, I manage to reach a fairly good policy that allows balancing the ball on the plate. With SAC, however, I struggle a lot and I believe it depends on the parameters.

Training is done by placing a ring around the plate to prevent the ball from falling off; episodes never end before 128 steps (I’ve set it up this way), which make up a single episode.

At the moment this is the definition:

SAC( policy="MlpPolicy", env=env, learning_rate=cosine_lr_schedule(3e-4), buffer_size=2**17, # Total steps learning_starts=4096, train_freq=128, gradient_steps=64, batch_size=256, policy_kwargs={"net_arch": [64, 64]}, gamma=0.99, n_steps=128, ent_coef="auto", target_entropy="auto", target_update_interval=1, verbose=0, tensorboard_log="./tensorboard_training_logs/", ) 

The agent tends to take very small actions, it almost learns to stay still. Could anyone explain why?

Find training metrics here:
https://postimg.cc/Q9W5b7P4
https://i.postimg.cc/htnpbjhq/SACLosses.png
These are some runs with different parameters; despite the best one (light blue) managing to reach the reward achieved by PPO (grey), in inference it fails to stabilize the ball unlike PPO which does it very accurately. Training is done on a real robot, it is very time consuming. For this reason some runs have been interrupted 🙁