I’m currently in the process of trying to teach an agent to play a homemade version of Suika Game. This a a highscore game some of you might know, in which you drop fruits of different sizes into a basket. Two same fruits that touch, merge together to the next bigger fruit and award score based on the type of the fruit. The game ends when the basket overflows.
I built the game in Godot and use the Godot-RL-Agents addon to communicate between godot and stablebaselines3.
This is the first time i am doing something with reinforcement learning, but i already had some success. The agent with the current settings managed to improve quite a bit over just random play. The training reward over almost 20 milion timesteps can be seen here:
https://preview.redd.it/6uidd8wegfbh1.png?width=1152&format=png&auto=webp&s=def29a694d920038ed9af940794e08d7d921fa13
policy_kwargs = dict( net_arch=dict( pi=[256, 256], vf=[256, 256] ) ) model: PPO = PPO( "MultiInputPolicy", env, ent_coef=args.ent_coef, verbose=2, n_steps=args.n_steps, tensorboard_log=args.experiment_dir, learning_rate=learning_rate, batch_size=args.batch_size, clip_range=args.clip_range, policy_kwargs=policy_kwargs, )
This is the model I currently use.
Learning rate for the first half of training was 5e-5 and for the later half 1e-5.
n-steps = 256.
entropy-coefficient = 1e-4
clip range: 0.2
batch size = 64
Does one step every 120 frames in game (60fps), so physics can settle mostly. This should optimally be lowered to like 30 frames, as a player can drop a fruit every 0.5 seconds.
For observation it gets:
the current held fruit type and radius, the upcoming fruit type and radius, the score, the boundaries of the basket left, right, bottom and top and the y position of the highest fruit, all scaled to be within 0 to 1.
And
For each fruit on the board, sorted by y-position:
type
position x,y
linear velocity x,y
radius
also all scaled to be within 0-1
As reward it gets:
+score gained from merges/max fruit score(=55)
+0.02 for each drop
+0.1 for each merge
-10 for game over
– 0.02 for choosing a drop position too close to the wall, where the fruit would clip into the wall shortly
-0.06 for dropping at the same position (+/- 1 % of total width) more than 2 times
It’s only action is to choose an x position to drop. This is a contiuous action from -1 to 1 and gets mapped to the width of the basket.
Now i am at the point where the objective becomes squeezing out better performance.
The agent definitely learns some strategy and is capable of decent scores, but not reliably. Training itself has plateaued and basically no gains were made over another 5mil steps. The reached scores form a relatively bell shaped curve with a low number of games ending with scores below 950, most of the games ending somewhere between 1000 and 1800, some games over 1800 and even single games where scores of more then 2500 were reached. Average and median score is just below 1400. Random games average around a score of 800, so there is some improvement.
The best score i reached myself in 20-30 games was around 2200 for reference.
Does anyone have some advice on how the proceed from here?
I still have some ideas of what to try, but i dont really know what would have the best potential:
– many atari games get the image of the game screen as the observation. with cnn policy
– change the observation. especially the fruits on the board. not sure if sorting the list the way i do is beneficial.
– use dicrete action space and deep-q network instead of PPO (already tried with ppo to less success as with contiuous action)
– increase gamma for weighing rewards in the future more
– stacking multiple observation (already tried, did pretty bad in this setup)
– starting with randomly prefilled boards in training
– modifying the reward
If you have any questions or suggestion, feel free to share.
