High variance returns, are they normal?

Using SAC, and trying to use curriculum learning to advance training slowly. Training advances when the moving average plateaus, however, often when it plateaus it is actually not the optimal solution yet when I look at the variance, there are many instances where the episode returns an optimal solution. How can I converge to this optimal instead? Or should I accept that this is inherent to RL?