When Chaos Wins: noisy net eval with noise off gave wildly inconsistent results. Turning it back on fixed everything.
Running a Rainbow DQN ablation on Snake (C51 + dueling + noisy nets). When I evaluated checkpoints with noise off (mean weights, sigma zeroed out, the standard approach), the scores were all over the place. Some checkpoints averaged 78, others averaged 18. Training curve at those same points was perfectly stable. First instinct was a bug. Checked everything. It wasn’t. The worst case was at ep450K. Deterministic eval produced a bimodal distribution: ~25% of episodes scored near zero, […]