On Adversarial RL

Zhang et al. paper’s introducing the SA-MDP framework (2020) (state adversarial MDP) argues that an attack using the critic network (V(s)) is expected and supposed to produce a weaker attack than an attack using the actor network (pi(s)) itself to generate perturbation on agent observations. A claim supported by their empirical results using different single-agent simulation environment.

However, I’m consistently finding the opposite when comparing both attacks on multi-agent PPO policies trained on some scenarios from the VMAS library. Policies are IPPO (Independent PPO, to not confuse with MAPPO) and GPPO (Graph Independent PPO, see Bettini et al. 2023 arxiv) with their heterogeneous versions. The PGD attack is adapted to continuous policies using the KL divergence closed form.

Am I doing something wrong or this is actually something that doesn’t contradicts Zhang et al’s findings given the difference of the context?