[D] Lessons learned when trying to rely on G-CTR-style guarantees in practice

Following up on earlier discussions around AI evals and static guarantees.

In some recent work, we looked at G-CTR-style approaches and tried to understand where they actually help in practice — and where they quietly fail.

A few takeaways that surprised us:

– static guarantees can look strong while missing adaptive failure modes

– benchmark performance ≠ deployment confidence

– some failure cases only show up when you stop optimizing the metric itself

Paper for context: https://arxiv.org/abs/2601.05887

Curious how others here are thinking about evals that don’t collapse once systems are exposed to non-iid or adversarial conditions.