[D] Evaluating AI Agents for enterprise use: Are standardized benchmarks (Terminal, Harbor, etc.) actually useful for non-tech stakeholders?

I’ve been assigned to vet potential AI agents for our ops team. I’m trying to move away from “vibes-based” evaluation (chatting with the bot manually) to something data-driven.

I’m looking at frameworks like Terminal Bench or Harbor.

My issue: They seem great for measuring performance (speed, code execution), but my stakeholders care about business logic and safety (e.g., “Will it promise a refund it shouldn’t?”).

Has anyone here:

Actually used these benchmarks to decide on a purchase?
Found that these technical scores correlate with real-world quality?
Or do you end up hiring a specialized agency to do a “Red Team” audit for specific business cases?

I need something that produces a report I can show to a non-technical VP. Right now, raw benchmark scores just confuse them.