Building a Critic-Agent Loop: Scores, Refinement, and Guardrails

Author(s): Nitingummidela Originally published on Towards AI. Building a Critic-Agent Loop: Scores, Refinement, and Guardrails A friendlier take on the guarded critic-agent loop — Worker Bot drafts, Critic Bot scores it, and only passing work ships; anything that fails three times gets escalated to a human instead of looping forever. This is the hands-on companion to Part 1: The Critic Agent — Teach Your AI to Check Its Own Work. There, we covered the idea: a separate critic model reviews the worker’s answer, scores it, and sends it back to fix specific flaws. If you have not read it, start there for the why and when. This piece is how — we build the loop end to end. The problem we are solving Picture a concrete example: an LLM that scans code and reports security vulnerabilities. In its simplest form it makes one pass. Feed it a file, it emits a finding: “SQL injection on line 44.” Most of the time it’s right. But every so often, it is confidently wrong, it flags a query that is actually parameterized, or latches onto a line that is only a comment or a test fixture. In a single-pass design, that false positive goes straight to a human, who wastes twenty minutes proving the code is safe. A few of those, and people stop trusting the tool. One way to improve the consistency and accuracy is to add a second model that reviews each finding before it ships and sends the bad ones back to be fixed. Simple enough as an idea. The trouble is that a naive loop breaks in four predictable ways: The critic rubber-stamps. If it shares the worker’s blind spots, it approves the same mistakes, giving you false confidence instead of a real review. The loop never ends. A stubborn answer bounces between worker and critic forever, burning time and money. A failure still ships. When nothing passes, a careless implementation quietly returns the last (still-wrong) answer as if it were fine. It costs too much. Running a critic on every request doubles your model calls, whether the answer needed review or not. So “add a critic” is not enough. What we actually need is a guarded loop, one that verifies before it ships, but also terminates, escalates honestly, controls cost, and cannot be fooled by a critic that just agrees with itself. That is what we build below, one guardrail at a time. The code is Python and model-agnostic; worker_llm and critic_llm are whatever clients you use. What we’ll build The critic prompt — teach a second model to judge, not redo the work. A critique you can branch on — turn the review into a machine-readable decision. The capped refine loop — wire it together, with a hard stop and honest escalation. Better feedback — make the loop converge by passing back exact issues. The confidence gate — only pay for the critic when it matters. Keep the critic independent — the guardrail that makes the whole thing trustworthy. Then we cover tools and security. Step 1 — The critic prompt We start with the critic itself, because everything downstream depends on the quality of its review. The critic needs a fundamentally different prompt from the worker. The worker’s prompt says, “Do the task.” The critic says, “Find what is wrong with this answer, and do not redo the task.” That distinction matters: a critic that re-scans the code from scratch is just a second worker, and it will invent its own new mistakes. We want it focused on judging the finding in front of it. Give it concrete checks and demand structured output, so we can act on the result in code rather than parse prose: You are reviewing another model’s security finding. Do NOT re-scan the code.Judge only the finding below against these checks:1. Evidence: does user input actually reach the sink, or is the cited line a comment, a test fixture, or already parameterized/sanitized?2. Correctness: is the vulnerability class right for the evidence?3. Completeness: did it miss an obviously related issue in the same snippet?Return JSON only:{“score”: 0-10, “verdict”: “pass” | “fail”, “issues”: [“…”, “…”]}Finding to review:{finding}Code evidence:{evidence} Two choices are doing real work here. The concrete checks turn a vague “is this good?” into specific, catchable failure modes. Our false positives came from comments and parameterized queries, so those are exactly what we ask the critic to look for. And the JSON output means the next step can branch on a verdict instead of interpreting free text. Step 2 — A critique you can branch on The critic now returns JSON, but a raw dict is fragile to build on. Parse it into a typed object, because the entire loop keys off this one result: from dataclasses import dataclass@dataclassclass Critique: score: float # 0–10, from the critic verdict: str # “pass” | “fail” issues: list[str] # specific, actionable notes for the workerPASS_THRESHOLD = 8.0def passes(c: Critique) -> bool: # require BOTH: a verdict of pass AND a score over the bar. # two independent signals guard against a fluke high score. return c.verdict == “pass” and c.score >= PASS_THRESHOLD Notice that passes() demands both a pass verdict and a numeric score over the bar. This is deliberate, and it is our first small guardrail: two independent signals are much harder to fluke than one. The numeric threshold also gives us a single knob to tune strictness later without rewriting the prompt. With a clean Critique and a clear pass rule, we can finally wire the loop. Step 3 — The capped refine loop This is the heart of the pattern, and where three of our four failure modes get their guardrails. Produce an answer, critique it, and if it fails, feed the specific issues back to the worker and try again, but only up to a hard limit: MAX_ITERS = 3def solve_with_critic(task, worker_llm, critic_llm): answer = worker_llm.produce(task) best = answer for attempt in range(MAX_ITERS): critique = critic_llm.review(task, answer) # returns a Critique […]