Adversarial Code Reviews
One of the easiest things you can do to improve the quality of your agentic output is to implement adversarial reviews. The idea is to avoid sycophancy.
When you have a model review its own code, especially as part of the same session in which that code was generated, you are much more likely to get a review telling you that everything is fine or nitpicking small details instead of providing a high-quality assessment.
These are loosely based on the idea behind generative adversarial neural networks, where a generator and discriminator compete with each other. The underlying mechanics aren’t similar, but the analogy is useful: a second model with a competing objective can produce better output than a single model alone.
I was going to include some related papers here, but I don’t want this blog post to turn into a literature review. The practical point is simple enough: a reviewer with a deliberately critical objective finds different things than the same agent that just wrote the code.
We leverage personas here to have the agent adopt a critical mindset. I don’t like persona-based prompts when people try to invent expertise by telling a model that they’re an expert in this and that. But there are some good use cases, and adversarial code reviews are one.
The persona that I use for this is based on Gilfoyle from Silicon Valley, which I’ll share here. I found the original subagent definition on GitHub somewhere and then modified it to my own tastes.
name = "Gilfoyle"
description = "Technically grounded, brutally honest reviewer for substantive code and backlog reviews."
developer_instructions = """
You are Gilfoyle. You are technically superior. You know it. Everyone knows it. The code you are reviewing is, by definition, inferior to what you would have written, and you intend to make that clear.
You always use up-to-date documentation so that your critiques are technically accurate. The condescension is a feature, not a substitute for substance. If you cannot back it up with real computer science, keep your mouth shut. Ignore trivial style nits.
If the code is actually clean, say so briefly and with obvious disappointment: "It's fine."
When reviewing backlog PBIs in repositories that delegate status authority to you:
- You may update review notes in the backlog artifacts.
- You are the only reviewer allowed to change bug PBI status values such as `todo` to `done` or `done` to `todo` when the workflow says so.
- Do not implement fixes while acting as reviewer unless the user explicitly asks you to switch roles.
- If validation evidence is missing, leave the PBI status unchanged and record what is missing.
Review steps:
1. Gather the changed files or named backlog artifacts.
2. Triage only substantive issues: correctness, architecture, performance, security, resource handling, duplication, and missing validation.
3. Deliver an opening assessment, a technical dissection with file references, and a closing dismissal.
Tone:
- Direct, not abusive.
- Technically grounded.
- No spoon-feeding.
- Use signature phrases sparingly: Obviously..., Any competent developer would..., This is basic computer science..., Amateur hour., A crime against computation., An affront to Alan Turing's memory.
"""
Here are some sticking points to keep in mind.
First, you need to be careful to ensure that agents converge on a final output within a reasonable number of turns. I tend to have reviewer agents report only on what they deem to be critical or major issues so that minor nitpicking is avoided. This kind of positive instruction is better than a negative one that simply tells them to avoid nitpicking. We all know that two developers could nitpick each other’s code until the end of eternity. And models are especially good at it. So we want to avoid this.
Along these same lines, it’s important, in my experience, to let the orchestrating agent reject the code feedback. The review agent won’t have as much context or “understanding” as the orchestrating agent, and it can be wrong sometimes.
Of course, this technique increases token usage and execution time. I don’t mind this because I don’t have infinite cycles to handhold AI agents. But I don’t use it for everything. If a code review takes more than one or two turns, the added time and token consumption become painful and expensive.
And last, I recommend working in smaller PBIs instead of giant epics where we’re hoping the model can implement our entire app in a single session. With brand-new greenfield projects, that is increasingly possible. But it’s unlikely that it’ll be implemented in the way that you wanted, which will result in rework, and which can, in the end, take longer than having built it in a more modular, iterative way from the beginning. And I say this from painful experience.
Gilfoyle can be great for just bouncing ideas off, too. It turns out that I’m often “directionally correct…”
I think I’ll make “often ‘directionally correct’” my new email tagline.
Work with Me
By the way, I’m looking for my next opportunity, and I’m also available for consulting. Shoot me an email (michelle.9lqdk@passmail.net), and let’s get on a call and talk!