loading
loading
Improving Code Review Recommendations with GEPA False positives are an obvious problem in AI code review. But there is another source of noise: a finding can be technically reasonable and still be something a particular team will repeatedly ignore. At Qodo, we personalize code review using a team’s own history to ensure that findings are not just high precision, but relevant to the team. Qodo’s recommendation agent runs under the hood, using the team’s PR review history to prioritize new findings. It considers which suggestions led to code changes and which were passed over. To optimize its system prompt, we adapted GEPA, an algorithm that iteratively improves prompts by analyzing an agent’s outputs and evaluation feedback. The prompt we shipped cut the agent’s average cost roughly in half and latency by 25% in a production A/B test, while preserving precision and improving recall. In this blog, we explain how we adapted GEPA to improve the recommendation agent’s instructions, evaluated the results, and tested the changes in production. What the recommendation agent does The recommendation agent runs after the review agent produces its findings. It receives the findings, the pull request diff, and repository context, then searches a preprocessed index of past PR suggestions and the team’s responses. That history helps it predict which findings are relevant to this team. For each finding, the recommendation agent predicts whether the team is likely to act on it, rather than judging technical correctness alone. The recommendation agent returns “yes”, “no”, or “maybe”, along with reasoning and references to relevant past PRs. In Qodo’s PR review, reviewers can see the predictions marked as star ratings: three stars for “yes”, two for “maybe”, and one for “no”. Improving consistency through prompt optimization We saw an opportunity to make the agent’s recommendations more consistent by optimizing its system prompt. With the current production prompt, running the agent twice on the same PR could produce different labels. On an evaluation set of 200 PRs, 20.7% of labels soft-flipped, shifting between “yes” and “maybe”, or “maybe” and “no”, while 4.3% switched directly between “yes” and “no”. These disagreements suggested room to improve how the agent used evidence and made decisions. We focused on its system instructions, keeping the same task, tools, and output labels; no model weights were trained. Creating the dataset We used a Fable agent to audit internal PR reviews in Qodo, with one agent per PR and access to the full review history and follow-up commits. The resulting dataset contained about 400 PRs from five repositories, split chronologically into roughly 120 training, 80 validation, and 200 test PRs. Each example has up to 10 findings, each associated with a boolean acceptance label. Adapting GEPA to noisy agents GEPA uses feedback from agent runs to propose better instructions. Long agent traces limit how many examples a reflector can inspect together. With noisy behavior, a small batch can make an accidental outcome look like a general lesson, so we made our own version of GEPA, with two steps: - First, reflect on two runs of each example: For each PR, we run the recommendation agent twice. An LLM analyzes the full traces and outputs a structured assessment of what went right, what went wrong, and what might explain any difference between the runs. - Aggregate the reflections across the batch: The structured analyses across the entire training set are aggregated in a single LLM call with high reasoning effort. This lets it look for patterns across the entire training set without needing to process every full trace at once. A single run can hide unstable behavior behind a plausible answer. Comparing two runs lets the reflector inspect where the agent gathered different evidence or reached different decisions on the same PR. Starting from a simple 43-word prompt that describes the agent’s goal only, we ran a single optimization step with a batch of the entire training set. For regularization, we instructed the LLM to write at most 200 words for each structured reflection and at most 1,000 words for the final prompt. The successful recipe took one wide training step across the training set, rather than a long sequence of small edits. We generated multiple candidates and compared recommendation quality, cost, latency, and stability offline. This made prompt optimization a measurable engineering process with explicit tradeoffs. Offline evaluation narrowed the candidates. The selected prompt was also more consistent on offline validation: hard flips fell from 8.2% to 2.2%, and total flips from about 27% to 20%, relative to the 43-word starting prompt. The decision to ship depended on whether the selected prompt helped on real review traffic. The production result We compared the selected prompt with the existing production prompt in a 50/50 A/B test assigned by PR, across all of our customers. The test was designed to change only the system prompt, keeping the model, tools, and UI behavior fixed. Of the findings developers implemented, the share labeled “no” (i.e. miss rate) fell from 7.83% to 4.46%. The quality analysis covered PRs with available attribution labels and at least one implemented finding, so the result applies to that subset of PRs. This metric measures missed developer action, not the rate of incorrect review suggestions. Mean recommendation agent cost per review run fell from $0.24 to $0.12, and mean latency from 40 to 30 seconds, across 27,582 review runs. These savings are for the recommendation agent component of code review. Precision was comparable, with no statistically significant difference. The practical result was better retention of findings developers acted on, at substantially lower serving cost. We promoted the prompt to production. The revised prompt gave the agent clearer guidance on finding relevant historical evidence, stopping redundant searches, and handling uncertainty. It also sharpened the distinction between predicting a team’s response and judging code quality. Lessons - Try a minimal prompt. It can be easier to optimize a minimal prompt than a production prompt that has already been optimized manually. - Learn from the agent runs. The additional signal in training comes from the agent’s tool choices, evidence gathering, and inconsistent decisions. Choose examples that expose these behaviors and reflect on the full agent traces, not only inputs / outputs. - Measure noise before scaling up experiments. Run the agent on the validation set multiple times to understand the variance before relying on the validation score to make decisions. - Run each recipe more than once and compare candidates. Instead of searching for the optimal stable recipe, just run a simple one multiple times and take advantage of the LLM’s stochastic behavior.
document fetched 2026-10-24 11:26 · canonical https://qodo.ai/blog/improving-code-review-recommendations-with-gepa