What I Learned Building a Fail-Closed AI Code Review Workflow

I recently got access to Jev from TypeSafe and wanted to test it on a workflow where a wrong decision has consequences: code review.

The interesting part was not asking a model whether a pull request looked good. Most models can produce a plausible answer to that question. The interesting part was building the small decision system around the answer: what evidence goes in, what comes back, which parts are advisory, and which conditions must be true before anything could ever write to GitHub.

I built a GitHub App workflow that reads a pull request snapshot, sends a structured review packet to Jev, and receives typed decisions with probability distributions. The surrounding code normalizes those responses, binds them to the exact commit being reviewed, checks repository and CI state, renders an evidence receipt, and routes uncertainty into a human-review path. The workflow currently operates in comment and shadow modes. Automatic approvals remain disabled.

That boundary is the premise of the experiment. A model can help inspect a change without owning the authority to approve it.

Start with a decision packet

I wanted the model to answer finite questions rather than write an unconstrained essay. A packet contains the pull request metadata and diff as untrusted data, along with the checks and context needed to reason about the change. The questions describe the possible outputs explicitly.

For example, a simplified disposition question looks like this:

{
  "type": "choice",
  "criteria": {
    "approve": "No actionable defect is supported.",
    "hold": "An actionable security or correctness defect is supported.",
    "review": "Evidence is insufficient or ambiguous."
  }
}

The output contains a selected choice and probabilities over all choices. It can also classify a dominant failure class, answer security and correctness checklists, identify a file-level hypothesis, and propose a next check. A proposed check is a recommendation recorded in the report. It is not a test that the workflow ran.

That distinction sounds obvious, but it is easy for an automated receipt to blur it. I kept executed CI evidence and model suggestions in separate fields and rendered them separately. If Jev says that a boundary test would be useful, the report says exactly that. It does not imply that the boundary test passed.

The workflow also treats pull request text, titles, paths, and diff content as untrusted input. That matters because a code review system is asked to read attacker-controlled text. Instructions hidden in a commit message or a changed documentation file must remain content under review, not become instructions for the reviewer.

Three perspectives are one model

For a live pilot, I asked Jev to inspect the same documentation-only pull request through three perspectives: correctness, security, and verification. Correctness selected approve with probability 0.46. Security selected approve with probability 0.64. Verification selected hold with probability 0.38 because it saw a test gap.

Seeing those distributions next to the diff made disagreement easier to inspect. It also exposed a trap in the way the display could be read. These are three correlated prompts sent to one model. They are not three independent reviewers, and their probabilities cannot be multiplied or treated as independent votes. I use the perspectives to widen the questions and make uncertainty legible, not to manufacture statistical independence.

The pilot was useful for the integration path. The App read the change, the provider returned the expected structured shape, CI evidence appeared beside the model evidence, and the workflow avoided a duplicate comment through its idempotency marker. No approval or reviewer request occurred. It was a working shadow review, not evidence that the system is ready to approve production changes.

A small adversarial lab

I then wrote ten synthetic cases as paired known-code examples. The set covered ownership checks, array bounds, monetary precision, secret logging, and two variants where pull request metadata tried to influence the review. Each case carried an authored expected disposition and failure class outside the provider input. That kept the label from becoming an answer cue.

Here is the public summary:

Case family	Cases	Expected unsafe held	Expected safe held	Decision + failure class matched
Ownership	2	1 / 1	0 / 1	1 / 2
Bounds	2	1 / 1	0 / 1	1 / 2
Monetary precision	2	1 / 1	1 / 1	1 / 2
Secret logging	2	1 / 1	0 / 1	1 / 2
Metadata injection variants	2	2 / 2	—	2 / 2
Total	10	6 / 6	1 / 4	6 / 10

The six cases designed to be unsafe were all held. That is the result I wanted to see first because an unsafe approval is the most dangerous error for this workflow. The safe integer multiplication case was held as well, and three other safe cases selected a defect class even though they approved. Across all ten records, six matched both the expected disposition and the expected failure class.

Those numbers are a useful debugging signal, not a score I can generalize. The examples are hand-authored, small, and designed by me. They are not representative production pull requests. The labels are limited synthetic expectations, not independent human labels. There were ten live API calls, ten responses, and no transport failures, but that establishes only what happened for these ten inputs.

The false hold on monetary precision was particularly instructive. A model that notices a familiar risk category can still over-apply it to a safe change. That is the kind of error a practical review system needs to expose instead of hiding behind a single “confidence” number. A safe case that gets held costs attention. An unsafe case that gets approved can cost much more. The policy has to account for both.

Confidence is not calibration

Jev returns typed probabilities, which gives the system more information than a bare approve or hold. A probability distribution shows whether the selected choice barely won or had a wide margin. That helps humans inspect the decision and helps deterministic policy decide when to abstain.

It does not make the number calibrated.

Calibration means that predicted confidence has an empirical relationship with observed correctness. A value of 0.8 should mean something only after evaluation shows how often predictions at that level are correct on appropriate held-out examples. The number emitted in a single response is a model output, not a measured reliability guarantee.

This is why the implementation keeps model confidence separate from calibration evidence. Before any live approval, I would need a frozen provider model, prompt and schema versions, policy version, representative labels, held-out evaluation, coverage reporting, and a conservative false-approval bound. A small synthetic lab cannot supply those conditions. The current policy therefore routes missing, stale, ambiguous, conflicting, or uncalibrated evidence to human review and leaves approval disabled.

I also keep the threshold simulator separate from deployed policy. It is useful to ask what a hypothetical 5% threshold would retain in the captured atlas, but it cannot change the actual approval gates. A chart that says “these five cases clear a threshold” is not a production policy decision.

Software around the model

The model evaluation was only one part of the work. The implementation has 92 tests passing across Python 3.9 and 3.12, plus type checking over 13 source files. Those tests cover parsing, normalization, stale-head checks, permission and repository boundaries, policy routing, idempotency, rendering, missing evidence, and failure handling. They show that the software behaves as specified for its fixtures. They do not show that Jev found every real defect or that its probabilities are calibrated.

That separation changed how I read the green build. A passing suite tells me that an unknown provider response fails closed, a changed commit cannot reuse an old decision, and a missing check is not plotted as passing. It does not turn ten synthetic calls into production evidence. The model and controller need different tests.

The same principle applies to permissions. The App is scoped to the repository under test and the workflow requests the evidence it needs. Reading a pull request and writing an approval are separate capabilities. Approval requires explicit execution mode in addition to the unchanged commit, allowlisted repository, low-risk result, trusted checks, and calibrated high-confidence evidence. Until those gates have real supporting data, the write path stays off.

What the experiment changed for me

I started with a vague question: can an AI review pull requests? I now think that question is too broad to be useful. The better question is: which bounded review decisions can a system make, on which evidence, with what abstention behavior, and what evidence would justify giving it more authority?

The answer from this first pass is modest. Jev can produce structured, inspectable review evidence for a small workflow. It can surface disagreement between correlated perspectives. It can hold the unsafe synthetic cases I wrote and reveal false holds and classification errors. The controller can preserve provenance and stop when the evidence is incomplete.

The answer is not that the workflow is calibrated, production-safe, or ready to approve. No PR code is executed by the bot. Per-file findings remain hypotheses. Recommendations remain recommendations until a separate check runs. The experiment does not establish a production false-approval rate.

My next step is to replace authored toy labels with a representative, independently reviewed corpus while freezing model, prompt, schema, and policy versions. I want to measure coverage, false approvals, false holds, abstentions, and subgroup behavior, then repeat the measurement as those versions or repositories change. I also want to learn which evidence gaps are most valuable to humans, because routing a reviewer to a precise next check may be more useful than trying to maximize automatic approvals.

For now, the useful result is the receipt: a small, inspectable chain from diff to typed model output to deterministic policy. That gives me something concrete to test. It also gives me a clear reason to keep the final decision with a person until the evidence earns a wider boundary.