The model is a commodity. The process around it is not. Your AI pipeline will only stop the failures you have explicitly told it to stop. The one reliable source for what fails in your codebase is your own review history.
So I went and got mine.
The audit
Over three months I worked through more than 100 PR review comments across two production services I contribute to. I copied every non-trivial comment into a spreadsheet, tagged the underlying failure, and looked for repetition. One-off stylistic preferences did not interest me. I wanted the things that kept coming back.
Sixteen categories survived the cut. They split cleanly by service.
Service A: Java / Spring Boot
| Category | Notes | Severity |
|---|---|---|
| Concurrency / transactions | Missing @Transactional, check-then-act gaps, Hibernate flush timing, batch input dedup | 4 instances |
| Entity copy gaps | Missing identity/tracking fields in copy constructors | 2 BLOCKERs |
| Data shape | Wrong nested object read, write payload pollution, stored vs. derived | Recurring |
| Migration safety | Data assumption not validated before transform; partial index scope too narrow | High |
| Model design | Wrong relationship type, wrong unique-constraint scope | High |
| Dead code | Stale abstractions left after refactor | Medium |
| Test quality | Asserting on infrastructure state instead of behavior | Medium |
| Defensive overreach | Null guards on data that cannot be null per domain model | Low |
Service B: Rails / React
| Category | Notes | Severity |
|---|---|---|
| Logic errors | .some() vs .every(), wrong property path, wrong operator | 9 instances |
| Duplicate logic | Same utility in 2-3 files instead of a shared module | 4 instances |
| Scope creep | Unrelated bug fix included; intentionally-removed feature re-added | High |
| Dead code | Unused props, unreferenced schema fields | Medium |
| Controller bloat | Transforming/filtering in the controller instead of passing through | Medium |
| UI state misuse | Business-action guards keyed on display state, not actual data | High |
| Feature flag gaps | New UI shipped without feature-flag registration | BLOCKER |
| Code hygiene | Redundant class names, missing params in mailer links | Low |
Two things are worth calling out. First, the services barely overlap in failure modes. Service A fails on concurrency and entity-state bugs, which is what a strongly typed, transactional, ORM-heavy stack punishes. Service B fails on logic errors and duplication, which is what a dynamic, two-language stack punishes. Same engineer, different traps.
That is the whole argument. A generic AI code reviewer cannot know that .some() against .every() is the most common failure in this codebase, or that copy-constructor identity drift has shipped to production twice. My reviewers know both. Their comments are the ground truth.
The insight: four intervention layers
Sixteen categories is too many gates to operate. But they cluster.
- Correctness: logic errors, concurrency, entity copy gaps, data shape, defensive overreach. Wrong code.
- Design: model design, controller bloat, duplicate logic, UI state misuse, dead code. Right answer, wrong shape.
- Scope: scope creep, intentionally-removed-feature re-add, dead code from prior refactors. Wrong boundary.
- Safety: migrations, feature flags, test quality. Wrong blast radius if it ships.
Each layer fails at a different point in the development arc, so each layer needs its own gate. Catching scope creep at code review is too late, because the work is already done. Catching a concurrency bug at planning is too early, because no code exists to reason about.
| Layer | Fails earliest at | Pipeline phase |
|---|---|---|
| Scope | Requirements gathering | Phase 1, Scout |
| Design | Plan, before code | Phase 2, Plan |
| Safety | Plan stress-test | Phase 2.5, Grill |
| Correctness | Implementation + self-review | Phase 3 + Phase 4 |
| All | Final gate | Phase 5, Completion |
The pipeline: disciplined-development
Six phases, each non-skippable, all built on Claude Code. The discipline is the product. The model is incidental.
Phase 1: Requirements (dd-scout)
Fetches the ticket, parent epic, sibling tickets, all comments, and any linked specs or ADRs. Produces a structured requirements brief: in-scope, out-of-scope, open questions, acceptance criteria, related decisions.
Gate: blocks on any UNKNOWN. You cannot proceed to Phase 2 with unresolved questions.
This kills scope creep before it has a surface to grow on. One failure taught me that Phase 1 had to exist: a feature we removed on purpose came back by accident. The only way to keep a removed feature out is to know, in writing, that it was removed.
Phase 2: Plan
The implementation plan has four required sections. A reuse inventory lists what already exists. A data-flow map shows where state enters, transforms, and lands. A NOT-DO list names the things a reasonable engineer might do that this ticket excludes. Intentionally Removed Items lists features deleted in earlier tickets that must not come back.
User-approved before any code is touched. The reuse inventory is the direct response to four "you duplicated this utility" comments. The NOT-DO list is the direct response to scope creep.
Phase 2.5: Stress-test (grill-me / grill-with-docs)
A sequential design interview, one question at a time, probing the plan adversarially. It routes automatically to grill-with-docs when Phase 1 surfaced a sensitive path such as auth, migrations, or payments. That variant pulls the relevant ADRs and internal docs into context.
This is where migration-safety and feature-flag failures get caught. The questions write themselves once you know what has broken before. What data assumption are you making about column X? How did you validate it? What is the partial-index scope, and what happens to the rows outside it?
Phase 3: Implement
TDD is enforced, and every file gets a dedup check. Whenever two or more files would hold the same logic, the pipeline extracts a shared module before it continues. That check exists because four reviewers said the same thing four times.
Phase 4: Self-review
Thirteen ordered checks executed by four parallel sub-agents. Findings classified:
- AUTO: fix immediately, no confirmation needed
- BATCH: collect, fix together, single confirmation
- MANUAL: surface to me with context, I decide
Results saved to Obsidian before Phase 5 is allowed to begin. The check list is the audit table, one-to-one. Every gate has a reviewer comment behind it.
Phase 5: Completion
Evidence is required. "Tests pass" does not count; the captured command output does. "Lint is clean" does not count; the linter's stdout does. dd-reviewer then runs a full 13-check pass against the diff and returns go or no-go. Only then does the PR open.
[dd-reviewer] 13/13 checks complete
correctness: pass
design: pass
scope: pass (NOT-DO list honored)
safety: pass (feature flag registered, migration validated)
-> GOThe supporting stack
Briefly, because the gates are what matters here.
- ast-grep + LSP for code search. ~70% token reduction versus grep on the same questions, because structural queries return structural answers.
- difft for structural diffs that sub-agents can actually read.
- shellcheck as a PostToolUse hard guard on Bash. The agent cannot run a shell command that shellcheck rejects.
- Claude Code sandbox mode for execution boundaries. Filesystem and network allowlists.
- Slack async observability so long-running phases don't require babysitting a terminal.
- Custom Bitbucket MCP server, because no official one existed.
- Three-tier memory: LLM Wiki (Karpathy's pattern) for durable facts, Obsidian vault for project state, JSONL session knowledge graph for in-flight context.
- Three auditors:
lint-memory,lint-skills,system-gc. They run against~/.claude/and flag drift.
All of this lives in ~/.claude/. None of it touches company code.
Honest assessment
Roughly 10 of my own tickets have gone through the full pipeline end to end. Separately, I have run dd-reviewer against more than 30 external open-source PRs as a learning loop. Running the 13 checks against code I did not write shows whether the gates fire on real failures or only on mine.
The gaps, plainly:
- N = 10 is a pilot. It is enough to know the gates fire correctly on the failures they were built for. It is not enough to claim a defect-rate reduction, so I am not claiming one.
- Single-user. Every calibration decision was mine. A second engineer would tune the AUTO/BATCH/MANUAL thresholds differently.
- Ceremony versus throughput. Six phases with hard gates is slower than writing a one-line CSS fix by hand, so I skip the pipeline for trivial tickets. That is deliberate, and it also means this is not a universal answer.
- The audit is a snapshot. Three months, two services, one engineer. Reviewer composition changes. Codebases evolve. I'll re-run the audit next quarter.
I am not claiming this makes Claude Code better. It makes my use of Claude Code catch the failures my reviewers have flagged, reliably. That is a smaller claim, and it is the one I am willing to defend.
The methodology
Forget my pipeline. The transferable thing is the method.
- Pull your last 90 days of PR review comments. All of them. Not just the BLOCKERs.
- Tag the underlying failure rather than the surface comment. "You forgot
@Transactional" and "this isn't atomic across the batch" describe the same failure. - Count. Anything with three or more instances is a pattern. Anything with one is a preference.
- Cluster into intervention layers. Where in the development arc does each pattern first become catchable?
- Build a gate per layer, not per pattern. Sixteen gates is unworkable. Four or five is operable.
- Make the gates non-skippable. A gate you can skip when you're tired is not a gate.
Your output will not look like mine. Your reviewers catch different things and your stack has different traps. That is exactly why it works: a pipeline built on someone else's review history is just another generic tool.
Audit your own review history. Build gates against what you actually ship wrong. The model will keep getting better on its own. The process will not.