The CI for PR feedback. It checks whether every review comment was actually addressed (by a code change, a reply, or a resolved thread) and blocks the merge if anything is still pending.
Not a review generator (that's CodeRabbit, Copilot Review). Not a fix bot (that's Claude Code, Cursor). The verifier that closes the loop.
go install github.com/alejandroSuch/review-replay/cmd/review-replay@latest
Three layers around a PR. Generation and implementation are crowded. The one that closes the loop wasn't:
An agent or human leaves the review comments.
Code gets pushed claiming "I addressed it".
An independent check reads the thread + the code and decides, per comment.
Drop the Action into a workflow, make it a required check, and a PR can't merge while a reviewer's concern is unanswered. Re-runs on every push, so the gate clears the moment the fix lands.
# re-check whenever the PR gets new commits on: pull_request: types: [synchronize, opened, reopened] jobs: verify: runs-on: ubuntu-latest permissions: { pull-requests: read, contents: read } steps: - uses: alejandroSuch/review-replay/action@main with: mode: check # exit 1 on any pending comment api-key: ${{ secrets.OPENROUTER_API_KEY }} # reviewer: copilot # optional: gate only on Copilot's review
| Status | Comment | Evidence |
|---|---|---|
| addressedrule | extract this into a helper | thread resolved by reviewer |
| addressedllm | rename tmp to something descriptive | e4f5g6h "Renamed to pendingWriteBuffer." |
| partial | add a test for the error path | i7j8k9l happy-path test only |
| pendingrule | this should be debounced | no change since comment |
| needs-discussion | should we cache this at all? | question, not a request |
Every verdict ships with its evidence (the commit, the resolved thread, or the reply) so a human can sanity-check in one click. rule verdicts are deterministic and free; only the ambiguous ones cost an LLM call.
Same engine, no CI needed. Run it before pinging the reviewer, or before a second-pass review, straight from the shell. --post drafts the replies and posts them for you (interactive by default).
$ review-replay acme/api#482 acme/api#482 · head a7c1e3d 3 addressed · 1 partial · 1 pending · 1 needs-discussion # │ author │ status │ conf │ evidence │ comment ───┼────────┼───────────────────┼──────┼──────────┼────────────────────────── 1 │ alice │ addressed │ 1.00 │ - │ extract this into a helper 2 │ alice │ addressed │ 0.92 │ e4f5g6h │ rename `tmp` to descriptive 3 │ bob │ partial │ 0.74 │ i7j8k9l │ add a test for the error path 4 │ alice │ pending │ 0.95 │ - │ this should be debounced 5 │ carol │ needs-discussion │ 0.61 │ - │ should we cache this at all? # exit 1: 1 pending (use --reviewer copilot to gate on one reviewer)
GitHub marks a comment "outdated" when its line range shifts. That's not the same as resolved: a cosmetic refactor outdates a comment without fixing the bug; a correct fix in another file leaves it looking pending. review-replay reads intent, not line numbers.
Review threads (with resolved state), review summaries, issue comments, and the commits that came after each one, via the GitHub API.
Per comment: did the region change at HEAD? Which later commits touched the file? Any replies? Chitchat and self-reviews filtered out.
Thread resolved by its opener → addressed. No change, no replies, no commits → pending. No LLM call.
The ambiguous ones go to the LLM of your choice (OpenRouter, OpenAI, Anthropic, Gemini) with a strict rubric and a confidence score.
Status + confidence + the commit or thread that proves it + a draft reply. Table, --json, or a CI exit code.
No DB, no web app. Auth from gh / GITHUB_TOKEN. Only the diffs and comments needed for classification reach your LLM provider.
Every comment lands in one of four buckets. The call is about whether the reviewer's intent was satisfied, not whether a line moved.
| Status | When it applies |
|---|---|
| addressed | A later commit makes the change the reviewer asked for (the diff moves in that direction). Also counts if the reviewer who opened the thread resolved it, a question got an answer in the thread, or it was a non-blocking nit the reviewer pre-blessed skipping. |
| partial | Some of the blocking asks landed, not all (e.g. "add tests" → only the happy path). |
| pending | Nothing relevant changed since the comment, or the change doesn't go in the asked direction. |
| needs-discussion | A question or a judgment call with no answer yet, not a concrete request. |
GitHub gives you a few signals that look like "addressed" but mean different things. Here's what each one actually buys you:
| Signal on GitHub | What it really means | What review-replay does |
|---|---|---|
Comment went outdated | The line range shifted | Ignored. A cosmetic refactor outdates a comment without fixing it; a fix in another file leaves it looking pending. Reads intent, not line numbers. |
A reply saying "done" | A claim, not proof | Counts as addressed only if the diff moved in the asked direction, or the reply names a commit SHA it can match. |
| Thread resolved by the reviewer who opened it | The person who raised it is satisfied | addressed deterministically: a short-circuit rule, no LLM call. |
| Resolved by someone else, or never resolved | Weak or no signal: many teams never click Resolve | Treated as a hint at most. The verdict comes from the code + the thread. |
A single Go binary for local and CLI use, the composite Action for CI, and ready-made rules for the popular AI assistants.
# prebuilt binary (linux/macOS/windows × amd64/arm64): github.com/alejandroSuch/review-replay/releases/latest # or with Go: go install github.com/alejandroSuch/review-replay/cmd/review-replay@latest # auth from your gh session: export GITHUB_TOKEN="$(gh auth token)" export OPENROUTER_API_KEY=<key> # or OPENAI / ANTHROPIC / GEMINI review-replay owner/repo#42
- uses: alejandroSuch/review-replay/action@main
with:
mode: check
api-key: ${{ secrets.OPENROUTER_API_KEY }}
Mark the job a required check to block merge on pending feedback.
Ready-to-copy rules and skills, one per tool. Same verification, invoked in chat: "did I address the reviews on this PR?"