Verify PR Descriptions with an LLM

Part of the DeepSeek Harness PR Review guide series

Most PR descriptions are approximations. Some are outdated by the time the PR is ready for review, others describe intent that never made it into the code, and a few are simply wrong. The fix is not to trust the description — it's to verify it claim-by-claim against the actual code.

The claim-splitting approach

A PR description is a list of assertions: "Adds X", "Fixes Y", "Refactors Z". Each assertion is a claim that can be checked by reading code. The LLM splits the description into these verifiable claims first:

[
  {"id": "C1", "text": "Adds Google sign-in via /auth/mobile",
   "category": "feature", "files": ["auth_api.dart"]},
  {"id": "C2", "text": "Fixes logo layout bug (full width)",
   "category": "bugfix", "files": ["sign_in_screen.dart"]}
]

Then each claim is verified

A DeepSeek Harness agent reads the actual code in the PR's workspace and returns one of four verdicts per claim, with evidence:

Matches — the code does what the description says
Partial — some of the claim is true, but not all
Does not match — the description and the code disagree
Unverified — cannot be confirmed (e.g. the behavior lives in an external service)

Every verdict carries file:line evidence, so a reviewer can jump straight to the code and judge for themselves.

A real-world example

A PR claims: "The logo layout bug is fixed: the logo no longer renders at full width." The agent reads the UI file and finds the layout was indeed changed — but the base branch never had the bug the description references. The verdict: Partial, with a note explaining the discrepancy. That single finding saves a reviewer from approving a description that quietly describes work that was never done.

What happens to unverified claims

When a claim cannot be verified from the repo alone (backend behavior, external service, missing context), the agent marks it Unverified and adds a ≤20-word question for a human. The tool never guesses — uncertain findings always surface as questions, never as silent conclusions.

Why claim-level verification beats a summary

A summary review produces a vague "looks good to me". Claim-level verification produces a checklist: 9 claims verified against code, 1 partial, 1 unverified — each with evidence. That checklist is what a reviewer actually needs to make a decision, and it is exactly what gets posted back on the PR.