Verify the patch, challenge the story
Find counterexamples, debug from evidence, and protect the boundaries an agent can cross.
BY THE END, YOU CAN
- Design checks that challenge the implementation’s assumptions
- Debug an AI-assisted change from a reproducible failure
- Review dependencies, sensitive data, and agent permissions
Diagram description: Patch → Your own test; Patch → Reproduce the bug; Patch → Read the diff; Patch → Check dependencies; Reproduce the bug → Merge; Read the diff → Shrink the failure
Build an independent reason to trust the result
A generated function and generated tests can share the same misunderstanding.
Derive expected results from the reviewed contract first. For Relay’s retry delay, check zero, the last value below the cap, the first capped value, huge attempts, negative input, and True. Python booleans are integers for isinstance checks, so the rejection policy needs deliberate code.
Inspect branches and side effects beyond the happy path. Run the documented checks and distinguish “passed”, “failed”, and “not run”. A test result is evidence for the cases and environment exercised, not proof that all possible behavior is correct.
Reduce the failure before requesting another patch
Suppose an assistant computes 2 ** attempt before applying the cap.
A huge attempt value needlessly creates a giant integer. Reproduce that behavior with a controlled case, state the hypothesis, and choose a change that predicts a measurable improvement. Test the correction against both the new case and the original contract.
Avoid stacking unrelated generated fixes until the symptom disappears. Keep a small diff and explain the causal chain: input, faulty assumption, effect, correction. Review any test deletion or weakened assertion with particular care.
Review the execution and supply boundaries
Check unfamiliar dependencies against their official project documentation and package metadata before installation.
Confirm purpose, version, provenance, license obligations, and maintenance fit; a plausible package name is insufficient. Inspect installation scripts and granted capabilities. Use isolated development environments, minimal credentials, and explicit authorization for production effects.
Do not paste private keys or customer records into a debugging chat. External content can attempt prompt injection, and an additional AI reviewer can also miss problems. Combine human review, appropriate automated checks, and access controls; ask a specialist to review risks beyond your expertise.
COMMON FAILURE MODE
Using the same assistant’s “looks correct” response as the only review, or accepting passing tests after the expected behavior has been weakened.
TRY IT YOURSELF
Review a retry implementation that uses isinstance(attempt, int), computes 2 ** attempt unconditionally, and catches every exception. Name three risks, then propose a minimal contract-based test for each.
Make the call in the laboratory.
The two-question checkpoint, your notes, and your saved progress are in the laboratory. Free, with no account.
Open in the laboratory ↗CONNECT IT TO RELAY
Review the AI delivery submission patch. Challenge duplicate keys, changed payloads, invalid input before reservation, and mutation of the original payload. Record a counterexample, minimal correction, actual test results, and the remaining concurrency limitation.