EPISTEMOLOGY & DEBUGGING
Epistemic Debugging: 12 Canonical Falsification Probes to Force AI Agents into Ground Reality
Applying Sir Karl Popper's falsificationism to autonomous coding agents. Replacing conversational apologies with 12 canonical ground-truth probes.
The Core Defect of LLM Debugging: Superstitious Patches
When an autonomous AI agent encounters a failing test, it generates superstitious patches. It wraps database queries in unnecessary try/catches, converts synchronous helpers to async without awaiting them, and renames variables randomly.
Why? Because LLMs optimize for token plausibility, not epistemic truth. To an autoregressive model, a plausible-sounding explanation that is completely false looks nearly identical to reality.
No theory can be proven by accumulating confirmatory examples; it can only be corroborated by failing to be falsified through rigorous experiments designed specifically to break it. In ctrl-alt-pray, when an agent fails twice, it is forbidden from writing speculative code. It must execute a falsification probe.
Deep Dive: 4 Crucial Falsification Probes
1. Rite 01: wrong-altar (The Stale Build Detector)
The Symptom: The agent modifies source code in src/, runs tests, and receives the exact same error with identical stack trace line numbers across multiple turns.
The Falsification Probe: Inject a unique runtime marker into the file:
console.log("RUNNING_CHECK_" + crypto.randomUUID());Run the test. If this marker does not appear in stdout, the hypothesis "I am testing my modified code" is falsified in 1 step. The agent is forced to inspect build artifacts (e.g. dist/, bundler caches) rather than mutating source logic.
2. Rite 02: check-the-check (The Poison Chalice)
The Symptom: A critical production bug is reported, but running npm test reports green (PASS).
The Falsification Probe: Inject an intentionally failing assertion directly into the business logic:
throw new Error("POISON_ASSERTION_VERIFICATION");If tests still pass, the test suite is invalid (swallowing errors or testing wrong mocks). The agent must repair the test runner before touching production code.
3. Rite 04: api-ground-truth (The Hallucination Squelcher)
The Symptom: Agent encounters TypeError: client.fetchUser is not a function and starts guessing method names (getUser, queryUser).
The Falsification Probe: Forbid guessing. Execute a 1-line runtime reflection probe:
node -e "import('./client.js').then(m => console.log(Object.keys(m.default || m)))"Inspect true runtime exports before writing a single line of invocation code.
4. Rite 08: minimal-counterexample (Payload Bisection)
The Symptom: A 200-field JSON schema fails validation, and the agent speculates across 50 fields.
The Falsification Probe: Payload Bisection. Halve the payload repeatedly until the atomic failure invariant is isolated in ≤5 steps.
Clean-Context Resurrection (ctrl-alt-pray resurrect)
When an autonomous session has degraded over 25 turns, continuing in that context poisons model reasoning. The conversation history is saturated with dead stack traces and polite apologies.
Running ctrl-alt-pray resurrect [session_id] condenses the entire ledger into a compact, pristine Resurrection Packet:
- The Inviolable Goal: High-level functional requirement.
- Verified Ground Truth: Facts established by command exit codes.
- Ruled-Out Hypotheses (FORBIDDEN TO RECYCLE): Defeated approaches that must never be repeated.
- Active Bounded Probe: Exactly one testable experiment.
The 12 Canonical Recovery Recipes
| # | Strategy | Trigger Symptom | Prescribed Probe |
|---|---|---|---|
| 01 | wrong-altar |
Edits produce zero output change | Inject unique runtime marker (RUNNING_CHECK_UUID) |
| 02 | check-the-check |
Tests stay green despite known bug | Inject deliberate failing assertion (assert(1 === 2)) |
| 03 | ghost-terminal-breaker |
Command hangs >15s; pipe deadlock | Inspect process tree; cascade kill orphaned child PIDs |
| 04 | api-ground-truth |
TypeError: is not a function |
Run runtime reflection: node -e "console.log(Object.keys(...))" |
| 05 | clean-slate-rollback |
≥4 dirty files; messy diffs | git stash push -u; re-run baseline test |
| 06 | environment-triage |
command not found, EACCES |
Verify binary path (which), permissions (ls -la) |
| 07 | assumption-audit |
Hypothesis treated as fact | Execute single probe strictly designed to disprove it |
| 08 | minimal-counterexample |
Huge 10MB payload; noisy failure | Halve input payload repeatedly until atomic failure isolated |
| 09 | divide-and-conquer |
Multi-step pipeline failure | Log payload state exactly midway between entry and failure |
| 10 | controlled-substitution |
Plausible causes separated by good input | Hold variables constant; swap suspect component with twin |
| 11 | boundary-check |
Subsystem boundary ambiguous | Compare inputs and outputs across boundary before editing code |
| 12 | human-checkpoint |
Ambiguous product requirement | Formulate one concrete multiple-choice question to human |
Enforce Epistemic Rigor in Your Agents
Equip your coding agent with the 12 Canonical Rites in 10 seconds.
npx ctrl-alt-pray init