klappy.dev

The StylebookLedger

PR #177 Validation Review

Audit Gates Are Managed Agents (klappy.dev)

SummaryWhat Was Reviewed and Found

Mode: validation. Reviewer: fresh session, no creation context for the PR (legitimate context break per klappy://canon/principles/verification-requires-fresh-context). Scope: single file change at canon/constraints/audit-gates-are-managed-agents.md, head 00c67009, base main.

The PR proposes to canonize a prescriptive rule that has been implicit in klappy://canon/methods/governance-validation-via-agents — that mechanical pattern matchers cannot serve as merge gates for governance audits when the check requires LLM-grade judgment. The motivation is a concrete downstream incident (an AMS PR that shipped exactly the forbidden architecture) plus a prior in-repo incident at canon/constraints/canon-integration-audit §Summary Gap 3.

The review ran the full gauntlet: time-anchor, bootstrap, version, URI resolution, audit, tier-1 challenge, factual cross-check on the cited prior incident, frontmatter conformance, writing-canon checklist, and CI signal interpretation.


Observations (O)

  • CI signals on 00c67009: Cursor Bugbot completed/success; Canon Quality oddkit_audit on writings/ scope passed (39 files, 0 findings); "Reference integrity audit" check shows completed/failure BUT investigation of the job step list shows the failure is in the Sticky comment — audit results step (GitHub Actions sticky-comment infrastructure), not the audit itself. The audit step succeeded.
  • Default audit scope mismatch: oddkit_audit defaults to writings/, so the new file at canon/constraints/ is not within scope of the CI audit. Explicit audit on the file path returned 0 findings (audit reads from main, where the new file does not yet exist — the audit is inherently blind to PR-only files).
  • Cited URI resolution: All 9 klappy:// URIs cited in the doc resolve cleanly via oddkit_resolve:
    • klappy://canon/methods/governance-validation-via-agents ✓
    • klappy://canon/methods/reference-integrity-audit ✓
    • klappy://canon/constraints/canon-integration-audit ✓
    • klappy://canon/principles/vodka-architecture ✓
    • klappy://canon/values/axioms ✓
    • klappy://canon/constraints/borrow-evaluation-before-implementation ✓
    • klappy://canon/constraints/frontmatter-validation-before-merge ✓
    • klappy://canon/meta/writing-canon ✓
    • klappy://canon/constraints/no-irreversible-action-without-epistemic-justification (frontmatter complements only) ✓
  • Tier-1 challenge: oddkit_challenge mode canon-tier-1 returned CHALLENGED with block_until_addressed: false. Matched challenge types: pattern-coinage, principle-extraction. Five generic missing-prerequisites flags surfaced; all five are addressed in the doc body (see Learnings).
  • Factual cross-check: The doc claims at L129 that canon-integration-audit §Summary Gap 3 records a same-session Python frontmatter validator passing all four PR files as compliant while an independent Managed Agent caught a derives_from shape violation. Direct retrieval of the cited section confirms this characterization is faithful — the actual canon describes "field presence and enum values" being checked while the schema's str vs list requirement was missed.
  • Frontmatter conformance: All native-YAML rules per klappy://canon/meta/frontmatter-schema honored — tier: 1 (unquoted int), date: 2026-05-07 (unquoted ISO date), epoch: E0008.5 (unquoted simple identifier), derives_from/complements/governs as quoted comma-separated strings (the precise lesson Gap 3 motivated — author practiced what they preach).
  • Writing Canon checklist: Title names concept + stance ("...Are Managed Agents, Not Pattern Matchers"); blockquote with compressed argument present; Summary — Pattern Matchers Cannot Replace Judgment is a descriptive subtitle; the 14 section headers tell the story when scanned (binding conditions → what gate must be → what mechanical may do → what's forbidden → what's not → migration → worked anti-pattern → failure modes → retraction → prior art → reversibility → relationships → see also); Axiom 4 explicitly cited at L182.
  • PR shape: No reviews; one bot comment from Canon Quality; mergeable=True with state unstable (the unstable state is from the false-failure on Reference integrity audit's sticky-comment step).
  • Release-validation-gate Rule 2 applicability: The rule binds for PRs touching orchestrate.ts / matchers / governance reads / envelope / action behavior in klappy/oddkit. This PR is canon-only in klappy/klappy.dev and touches none of those code surfaces. Rule 2 does not bind.

Learnings (L)

  • The "Reference integrity audit" CI check produces a false-failure signal when the audit succeeds but the sticky-comment posting step fails. Reviewers must read the step list to disambiguate audit-failed from comment-failed. Candidate canon improvement: the workflow could fail-soft on the sticky-comment step or surface the audit verdict in the workflow step summary even when commenting fails. (Captured for follow-up; not a blocker for this PR.)
  • Default audit scope (writings/) does not cover canon/ additions. For canon-doc PRs, automated audit signal is informational at best — manual URI resolution per cited reference is the actual integrity check. The audit-gates-are-managed-agents constraint itself reinforces this: pattern-matching scope rules cannot replace LLM-grade integration checks.
  • The doc internalizes its own thesis. The Tier-1 challenge engine flagged five generic prereqs; the doc had already addressed all five inline before the challenge ran. Pattern-coinage prereqs (prior-art comparison) addressed in §Prior Art and Why a New Name with three explicit comparisons (Fowler smart-endpoints/dumb-pipes; OPA policy-as-code vs policy-as-data; linters-as-advisory vs type-checkers-as-gates). Principle-extraction prereqs (multi-case anchoring) addressed via two recorded cases (the AMS PR and canon-integration-audit Gap 3) with explicit framing of the second incident as "the same lesson surfacing a second time at higher stakes." Retraction conditions and reversibility named in dedicated sections.
  • Generic missing-prereq flags from oddkit_challenge are not the same as actual gaps. The challenge engine returns the prereq checklist for the matched claim type without reading the doc body. Reviewer must cross-check each prereq against the doc — most well-prepared tier-1 docs will pre-address them.

Decisions (D)

  • Verdict: ACCEPT for merge. All blocking checks pass; CI false-failure on sticky-comment step is non-blocking; release-validation-gate Rule 1 (Bugbot completed) satisfied; Rule 2 (MA dispatch for code surfaces) does not apply; Rule 3 (canon outranks session) honored.
  • No iteration findings. No edits required to merge.
  • Follow-up after merge as already scoped in PR body §Follow-up after merge: comment on the AMS PR with supersession path; optional AMS-side adoption pointer with project-specific drift surfaces.

Constraints honored (C)

  • klappy://canon/constraints/release-validation-gate — Rule 1 satisfied (Bugbot completed/success); Rule 2 inapplicable (no code surfaces touched); Rule 3 honored (no handoff or session ledger overrode canon during this review).
  • klappy://canon/constraints/frontmatter-validation-before-merge — frontmatter passes the native-YAML conventions and the known-crash-pattern table.
  • klappy://canon/constraints/canon-integration-audit — Gap 1 (concept has canon home: this is the home), Gap 2 (adjacent canon cited: 5 in derives_from, 2 in complements, 9 in body), Gap 3 (validator-completeness exercised by independent fresh-session review).
  • klappy://canon/principles/verification-requires-fresh-context — review performed in a fresh session with no creation context for the PR.

Handoff (H)

  • To operator: Squash-merge PR #177. Then proceed with the follow-up actions in the PR body — comment on the AMS PR with the supersession path; optionally land an AMS-side adoption pointer with project-specific drift surfaces.
  • To future reviewers: When CI shows "Reference integrity audit" failure on a canon-doc PR with no other red flags, check the job step list before treating it as a real audit failure — the sticky-comment posting step has been brittle.
  • Candidate canon edit (deferred, not blocking this PR): Either widen the default oddkit_audit scope to include canon/, or document the scope limitation as a known reviewer-must-do-manual-check item in the canon-integration-audit doc.

Markdown · RSS · built from klappy/klappy.dev@b750d7d