--- title: Testing strategy description: Twenty-three gates, seven suites, five smoke tests and two audits — and what each one actually proves. --- ```bash npm run verify ``` One command, 23 stages, in the order a reviewer would run them: static checks first because they are cheap, then the audits that enforce the product's hard boundaries, then the test suites, then smoke tests that drive real processes over real HTTP. :::check Skipped stages are named, not hidden Optional stages — the Python SDK, the console, the console smoke test — are **skipped with an explicit reason** rather than silently omitted. A green run that quietly dropped four stages is worse than a red one, because it tells you something false. If the console smoke test says the stack is not running, that is the output you need to see. ::: ## The stages | | Stage | Proves | | --- | --- | --- | | 1 | Typecheck | Every package compiles | | 2 | Lint | Zero warnings tolerated | | 3 | Format check | A diff here means someone skipped the hook | | 4 | **Scope audit** | No wallet, marketplace, payment, settlement, ledger or contract code exists | | 5 | **Quality audit** | No TODOs for required features, mock data, hardcoded metrics or auth bypasses | | 6 | Migration check | Every migration applies cleanly to an empty database | | 7 | Unit tests | Pure logic | | 8 | Integration tests | Services against a real database | | 9 | Contract tests | The HTTP surface matches the published contract | | 10 | Policy tests | The evaluator is deterministic and total | | 11 | Security tests | Every attack case from the threat model is refused | | 12 | End-to-end tests | The documented journeys complete | | 13 | Chaos tests | Dependency loss fails closed; recovery restores enforcement | | 14 | Python SDK tests | A first-class client, verified too | | 15 | Control plane smoke | Signup through enforcement, real database | | 16 | Gateway smoke | Agent auth, replay refusal, enforcement, revocation | | 17 | JS SDK smoke | The published surface over real HTTP, including fail-closed | | 18 | Python SDK smoke | Cross-language signing agrees with the server | | 19 | CLI smoke | Every command against real processes, including secret hygiene | | 20 | Infrastructure validation | Container, Kubernetes, Helm, Terraform are well formed | | 21 | Console tests | Error, empty and permission-denied stay distinguishable | | 22 | Console build | The browser app typechecks and bundles | | 23 | Console smoke | Every console page against a running stack, through the browser origin | ```bash npm run verify -- --bail # stop at the first failure npm run verify -- --only=policy # one stage VERIFY_QUIET=1 npm run verify # capture output, print only on failure ``` ## The two audits These are the unusual ones, and the reason they exist is that a principle without a gate is a preference. ### Scope audit ```bash npm run scope:audit ``` Fails the build if an out-of-scope concept appears as an implementation. Two different checks, because keyword matching alone is easy to evade: **Naming.** Table names, service class names, route paths, identifiers, nav labels. A `WalletService`, a `/v1/payments` route, a `CREATE TABLE settlements`, a `seller_id` column. Comments and prose that *state* the boundary are allowed, which is how [the scope page](/scope) can exist. A negated mention is documentation, not an implementation. **Shape.** A wallet could be written without the word "wallet", but not without somewhere to put a balance. So the audit also asserts no migration declares a `balance` column, and that any table with an `amount` column is on an explicit allow-list of policy and decision tables. An amount column on a table that is not a policy or decision table fails the build **with that table named**. ### Quality audit ```bash npm run quality:audit ``` Fails on the markers of unfinished or faked work. The emphasis is security-relevant fakery — a `return true` inside an authorization function is not a style problem, it is a vulnerability. | Rule | Catches | | --- | --- | | `todo-for-required-feature` | `TODO`, `FIXME`, "not implemented" | | `placeholder-ui` | "coming soon", `alert('…')`, lorem ipsum, prose placeholders | | `mock-data-in-production` | `mockX`, `fakeX`, `MOCK_`, `sampleData`, "hardcoded" | | `authorization-bypass` | `function can…() { return true }`, `skipAuthorization`, `ALLOW_ALL`, `if (true) return` | | `fail-open` | `catch { return true }`, "fail open" | | `hardcoded-metric` | A dashboard number that is a literal | | `secret-in-source` | Private key blocks, high-entropy secret assignments, provider key formats | | `console-logging` | Server code bypassing the structured logger (warning) | | `empty-catch` | Silently swallowed errors (warning) | The rules have carefully scoped exemptions, each with a reason in the source: test code legitimately contains stubs; the CLI, scripts, process entry points, the browser console and the examples are places where **stdout is the interface**; a policy condition's `value` is a threshold rather than a fabricated metric. :::note Exemptions with reasons, not blanket suppressions A rule that fires on legitimate code trains engineers to ignore it, which is worse than not having the rule. Each exemption in `quality-audit.ts` carries a comment explaining the legitimate case it allows. ::: ## The suites ### `tests/unit` Pure logic with no I/O. Includes the database driver's result normalisation — the multi-statement array case that once left a schema applied but unrecorded — and the CLI's `.env.local` writer, which once stacked a header on every re-run because the filter removed managed *keys* and comment lines have no `=`. ### `tests/integration` Services against a real database. The job queue's claim path, dead-lettering and overlapping-tick guard. The credential broker's sealing, leasing and the material-key validation. ### `tests/contract` 50 cases asserting the HTTP surface matches the published contract: status codes, envelope shapes, error codes, required headers. These are what stop an accidental breaking change. ### `tests/policy` 31 cases on the evaluator: every operator against every input type, precedence, effect merging, the default decision, and — importantly — that no operator ever throws. Totality is a property you have to test for, because the failure mode is an exception nobody specified. ### `tests/security` 49 cases, one per attacker goal in [the threat model](/secure/threat-model). Cross-tenant access through every surface that accepts an id, delegation escalation, replay, stale generations, credential exposure, step-up bypass. :::security A threat model without tests is a wish list Every row in the threat model has a case here. That is what lets the security pages state behaviour as fact rather than as intent. ::: ### `tests/e2e` 29 cases walking the documented journeys: signup through first protected action, OBSERVE to ENFORCE, the approval loop, revocation and its cascade. ### `tests/chaos` 13 cases. Dependency loss must fail closed, and recovery must restore enforcement. This is the suite behind [Fail closed](/secure/fail-closed) — without it, that page would be a claim. ```bash npm run test:chaos npm run test:dr # the disaster recovery subset ``` ## The smoke tests Real processes, real HTTP, real decisions. Not mocked anything. | | Covers | | --- | --- | | `smoke.ts` | Signup through enforcement, in-process, against a real database | | `smoke-gateway.ts` | Agent authentication, replay refusal, enforcement, revocation | | `smoke-sdk.ts` | The published JS SDK surface, including fail-closed behaviour | | `smoke-python-sdk.ts` | Cross-language signing and canonicalisation agree with the server | | `smoke-cli.ts` | Every CLI command, including that no command prints a secret | | `smoke-console.ts` | 81 checks across every console page, through the browser origin | :::note Redirect smoke output to a file These write with `console.log` while the services log through pino. The two interleave badly on a terminal: ```bash npm run smoke:gateway > /tmp/smoke.log 2>&1; grep -v '"level"' /tmp/smoke.log ``` ::: `smoke-console.ts` drives a **running** stack, so it cannot start anything itself. It is skipped with a reason when the stack is not up — and it is the only stage that proves the proxy, the session cookie and the page-to-endpoint wiring all line up. ## Console tests Eighteen cases. The one worth calling out asserts that **empty, error and permission-denied remain three distinguishable states**. "No agents yet", "we could not load agents" and "you cannot see agents" must look different. Conflating them sends people debugging the wrong thing, and it is the kind of regression that is easy to introduce and invisible in review. ## Test isolation `scripts/lib/isolated-env.ts` pins a test run to its own database and clears ambient `VEEFORE_*` variables — **twice**, before and after loading `.env`. :::warning Ambient variables broke three suites Python's "missing config raises" test saw real credentials in the environment and did not raise. SDK smoke tests authenticated as the wrong agent. Shell sessions persist variables between commands, and a stale `VEEFORE_AGENT_ID` makes everything subtly wrong rather than obviously broken. The helper also **verifies that pinning took effect** rather than assuming it. Isolation you have not verified is isolation you do not have — which is how the smoke tests once wrote five organizations into the shared database. ::: ## Docs are in the gate too ```bash npm run docs:check ``` Fails on a broken internal link, an anchor pointing at a heading that does not exist, a page missing from the navigation, a page in the navigation with no file, an API group added to the catalogue without an explanation, or an error code with no documented status group. The last two matter most: they mean a new API surface cannot ship undocumented, and a new error code cannot ship unexplained. ## Related :::cards :::card Gate your pipeline | /build/ci-gate Running these in CI. ::: :::card Threat model | /secure/threat-model What `tests/security` asserts. ::: :::card Fail closed | /secure/fail-closed What `tests/chaos` asserts. ::: :::card Repository layout | /architecture/repository-layout Where the suites live. ::: :::