Testing strategy

Twenty-three gates, seven suites, five smoke tests and two audits — and what each one actually proves.

bash
npm run verify

One command, 23 stages, in the order a reviewer would run them: static checks first because they are cheap, then the audits that enforce the product's hard boundaries, then the test suites, then smoke tests that drive real processes over real HTTP.

Skipped stages are named, not hidden

Optional stages — the Python SDK, the console, the console smoke test — are skipped with an explicit reason rather than silently omitted.

A green run that quietly dropped four stages is worse than a red one, because it tells you something false. If the console smoke test says the stack is not running, that is the output you need to see.

#The stages

	Stage	Proves
1	Typecheck	Every package compiles
2	Lint	Zero warnings tolerated
3	Format check	A diff here means someone skipped the hook
4	Scope audit	No wallet, marketplace, payment, settlement, ledger or contract code exists
5	Quality audit	No TODOs for required features, mock data, hardcoded metrics or auth bypasses
6	Migration check	Every migration applies cleanly to an empty database
7	Unit tests	Pure logic
8	Integration tests	Services against a real database
9	Contract tests	The HTTP surface matches the published contract
10	Policy tests	The evaluator is deterministic and total
11	Security tests	Every attack case from the threat model is refused
12	End-to-end tests	The documented journeys complete
13	Chaos tests	Dependency loss fails closed; recovery restores enforcement
14	Python SDK tests	A first-class client, verified too
15	Control plane smoke	Signup through enforcement, real database
16	Gateway smoke	Agent auth, replay refusal, enforcement, revocation
17	JS SDK smoke	The published surface over real HTTP, including fail-closed
18	Python SDK smoke	Cross-language signing agrees with the server
19	CLI smoke	Every command against real processes, including secret hygiene
20	Infrastructure validation	Container, Kubernetes, Helm, Terraform are well formed
21	Console tests	Error, empty and permission-denied stay distinguishable
22	Console build	The browser app typechecks and bundles
23	Console smoke	Every console page against a running stack, through the browser origin

bash
npm run verify -- --bail           # stop at the first failure
npm run verify -- --only=policy    # one stage
VERIFY_QUIET=1 npm run verify      # capture output, print only on failure

#The two audits

These are the unusual ones, and the reason they exist is that a principle without a gate is a preference.

#Scope audit

bash
npm run scope:audit

Fails the build if an out-of-scope concept appears as an implementation. Two different checks, because keyword matching alone is easy to evade:

Naming. Table names, service class names, route paths, identifiers, nav labels. A WalletService, a /v1/payments route, a CREATE TABLE settlements, a seller_id column.

Comments and prose that state the boundary are allowed, which is how the scope page can exist. A negated mention is documentation, not an implementation.

Shape. A wallet could be written without the word "wallet", but not without somewhere to put a balance. So the audit also asserts no migration declares a balance column, and that any table with an amount column is on an explicit allow-list of policy and decision tables.

An amount column on a table that is not a policy or decision table fails the build with that table named.

#Quality audit

bash
npm run quality:audit

Fails on the markers of unfinished or faked work. The emphasis is security-relevant fakery — a return true inside an authorization function is not a style problem, it is a vulnerability.

Rule	Catches
`todo-for-required-feature`	`TODO`, `FIXME`, "not implemented"
`placeholder-ui`	"coming soon", `alert('…')`, lorem ipsum, prose placeholders
`mock-data-in-production`	`mockX`, `fakeX`, `MOCK_`, `sampleData`, "hardcoded"
`authorization-bypass`	`function can…() { return true }`, `skipAuthorization`, `ALLOW_ALL`, `if (true) return`
`fail-open`	`catch { return true }`, "fail open"
`hardcoded-metric`	A dashboard number that is a literal
`secret-in-source`	Private key blocks, high-entropy secret assignments, provider key formats
`console-logging`	Server code bypassing the structured logger (warning)
`empty-catch`	Silently swallowed errors (warning)

The rules have carefully scoped exemptions, each with a reason in the source: test code legitimately contains stubs; the CLI, scripts, process entry points, the browser console and the examples are places where stdout is the interface; a policy condition's value is a threshold rather than a fabricated metric.

Exemptions with reasons, not blanket suppressions

A rule that fires on legitimate code trains engineers to ignore it, which is worse than not having the rule. Each exemption in quality-audit.ts carries a comment explaining the legitimate case it allows.

#The suites

#`tests/unit`

Pure logic with no I/O. Includes the database driver's result normalisation — the multi-statement array case that once left a schema applied but unrecorded — and the CLI's .env.local writer, which once stacked a header on every re-run because the filter removed managed keys and comment lines have no =.

#`tests/integration`

Services against a real database. The job queue's claim path, dead-lettering and overlapping-tick guard. The credential broker's sealing, leasing and the material-key validation.

#`tests/contract`

50 cases asserting the HTTP surface matches the published contract: status codes, envelope shapes, error codes, required headers. These are what stop an accidental breaking change.

#`tests/policy`

31 cases on the evaluator: every operator against every input type, precedence, effect merging, the default decision, and — importantly — that no operator ever throws. Totality is a property you have to test for, because the failure mode is an exception nobody specified.

#`tests/security`

49 cases, one per attacker goal in the threat model. Cross-tenant access through every surface that accepts an id, delegation escalation, replay, stale generations, credential exposure, step-up bypass.

A threat model without tests is a wish list

Every row in the threat model has a case here. That is what lets the security pages state behaviour as fact rather than as intent.

#`tests/e2e`

29 cases walking the documented journeys: signup through first protected action, OBSERVE to ENFORCE, the approval loop, revocation and its cascade.

#`tests/chaos`

13 cases. Dependency loss must fail closed, and recovery must restore enforcement. This is the suite behind Fail closed — without it, that page would be a claim.

bash
npm run test:chaos
npm run test:dr      # the disaster recovery subset

#The smoke tests

Real processes, real HTTP, real decisions. Not mocked anything.

	Covers
`smoke.ts`	Signup through enforcement, in-process, against a real database
`smoke-gateway.ts`	Agent authentication, replay refusal, enforcement, revocation
`smoke-sdk.ts`	The published JS SDK surface, including fail-closed behaviour
`smoke-python-sdk.ts`	Cross-language signing and canonicalisation agree with the server
`smoke-cli.ts`	Every CLI command, including that no command prints a secret
`smoke-console.ts`	81 checks across every console page, through the browser origin

Redirect smoke output to a file

These write with console.log while the services log through pino. The two interleave badly on a terminal:

bash
npm run smoke:gateway > /tmp/smoke.log 2>&1; grep -v '"level"' /tmp/smoke.log

smoke-console.ts drives a running stack, so it cannot start anything itself. It is skipped with a reason when the stack is not up — and it is the only stage that proves the proxy, the session cookie and the page-to-endpoint wiring all line up.

#Console tests

Eighteen cases. The one worth calling out asserts that empty, error and permission-denied remain three distinguishable states.

"No agents yet", "we could not load agents" and "you cannot see agents" must look different. Conflating them sends people debugging the wrong thing, and it is the kind of regression that is easy to introduce and invisible in review.

#Test isolation

scripts/lib/isolated-env.ts pins a test run to its own database and clears ambient VEEFORE_* variables — twice, before and after loading .env.

Ambient variables broke three suites

Python's "missing config raises" test saw real credentials in the environment and did not raise. SDK smoke tests authenticated as the wrong agent. Shell sessions persist variables between commands, and a stale VEEFORE_AGENT_ID makes everything subtly wrong rather than obviously broken.

The helper also verifies that pinning took effect rather than assuming it. Isolation you have not verified is isolation you do not have — which is how the smoke tests once wrote five organizations into the shared database.

#Docs are in the gate too

bash
npm run docs:check

Fails on a broken internal link, an anchor pointing at a heading that does not exist, a page missing from the navigation, a page in the navigation with no file, an API group added to the catalogue without an explanation, or an error code with no documented status group.

The last two matter most: they mean a new API surface cannot ship undocumented, and a new error code cannot ship unexplained.

Gate your pipeline

Running these in CI.

Threat model

What tests/security asserts.

Fail closed

What tests/chaos asserts.

Repository layout

Where the suites live.

Testing strategy

#The stages

#The two audits

#Scope audit

#Quality audit

#The suites

#tests/unit

#tests/integration

#tests/contract

#tests/policy

#tests/security

#tests/e2e

#tests/chaos