ADR 0012: Audit architecture
A keyed hash chain in PostgreSQL, append-only in practice, with redaction applied on ingest rather than on output.
#Context
A security product's audit trail has to support a claim stronger than "here are some logs". Specifically: this record has not been altered since it was written, and the alteration would be detectable.
It also has to be queryable, retained, exportable as evidence, and free of secrets — including secrets nobody thought about when the log line was written.
#Decision
A keyed hash chain in PostgreSQL.
texth(n) = HMAC(VEEFORE_AUDIT_CHAIN_SECRET, h(n-1) ‖ content(n))
Each event's hash covers its own content and the previous event's hash. Altering one event invalidates every event after it, and verification names the first divergent position rather than reporting a boolean.
Append-only in practice. Nothing in the application updates or deletes an
audit row, outside the retention process — which requires
audit:configure_retention, is step-up gated, and writes a CRITICAL audit
event.
Redaction applied on ingest. Credential material, private keys, session
tokens and tool-declared sensitive_parameters are stripped before the record
is constructed.
Exports are watermarked and hashed, require a reason, and taking one is itself an audit event.
Correlation ids on everything. request_id and trace_id appear on the
API response, the action record, the audit event and every log line.
Four severities, with CRITICAL reserved for: agent revocation, emergency
controls, break-glass, reducing enforcement, changing retention.
#Alternatives
Plain append-only logging, no chain. Simpler and faster. Supports "here are some logs" and not "this has not been altered". For a product whose output is authorization decisions, the stronger claim is the point.
Unkeyed hash chain. Detects accidental corruption. Anyone who can write to the table can recompute a plain hash chain consistently, so it detects mistakes and not tampering. The key is what makes the chain require a second compromise.
A Merkle tree with periodic published roots. Stronger: publishing a root externally means even a full consistent rewrite is detectable. It is also substantially more machinery, and it requires an external publication channel whose availability becomes part of the audit story.
The practical gap is closed more cheaply by shipping events to an append-only sink, which is recommended rather than built.
Write audit to a separate append-only store rather than PostgreSQL. Stronger isolation between the operational database and the trail. Also a second datastore to operate, a second thing to back up and restore consistently, and a window where a decision is recorded and its audit event is not.
PostgreSQL in the same transaction means the decision and its record commit together.
Blockchain or an external notary. Removes the need to trust the operator. Adds an external dependency on the hot path, an availability coupling, and a cost per event. Out of proportion to the threat, and explicitly out of scope.
Redact on output rather than on ingest. More flexible — the raw value is available if you later need it. It is also strictly weaker: an output filter protects only the paths it covers, and every new log statement is a new path. It means the secret is in the database, and log shipping, retention and backups all become places it can leak.
Rejected. There should be no code path where a secret is written and later filtered.
#Consequences
VEEFORE_AUDIT_CHAIN_SECRET must live outside the database. If an attacker
has both database write access and the chain secret, the chain can be rewritten
consistently and verification passes. Keeping the secret in a secret manager is
what makes tampering require two separate compromises.
The chain proves integrity, not completeness. If events were never written — because the audit pipeline was down — the chain over what was written is perfectly valid. Coverage gaps and chain breaks are different failures, and the distinction has to be explained. → Audit integrity
AUDIT_SERVICE_UNAVAILABLE is the most serious dependency failure. The
other 503s mean work is not happening, which is recoverable. Audit failure
means evidence is being lost now, and it does not come back on recovery.
Alert on veefore_audit_pipeline_errors_total with zero tolerance — not a
threshold.
Verification has to be scheduled, not reactive. A check run only after an incident tells you nothing about when the problem started. Run daily, so a failure has a date and the date narrows the investigation.
Rotating the chain secret breaks verification of the existing chain. Export and archive first, record the rotation point, verify forward from it. A project.
Two tables grow without bound. actions and audit_events, roughly
proportional to protected action volume. Retention is the answer and shortening
it destroys evidence, which is why it is high-impact and CRITICAL-audited.
A masked field is masked in audit too. A MASK policy decision redacts the
value in the forwarded request and in the trail — a direct consequence of
redacting on ingest rather than on output. There is no version of the record
that contains the unmasked value.