Runbook: dependency outage

Protected actions are being refused. That is the design. Here is how to confirm it and restore service.

Trigger. Agents are receiving 503s. Protected actions are refused.

This is fail-closed working, not a bug

A dependency enforcement relies on is unavailable, so protected actions are refused rather than waved through. There is no mode in which an error becomes an allow.

An authorization system that permits everything when it cannot evaluate is worse than no authorization system, because you believe you have one. This behaviour is tested in the chaos suite — npm run test:chaos asserts that dependency loss fails closed and that recovery restores enforcement.

Resist the urge to build a bypass. A code path that proceeds when Veefore cannot evaluate is a code path an attacker will find and trigger deliberately.

#Identify which dependency

The error code says which one.

Code	Dependency	Consequence
`IDENTITY_SERVICE_UNAVAILABLE`	Identity verification	No agent can authenticate
`POLICY_SERVICE_UNAVAILABLE`	Policy evaluation	No protected action can be decided
`CREDENTIAL_SERVICE_UNAVAILABLE`	Credential broker	Actions needing a credential cannot execute
`APPROVAL_SERVICE_UNAVAILABLE`	Approval service	Approval-gated actions cannot proceed
`AUDIT_SERVICE_UNAVAILABLE`	Audit pipeline	Events are not being recorded
`DEPENDENCY_UNAVAILABLE`	Something else	Generic

`AUDIT_SERVICE_UNAVAILABLE` is the most serious of these

The others mean actions are not happening, which is recoverable — the work waits.

Audit failure means evidence is being lost right now, and lost evidence does not come back when the service recovers. Prioritise it above everything else on this page.

#Check health

bash
curl -s http://localhost:4000/health/ready
curl -s http://localhost:4010/health/ready

/health/ready genuinely checks the database. If it is red, the database is the problem and everything else follows from it.

bash
curl -s http://localhost:4000/health/live

Live but not ready means the process is fine and a dependency is not.

#It is usually the database

Nine times in ten.

bash
psql "$DATABASE_URL" -c 'select 1'
psql "$DATABASE_URL" -c 'select count(*) from pg_stat_activity'

Cause	Symptom	Fix
Connection exhaustion	`remaining connection slots` errors	Raise `max_connections`, lower `DATABASE_POOL_MAX`, or add PgBouncer
Statement timeouts	`veefore_db_query_duration_ms` p99 at the timeout	Find the slow query; check for a missing index after a data growth spike
Failover in progress	Connections refused, then recovering	Wait. Both planes reconnect
Disk full	Write errors	Reclaim space. Check audit table growth
Network partition	Timeouts rather than refusals	Infrastructure

Remember DATABASE_POOL_MAX is per process instance. Three control plane instances, two gateway instances and a worker at 20 each is 120 connections. Scaling out multiplies your connection count; size PostgreSQL for the total.

The embedded driver can present as this

If DATABASE_URL is unset, processes fall back to the embedded PGlite driver — which is per-process. Each process gets a private, empty database.

The symptom is not a clean outage: logins work in one process and fail in another, an agent you just created does not exist, a policy you activated does not apply. Check DATABASE_URL is actually set in every process's environment.

#Then check Redis, if you use it

bash
redis-cli -u "$REDIS_URL" ping

Redis backs rate limiting and nonce tracking in a multi-node deployment. Losing it degrades rather than stops: each instance falls back to its own in-process store, so the effective rate limit becomes per-instance and nonce replay protection weakens.

Not an outage. Worth noticing, because it is a quiet reduction in a security property.

#What to do while it is down

Do not bypass

There is no bypass flag, and building one in your agent is the wrong fix. Your agents should be treating 503 as "retry with backoff", which they will do correctly if they follow Errors and retries.

Do not drop the enforcement mode

It would not help — the mode change itself needs the control plane and the database. And if it did work, it would mean agents executing without authorization during an outage, which is the scenario fail-closed exists to prevent.

Communicate the scope

"Protected actions are being refused" is accurate and useful. Your agents are not broken; they are correctly declining to act without authorization.

Agents doing work that does not route through Veefore are unaffected, which is usually most of what they do.

Fix the dependency

Normal operational work. Nothing Veefore-specific.

#After recovery

Confirm enforcement is back

bash
curl -s http://localhost:4000/health/ready
curl -s http://localhost:4010/health/ready

curl -X POST $CP/v1/policies/simulate \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{ "agentId": "agt_01JAV…", "tool": "refund_customer", "amount": 150000 }'

Simulate something you know should be refused. A simulation that returns the expected refusal proves evaluation is working, not merely that the process is up.

Verify the audit chain

bash
veefore audit verify

If audit writes failed, there may be a gap. The chain verification tells you whether the stored sequence is intact; a gap in coverage is a different thing from a break in the chain, and the window is what you lost.

Record the window on an incident. "Audit coverage is incomplete between 02:14 and 02:48 UTC" is a usable statement. Discovering it six months later is not.

Check the job queues

promql
veefore_job_queue_depth

Webhook fan-out, usage rollups, risk baseline recomputation and notifications all queue during an outage. They drain on their own, but a queue that is still growing means the worker is not running.

Abandoned jobs are reclaimed and dead-lettered after repeated failure, so a job that was mid-flight when the process died does not wedge the queue. A stalled queue with no worker is the failure mode that does not page anyone — monitor depth.

Expect a risk spike

Velocity and history baselines were disrupted. Agents resuming after an outage will trip VELOCITY_ANOMALY and TIME_ANOMALY for a while as they catch up on queued work.

This is correct behaviour. It will also generate approval requests. Expect it, and do not retune weights in response to a one-off.

#Reduce the chance of recurrence

Run more than one of everything. Both planes are stateless; session state and the job queue are in the database.

Alert on readiness, not just liveness. A process that is live and not ready will be in your load balancer's rotation if your probe is pointed at the wrong endpoint.

Alert on audit pipeline errors with zero tolerance. Evidence loss is not a threshold thing.

Size the connection pool for your instance count, not for one process.

Test it. npm run test:chaos asserts fail-closed behaviour and recovery. Run it as part of npm run verify; it is the suite that proves this page is accurate.

Fail closed

The principle and where it applies.

Database

Pooling, timeouts, restores.

Observability

The alerts that fire here.

Errors and retries

How agents should behave.