---
title: 'Runbook: dependency outage'
description: Protected actions are being refused. That is the design. Here is how to confirm it and restore service.
label: Dependency outage
---

**Trigger.** Agents are receiving `503`s. Protected actions are refused.

:::security This is fail-closed working, not a bug
A dependency enforcement relies on is unavailable, so protected actions are
**refused** rather than waved through. There is no mode in which an error
becomes an allow.

An authorization system that permits everything when it cannot evaluate is worse
than no authorization system, because you believe you have one. This behaviour
is tested in the chaos suite — `npm run test:chaos` asserts that dependency loss
fails closed and that recovery restores enforcement.

Resist the urge to build a bypass. A code path that proceeds when Veefore cannot
evaluate is a code path an attacker will find and trigger deliberately.
:::

## Identify which dependency

The error code says which one.

| Code | Dependency | Consequence |
| --- | --- | --- |
| `IDENTITY_SERVICE_UNAVAILABLE` | Identity verification | No agent can authenticate |
| `POLICY_SERVICE_UNAVAILABLE` | Policy evaluation | No protected action can be decided |
| `CREDENTIAL_SERVICE_UNAVAILABLE` | Credential broker | Actions needing a credential cannot execute |
| `APPROVAL_SERVICE_UNAVAILABLE` | Approval service | Approval-gated actions cannot proceed |
| `AUDIT_SERVICE_UNAVAILABLE` | Audit pipeline | Events are not being recorded |
| `DEPENDENCY_UNAVAILABLE` | Something else | Generic |

:::danger `AUDIT_SERVICE_UNAVAILABLE` is the most serious of these
The others mean actions are not happening, which is recoverable — the work waits.

Audit failure means **evidence is being lost right now**, and lost evidence does
not come back when the service recovers. Prioritise it above everything else on
this page.
:::

## Check health

```bash
curl -s http://localhost:4000/health/ready
curl -s http://localhost:4010/health/ready
```

`/health/ready` genuinely checks the database. If it is red, the database is the
problem and everything else follows from it.

```bash
curl -s http://localhost:4000/health/live
```

Live but not ready means the process is fine and a dependency is not.

## It is usually the database

Nine times in ten.

```bash
psql "$DATABASE_URL" -c 'select 1'
psql "$DATABASE_URL" -c 'select count(*) from pg_stat_activity'
```

| Cause | Symptom | Fix |
| --- | --- | --- |
| Connection exhaustion | `remaining connection slots` errors | Raise `max_connections`, lower `DATABASE_POOL_MAX`, or add PgBouncer |
| Statement timeouts | `veefore_db_query_duration_ms` p99 at the timeout | Find the slow query; check for a missing index after a data growth spike |
| Failover in progress | Connections refused, then recovering | Wait. Both planes reconnect |
| Disk full | Write errors | Reclaim space. Check audit table growth |
| Network partition | Timeouts rather than refusals | Infrastructure |

Remember `DATABASE_POOL_MAX` is **per process instance**. Three control plane
instances, two gateway instances and a worker at 20 each is 120 connections.
Scaling out multiplies your connection count; size PostgreSQL for the total.

:::warning The embedded driver can present as this
If `DATABASE_URL` is unset, processes fall back to the embedded PGlite driver —
which is per-process. Each process gets a private, empty database.

The symptom is not a clean outage: logins work in one process and fail in
another, an agent you just created does not exist, a policy you activated does
not apply. Check `DATABASE_URL` is actually set in every process's environment.
:::

## Then check Redis, if you use it

```bash
redis-cli -u "$REDIS_URL" ping
```

Redis backs rate limiting and nonce tracking in a multi-node deployment. Losing
it degrades rather than stops: each instance falls back to its own in-process
store, so the effective rate limit becomes per-instance and nonce replay
protection weakens.

Not an outage. Worth noticing, because it is a quiet reduction in a security
property.

## What to do while it is down

:::steps

:::step Do not bypass

There is no bypass flag, and building one in your agent is the wrong fix. Your
agents should be treating `503` as "retry with backoff", which they will do
correctly if they follow [Errors and retries](/build/errors-and-retries).
:::

:::step Do not drop the enforcement mode

It would not help — the mode change itself needs the control plane and the
database. And if it did work, it would mean agents executing without
authorization during an outage, which is the scenario fail-closed exists to
prevent.
:::

:::step Communicate the scope

"Protected actions are being refused" is accurate and useful. Your agents are
not broken; they are correctly declining to act without authorization.

Agents doing work that does not route through Veefore are unaffected, which is
usually most of what they do.
:::

:::step Fix the dependency

Normal operational work. Nothing Veefore-specific.
:::

:::

## After recovery

:::steps

:::step Confirm enforcement is back

```bash
curl -s http://localhost:4000/health/ready
curl -s http://localhost:4010/health/ready

curl -X POST $CP/v1/policies/simulate \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{ "agentId": "agt_01JAV…", "tool": "refund_customer", "amount": 150000 }'
```

Simulate something you know should be refused. A simulation that returns the
expected refusal proves evaluation is working, not merely that the process is
up.
:::

:::step Verify the audit chain

```bash
veefore audit verify
```

If audit writes failed, there may be a gap. The chain verification tells you
whether the stored sequence is intact; a gap in *coverage* is a different thing
from a break in the chain, and the window is what you lost.

Record the window on an incident. "Audit coverage is incomplete between 02:14
and 02:48 UTC" is a usable statement. Discovering it six months later is not.
:::

:::step Check the job queues

```promql
veefore_job_queue_depth
```

Webhook fan-out, usage rollups, risk baseline recomputation and notifications
all queue during an outage. They drain on their own, but a queue that is still
growing means the worker is not running.

Abandoned jobs are reclaimed and dead-lettered after repeated failure, so a job
that was mid-flight when the process died does not wedge the queue. A stalled
queue with no worker is the failure mode that does not page anyone — monitor
depth.
:::

:::step Expect a risk spike

Velocity and history baselines were disrupted. Agents resuming after an outage
will trip `VELOCITY_ANOMALY` and `TIME_ANOMALY` for a while as they catch up on
queued work.

This is correct behaviour. It will also generate approval requests. Expect it,
and do not retune weights in response to a one-off.
:::

:::

## Reduce the chance of recurrence

**Run more than one of everything.** Both planes are stateless; session state
and the job queue are in the database.

**Alert on readiness, not just liveness.** A process that is live and not ready
will be in your load balancer's rotation if your probe is pointed at the wrong
endpoint.

**Alert on audit pipeline errors with zero tolerance.** Evidence loss is not a
threshold thing.

**Size the connection pool for your instance count**, not for one process.

**Test it.** `npm run test:chaos` asserts fail-closed behaviour and recovery.
Run it as part of `npm run verify`; it is the suite that proves this page is
accurate.

## Related

:::cards

:::card Fail closed | /secure/fail-closed
The principle and where it applies.
:::

:::card Database | /operate/database
Pooling, timeouts, restores.
:::

:::card Observability | /operate/observability
The alerts that fire here.
:::

:::card Errors and retries | /build/errors-and-retries
How agents should behave.
:::

:::