---
title: Upgrades
description: Forward-only migrations, the rollout order, and what cannot be rolled back.
---

## Before you start

```bash
npm run migrate:check     # every migration applies cleanly to an empty database
npm run verify            # the full gate: 23 stages
veefore audit verify  # the chain is intact before you touch anything
```

Take a database backup, and confirm the four keys are backed up with it:

```bash
pg_dump --format=custom --no-owner --no-privileges veefore > pre-upgrade.dump
```

:::danger A database backup alone is not a restorable backup
Credential material is sealed with `VEEFORE_DATA_ENCRYPTION_KEY`, and the audit
chain is keyed with `VEEFORE_AUDIT_CHAIN_SECRET`. Neither is in the database.

Restore without them and the credentials are unreadable and the chain is
unverifiable. Back up all four keys alongside the dump, and test a restore that
uses both. A backup you have not restored is a hypothesis.
:::

## Rollout order

```bash
npm run migrate                       # 1. forward-only, before anything restarts
# 2. control plane, rolling
# 3. gateway, rolling
# 4. worker, any time
# 5. console static assets, last
```

Migrations run first because they are designed to be safe against the previous
version — the old code keeps working against the new schema. That is what makes
a rolling restart possible rather than requiring a maintenance window.

Gateway after the control plane, so the control plane is already on the new
schema when the gateway starts depending on it.

:::warning Never edit an applied migration
Add a new one. An edited migration has already run everywhere it was going to
run, so the edit applies only to new environments — and your environments
diverge silently, with nothing to indicate it.

`npm run migrate:check` catches this by applying every migration to a throwaway
database and verifying it comes out clean. Run it in CI.
:::

## What rolls back and what does not

| | Rolls back |
| --- | --- |
| Application code | Yes — redeploy the previous image |
| Console assets | Yes |
| Migrations | **No.** Forward-only by design |
| Enforcement mode | Yes, with a reason and step-up |
| Policy versions | Yes — rollback creates a new version with the old content |
| Agent key rotation | **No.** The old key is gone |
| Credential rotation | **No.** The old material is gone |
| Revocation | **No.** Terminal |

Migrations are forward-only because a down-migration that drops a column drops
the data in it, and for an audit trail that is not a rollback — it is deletion.
If a schema change must be undone, the route is a new forward migration that
undoes it, which leaves a record.

## Rolling back the application

Redeploy the previous image. The schema stays where it is, which is fine because
migrations are written to be safe against the previous version.

What to check after rolling back:

- `/health/ready` on both planes
- A simulation returning the decision you expect
- `veefore audit verify`
- Job queue depth draining

## Zero-downtime

Both planes are stateless — session state and the job queue are in the database
— so a rolling restart is safe with no coordination.

Three things to get right:

**Drain before terminating.** Set `terminationGracePeriodSeconds` long enough
for in-flight protected actions to finish. The egress timeout is 15 seconds by
default, so 30 is a reasonable floor.

**Readiness probes on `/health/ready`.** It genuinely checks the database, so a
pod whose connection is gone leaves rotation rather than serving errors.

**Workers can restart at any point.** In-flight jobs are reclaimed, and
repeatedly failing jobs are dead-lettered rather than retried forever.

## After the upgrade

```bash
curl -s http://localhost:4000/v1/meta/info     # version and both public URLs
curl -s http://localhost:4000/health/ready
curl -s http://localhost:4010/health/ready
veefore doctor --strict
veefore audit verify
```

Then check the two things that break quietly:

```promql
veefore_job_queue_depth                        # draining?
increase(veefore_audit_pipeline_errors_total[10m])   # must be zero
```

And confirm a real decision:

```bash
curl -X POST $CP/v1/policies/simulate \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{ "agentId": "agt_01JAV…", "tool": "refund_customer", "amount": 150000 }'
```

A simulation returning the expected refusal proves evaluation works, not merely
that the process started.

## SDK upgrades

The SDKs and the server are versioned independently, and the wire contract is
stable within `v1`. An older SDK works against a newer server.

`veefore doctor` reports SDK version drift as a finding, because drift that is
tolerated for a long time becomes drift that is relied on. Agents report their
SDK name and version at handshake, which is where that check gets its data.

Upgrade SDKs on your own schedule. Upgrade them when `doctor` complains.

## Breaking changes

Within `v1`, the wire contract does not break. New fields are added; existing
fields keep their meaning.

If a breaking change is ever required it arrives as `v2`, served alongside `v1`,
with `v1` supported through a deprecation window. There is no silent change to
the meaning of an existing field — that is the one thing a client cannot defend
against.

## Enforcement mode is data

Deploying does not change any environment's mode. It lives in the database and
changes only through `POST /v1/environments/{id}/gateway-mode` or
`veefore deploy`, with a reason, step-up authentication, and a `CRITICAL` audit
event if it reduces enforcement.

Deliberate: a mode that could be changed by a deploy could be changed by a
misconfigured pipeline.

## Related

:::cards

:::card Database | /operate/database
Migrations, backups, restores.
:::

:::card Deployment | /operate/deployment
Containers, Kubernetes, rollout.
:::

:::card Dependency outage | /operate/runbook-dependency-outage
If something goes wrong mid-upgrade.
:::

:::card Testing strategy | /architecture/testing-strategy
What `npm run verify` proves.
:::

:::