Upgrades

Forward-only migrations, the rollout order, and what cannot be rolled back.

#Before you start

bash
npm run migrate:check     # every migration applies cleanly to an empty database
npm run verify            # the full gate: 23 stages
veefore audit verify  # the chain is intact before you touch anything

Take a database backup, and confirm the four keys are backed up with it:

bash
pg_dump --format=custom --no-owner --no-privileges veefore > pre-upgrade.dump

A database backup alone is not a restorable backup

Credential material is sealed with VEEFORE_DATA_ENCRYPTION_KEY, and the audit chain is keyed with VEEFORE_AUDIT_CHAIN_SECRET. Neither is in the database.

Restore without them and the credentials are unreadable and the chain is unverifiable. Back up all four keys alongside the dump, and test a restore that uses both. A backup you have not restored is a hypothesis.

#Rollout order

bash
npm run migrate                       # 1. forward-only, before anything restarts
# 2. control plane, rolling
# 3. gateway, rolling
# 4. worker, any time
# 5. console static assets, last

Migrations run first because they are designed to be safe against the previous version — the old code keeps working against the new schema. That is what makes a rolling restart possible rather than requiring a maintenance window.

Gateway after the control plane, so the control plane is already on the new schema when the gateway starts depending on it.

Never edit an applied migration

Add a new one. An edited migration has already run everywhere it was going to run, so the edit applies only to new environments — and your environments diverge silently, with nothing to indicate it.

npm run migrate:check catches this by applying every migration to a throwaway database and verifying it comes out clean. Run it in CI.

#What rolls back and what does not

	Rolls back
Application code	Yes — redeploy the previous image
Console assets	Yes
Migrations	No. Forward-only by design
Enforcement mode	Yes, with a reason and step-up
Policy versions	Yes — rollback creates a new version with the old content
Agent key rotation	No. The old key is gone
Credential rotation	No. The old material is gone
Revocation	No. Terminal

Migrations are forward-only because a down-migration that drops a column drops the data in it, and for an audit trail that is not a rollback — it is deletion. If a schema change must be undone, the route is a new forward migration that undoes it, which leaves a record.

#Rolling back the application

Redeploy the previous image. The schema stays where it is, which is fine because migrations are written to be safe against the previous version.

What to check after rolling back:

/health/ready on both planes
A simulation returning the decision you expect
veefore audit verify
Job queue depth draining

#Zero-downtime

Both planes are stateless — session state and the job queue are in the database — so a rolling restart is safe with no coordination.

Three things to get right:

Drain before terminating. Set terminationGracePeriodSeconds long enough for in-flight protected actions to finish. The egress timeout is 15 seconds by default, so 30 is a reasonable floor.

Readiness probes on /health/ready. It genuinely checks the database, so a pod whose connection is gone leaves rotation rather than serving errors.

Workers can restart at any point. In-flight jobs are reclaimed, and repeatedly failing jobs are dead-lettered rather than retried forever.

#After the upgrade

bash
curl -s http://localhost:4000/v1/meta/info     # version and both public URLs
curl -s http://localhost:4000/health/ready
curl -s http://localhost:4010/health/ready
veefore doctor --strict
veefore audit verify

Then check the two things that break quietly:

promql
veefore_job_queue_depth                        # draining?
increase(veefore_audit_pipeline_errors_total[10m])   # must be zero

And confirm a real decision:

bash
curl -X POST $CP/v1/policies/simulate \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{ "agentId": "agt_01JAV…", "tool": "refund_customer", "amount": 150000 }'

A simulation returning the expected refusal proves evaluation works, not merely that the process started.

#SDK upgrades

The SDKs and the server are versioned independently, and the wire contract is stable within v1. An older SDK works against a newer server.

veefore doctor reports SDK version drift as a finding, because drift that is tolerated for a long time becomes drift that is relied on. Agents report their SDK name and version at handshake, which is where that check gets its data.

Upgrade SDKs on your own schedule. Upgrade them when doctor complains.

#Breaking changes

Within v1, the wire contract does not break. New fields are added; existing fields keep their meaning.

If a breaking change is ever required it arrives as v2, served alongside v1, with v1 supported through a deprecation window. There is no silent change to the meaning of an existing field — that is the one thing a client cannot defend against.

#Enforcement mode is data

Deploying does not change any environment's mode. It lives in the database and changes only through POST /v1/environments/{id}/gateway-mode or veefore deploy, with a reason, step-up authentication, and a CRITICAL audit event if it reduces enforcement.

Deliberate: a mode that could be changed by a deploy could be changed by a misconfigured pipeline.

Database

Migrations, backups, restores.

Deployment

Containers, Kubernetes, rollout.

Dependency outage

If something goes wrong mid-upgrade.

Testing strategy

What npm run verify proves.