Operate / Runbooks

Troubleshooting

From a symptom to a cause. The failures people actually hit, including several that look like bugs and are not.

#Start here, always

bashCopy
veefore doctor --strict --verbose
veefore status
veefore logs --limit 50

doctor audits the whole posture and a surprising share of problems are already in its output. If you read nothing else on this page, run it first.

#Agents cannot authenticate

#A bare 401 and the key is correct

Almost certainly the audience. The gateway compares aud against its configured public URL as a string, so http://127.0.0.1:4010 and http://localhost:4010 are the same host and different values.

bashCopy
curl -s http://localhost:4010/.well-known/veefore-platform-key

Use the audience from that response verbatim. veefore init fetches it rather than assuming, which is why a hand-written config is the usual culprit.

This is the single most common onboarding failure.

#IDENTITY_GENERATION_STALE

The key was rotated. The agent is using the old one.

Re-read VEEFORE_AGENT_PRIVATE_KEY and VEEFORE_AGENT_KEY_ID from your configuration source. A long-running process that cached config at startup needs a path for this.

#NONCE_REPLAYED

An assertion was reused. Generate a fresh nonce per assertion. From an SDK, this usually means an assertion object is being cached.

In a multi-node deployment without REDIS_URL, nonce tracking is per-instance — so replay protection is weaker than you think, and this error can appear inconsistently.

#TIMESTAMP_OUT_OF_WINDOW

Clock drift. Check NTP on the agent's host.

#AGENT_IDENTITY_REVOKED

Terminal. Somebody revoked it. Check the audit trail for who and why:

bashCopy
veefore audit list --type agent.revoked

#Nothing works and no rule looks relevant

A deny-by-default policy bound to a broad scope.

bashCopy
veefore policies list

Look for a policy whose defaultDecision is BLOCK and whose binding scope is ENVIRONMENT or wider. Because strictness beats priority, it denies everything in that scope regardless of what any other policy says.

The tell is in the explanation: the deciding rule is reported as Policy default rather than as something somebody wrote.

Fix: bind it to the AGENT or TOOL it was meant to constrain.

#An agent you just created does not exist

The embedded database. If DATABASE_URL is unset, processes fall back to PGlite, which is per-process — the control plane, the gateway and any script each get a private, empty database.

bashCopy
env | grep DATABASE_URL

Symptoms are confusing rather than obviously broken: logins work in one process and fail in another, a policy you activated does not apply, a seed script appears to succeed and leaves nothing behind.

Set DATABASE_URL for anything multi-process.

#Something is behaving as a different agent or organization

Ambient environment variables. Shell sessions persist variables between commands, and an VEEFORE_AGENT_ID exported an hour ago will quietly make everything authenticate as the wrong agent.

bashCopy
env | grep Veefore

This also breaks test suites in ways that look like real failures. The repository's isolation helper clears the ambient agent variables deliberately, twice, for exactly this reason.

#The console appears completely broken

CORS. If you changed the console's port or host and did not update CORS_ALLOWED_ORIGINS, the browser refuses every request and there is no useful error in the UI.

iniCopy
VEEFORE_CONSOLE_PORT=5173
CORS_ALLOWED_ORIGINS=http://localhost:5173

Check the browser's network tab for the actual cause before anything else.

#The console is pointing at the wrong API

VITE_API_BASE_URL and VITE_GATEWAY_BASE_URL are baked in at build time. Changing them requires a rebuild and redeploy of the console; restarting a server does nothing.

#Veefore allowed it and the downstream returned 401

Veefore permitted the action and brokered a credential; the downstream rejected what it received.

bashCopy
veefore logs --limit 20

A CREDENTIAL stage of PASS with an EXECUTION stage carrying http_401 narrows it to three causes:

1. Wrong material key for the credential type. API_KEY reads api_key, apiKey, key or value. Now refused at registration and rotation, but an older credential may predate the check — rotate it and the check runs.
2. Wrong header. Your downstream wants x-api-key, you configured authorization, or the reverse.
3. Missing prefix. authorization usually needs Bearer with the trailing space.

Full table.

#DESTINATION_BLOCKED on a host that is in the allowlist

Three possibilities:

• Plaintext HTTP to a non-loopback host. HTTPS is required unless the host is a loopback address.
• A redirect. redirect: 'error' — redirects are refused, not followed. Point endpoint_url at the final location.
• The allowlist is on a different tool. The check is against this tool's own destination_allowlist.

#Webhooks are not arriving

bashCopy
env | grep GATEWAY_ALLOW_REAL_EGRESS

With GATEWAY_ALLOW_REAL_EGRESS=false — the default — delivery is not attempted and is recorded as failed with an explicit reason. That is deliberate: a delivery reported as successful when nothing was sent would be a lie in the record you would later rely on.

If egress is on and deliveries still fail:

• Signature verification failing on your side. You are almost certainly re-serialising the JSON. The signature covers the exact bytes sent — capture the raw body before parsing.
• Your endpoint is slow. Deliveries time out at 10 seconds. Return 2xx immediately and queue the work.
• A redirect. Refused, not followed.

#Queues are backing up

promqlCopy
veefore_job_queue_depth

Is a worker running? With no worker, webhooks never deliver and usage never rolls up, and nothing visibly breaks — which is why this needs an alert rather than a glance.

bashCopy
node dist/control-plane/worker.js

Job claiming is tenant-safe and guarded against overlapping poll ticks, so several workers can run concurrently. Repeatedly failing jobs are dead-lettered rather than retried forever.

#Approvals are expiring

promqlCopy
histogram_quantile(0.95, rate(veefore_approval_decision_duration_seconds_bucket[1h]))

If p95 exceeds your rule TTLs, approvals are expiring systematically. Each one is an action that silently did not happen.

Fix one of: the TTL, the staffing, or the rule that fires too often. See Approvals queue.

#Everything is 503

A dependency is unavailable and protected actions are being refused. That is fail-closed working. See Dependency outage.

#A new agent trips risk signals on everything

Correct. NEW_TOOL, NEW_RESOURCE, NEW_RECIPIENT and UNUSUAL_DESTINATION are baseline signals — they compare against what this agent has done before, and a new agent has no baseline.

Run new agents in OBSERVE or SIMULATE long enough to establish one before promoting. Do not retune weights in response.

#Rate limits firing unexpectedly

In a multi-node deployment without REDIS_URL, each instance keeps its own rate limit and nonce state. So the effective limit is multiplied by your instance count, and behaviour is inconsistent depending on which instance a request lands on.

iniCopy
REDIS_URL=redis://…

#The gateway is healthy but has no data

Historically caused by the embedded driver booting a fresh empty database after being closed, which left readiness green with nothing behind it. close() is now terminal and readiness reflects reality.

If you see this on a current version, check DATABASE_URL in the gateway's environment specifically — not in the control plane's.

#Migrations "failed" but the tables exist

Historically caused by node-postgres returning an array of results for multi-statement SQL: the driver read .rows off the array, got undefined, and threw after the DDL had committed. The schema was applied but not recorded.

Fixed. If you have an environment stuck in that state from an older version, compare schema_migrations against what actually exists before re-running.

#The CLI connects to the wrong Veefore

veefore prints where it connected and why, for example Veefore: https://veefore.your-company.com (last sign-in). It looks, in order, at --control-plane, the project's veefore.config.json, VEEFORE_CONTROL_PLANE_URL, the address you last signed in to, Veefore running on this machine, and finally hosted Veefore.

• Self-hosted, but it went to hosted Veefore. Nothing named your address yet. Run veefore --control-plane https://veefore.your-company.com once.
• It refuses an address with "not a Veefore control plane". Something else answers there, often another local service on the same port. Use the address your administrator set as CONTROL_PLANE_PUBLIC_URL.
• Browser sign-in opens the wrong console, or a page that does not load. On a self-hosted install, CONSOLE_PUBLIC_URL must be the address people open the console at.

#A CLI command looks hung

It probably is not. Piping CLI output through tail or grep can make a fast command appear to hang because nothing is flushed until it completes.

Verify before concluding:

bashCopy
node -e "const {spawn}=require('node:child_process');const p=spawn('npx',['veefore','status'],{stdio:'inherit'});const t=setTimeout(()=>{p.kill();process.exit(1)},10000);p.on('exit',c=>{clearTimeout(t);process.exit(c??1)})"

Relatedly: smoke-test output interleaves badly with the services' structured logs. Redirect to a file and filter:

bashCopy
npm run smoke:gateway > /tmp/smoke.log 2>&1; grep -v '"level"' /tmp/smoke.log

#Still stuck

Gather these and see Getting help:

• request_id from the error body or the x-request-id header
• The error.code, not the message
• The enforcement mode, from veefore status
• veefore doctor --strict output
• Versions: veefore --version, your SDK, and GET /v1/meta/info

#Related