--- title: Troubleshooting description: From a symptom to a cause. The failures people actually hit, including several that look like bugs and are not. --- ## Start here, always ```bash veefore doctor --strict --verbose veefore status veefore logs --limit 50 ``` `doctor` audits the whole posture and a surprising share of problems are already in its output. If you read nothing else on this page, run it first. ## Agents cannot authenticate ### A bare `401` and the key is correct Almost certainly the **audience**. The gateway compares `aud` against its configured public URL **as a string**, so `http://127.0.0.1:4010` and `http://localhost:4010` are the same host and different values. ```bash curl -s http://localhost:4010/.well-known/veefore-platform-key ``` Use the `audience` from that response verbatim. `veefore init` fetches it rather than assuming, which is why a hand-written config is the usual culprit. This is the single most common onboarding failure. ### `IDENTITY_GENERATION_STALE` The key was rotated. The agent is using the old one. Re-read `VEEFORE_AGENT_PRIVATE_KEY` and `VEEFORE_AGENT_KEY_ID` from your configuration source. A long-running process that cached config at startup needs a path for this. ### `NONCE_REPLAYED` An assertion was reused. Generate a fresh nonce per assertion. From an SDK, this usually means an assertion object is being cached. In a multi-node deployment without `REDIS_URL`, nonce tracking is per-instance — so replay protection is weaker than you think, and this error can appear inconsistently. ### `TIMESTAMP_OUT_OF_WINDOW` Clock drift. Check NTP on the agent's host. ### `AGENT_IDENTITY_REVOKED` Terminal. Somebody revoked it. Check the audit trail for who and why: ```bash veefore audit list --type agent.revoked ``` ## Nothing works and no rule looks relevant A **deny-by-default policy bound to a broad scope**. ```bash veefore policies list ``` Look for a policy whose `defaultDecision` is `BLOCK` and whose binding scope is `ENVIRONMENT` or wider. Because strictness beats priority, it denies everything in that scope regardless of what any other policy says. The tell is in the explanation: the deciding rule is reported as `Policy default` rather than as something somebody wrote. Fix: bind it to the `AGENT` or `TOOL` it was meant to constrain. ## An agent you just created does not exist **The embedded database.** If `DATABASE_URL` is unset, processes fall back to PGlite, which is per-process — the control plane, the gateway and any script each get a private, empty database. ```bash env | grep DATABASE_URL ``` Symptoms are confusing rather than obviously broken: logins work in one process and fail in another, a policy you activated does not apply, a seed script appears to succeed and leaves nothing behind. Set `DATABASE_URL` for anything multi-process. ## Something is behaving as a different agent or organization **Ambient environment variables.** Shell sessions persist variables between commands, and an `VEEFORE_AGENT_ID` exported an hour ago will quietly make everything authenticate as the wrong agent. ```bash env | grep Veefore ``` This also breaks test suites in ways that look like real failures. The repository's isolation helper clears the ambient agent variables deliberately, twice, for exactly this reason. ## The console appears completely broken **CORS.** If you changed the console's port or host and did not update `CORS_ALLOWED_ORIGINS`, the browser refuses every request and there is no useful error in the UI. ```ini VEEFORE_CONSOLE_PORT=5173 CORS_ALLOWED_ORIGINS=http://localhost:5173 ``` Check the browser's network tab for the actual cause before anything else. ## The console is pointing at the wrong API `VITE_API_BASE_URL` and `VITE_GATEWAY_BASE_URL` are **baked in at build time**. Changing them requires a rebuild and redeploy of the console; restarting a server does nothing. ## Veefore allowed it and the downstream returned `401` Veefore permitted the action and brokered a credential; the downstream rejected what it received. ```bash veefore logs --limit 20 ``` A `CREDENTIAL` stage of `PASS` with an `EXECUTION` stage carrying `http_401` narrows it to three causes: 1. **Wrong material key for the credential type.** `API_KEY` reads `api_key`, `apiKey`, `key` or `value`. Now refused at registration and rotation, but an older credential may predate the check — rotate it and the check runs. 2. **Wrong `header`.** Your downstream wants `x-api-key`, you configured `authorization`, or the reverse. 3. **Missing `prefix`.** `authorization` usually needs `Bearer ` with the trailing space. [Full table](/build/broker-a-credential). ## `DESTINATION_BLOCKED` on a host that is in the allowlist Three possibilities: - **Plaintext HTTP to a non-loopback host.** HTTPS is required unless the host is a loopback address. - **A redirect.** `redirect: 'error'` — redirects are refused, not followed. Point `endpoint_url` at the final location. - **The allowlist is on a different tool.** The check is against *this tool's own* `destination_allowlist`. ## Webhooks are not arriving ```bash env | grep GATEWAY_ALLOW_REAL_EGRESS ``` With `GATEWAY_ALLOW_REAL_EGRESS=false` — the default — delivery is **not attempted** and is recorded as failed with an explicit reason. That is deliberate: a delivery reported as successful when nothing was sent would be a lie in the record you would later rely on. If egress is on and deliveries still fail: - **Signature verification failing on your side.** You are almost certainly re-serialising the JSON. The signature covers the exact bytes sent — capture the raw body before parsing. - **Your endpoint is slow.** Deliveries time out at 10 seconds. Return `2xx` immediately and queue the work. - **A redirect.** Refused, not followed. ## Queues are backing up ```promql veefore_job_queue_depth ``` **Is a worker running?** With no worker, webhooks never deliver and usage never rolls up, and nothing visibly breaks — which is why this needs an alert rather than a glance. ```bash node dist/control-plane/worker.js ``` Job claiming is tenant-safe and guarded against overlapping poll ticks, so several workers can run concurrently. Repeatedly failing jobs are dead-lettered rather than retried forever. ## Approvals are expiring ```promql histogram_quantile(0.95, rate(veefore_approval_decision_duration_seconds_bucket[1h])) ``` If p95 exceeds your rule TTLs, approvals are expiring systematically. Each one is an action that silently did not happen. Fix one of: the TTL, the staffing, or the rule that fires too often. See [Approvals queue](/operate/approvals-queue). ## Everything is `503` A dependency is unavailable and protected actions are being refused. That is fail-closed working. See [Dependency outage](/operate/runbook-dependency-outage). ## A new agent trips risk signals on everything Correct. `NEW_TOOL`, `NEW_RESOURCE`, `NEW_RECIPIENT` and `UNUSUAL_DESTINATION` are **baseline** signals — they compare against what *this agent* has done before, and a new agent has no baseline. Run new agents in `OBSERVE` or `SIMULATE` long enough to establish one before promoting. Do not retune weights in response. ## Rate limits firing unexpectedly In a multi-node deployment without `REDIS_URL`, each instance keeps its own rate limit and nonce state. So the effective limit is multiplied by your instance count, and behaviour is inconsistent depending on which instance a request lands on. ```ini REDIS_URL=redis://… ``` ## The gateway is healthy but has no data Historically caused by the embedded driver booting a fresh empty database after being closed, which left readiness green with nothing behind it. `close()` is now terminal and readiness reflects reality. If you see this on a current version, check `DATABASE_URL` in the gateway's environment specifically — not in the control plane's. ## Migrations "failed" but the tables exist Historically caused by node-postgres returning an **array** of results for multi-statement SQL: the driver read `.rows` off the array, got `undefined`, and threw *after* the DDL had committed. The schema was applied but not recorded. Fixed. If you have an environment stuck in that state from an older version, compare `schema_migrations` against what actually exists before re-running. ## The CLI connects to the wrong Veefore `veefore` prints where it connected and why, for example `Veefore: https://veefore.your-company.com (last sign-in)`. It looks, in order, at `--control-plane`, the project's `veefore.config.json`, `VEEFORE_CONTROL_PLANE_URL`, the address you last signed in to, Veefore running on this machine, and finally hosted Veefore. - **Self-hosted, but it went to hosted Veefore.** Nothing named your address yet. Run `veefore --control-plane https://veefore.your-company.com` once. - **It refuses an address with "not a Veefore control plane".** Something else answers there, often another local service on the same port. Use the address your administrator set as `CONTROL_PLANE_PUBLIC_URL`. - **Browser sign-in opens the wrong console, or a page that does not load.** On a self-hosted install, `CONSOLE_PUBLIC_URL` must be the address people open the console at. ## A CLI command looks hung It probably is not. Piping CLI output through `tail` or `grep` can make a fast command appear to hang because nothing is flushed until it completes. Verify before concluding: ```bash node -e "const {spawn}=require('node:child_process');const p=spawn('npx',['veefore','status'],{stdio:'inherit'});const t=setTimeout(()=>{p.kill();process.exit(1)},10000);p.on('exit',c=>{clearTimeout(t);process.exit(c??1)})" ``` Relatedly: smoke-test output interleaves badly with the services' structured logs. Redirect to a file and filter: ```bash npm run smoke:gateway > /tmp/smoke.log 2>&1; grep -v '"level"' /tmp/smoke.log ``` ## Still stuck Gather these and see [Getting help](/support): - `request_id` from the error body or the `x-request-id` header - The `error.code`, not the message - The enforcement mode, from `veefore status` - `veefore doctor --strict` output - Versions: `veefore --version`, your SDK, and `GET /v1/meta/info` :::danger Never include a private key, credential or session token Not in an issue, not in a chat message, not in a redacted screenshot. If one has already been sent anywhere, rotate it. ::: ## Related :::cards :::card Error codes | /reference/errors Every code with its meaning. Generated. ::: :::card Getting help | /support What to gather and where to go. ::: :::card FAQ | /faq Several of these, in question form. ::: :::card Observability | /operate/observability The metrics quoted above. ::: :::