---
title: Observability
description: Every metric Veefore exports, the traces, the log shape, and the six alerts worth having.
---

Three signals, all on by default except tracing.

```ini
METRICS_ENABLED=true
OTEL_ENABLED=true
OTEL_SERVICE_NAME=veefore
LOG_LEVEL=info
```

## Metrics

Prometheus format at `/metrics` on both planes.

### Enforcement

| Metric | Type | Labels |
| --- | --- | --- |
| `veefore_gateway_requests_total` | counter | `organization` `environment` `protocol` `mode` `decision` `outcome` |
| `veefore_gateway_request_duration_ms` | histogram | `protocol` `mode` `outcome` |
| `veefore_policy_decisions_total` | counter | `environment` `decision` `policy` |
| `veefore_policy_evaluation_duration_ms` | histogram | `environment` `decision` |
| `veefore_blocked_requests_total` | counter | `organization` `environment` `reason` |

`veefore_gateway_requests_total` carries both `mode` and `decision`, which is
what lets you see divergence in `OBSERVE` as a time series rather than by
reading logs.

### Identity

| Metric | Type | Labels |
| --- | --- | --- |
| `veefore_authentication_duration_ms` | histogram | `kind` `result` |
| `veefore_authentication_failures_total` | counter | `kind` `reason` |
| `veefore_agent_tokens_issued_total` | counter | `kind` |

`veefore_authentication_failures_total` by `reason` is the one to watch. A spike
in `nonce_replayed` or `timestamp_out_of_window` is not a user error at scale —
it is either a broken client or someone replaying captured traffic.

### Approvals, credentials, tools

| Metric | Type | Labels |
| --- | --- | --- |
| `veefore_approval_decision_duration_seconds` | histogram | `organization` `decision` |
| `veefore_credential_leases_total` | counter | `provider` `type` |
| `veefore_credential_broker_errors_total` | counter | `provider` `operation` |
| `veefore_tool_failures_total` | counter | `tool` `kind` `reason` |

`veefore_approval_decision_duration_seconds` is your approval SLA, measured
rather than asserted. If the p95 is longer than your rule TTLs, your approvals
are expiring systematically.

### Audit and jobs

| Metric | Type | Labels |
| --- | --- | --- |
| `veefore_audit_events_total` | counter | `event_type` `severity` |
| `veefore_audit_pipeline_errors_total` | counter | — |
| `veefore_job_queue_depth` | gauge | `queue` |
| `veefore_job_runs_total` | counter | `queue` `result` |

### Infrastructure

| Metric | Type | Labels |
| --- | --- | --- |
| `veefore_http_requests_total` | counter | `method` `route` `status` |
| `veefore_http_request_duration_ms` | histogram | `method` `route` |
| `veefore_db_query_duration_ms` | histogram | `operation` |
| `veefore_db_errors_total` | counter | `operation` |

:::warning `organization` as a label is a cardinality decision
Fine for tens or hundreds of tenants. At thousands it will hurt your Prometheus.
Drop the label with a relabel rule if you operate at that scale, and get
per-tenant breakdown from the audit trail instead.
:::

## The six alerts worth having

```yaml
groups:
  - name: veefore
    rules:
      # Enforcement cannot evaluate. Protected actions are being refused.
      - alert: VeeforeDependencyUnavailable
        expr: |
          sum(rate(veefore_gateway_requests_total{outcome="FAILED"}[5m]))
            / sum(rate(veefore_gateway_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: Over 5% of protected actions are failing. Check /health/ready.

      # The audit trail is the thing you cannot reconstruct.
      - alert: VeeforeAuditPipelineErrors
        expr: increase(veefore_audit_pipeline_errors_total[10m]) > 0
        for: 0m
        labels: { severity: critical }
        annotations:
          summary: Audit writes are failing. Evidence is being lost right now.

      # A stalled queue breaks nothing visibly, which is why it needs an alert.
      - alert: VeeforeJobQueueStalled
        expr: veefore_job_queue_depth > 100
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Queue {{ $labels.queue }} is backing up. Is a worker running?"

      # Either a broken client or replayed traffic.
      - alert: VeeforeAuthFailureSpike
        expr: sum(rate(veefore_authentication_failures_total[5m])) > 1
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: Sustained agent authentication failures.

      # Approvals are expiring because nobody is working the queue.
      - alert: VeeforeApprovalsNotBeingWorked
        expr: |
          histogram_quantile(0.95,
            rate(veefore_approval_decision_duration_seconds_bucket[1h])) > 900
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: p95 approval time exceeds the default 900s TTL.

      # The gateway is on the latency path for every protected action.
      - alert: VeeforeGatewayLatency
        expr: |
          histogram_quantile(0.99,
            rate(veefore_gateway_request_duration_ms_bucket[5m])) > 500
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: Gateway p99 decision latency above 500ms.
```

:::security Do not alert on blocked requests
`veefore_blocked_requests_total` rising means enforcement is working. Paging on
it teaches the on-call to see refusals as incidents, and the natural response to
an alert you do not want is to make the rule looser.

Alert on *changes in the rate* if you want a signal, or review it on a schedule.
A refusal is not an outage.
:::

What is worth watching but probably not paging on: refusal rate dropping to
zero. That can mean your agents fixed their behaviour, or that enforcement
stopped being applied.

## Traces

OpenTelemetry, exported through the standard `OTEL_EXPORTER_*` environment
variables.

A protected action produces one span per pipeline stage, named for the stage:
`REQUEST`, `AUTHENTICATION`, `IDENTITY`, `LIFECYCLE`, `CAPABILITY`,
`DELEGATION`, `POLICY`, `RISK`, `APPROVAL`, `CREDENTIAL`, `EXECUTION`, `RESULT`.

That decomposition is the fastest way to answer "why was that slow": the stage
with the time in it is the stage with the problem. Usually it is `EXECUTION`,
which means your downstream, not Veefore.

Trace context propagates to the downstream on real egress, so a single trace
spans the agent's request, the Veefore decision and your service's handling.

## Logs

Structured JSON through pino. Every line carries `requestId` and `traceId`.

```json
{
  "level": "info",
  "time": 1759492684123,
  "requestId": "req_01JAV7YQ4T2K9M3E8H6R5N1XZ0",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "event": "gateway.action_decided",
  "agentId": "agt_01JAV…",
  "tool": "refund_customer",
  "decision": "REQUIRE_APPROVAL",
  "mode": "ENFORCE",
  "latencyMs": 38,
  "msg": "Protected action requires approval"
}
```

:::security Redaction happens on the way in
Credential material, private keys, session tokens and parameters listed in a
tool's `sensitive_parameters` are redacted before the line is constructed.

There is no code path where a secret is written to a log and later filtered,
which means log shipping, retention and backups cannot leak what was never
written.
:::

Index on `requestId` and `traceId`. Given a `request_id` from an error response
you then have every log line, the action record and the audit event in one
query.

:::warning `debug` in production is expensive and sticky
`debug` logs request and response shapes. Redaction still applies, but the
volume costs money and a log retained for 90 days is a much larger surface than
one retained for 7. Raise it for an investigation; lower it afterwards.
:::

## What Veefore gives you that metrics cannot

Metrics tell you a decision happened. The **action record** tells you why:

```bash
veefore logs --decision BLOCK --limit 50
curl -s $CP/v1/actions/act_01ABC -H "Authorization: Bearer $KEY"
```

Policy and version, the deciding rule, the per-condition trace, the resolved
authority, the delegation chain, the risk signals with their explanations, the
approval and who decided it, the credential lease by reference, the downstream
status.

Do not try to reproduce that in a metric. Alert on rates; investigate in the
action record.

## Product analytics

Veefore tracks adoption without collecting agent content: time to first agent,
time to first protected action, `OBSERVE`→`SIMULATE` and `SIMULATE`→`ENFORCE`
conversion, policies created.

Those are the numbers that tell you whether the thing is being adopted or just
installed — an organization with fifty agents still in `OBSERVE` has not
adopted anything.

## Related

:::cards

:::card Run the stack | /operate/run-the-stack
Health endpoints and process responsibilities.
:::

:::card Audit | /concepts/audit
The complete record behind every decision.
:::

:::card Troubleshooting | /operate/troubleshooting
From a symptom to a cause.
:::

:::card Dependency outage | /operate/runbook-dependency-outage
When the alerts above fire.
:::

:::