Operate / Run Veefore

Observability

Every metric Veefore exports, the traces, the log shape, and the six alerts worth having.

Three signals, all on by default except tracing.

iniCopy
METRICS_ENABLED=true
OTEL_ENABLED=true
OTEL_SERVICE_NAME=veefore
LOG_LEVEL=info

#Metrics

Prometheus format at /metrics on both planes.

#Enforcement

veefore_gateway_requests_total carries both mode and decision, which is what lets you see divergence in OBSERVE as a time series rather than by reading logs.

#Identity

veefore_authentication_failures_total by reason is the one to watch. A spike in nonce_replayed or timestamp_out_of_window is not a user error at scale — it is either a broken client or someone replaying captured traffic.

#Approvals, credentials, tools

veefore_approval_decision_duration_seconds is your approval SLA, measured rather than asserted. If the p95 is longer than your rule TTLs, your approvals are expiring systematically.

#Audit and jobs

#Infrastructure

#The six alerts worth having

yamlCopy
groups:
  - name: veefore
    rules:
      # Enforcement cannot evaluate. Protected actions are being refused.
      - alert: VeeforeDependencyUnavailable
        expr: |
          sum(rate(veefore_gateway_requests_total{outcome="FAILED"}[5m]))
            / sum(rate(veefore_gateway_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: Over 5% of protected actions are failing. Check /health/ready.

      # The audit trail is the thing you cannot reconstruct.
      - alert: VeeforeAuditPipelineErrors
        expr: increase(veefore_audit_pipeline_errors_total[10m]) > 0
        for: 0m
        labels: { severity: critical }
        annotations:
          summary: Audit writes are failing. Evidence is being lost right now.

      # A stalled queue breaks nothing visibly, which is why it needs an alert.
      - alert: VeeforeJobQueueStalled
        expr: veefore_job_queue_depth > 100
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Queue {{ $labels.queue }} is backing up. Is a worker running?"

      # Either a broken client or replayed traffic.
      - alert: VeeforeAuthFailureSpike
        expr: sum(rate(veefore_authentication_failures_total[5m])) > 1
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: Sustained agent authentication failures.

      # Approvals are expiring because nobody is working the queue.
      - alert: VeeforeApprovalsNotBeingWorked
        expr: |
          histogram_quantile(0.95,
            rate(veefore_approval_decision_duration_seconds_bucket[1h])) > 900
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: p95 approval time exceeds the default 900s TTL.

      # The gateway is on the latency path for every protected action.
      - alert: VeeforeGatewayLatency
        expr: |
          histogram_quantile(0.99,
            rate(veefore_gateway_request_duration_ms_bucket[5m])) > 500
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: Gateway p99 decision latency above 500ms.

What is worth watching but probably not paging on: refusal rate dropping to zero. That can mean your agents fixed their behaviour, or that enforcement stopped being applied.

#Traces

OpenTelemetry, exported through the standard OTEL_EXPORTER_* environment variables.

A protected action produces one span per pipeline stage, named for the stage: REQUEST, AUTHENTICATION, IDENTITY, LIFECYCLE, CAPABILITY, DELEGATION, POLICY, RISK, APPROVAL, CREDENTIAL, EXECUTION, RESULT.

That decomposition is the fastest way to answer "why was that slow": the stage with the time in it is the stage with the problem. Usually it is EXECUTION, which means your downstream, not Veefore.

Trace context propagates to the downstream on real egress, so a single trace spans the agent's request, the Veefore decision and your service's handling.

#Logs

Structured JSON through pino. Every line carries requestId and traceId.

jsonCopy
{
  "level": "info",
  "time": 1759492684123,
  "requestId": "req_01JAV7YQ4T2K9M3E8H6R5N1XZ0",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "event": "gateway.action_decided",
  "agentId": "agt_01JAV…",
  "tool": "refund_customer",
  "decision": "REQUIRE_APPROVAL",
  "mode": "ENFORCE",
  "latencyMs": 38,
  "msg": "Protected action requires approval"
}

Index on requestId and traceId. Given a request_id from an error response you then have every log line, the action record and the audit event in one query.

#What Veefore gives you that metrics cannot

Metrics tell you a decision happened. The action record tells you why:

bashCopy
veefore logs --decision BLOCK --limit 50
curl -s $CP/v1/actions/act_01ABC -H "Authorization: Bearer $KEY"

Policy and version, the deciding rule, the per-condition trace, the resolved authority, the delegation chain, the risk signals with their explanations, the approval and who decided it, the credential lease by reference, the downstream status.

Do not try to reproduce that in a metric. Alert on rates; investigate in the action record.

#Product analytics

Veefore tracks adoption without collecting agent content: time to first agent, time to first protected action, OBSERVE→SIMULATE and SIMULATE→ENFORCE conversion, policies created.

Those are the numbers that tell you whether the thing is being adopted or just installed — an organization with fifty agents still in OBSERVE has not adopted anything.

#Related