Operations

Troubleshooting

The top ten failure modes from the operations runbook — symptom, cause, fix — with expanded recovery procedures for the five that hurt most, and the SLOs they protect.

Use this page the way the on-call runbook is used: find your failure signature in the index table below, apply the immediate actions to stabilise, then work the root-cause steps and remediation in the matching expanded section. Most failures here are self-healing once the underlying cause is removed — the circuit breaker recovers on its own, retries absorb transient 429s — so the goal of the immediate actions is to stop the bleeding, not to bypass the protection layer.

Failure index

Failure 1 — REPLICATE_API_TOKEN not set

The single most common failure. Every agent invocation returns {"error": "REPLICATE_API_TOKEN is not set"} and CLI agents run exits 1 with a red error message.

Immediate actions

export REPLICATE_API_TOKEN=<your-token>
replicate-agent status

replicate-mcp-agents v0.6.0

  API Token  ✓  r8_x...xxxx
  Agents     ✓  6 registered

Root cause

Check the process environment with env | grep REPLICATE — the token must be set in the same shell (and the same service unit) as the process that failed. Check your .env file or secret-manager configuration, and verify the token itself is valid:

curl -H "Authorization: Token $REPLICATE_API_TOKEN" \
  https://api.replicate.com/v1/account

Remediation

Set the token in the runtime environment where the process actually runs — the systemd unit, Docker --env-file, or a Kubernetes Secret — and add a SecretManager.validate_replicate_token() check to your startup probes so a missing token fails fast instead of at the first model call.

Failure 2 — Circuit breaker stuck OPEN

All invocations for one specific model fail immediately with a circuit-open error while other models work, and the replicate_mcp.circuit_breaker.trips counter is elevated. The breaker opens after 5 consecutive failures and stays OPEN for recovery_timeout (60 s by default) before probing again.

Immediate actions

If you need the model back before the timeout elapses, reset its breaker:

# Reset the breaker for a specific model
from replicate_mcp.server import _executor
breaker = _executor.circuit_breaker("meta/meta-llama-3-70b-instruct")
breaker.reset()

Root cause

Check the Replicate status page first — a stuck-open breaker almost always means the upstream model is genuinely failing. Then check the breaker state in the OTEL "Circuit Breakers" dashboard panel, and confirm whether recovery_timeout has actually elapsed since the trip.

Remediation

If Replicate is healthy, do nothing: the breaker auto-recovers through HALF-OPEN after the timeout (2 successful probes close it). If Replicate is degraded, steer the router away from the failing model so traffic lands on an alternative:

from replicate_mcp.server import _router
_router.record_outcome("meta/meta-llama-3-70b-instruct",
                       latency_ms=60000, cost_usd=0, success=False)

Failure 3 — Replicate rate limiting (429)

Logs show 429 Too Many Requests, OTEL traces show high retry counts, and replicate_mcp.error.count climbs. Your request rate is bunching above the account's limit — retries alone make this worse, because each retry adds load.

Immediate actions

Cut concurrency and add a rate limiter in front of the API:

from replicate_mcp.agents.execution import AgentExecutor
from replicate_mcp.ratelimit import TokenBucket

bucket = TokenBucket(rate=5.0, capacity=10.0)   # 5 req/s, burst of 10
executor = AgentExecutor(max_concurrency=3, rate_limiter=bucket)

Root cause

Review the max_concurrency setting on every AgentExecutor in the deployment (the default is 10 per executor — several executors multiply). Inspect the OTEL latency histogram for request bunching, and if sustained throughput genuinely needs to be higher, contact Replicate support about raising the account limit.

Remediation

Make the rate_limiter part of the production configuration rather than an incident response, and use the RateLimiter registry for per-model buckets when different models have different limits. The framework already classifies 429s as RateLimitError (retryable with back-off), so a correctly sized bucket plus the default jittered retry absorbs bursts cleanly. Details in Configure resilience and caching.

Failure 4 — Checkpoint corruption or partial write

Resuming a workflow fails with json.JSONDecodeError, or a checkpoint is reported missing despite the file existing on disk. Checkpoint writes are atomic (os.replace()), so corruption indicates an OS-level problem — usually a full disk or a process killed mid-write.

Immediate actions

# 1. List known checkpoint sessions
CheckpointManager.list_sessions()

# 2. Inspect the suspect file
#    cat <checkpoint_dir>/<session_id>.json

# 3. Delete the corrupted checkpoint and restart the workflow
ckpt.delete(session_id)

Root cause

Look for .tmp files left in the checkpoint directory — they are the residue of incomplete atomic writes. Check disk space with df -h, and check for kill signals during writes with dmesg | grep oom.

Remediation

Add a disk-space alert at 80% full, and clean up orphaned temp files on a schedule:

find <checkpoint_dir> -name '*.tmp' -mmin +60 -delete

Failure 5 — MCP server not connecting to Claude Desktop

Claude Desktop shows "MCP server not connected" or "Tool not available", and the replicate-mcp-server process exits immediately when the client launches it. This is the highest-impact configuration failure because it takes every agent offline for the client at once.

Immediate actions

Run the server manually and read stderr — the failure reason is almost always printed there:

REPLICATE_MCP_ENV=dev replicate-mcp-server 2>&1

Then validate the client's MCP configuration block:

{
  "mcpServers": {
    "replicate": {
      "command": "replicate-mcp-server",
      "env": {"REPLICATE_API_TOKEN": "r..."}
    }
  }
}

Root cause

Verify the executable is on the PATH the client sees (which replicate-mcp-server), check the Python/Poetry environment with poetry env info, and confirm the MCP SDK version with pip show mcp.

Remediation

Ensure the virtualenv that owns the executable is visible to Claude Desktop's shell environment (an absolute path in command is the robust fix), and keep the MCP SDK pinned to mcp >=1.20.0,<2.0.0 as the package's dependency constraint requires. Full client wiring is covered in Serve agents over MCP.

Service level objectives

The published SLOs define what "healthy" means for the failures above. Targets come in two grades — A is the committed baseline, A+ is the stretch target.

The error-budget burn policy decides how loudly to react when an SLO is being consumed faster than planned:

Next steps

• Configure resilience and caching — tune the breakers, retries, and rate limits behind failures 2 and 3.
• Scale out with worker nodes — worker health checks and circuit-aware failover for distributed deployments.
• CLI reference — doctor, status, and the audit commands used throughout this page.