replicate-mcp-agents routes every call across Replicate's model marketplace with bandit algorithms, isolates failures with circuit breakers, and speaks MCP natively — so Claude Desktop and Cursor invoke your agents with zero wiring.
Replicate hosts 50,000+ models. Choosing among them — and surviving their outages — is usually manual heuristics and brittle glue code. replicate-mcp-agents is a Python framework that turns model selection, failure handling, and agent serving into a production-grade, self-tuning layer.
Cost-aware bandit routing. UCB1 explores systematically for the first 20 calls, then Thompson Sampling exploits — scalarizing cost, latency, and quality into one utility. For teams running 10k+ invocations a day, cost-aware selection alone can cut inference spend by 20–40%.
Per-model failure isolation. A three-state circuit breaker (CLOSED → OPEN → HALF_OPEN) wraps every model, with decorrelated-jitter retries. One flaky model never cascades into the rest of your pipeline.
MCP-native, not HTTP-first. Every registered agent is automatically an MCP tool over stdio, SSE, or Streamable HTTP. Claude Desktop, Cursor, and any MCP client call your Replicate agents with zero additional wiring.
A fluent SDK that stays out of the way. Register an agent with one decorator, compose DAG workflows with parallel fan-out in a builder, run them from the CLI.
# pip install replicate-mcp-agents
from replicate_mcp import agent, CostAwareRouter
@agent(
model="meta/meta-llama-3-8b-instruct",
description="Fast chat for general queries",
tags=["chat", "fast"],
)
def llama_chat(prompt: str) -> dict:
return {"prompt": prompt}
# the router learns from every outcome
router = CostAwareRouter(strategy="thompson_multi")
chosen = router.select_model(candidates)
router.record_outcome(chosen,
latency_ms=812, cost_usd=0.002, success=True)Fifteen stable subsystems ship in v0.8.0. These six carry the weight.
UCB1 for the first 20 calls, Thompson Sampling after. thompson_multi samples a Gaussian posterior over scalarized cost + latency + quality utility.
Hard SLA pre-filtering before the bandit selects: FAST < 2,000 ms, BALANCED < 5,000 ms / $0.05, QUALITY ≥ 0.9 quality floor.
Per-model FSM trips after 5 failures, probes recovery after 60 s. AWS-style decorrelated jitter prevents thundering herds.
Local threaded workers plus HTTP worker nodes on port :7999, least-loaded dispatch, health checks, and worker-level circuit breakers with coordinator failover.
Lifecycle hooks with 3 built-in guardrails — PII masking, content filtering, cost caps — discovered via Python entry points. OTEL spans and metrics out of the box.
Content-addressed result cache with LRU / TTL / FIFO eviction and background cleanup. Pin model versions EXACT for deterministic inference.
Every invocation emits OpenTelemetry spans and metrics — invocation.count, latency and cost histograms, circuit-breaker trips — plus structured audit records you can tail from the CLI. Published SLOs put numbers on it: 99.5% availability, <1% error rate, ≤90 s breaker recovery.
Register the stdio server in mcp_config.json — agents appear in the tool palette.
Same MCP server, same zero wiring — invoke Replicate agents from your editor.
Model discovery auto-registers from the 50,000+ model marketplace, with version pinning.
Spans + metrics over OTLP gRPC; null-safe when the SDK is absent. Optional extra.
Prompt management, tracing, and evaluations via LatitudePlugin — zero-config from env vars.