Guides

Configure resilience and caching

Tune the four layers that stand between your agents and a flaky upstream: per-model circuit breakers, jittered retries, token-bucket rate limiting, and an opt-in result cache — then pin model versions so the answers stay deterministic.

Route map

1. 1

   Tune the circuit breaker

   Understand the CLOSED → OPEN → HALF_OPEN state machine and wire a custom CircuitBreakerConfig into the executor.

2. 2

   Configure retries and rate limits

   Set back-off parameters, learn which exceptions are retried, and cap outgoing request rate with a TokenBucket.

3. 3

   Enable result caching

   Attach a ResultCache, pick an eviction policy that matches your workload, and read the hit-rate stats.

4. 4

   Pin model versions

   Use VersionPinningMode.EXACT so discovery refreshes never silently swap the model under you.

Why per-model circuit breaking matters

A circuit breaker is a small state machine that watches the calls to one downstream dependency and stops sending traffic when that dependency is clearly broken. AgentExecutor keeps one breaker per Replicate model (created lazily by executor.circuit_breaker(model_id)), so one flaky model fails fast and alone — it cannot queue up retries, exhaust your concurrency semaphore, and drag every other agent in the process down with it.

Each breaker moves through three states:

• CLOSED — normal operation. Calls pass through; consecutive failures are counted, and any success resets the counter.
• OPEN — tripped after failure_threshold consecutive failures. Every call is rejected immediately with CircuitOpenError (the message includes an estimated retry time). After recovery_timeout seconds the breaker moves to HALF_OPEN.
• HALF_OPEN — probing. At most half_open_max_calls concurrent probe calls are allowed. success_threshold consecutive successes close the circuit; a single failure reopens it.

The defaults in CircuitBreakerConfig are deliberately conservative:

All four fields are optional — override only what you need. Passing a config to the executor applies it to every per-model breaker it creates:

from replicate_mcp.agents.execution import AgentExecutor
from replicate_mcp.resilience import CircuitBreakerConfig

cb_config = CircuitBreakerConfig(
    failure_threshold=3,     # trip sooner for an expensive model
    recovery_timeout=30.0,   # probe again after 30 s instead of 60 s
    half_open_max_calls=2,
    success_threshold=2,
)

executor = AgentExecutor(circuit_breaker_config=cb_config)

You can watch the state machine work without touching the Replicate API. Save this as breaker_test.py — it forces failures against a standalone breaker:

from replicate_mcp.resilience import (
    CircuitBreaker, CircuitBreakerConfig, CircuitOpenError,
)

breaker = CircuitBreaker("flaky-model", CircuitBreakerConfig(failure_threshold=3))

for i in range(4):
    try:
        breaker.pre_call()
        raise RuntimeError("simulated model failure")
    except CircuitOpenError as exc:
        print(f"call {i + 1}: rejected — {exc}")
    except RuntimeError:
        breaker.record_failure()
        print(f"call {i + 1}: failed (state: {breaker.state.value})")

python breaker_test.py

call 1: failed (state: closed)
call 2: failed (state: closed)
call 3: failed (state: open)
call 4: rejected — Circuit 'flaky-model' is OPEN (retry in ≈60s) — call rejected

The third consecutive failure trips the circuit, and the fourth call is rejected without ever reaching the (simulated) model. That rejection is the whole point: failing in microseconds instead of timing out in seconds is what keeps a bad model from cascading.

Retries with decorrelated jitter

Failures that get past the breaker are retried with exponential back-off. The parameters live in RetryConfig:

Note that AgentExecutor builds its internal RetryConfig from its own constructor arguments, which default to max_retries=2 and retry_base=0.5 — slightly tighter than the dataclass defaults above. Pass AgentExecutor(max_retries=..., retry_base=...) to change them.

The jitter exists for one reason: thundering herd avoidance. If a model blips and 50 callers all retry exactly 0.5 s later, the synchronized wave of retries can knock it over again. compute_retry_delay() implements the decorrelated-jitter formula from the AWS architecture blog — min(max_delay, base_delay × 2^attempt) plus a random offset of up to ±25% — so retries from independent callers spread out instead of arriving in lockstep.

Not every error deserves a retry. is_retryable_error() classifies an exception in strict order:

1. NonRetryableError subclasses are never retried — AuthenticationError (bad token) and ClientError (4xx bad request) will fail the same way every time.
2. RetryableError subclasses are always retried — RateLimitError (429) and ServerError (5xx) are transient by definition.
3. Anything else is retried only if it matches the retryable_exceptions tuple, which defaults to (Exception,).

Rate limiting with a token bucket

A token bucket throttles outgoing requests: tokens refill at rate per second up to capacity, each call consumes one, and a caller with no tokens available sleeps until refill. The capacity is your burst allowance — short spikes pass, sustained load is held at rate.

from replicate_mcp.agents.execution import AgentExecutor
from replicate_mcp.ratelimit import TokenBucket

# 5 requests/second sustained, bursts of up to 10
executor = AgentExecutor(rate_limiter=TokenBucket(rate=5.0, capacity=10.0))

The executor calls await rate_limiter.acquire() before every attempt — including retries — so back-off and rate limiting compose instead of fighting. The rate_limiter parameter is optional; with the default None no rate limiting is applied. Keep rate below your Replicate account's limit, not at it, so retries have headroom.

Result caching

ResultCache is a content-addressed, in-memory cache for complete invocation results. It is disabled by default — caching is opt-in because production workloads usually must not serve stale model output. Where it shines is development and prompt iteration, where the same (model, payload) pair gets re-run constantly and each run costs money and seconds.

from replicate_mcp.agents.execution import AgentExecutor
from replicate_mcp.cache import EvictionPolicy, ResultCache

cache = ResultCache(
    ttl_s=300.0,                  # default: entries expire after 5 minutes
    max_entries=500,              # default capacity
    policy=EvictionPolicy.LRU,    # default policy
)
executor = AgentExecutor(cache=cache)

Keys come from ResultCache.make_key(model_id, payload): the first 32 hex characters of a SHA-256 hash over the model ID plus the sorted-keys JSON of the payload. Dict insertion order does not matter, and only the hash is stored — never the payload itself. On a hit the cached chunk list is replayed instantly with the same streaming interface as a live call.

Choosing an eviction policy

• LRU (default) — evicts the entry accessed longest ago. Best for workloads with temporal locality: the prompts you just used are the ones you will use again.
• FIFO — evicts by insertion time regardless of access. Best for streaming workloads where newer entries are always more valuable than old ones.
• TTL — evicts only expired entries, never live ones. Best when freshness is the sole eviction criterion and every cached entry has similar value.
• LFU — reserved for a future release; selecting it currently falls back to FIFO-style eviction.

Verify the cache is doing work with its built-in counters. Save as cache_demo.py:

from replicate_mcp.cache import ResultCache

cache = ResultCache(ttl_s=300.0, max_entries=500)
key = cache.make_key("meta/meta-llama-3-8b-instruct", {"prompt": "hi"})

print(cache.get(key))                                  # miss
cache.put(key, [{"output": "Hello!", "done": True}])
print(cache.get(key))                                  # hit
print(cache.stats)

python cache_demo.py

None
[{'output': 'Hello!', 'done': True}]
{'size': 1, 'capacity': 500, 'hits': 1, 'misses': 1, 'hit_rate': 0.5, 'evictions': 0, 'policy': 'lru', 'ttl_s': 300.0}

The stats property returns that complete dict; hits, misses, hit_rate, and evictions are also exposed as individual properties for scraping into your own metrics.

Pin model versions

Model discovery refreshes the agent registry from the Replicate catalog, and by default (VersionPinningMode.LATEST) a refreshed model points at whatever version the API currently serves. For reproducible inference — evaluations, regression suites, anything audited — pin exact version hashes instead:

from replicate_mcp.discovery import DiscoveryConfig, VersionPinningMode

config = DiscoveryConfig(
    version_pinning=VersionPinningMode.EXACT,
    pinned_versions={"meta/llama-2-70b": "5c7854e8"},
)

Under EXACT mode a pinned model is registered as owner/name:versionhash (tagged pinned and exact-pin), and discovery refreshes skip updates for it — the version in your registry, your invocation records, and your audit log stays the hash you chose, byte for byte, no matter what the upstream publishes. MINOR mode is reserved and currently behaves like EXACT. Pinning applies only to discovery refreshes; agents you register manually are unaffected.

Verification

Common mistakes

Next steps

• Routing & QoS — how breaker trips and latency feed back into model selection.
• Scale out with worker nodes — the same circuit-breaker pattern applied to whole machines.
• Troubleshooting — symptom → cause → fix for CircuitOpenError, 429s, and stale-cache reports.