Concepts

Routing and QoS

How the framework learns which Replicate model to send each call to — balancing cost, latency, and quality with bandit algorithms — and how QoS tiers enforce hard SLA limits before the learning even starts.

This page is for engineers tuning model selection. You will learn the statistics the router keeps per model, the three selection strategies of CostAwareRouter, when UCB1 hands over to Thompson Sampling, and the exact thresholds behind the FAST, BALANCED, and QUALITY tiers. Everything here is implemented in routing.py and qos.py; the decision history is in ADR-005.

The bandit framing

The core idea is the multi-armed bandit: imagine a row of slot machines, each with an unknown payout rate, and a limited budget of pulls. Spend every pull on the machine that looks best so far and you may never discover a better one (pure exploitation); spend pulls evenly across all machines and you waste money on bad ones (pure exploration). A bandit algorithm balances the two — it keeps trying under-tested options just often enough to learn, while sending most traffic to the proven winner.

Here, each "machine" is a Replicate model and each "pull" is a real invocation with a real cost, latency, and outcome. The router never needs offline benchmarks: production traffic itself is the experiment, and every recorded outcome sharpens the next decision. Static rules ("always use the cheapest") stay wrong forever when a model degrades; a bandit notices and shifts traffic away.

What the router measures

Per model, the router maintains a ModelStats record built on exponential moving averages (EMAs) — running estimates where each new observation gets weight alpha and history gets the rest. The default ema_alpha is 0.3: 30% weight on the newest observation, 70% on the accumulated average. Three dimensions are tracked, each starting from a deliberately pessimistic-but-plausible prior so unobserved models neither dominate nor starve: ema_latency_ms starts at 5000.0, ema_cost_usd at 0.01, and ema_quality at 0.8. Alongside the EMAs sit Beta posterior parameters (ts_alpha, ts_beta, both starting at 1.0 — a uniform prior) counting successes and failures.

How much each dimension matters is your call, via RoutingWeights(cost, latency, quality). The defaults are cost=0.4, latency=0.3, quality=0.3; the router normalises internally, so the weights express relative priority rather than needing to sum to one.

The three strategies of CostAwareRouter

CostAwareRouter accepts a strategy argument with three values (default "thompson"):

"score" is deterministic: each candidate gets a weighted sum of its EMA cost, latency, and inverted quality, and the lowest score wins. It never explores, which makes it predictable — appropriate when consistency matters more than discovering a better model.

"thompson" is Thompson Sampling over a Beta posterior on binary success/failure: at selection time the router draws one random sample from each candidate's Beta(ts_alpha, ts_beta) distribution and the highest sample wins. Under-tested models have wide distributions, so they occasionally sample high and get a chance to prove themselves; reliable models have tight, high distributions and win most of the time. Exploration decays naturally as evidence accumulates.

"thompson_multi" exists because plain Thompson Sampling has a blind spot the project's own audit flagged (README §4.6, fixed in v0.7.0): the Beta posterior sees only success or failure, so a model that succeeds slowly and expensively looks identical to one that succeeds fast and cheap. The fix, recorded in ADR-005, scalarizes cost, latency, and quality into a single utility score in [0, 1] and runs Gaussian Thompson Sampling over that utility posterior instead — all three objectives now shape the explore/exploit balance, not just the success bit. Failures still hurt: a failed call's utility is halved before it updates the posterior.

UCB1 and the adaptive transition

UCB1Router implements Upper Confidence Bound 1, a deterministic alternative: it picks the model maximising empirical success rate plus an exploration bonus that grows for under-visited models (unvisited models score infinity, so every candidate is tried at least once). The bonus is scaled by exploration_c, default 1.0 — standard UCB1. Because there is no random sampling, batch runs are reproducible.

AdaptiveRouter combines the two: for the first explore_threshold total invocations — default 20 — it routes with UCB1, then switches to Thompson Sampling, syncing the accumulated Beta posteriors into its Thompson delegate. The reasoning is that UCB1's systematic exploration guarantees every model gets tested during cold start, preventing early over-exploitation of whichever model happened to win first; once priors are informed, Thompson Sampling's empirically stronger exploitation takes over. The active_strategy property tells you which phase the router is in.

QoS tiers

Quality-of-Service (QoS) tiers answer a different question than the bandit. The bandit asks "which candidate is best?"; a QoSPolicy asks "which candidates are acceptable at all?" — and it runs first. The policy is a pre-filter: candidates whose EMA statistics violate its caps are removed before the bandit ever sees them, so no amount of sampled optimism can route a call to a model that breaks your SLA. QoSPolicy.for_level() builds the three standard tiers:

You can also construct a QoSPolicy directly with custom max_latency_ms, max_cost_usd, min_quality, and min_success_rate caps; any constraint left as None is not enforced.

Putting it together

The full loop — build a multi-objective router, filter through a policy, select, and feed the outcome back — is a few lines:

from replicate_mcp.qos import QoSLevel, QoSPolicy
from replicate_mcp.routing import CostAwareRouter, RoutingWeights

router = CostAwareRouter(
    strategy="thompson_multi",
    weights=RoutingWeights(cost=0.5, latency=0.3, quality=0.2),
)
router.register_model("meta/meta-llama-3-8b-instruct",
                      initial_cost=0.002, initial_latency_ms=3000)
router.register_model("mistralai/mixtral-8x7b-instruct-v0.1",
                      initial_cost=0.001, initial_latency_ms=2000)

candidates = ["meta/meta-llama-3-8b-instruct",
              "mistralai/mixtral-8x7b-instruct-v0.1"]
policy = QoSPolicy.for_level(QoSLevel.BALANCED)
filtered = policy.filter_candidates(candidates, router.stats())

chosen = router.select_model(filtered)

# ... execute on Replicate, then close the feedback loop:
router.record_outcome(chosen, latency_ms=1840.0, cost_usd=0.0021,
                      success=True, quality=0.85)

The one rule that matters operationally: call record_outcome() after every invocation, including failures. A router that only hears about successes learns nothing, and select_model_explain() will show you per-candidate scores when you need to audit why a model was picked.

Next steps

• Architecture — see where routing sits in the request lifecycle and what wraps the call after selection.
• Resilience & caching — the circuit breakers and retries that run downstream of the routing decision.
• API reference — full signatures for CostAwareRouter, UCB1Router, AdaptiveRouter, and QoSPolicy.