LLM routing architecture
Every LLM call in the Nebbos platform — whether it originates from a Pearl agent, a Platform surface, a scheduled worker, or an operator-initiated action — routes through a single central component: the ResilientLLMRouter. The router applies cost caps, circuit breakers, prompt caching, attribution tuples, PII sanitization, and provider selection. Never call an LLM SDK directly from application code.
The architecture composes two orthogonal axes: the quality tier (what class of reasoning does this call need) and the execution provider (where does the call physically run). The two together determine the specific model + endpoint invoked.
System overview
Every arrow in the diagram is enforced by test-covered code. The direct-SDK-import trap at nebos-backend/llm/__init__.py catches any import anthropic / import ollama / from openai import ... outside llm/providers/*.py, emits a router=direct provenance counter, and logs the calling stack frame at WARNING.
The two-axis routing model
Two orthogonal decisions determine which model runs a call.
Axis 1 — Quality tier (what class of reasoning)
| Tier | Frontier default | Task shape | Max tokens |
|---|---|---|---|
| T1 | claude-haiku-4-5 | Classification, routing, risk assessment, event categorization | 200 |
| T2 | claude-sonnet-4-6 | Pearl reasoning, isolated code execution, pattern context, extraction | 800 |
| T3 | claude-opus-4-6 | Orchestrator synthesis, constitutional review, novel multi-step reasoning | 1500 |
The tier decision is made by the caller, per the task’s known shape. Every tier is declared in config/model_routing.toml; no tier name or model string is ever hardcoded in application code.
Axis 2 — Execution provider (where does it run)
executor_hint | Where the call runs | When to use |
|---|---|---|
"frontier" (default) | Anthropic hosted API (or Gemini per multi-provider Article IX) | Novel reasoning · cross-file architecture · adversarial review · quality-first paths |
"local" | Peer provider on the tenant’s own hardware (Ollama, vLLM, LM Studio) via LiteLLM gateway | Isolated well-specced execution · high-volume classification · sovereignty-scoped tasks |
"auto" | Complexity router decides per task_shape + spec_density + cross_file signal | Default for most call sites once the router is populated with task-shape metadata |
The two axes are independent — a T2 call can run either on Sonnet (frontier) or Qwen 3 Coder (local); a T1 call can run on Haiku or Phi-4. Each tier declares a peer_providers list in config/model_routing.toml; the router picks the first available peer for the tenant’s preferences.
Per-tenant configuration
Each tenant’s preferences live in a tenant_ai_credentials row (extending the per-user pattern from ADR-362). Frontmatter fields the router consults at call time:
{
"t1_peer_provider": {"provider": "ollama", "model": "phi-4"},
"t2_peer_provider": {"provider": "vllm", "model": "deepseek-coder-v3.1:16b"},
"t3_peer_provider": null,
"default_executor_hint": "auto",
"sovereignty_mode": "local_first_frontier_fallback"
}Resolution order at call time:
- Tenant override for tier
config/model_routing.tomlpeer_providerslist- Frontier default
sovereignty_mode: local_only fails closed if no local peer available for the tier — the call raises NoLocalProviderConfiguredError rather than silently falling back to frontier. This preserves the sovereignty contract for compliance-scoped tenants.
Call-time flow
The complexity router (when executor_hint="auto")
When the caller passes executor_hint="auto", a pure classifier decides between frontier / local / hybrid:
from typing import Literal
from llm.complexity_router import classify_task_complexity
decision: Literal["local_only", "local_with_frontier_review", "frontier_required"] = (
classify_task_complexity(
task_shape="code_generation", # per canonical task-shape taxonomy
spec_density_score=0.85, # 0.0 (ambiguous prose) → 1.0 (precise spec + types)
cross_file=False, # multi-file reasoning?
tenant_override=None, # per-tenant override from memory-config
)
)Routing rules (default table, tenant-overridable):
| Task shape | Default routing |
|---|---|
classification | local_only |
extraction | local_only |
summarization | local_with_frontier_review |
code_generation | local_with_frontier_review (frontier writes spec, local writes code) |
code_review | frontier_required (cross-file reasoning) |
reasoning_multi_step | frontier_required |
novel_synthesis | frontier_required |
Auto-escalation applies: when a local_only decision produces a low-confidence response or a Pydantic validation failure, the router auto-retries against the frontier and logs an escalation_event.
Invariants that hold across every call
Regardless of tier or executor_hint, every LLM call satisfies these invariants:
- Attribution tuple written to
ai_usage_events— even forcost_per_call_eur = 0.0local calls. Local execution appears in the ledger with zero token cost; capacity-planning tracks compute-time separately. - Kill-switch honored — the ADR-134 fleet/tenant/agent kill-switch halts ALL LLM calls for the affected scope, regardless of provider.
- Circuit breaker per-provider — 5 failures in 60s trips; 300s open before half-open probe. Each provider has its own breaker; a downed Ollama endpoint does not affect Anthropic calls.
- PII sanitization pre-call —
sanitize_for_llm(user_message)runs regardless of destination provider. Local execution is not an exemption from data-minimization. - DSPy compiled prompts — every prompt in the Orchestrator / Pearl reasoning layer is a compiled DSPy Signature module. Recompilation via
scripts/compile_dspy.pyproduces provider-specific artifacts (artifacts/dspy/gap_question_anthropic_v{N}.json,_ollama_v{N}.json, …). - Provenance counter — every call increments
nebos.llm.calls_by_router_provenance_total{router, model_id, caller_module, deployment}per ADR-594. Direct SDK imports that bypass the router still get counted asrouter="direct"via the import trap atllm/__init__.py. - Prompt-caching invariant — Anthropic path enforces
system=as list withcache_control(10× cost penalty otherwise). Providers with implicit KV caching (vLLM v0.6+ prefix cache, llama.cpp KV cache) benefit from identical prefixes automatically. - Server-authoritative cost — for cloud providers, cost is recomputed from
ai_model_rates(ADR-141/142) at the backend regardless of what the provider returns.response_costfrom LiteLLM is discarded.
The LiteLLM unified gateway (ADR-88 AD-7)
Both surfaces — internal fleet (Pearls, workers, orchestrator) and customer IDE (VS Code / Claude Code BYOK per ADR-679) — route through a single self-hosted LiteLLM proxy on the ADR-77 Managed-Agents substrate. LiteLLM natively supports 140+ providers; Nebbos adds:
- Per-tenant virtual keys minted at the gate (LiteLLM
/key/generateAPI) async_pre_call_hook— kill-switch check + budget ceiling check + attribution-key validation before every upstream callasync_log_success_event— writes toai_usage_eventswith server-authoritative cost recomputation- Byte-transparent
/anthropic/*passthrough for cache_control preservation
Adding a new local provider is thin: a llm/providers/{name}.py adapter satisfying the LLMProvider protocol + a [[tiers.tN.peer_providers]] entry in config/model_routing.toml + a LiteLLM config entry. ResilientLLMRouter itself is not modified.
Composition with the MCP substrate
The nebbos-mcp-local FastMCP server on the tenant hardware exposes three LLM tools that call directly into the tenant’s local provider (bypassing the cloud round-trip when the tenant’s sovereignty_mode permits):
local_llm_list_models— enumerate models on the tenant server (registry read)local_llm_call— execute a tier-scoped LLM call locally; carries the same attribution tuple; kick-up to frontier on ambiguitylocal_llm_status— VRAM used · throughput last 60s · escalation-to-frontier count today · cost saved vs frontier estimate
These tools compose with nebbos_brain_sync_posttooluse.py — every user’s local Pearl action becomes an audit signal in the shared brain, and Nebbos gets smarter every day.
Add a new task type — end-to-end workflow
- Add a
[tiers.tN.tasks.your_task_name]block toconfig/model_routing.tomlwithdescription,max_tokens, andjustification. If the task should default-run local, includemin_local_confidence. - Author a Pydantic
response_modelinagents/pearl/schemas/describing the expected output shape (Instructor validates automatically). - Author the DSPy
Signaturemodule and compile artifacts for every provider in the tier’s peer list viascripts/compile_dspy.py --tier tN --task your_task_name --peer-providers all. - Add test fixtures under
tests/llm/local/test_your_task.py— golden inputs, VCR cassette, confidence-threshold auto-escalation, cost-ledger row assertion. - Register in the caller:
router.call(system_prompt=..., user_message=..., response_model=..., model_tier="tN", executor_hint="auto", ...).
Failure modes + rollback
The router degrades gracefully across failure modes:
| Failure | Effect | Recovery |
|---|---|---|
| Voyage-4 embedding provider down | Embedding-consuming search paths return graceful empty result; retrieval falls back to BM25-only via tsvector | Automatic on provider recovery |
| Anthropic circuit tripped | T1/T2/T3 calls with executor_hint="frontier" return CircuitOpenError; caller can retry with executor_hint="local" if configured | 300s open; half-open probe |
| Ollama endpoint on tenant hardware unreachable | executor_hint="local" calls return LocalProviderUnavailableError; caller retries with executor_hint="frontier" unless sovereignty_mode="local_only" | Depends on tenant infra |
| Cost cap exceeded (STOPPED tier) | All LLM calls for tenant raise CostCapExceededError | Founder/tenant-admin action; kill-switch clearable via MCP tool |
| Prompt caching regression | check_prompt_caching.py CI check catches the loss; alert fires at 10× cost | Same-PR revert; test-enforced |
Reference documentation
- Configuration file:
nebos-backend/config/model_routing.toml - Router source:
nebos-backend/llm/resilience.py::ResilientLLMRouter - Provider protocol:
nebos-backend/llm/providers/protocol.py::LLMProvider - Prompt-caching CI check:
nebos-backend/scripts/check_prompt_caching.py - Direct-SDK-import trap:
nebos-backend/llm/__init__.py - Related ADRs: ADR-77 (Managed Agents), ADR-88 (owned data + agent plane), ADR-107 (token usage in DB), ADR-115 (graduation ladder), ADR-134 (AI gateway impl), ADR-141/142 (model rates + integer minor units), ADR-243 (credential routing), ADR-269 (token budget), ADR-289 (BYOK is the spine), ADR-332 (3D metered pricing), ADR-362 (per-user BYOK), ADR-594 (RLR provenance counter), ADR-679 (unified BYOK gateway)
Reading path
- Architecture overview — the four product surfaces this router serves.
- Security model — the tenant-isolation contract at the substrate.
- Data model — the
ai_usage_events+tenant_ai_credentialstables the router writes. - Row-level isolation — how RLS enforces tenant fences at the DB layer.