Skip to Content
FoundationsLLM routing architecture

LLM routing architecture

Every LLM call in the Nebbos platform — whether it originates from a Pearl agent, a Platform surface, a scheduled worker, or an operator-initiated action — routes through a single central component: the ResilientLLMRouter. The router applies cost caps, circuit breakers, prompt caching, attribution tuples, PII sanitization, and provider selection. Never call an LLM SDK directly from application code.

The architecture composes two orthogonal axes: the quality tier (what class of reasoning does this call need) and the execution provider (where does the call physically run). The two together determine the specific model + endpoint invoked.

System overview

Every arrow in the diagram is enforced by test-covered code. The direct-SDK-import trap at nebos-backend/llm/__init__.py catches any import anthropic / import ollama / from openai import ... outside llm/providers/*.py, emits a router=direct provenance counter, and logs the calling stack frame at WARNING.

The two-axis routing model

Two orthogonal decisions determine which model runs a call.

Axis 1 — Quality tier (what class of reasoning)

TierFrontier defaultTask shapeMax tokens
T1claude-haiku-4-5Classification, routing, risk assessment, event categorization200
T2claude-sonnet-4-6Pearl reasoning, isolated code execution, pattern context, extraction800
T3claude-opus-4-6Orchestrator synthesis, constitutional review, novel multi-step reasoning1500

The tier decision is made by the caller, per the task’s known shape. Every tier is declared in config/model_routing.toml; no tier name or model string is ever hardcoded in application code.

Axis 2 — Execution provider (where does it run)

executor_hintWhere the call runsWhen to use
"frontier" (default)Anthropic hosted API (or Gemini per multi-provider Article IX)Novel reasoning · cross-file architecture · adversarial review · quality-first paths
"local"Peer provider on the tenant’s own hardware (Ollama, vLLM, LM Studio) via LiteLLM gatewayIsolated well-specced execution · high-volume classification · sovereignty-scoped tasks
"auto"Complexity router decides per task_shape + spec_density + cross_file signalDefault for most call sites once the router is populated with task-shape metadata

The two axes are independent — a T2 call can run either on Sonnet (frontier) or Qwen 3 Coder (local); a T1 call can run on Haiku or Phi-4. Each tier declares a peer_providers list in config/model_routing.toml; the router picks the first available peer for the tenant’s preferences.

Per-tenant configuration

Each tenant’s preferences live in a tenant_ai_credentials row (extending the per-user pattern from ADR-362). Frontmatter fields the router consults at call time:

{ "t1_peer_provider": {"provider": "ollama", "model": "phi-4"}, "t2_peer_provider": {"provider": "vllm", "model": "deepseek-coder-v3.1:16b"}, "t3_peer_provider": null, "default_executor_hint": "auto", "sovereignty_mode": "local_first_frontier_fallback" }

Resolution order at call time:

  1. Tenant override for tier
  2. config/model_routing.toml peer_providers list
  3. Frontier default

sovereignty_mode: local_only fails closed if no local peer available for the tier — the call raises NoLocalProviderConfiguredError rather than silently falling back to frontier. This preserves the sovereignty contract for compliance-scoped tenants.

Call-time flow

The complexity router (when executor_hint="auto")

When the caller passes executor_hint="auto", a pure classifier decides between frontier / local / hybrid:

from typing import Literal from llm.complexity_router import classify_task_complexity decision: Literal["local_only", "local_with_frontier_review", "frontier_required"] = ( classify_task_complexity( task_shape="code_generation", # per canonical task-shape taxonomy spec_density_score=0.85, # 0.0 (ambiguous prose) → 1.0 (precise spec + types) cross_file=False, # multi-file reasoning? tenant_override=None, # per-tenant override from memory-config ) )

Routing rules (default table, tenant-overridable):

Task shapeDefault routing
classificationlocal_only
extractionlocal_only
summarizationlocal_with_frontier_review
code_generationlocal_with_frontier_review (frontier writes spec, local writes code)
code_reviewfrontier_required (cross-file reasoning)
reasoning_multi_stepfrontier_required
novel_synthesisfrontier_required

Auto-escalation applies: when a local_only decision produces a low-confidence response or a Pydantic validation failure, the router auto-retries against the frontier and logs an escalation_event.

Invariants that hold across every call

Regardless of tier or executor_hint, every LLM call satisfies these invariants:

  1. Attribution tuple written to ai_usage_events — even for cost_per_call_eur = 0.0 local calls. Local execution appears in the ledger with zero token cost; capacity-planning tracks compute-time separately.
  2. Kill-switch honored — the ADR-134 fleet/tenant/agent kill-switch halts ALL LLM calls for the affected scope, regardless of provider.
  3. Circuit breaker per-provider — 5 failures in 60s trips; 300s open before half-open probe. Each provider has its own breaker; a downed Ollama endpoint does not affect Anthropic calls.
  4. PII sanitization pre-call — sanitize_for_llm(user_message) runs regardless of destination provider. Local execution is not an exemption from data-minimization.
  5. DSPy compiled prompts — every prompt in the Orchestrator / Pearl reasoning layer is a compiled DSPy Signature module. Recompilation via scripts/compile_dspy.py produces provider-specific artifacts (artifacts/dspy/gap_question_anthropic_v{N}.json, _ollama_v{N}.json, …).
  6. Provenance counter — every call increments nebos.llm.calls_by_router_provenance_total{router, model_id, caller_module, deployment} per ADR-594. Direct SDK imports that bypass the router still get counted as router="direct" via the import trap at llm/__init__.py.
  7. Prompt-caching invariant — Anthropic path enforces system= as list with cache_control (10× cost penalty otherwise). Providers with implicit KV caching (vLLM v0.6+ prefix cache, llama.cpp KV cache) benefit from identical prefixes automatically.
  8. Server-authoritative cost — for cloud providers, cost is recomputed from ai_model_rates (ADR-141/142) at the backend regardless of what the provider returns. response_cost from LiteLLM is discarded.

The LiteLLM unified gateway (ADR-88 AD-7)

Both surfaces — internal fleet (Pearls, workers, orchestrator) and customer IDE (VS Code / Claude Code BYOK per ADR-679) — route through a single self-hosted LiteLLM proxy on the ADR-77 Managed-Agents substrate. LiteLLM natively supports 140+ providers; Nebbos adds:

  • Per-tenant virtual keys minted at the gate (LiteLLM /key/generate API)
  • async_pre_call_hook — kill-switch check + budget ceiling check + attribution-key validation before every upstream call
  • async_log_success_event — writes to ai_usage_events with server-authoritative cost recomputation
  • Byte-transparent /anthropic/* passthrough for cache_control preservation

Adding a new local provider is thin: a llm/providers/{name}.py adapter satisfying the LLMProvider protocol + a [[tiers.tN.peer_providers]] entry in config/model_routing.toml + a LiteLLM config entry. ResilientLLMRouter itself is not modified.

Composition with the MCP substrate

The nebbos-mcp-local FastMCP server on the tenant hardware exposes three LLM tools that call directly into the tenant’s local provider (bypassing the cloud round-trip when the tenant’s sovereignty_mode permits):

  • local_llm_list_models — enumerate models on the tenant server (registry read)
  • local_llm_call — execute a tier-scoped LLM call locally; carries the same attribution tuple; kick-up to frontier on ambiguity
  • local_llm_status — VRAM used · throughput last 60s · escalation-to-frontier count today · cost saved vs frontier estimate

These tools compose with nebbos_brain_sync_posttooluse.py — every user’s local Pearl action becomes an audit signal in the shared brain, and Nebbos gets smarter every day.

Add a new task type — end-to-end workflow

  1. Add a [tiers.tN.tasks.your_task_name] block to config/model_routing.toml with description, max_tokens, and justification. If the task should default-run local, include min_local_confidence.
  2. Author a Pydantic response_model in agents/pearl/schemas/ describing the expected output shape (Instructor validates automatically).
  3. Author the DSPy Signature module and compile artifacts for every provider in the tier’s peer list via scripts/compile_dspy.py --tier tN --task your_task_name --peer-providers all.
  4. Add test fixtures under tests/llm/local/test_your_task.py — golden inputs, VCR cassette, confidence-threshold auto-escalation, cost-ledger row assertion.
  5. Register in the caller: router.call(system_prompt=..., user_message=..., response_model=..., model_tier="tN", executor_hint="auto", ...).

Failure modes + rollback

The router degrades gracefully across failure modes:

FailureEffectRecovery
Voyage-4 embedding provider downEmbedding-consuming search paths return graceful empty result; retrieval falls back to BM25-only via tsvectorAutomatic on provider recovery
Anthropic circuit trippedT1/T2/T3 calls with executor_hint="frontier" return CircuitOpenError; caller can retry with executor_hint="local" if configured300s open; half-open probe
Ollama endpoint on tenant hardware unreachableexecutor_hint="local" calls return LocalProviderUnavailableError; caller retries with executor_hint="frontier" unless sovereignty_mode="local_only"Depends on tenant infra
Cost cap exceeded (STOPPED tier)All LLM calls for tenant raise CostCapExceededErrorFounder/tenant-admin action; kill-switch clearable via MCP tool
Prompt caching regressioncheck_prompt_caching.py CI check catches the loss; alert fires at 10× costSame-PR revert; test-enforced

Reference documentation

  • Configuration file: nebos-backend/config/model_routing.toml
  • Router source: nebos-backend/llm/resilience.py::ResilientLLMRouter
  • Provider protocol: nebos-backend/llm/providers/protocol.py::LLMProvider
  • Prompt-caching CI check: nebos-backend/scripts/check_prompt_caching.py
  • Direct-SDK-import trap: nebos-backend/llm/__init__.py
  • Related ADRs: ADR-77 (Managed Agents), ADR-88 (owned data + agent plane), ADR-107 (token usage in DB), ADR-115 (graduation ladder), ADR-134 (AI gateway impl), ADR-141/142 (model rates + integer minor units), ADR-243 (credential routing), ADR-269 (token budget), ADR-289 (BYOK is the spine), ADR-332 (3D metered pricing), ADR-362 (per-user BYOK), ADR-594 (RLR provenance counter), ADR-679 (unified BYOK gateway)

Reading path

  1. Architecture overview — the four product surfaces this router serves.
  2. Security model — the tenant-isolation contract at the substrate.
  3. Data model — the ai_usage_events + tenant_ai_credentials tables the router writes.
  4. Row-level isolation — how RLS enforces tenant fences at the DB layer.