Introduction
In the past year, the term LLM router has become a buzzword in the AI engineering community. Start‑ups, cloud providers, and enterprise teams alike are building sophisticated routing layers that sit between an application and one or more large‑language‑model (LLM) back‑ends. The promise is clear: route each prompt to the most appropriate model, balancing cost optimisation against performance tuning.
Yet, not all routers live up to that promise. In March 2024, Manifest, a well‑known LLM‑gateway provider, launched a rule‑based router that quickly gained traction. By September 2026, the company announced that the router would be fully deprecated. The decision was driven by a combination of mixed performance, escalating inference costs, and the inherent brittleness of inferring prompt complexity from surface features alone.
This article dives deep into why Manifest’s router failed, what the broader LLM router trend looks like, and how you can choose or build a routing layer that truly delivers on cost and quality. We’ll cover the core concepts, architectural patterns, failure modes, and practical alternatives—complete with code snippets, comparison tables, and a FAQ section to answer the most common questions.
---
1. What Is an LLM Router?
An LLM router is a middleware component that receives a user prompt (and optional metadata), decides which LLM should answer, and forwards the request. The decision can be based on a variety of signals:
| Type | Description | Typical Use‑Case |
|---|---|---|
| Rule‑Based | Hard‑coded thresholds (prompt length, keyword lists). | Quick to implement, deterministic. |
| Semantic | Embedding similarity to model‑specific “skill” vectors. | Handles nuanced domain knowledge. |
| Predictive | ML classifiers trained on historical prompt‑model pairs. | Learns complex patterns. |
| Cascading | Sequential fallback from cheap to expensive models. | Guarantees quality while controlling cost. |
| Cost‑Based | Uses real‑time pricing APIs to pick the cheapest viable model. | Direct cost optimisation. |
The router sits between the client layer (web app, mobile, API consumer) and the LLM providers (OpenAI, Anthropic, Azure, proprietary). It can be exposed as a REST endpoint, gRPC service, or integrated directly into a gateway product.
---
2. Manifest’s Rule‑Based Router: Architecture & Design
Manifest’s router was a classic rule‑based implementation. Below is a high‑level diagram of its architecture:
┌───────────────────────┐
│ Client / App Layer │
└─────────────┬─────────┘
│
▼
┌───────────────────────┐
│ Manifest Gateway API │
│ (Router Layer) │
└───────┬───────┬───────┘
│ │ │
▼ ▼ ▼
Tier‑1 Tier‑2 Tier‑3 Tier‑4
(Simple) (Standard) (Complex) (Reasoning)
│ │ │
▼ ▼ ▼
Model‑A Model‑B Model‑C Model‑D2.1 Core Components
| Component | Responsibility |
|---|---|
| Classifier | A lightweight rule engine that inspects prompt length, keyword presence, and user‑defined heuristics to assign a tier. |
| Model Mapping | Static mapping from tier to a specific LLM (e.g., Standard → GPT‑4‑Turbo, Complex → Claude‑3‑Sonnet). |
| Fallback Logic | If a tier’s model fails (timeout, error), the router automatically falls back to the next tier. |
| Observability | Metrics (latency, error rates, cost per request) are exposed via Prometheus and a dashboard. |
2.2 Sample Rule‑Based Classifier (Python)
from typing import List, Dict
# Simple rule set
RULES = {
"simple": {"max_tokens": 200, "keywords": ["hello", "hi"]},
"standard": {"max_tokens": 500, "keywords": ["define", "explain"]},
"complex": {"max_tokens": 1000, "keywords": ["compare", "contrast"]},
"reasoning": {"max_tokens": 2000, "keywords": ["why", "how"]},
}
def classify_prompt(prompt: str) -> str:
tokens = len(prompt.split())
for tier, rule in RULES.items():
if tokens <= rule["max_tokens"] and any(k in prompt.lower() for k in rule["keywords"]):
return tier
return "complex" # default fallback
# Example usage
prompt = "Explain the difference between supervised and unsupervised learning."
tier = classify_prompt(prompt)
print(f"Assigned tier: {tier}")The classifier is intentionally simple: it checks token count and the presence of a handful of keywords. While this works for a narrow set of prompts, it quickly breaks down when prompts are ambiguous, multi‑topic, or when new LLMs with different capabilities are added.
---
3. Why the Router Fell Short
3.1 Mixed Performance
- Misclassification: A prompt about “financial forecasting” might be 150 tokens but contain no keywords, causing it to be routed to the Simple tier and answered by a small model that lacks domain knowledge.
- Cold‑Start Latency: Some LLMs (e.g., Claude‑3‑Sonnet) have higher warm‑up times. If the router always routes Complex prompts to such models, overall latency spikes.
3.2 Escalating Inference Costs
- Static Tier‑Model Mapping: As pricing models changed (e.g., GPT‑4‑Turbo became cheaper), the router’s static mapping became suboptimal. The cost per token for Complex prompts remained high because the router still routed them to the most expensive model.
- No Real‑Time Pricing: The router did not query real‑time pricing APIs, so it could not adapt to dynamic cost fluctuations.
3.3 Difficulty of Inferring Complexity
- Surface Features Are Noisy: Prompt length and keyword presence are weak proxies for actual complexity. A short prompt can be highly technical, while a long prompt can be a simple greeting.
- Evolving Prompt Styles: Users increasingly embed code snippets, tables, or multi‑step instructions, which the rule set did not anticipate.
3.4 Operational Overhead
- Manual Rule Updates: Every new model or pricing change required a manual tweak to the rule set.
- Limited Observability: While metrics were collected, the router lacked a feedback loop to automatically adjust rules based on performance data.
---
4. The Broader LLM Router Trend
The LLM router trend is driven by three forces:
- Cost Pressure – Enterprises are looking to keep inference budgets under control.
- Model Heterogeneity – The ecosystem now includes dozens of models with varying strengths (e.g., code generation, reasoning, summarization).
- Performance Demands – Applications require consistent latency and high‑quality responses.
Start‑ups are experimenting with semantic and predictive routers that learn from data, while larger providers are offering cascading and cost‑based routing as part of their platform services. The trend is moving away from brittle rule‑based systems toward adaptive, data‑driven architectures.
---
5. Alternative Routing Strategies
| Strategy | How It Works | Pros | Cons |
|---|---|---|---|
| Semantic Routing | Compute embeddings for the prompt and compare against model “skill” vectors. | Handles nuanced domain knowledge; adapts to new models. | Requires embedding infrastructure; higher compute cost. |
| Predictive Routing | Train a classifier (e.g., XGBoost, neural net) on historical prompt‑model pairs. | Learns complex patterns; can incorporate many features. | Needs labeled data; retraining overhead. |
| Cascading Routing | Start with a cheap model; if quality is insufficient, fall back to a more capable one. | Guarantees quality; cost‑efficient. | Requires quality estimation logic; potential double latency. |
| Cost‑Based Routing | Query real‑time pricing APIs; pick the cheapest viable model that meets constraints. | Direct cost optimisation; dynamic. | Needs up‑to‑date pricing; may sacrifice quality for cost. |
| Hybrid (Rule + Predictive) | Combine simple rules for quick decisions with a fallback predictive model. | Balances speed and accuracy. | Complexity in orchestration. |
5.1 Semantic Routing Example
import openai
import numpy as np
# Pre‑computed skill vectors for each model
MODEL_SKILLS = {
"gpt-4-turbo": np.array([0.8, 0.1, 0.1]),
"claude-3-sonnet": np.array([0.2, 0.7, 0.1]),
"codex": np.array([0.1, 0.2, 0.7]),
}
def embed_prompt(prompt: str) -> np.ndarray:
response = openai.Embedding.create(
input=prompt,
model="text-embedding-ada-002"
)
return np.array(response["data"][0]["embedding"])
def semantic_router(prompt: str) -> str:
prompt_vec = embed_prompt(prompt)
# Cosine similarity
sims = {
model: np.dot(prompt_vec, skill) / (np.linalg.norm(prompt_vec) * np.linalg.norm(skill))
for model, skill in MODEL_SKILLS.items()
}
# Pick model with highest similarity
return max(sims, key=sims.get)
# Usage
router_choice = semantic_router("Generate a Python function to parse CSV files.")
print(f"Route to: {router_choice}")This approach uses embeddings to capture semantic similarity between the prompt and each model’s skill vector, allowing the router to make more informed decisions.
---
6. Cost‑Optimization in LLM Routing
6.1 Real‑Time Pricing APIs
Many providers expose pricing endpoints that return token rates for each model. A cost‑based router can query these APIs on each request or cache them for a short window.
import requests
import time
CACHE_TTL = 60 # seconds
price_cache = {}
last_fetch = 0
def fetch_prices():
global price_cache, last_fetch
if time.time() - last_fetch > CACHE_TTL:
resp = requests.get("https://api.openai.com/v1/pricing")
price_cache = resp.json()
last_fetch = time.time()
return price_cache
def cost_based_router(prompt_tokens: int) -> str:
prices = fetch_prices()
# Find cheapest model that can handle the prompt
viable = {m: p for m, p in prices.items() if p["max_tokens"] >= prompt_tokens}
return min(viable, key=viable.get)6.2 Dynamic Tier Adjustment
Instead of static tier‑model mapping, the router can adjust tiers based on recent cost and latency metrics. For example, if the Complex tier’s cost has risen by 20 % over the last week, the router can temporarily route Complex prompts to a cheaper Standard model and flag the issue for review.
---
7. Handling Router Failures
Even the best router can fail. Here are best practices for resilience:
| Failure Mode | Mitigation |
|---|---|
| Model Outage | Implement graceful fallback to the next tier; use health checks. |
| Latency Spike | Use a circuit breaker; route to a low‑latency model if thresholds are exceeded. |
| Cost Surge | Monitor cost per request; trigger alerts when thresholds are breached; temporarily disable expensive models. |
| Misclassification | Log misrouted requests; feed back into a predictive model for retraining. |
A robust router should expose a health endpoint that reports the status of each downstream model, allowing orchestration tools to detect and react to failures automatically.
---
8. Building a Future‑Proof Router
- Start with a Modular Design – Separate the classifier, cost engine, and fallback logic into independent services or modules.
- Ingest Observability Data – Collect latency, cost, and error metrics per model and per tier. Store them in a time‑series database.
- Implement Feedback Loops – Use the metrics to retrain predictive models or adjust semantic thresholds automatically.
- Adopt a Hybrid Approach – Combine rule‑based quick checks with a predictive or semantic engine for complex cases.
- Automate Model Lifecycle – When a new model is added or an existing one is retired, update the router configuration automatically via a CI/CD pipeline.
---
9. FAQs
What is an LLM router?
An LLM router sits between an application and one or more LLM providers, receiving a prompt, deciding which model should answer, and forwarding the request.
Why did Manifest deprecate its LLM router?
Manifest’s rule‑based router struggled with mixed performance, rising inference costs, and unreliable complexity inference, leading to its deprecation in September 2026.
What are the challenges of rule‑based LLM routers?
Rule‑based routers rely on hard‑coded thresholds like prompt length or keyword lists, which can misclassify prompts, fail to adapt to new models, and become costly when scaling.
How can I choose the right LLM router for my application?
Consider your cost budget, performance needs, and model diversity. Evaluate rule‑based, semantic, predictive, cascading, and cost‑based routing options and test them with real traffic.
What alternatives exist to rule‑based routers?
Semantic routing uses embeddings to match prompts to model skill vectors, predictive routing trains classifiers on historical data, cascading routes from cheap to expensive models, and cost‑based routing selects the cheapest viable model in real time.
---
10. Conclusion
The LLM router is no longer a niche experiment; it has become a strategic layer in modern AI stacks. Manifest’s experience underscores a critical lesson: rule‑based systems, while simple to deploy, are fragile in the face of evolving models, dynamic pricing, and complex prompt semantics.
Future‑proof routers must be adaptive, data‑driven, and cost‑aware. By combining semantic embeddings, predictive classifiers, cascading fallbacks, and real‑time pricing, engineers can build routing layers that deliver consistent quality while keeping inference budgets in check. As the LLM ecosystem continues to mature, the router will evolve from a simple decision engine into a sophisticated orchestration platform—one that balances performance, cost, and reliability at scale.
---