“In 2011, we used BGP Local Preference to route heavy ISP traffic away from expensive transit providers. In 2026, we use LLM Routers to steer prompts away from $15/M token frontier models for simple tasks.”

At Kyndryl, scaling enterprise GenAI workloads required optimizing token expenditure without compromising response quality.

Sending simple classification queries (e.g., “Is this ticket urgent?”) to a top-tier frontier model is the equivalent of routing local subnet traffic across an international submarine fiber link.

The Multi-Tier LLM Router Architecture

We designed a lightweight Prompt Complexity Analyzer that inspects incoming queries and routes them across model tiers:

• Tier 1 (Local / Edge Models): Simple lookup, classification, and formatting -> Routed to Ollama / Gemini Flash ($0.0001 / 1k ops).
• Tier 2 (Mid-Tier Code Models): Standard code generation, unit tests -> Routed to Claude 3.5 Haiku / Llama 3.1 70B ($0.001 / 1k ops).
• Tier 3 (Frontier Reasoning Models): Multi-step system architecture, root cause post-mortems -> Routed to Gemini 2.0 Pro / Claude 3.5 Sonnet ($0.015 / 1k ops).

[!NOTE] Task-Based Routing: Similar to BGP metric evaluation, the router calculates a “Complexity Score” based on token length, code syntax density, and context depth before selecting the target endpoint.

# # LLM Router Policy Configuration (YAML)
router_policy:
  default_target: "gemini-2.0-flash"
  rules:
    - match:
        max_tokens: 500
        keywords: ["format", "classify", "extract", "summarize"]
      target: "ollama-local-llama3"
      cost_weight: 0.1

    - match:
        keywords: ["refactor", "unit test", "python", "terraform"]
      target: "claude-3-5-haiku"
      cost_weight: 0.5

    - match:
        keywords: ["architect", "root cause", "post-mortem", "system design"]
      target: "gemini-2-0-pro"
      cost_weight: 1.0

Open-Source Traffic Steering

We have open-sourced a lightweight TypeScript library implementing this BGP-inspired routing logic, allowing you to easily configure dynamic LLM path selection.

View the code and examples on GitHub:
👉 virtualsachin/llm-cost-router

The Verdict

Key Takeaway

Route Prompts by Task Complexity.

Do not default all application queries to the most expensive model provider. Implementing an LLM Router reduces token costs by up to 65% while preserving frontier quality where it matters most.