Millwright

The self-hosted LLM router that cuts inference cost, not quality

Route Anthropic Messages and OpenAI Chat Completions to the model your policy allows: frontier where the work needs it, approved lower-cost models for the rest. Millwright decides from explicit policy, not prompt inspection, and records the reason for every route.

One endpoint applies your policy across approved providers. Each session keeps independent cheap, mid, and frontier affinity lanes, so concurrent workloads do not fight over one warm route.
LicenseApache-2.0
RuntimeOne Rust binary
DeploymentSelf-hosted
API surfacesMessages + Chat Completions

Find the expensive workloads and model the savings.

millwright analyze turns existing Anthropic, OpenAI-compatible, or Millwright logs into a local report: which workload cohorts cost the most and how selected candidate-model economics compare.

Every estimate includes its coverage, price source, and assumptions. Candidate costs are modeled, not realized savings. Quality remains explicitly unverified until you evaluate representative work.

millwright analyze
$ millwright analyze --input ./anthropic.jsonl \
--input-format anthropic --catalog ./models.json \
--candidates deepseek-v3,gemma-3,glm-4.6
scanned 42,318 requests · 2026-06-01 → 06-30 · coverage 96%
MEASURED SPEND billed · from your logs
frontier (claude-*) $••••••
operational leakage $•••• retries · failovers · aborts
SUBSTITUTION CANDIDATES modeled · not yet replayed
workload share candidate model quality
bulk edits 38% DeepSeek-V3 UNVERIFIED
test scaffolding 17% Gemma 3 UNVERIFIED
everyday coding 29% GLM-4.6 UNVERIFIED
every figure carries coverage · price source · assumptions
quality stays UNVERIFIED → evaluate before changing policy
Illustrative output. Observed figures come from your logs; candidate costs come from the catalog you select and remain explicitly modeled.

Route only within model sets you've approved.

Assign requests to cheap, mid, or frontier roles, then list the models each role may use. Millwright reuses a confirmed affinity route when it remains valid and otherwise selects the lowest-estimated-cost healthy model the resolved role permits.

Illustrative policy

An illustrative mid role for day-to-day coding and tool use. A valid affinity lane reuses its confirmed provider and model for the role-specific TTL.

Permitted models (Mid role)
Claude SonnetGLM-4.6Qwen3-CoderDeepSeek-V3
millwright route
class everyday-coding
routed → glm-4.6 policy: approved for mid role · affinity valid
  reason  retained confirmed warm route for mid role
  rejected claude-sonnet higher estimated request cost
  rejected gemma-3 not permitted for mid role
noruntime prompt classifier
noprompt inspection for routing
nomodels outside your approved role

Change the endpoint, not the developer workflow.

Keep compatible Anthropic Messages and OpenAI Chat Completions clients, including supported streaming and tool use. Route them to native Anthropic, AWS Bedrock, or approved OpenAI-compatible endpoints.

When a request cannot be translated faithfully, Millwright rejects it explicitly instead of silently changing its meaning.

client.py
# before: talking to the provider directly
client = Anthropic(
base_url="https://api.anthropic.com",
)
# after: same client, one line changes
client = Anthropic(
base_url="http://localhost:8080", # mw
)
# streaming and tool use unchanged

Drop into Claude Code and Codex.

Millwright speaks the APIs your agents already use, so nothing about the workflow changes. Point Claude Code at the Messages surface or Codex at Chat Completions, and the model picker you already use becomes the tier switch: your main work runs on the role you choose, background chores fall to the cheap role automatically, and policy floors keep planning and high-risk work on frontier.

Claude Code
# point Claude Code at Millwright
export ANTHROPIC_BASE_URL=http://localhost:8080
export ANTHROPIC_AUTH_TOKEN=sk-millwright-…
# the model picker is now the tier switch
export ANTHROPIC_MODEL=millwright-code # main work → mid
export ANTHROPIC_SMALL_FAST_MODEL=millwright-fast # background → cheap
~/.codex/config.toml
model = "millwright-code"
model_provider = "millwright"
[model_providers.millwright]
name = "Millwright"
base_url = "http://localhost:8080/v1"
env_key = "MILLWRIGHT_API_KEY"

Switch tiers mid-session with /model, or steer any harness you control with X-Millwright-Task-Type and X-Millwright-Risk headers. Full setup for Claude Code, Codex, OpenCode, and Pi is in the docs.

A policy-approved route, role-scoped affinity, and evidence you can inspect.

Each response exposes the chosen model, provider, trace ID, and route reason. Role-scoped affinity keeps cheap, mid, and frontier routes independent, and retryable failures may receive one alternate attempt. Configured storage retains detailed traces and cost evidence for completed inference calls.

illustrative trace API record
trace 9c41 task summarization role cheap
routed → deepseek-v3 provider deepseek
reason cheap role allowed for task type
alternative kimi-k2 candidate allowed by role policy
alternative gemma-3 candidate allowed by role policy
attempts deepseek ✓ (1) failover chain: none
latency 1.83s tokens in 4,102 / out 380
cost $•••• source catalog_priced
Engineering

Read the chosen model, provider, trace ID, and route reason from the response.

Finance

Inspect measured usage and cost provenance for completed inference calls.

Platform

Trace the resolved role, provider attempts, and alternatives the router considered.

Built for concurrent traffic.

Millwright runs as one self-hosted Rust binary. Its router never holds a lock across provider I/O, supported streams are forwarded incrementally, and a configurable global concurrency limit provides backpressure.

No classifier in the hot path.

Millwright routes from explicit task and risk metadata, model aliases, and your policy. It does not make a second model call to decide where the first one should go.

Streaming remains streaming.

Supported upstream streams are forwarded incrementally instead of being buffered into a complete response first.

Concurrent sessions stay concurrent.

Role-scoped affinity keeps separate cheap, mid, and frontier lanes under one session ID without adding a per-session lock or serializing upstream requests.

We don't promise a savings number.
Millwright shows its work.

Point Millwright at your own logs to measure current spend and model selected alternatives. Coverage, price sources, and assumptions remain visible; candidate quality remains unverified until you test a representative sample in an approved evaluation harness. Evidence over claims.