Millwrightv0.1

Documentation

Self-hosted LLM routing and spend control. Deterministic, inspectable, vendor-neutral, in one Rust binary between your AI applications and agents and the providers you choose.

Millwright sits in front of your AI applications and agents and sends each request to the cheapest model its resolved policy role allows. Provider credentials stay on the router, and requests go only to the provider endpoints you configure. Keep frontier models where quality matters and route routine work to cheaper ones, subject to the policy and evaluation bar you control.

OpenAI Chat Completions and Anthropic Messages surfaces come in, and configured providers go out. With storage available, completed inference calls add route and cost evidence to a ledger you can query. It is a routing data plane, not an agent orchestrator: no background scheduler, no hosted service, and no model call outside the request path.

LicenseApache-2.0

RuntimeOne Rust binary

DeploymentSelf-hosted

API surfacesMessages + Chat Completions

Quickstart

Build from source with Rust 1.97+. make build produces bin/millwright, and the included Dockerfile supports docker build -t millwright ..

bash

# install from source (Rust 1.97+)
cargo install --path . --locked

Run the interactive setup to pick providers, assign cheap / mid / frontier models, and generate separate workload and operator keys. Keys are shown once and stored only as hashes; init never collects your provider credentials.

bash

millwright init     # generates workload + operator keys (shown once)
millwright serve    # listens on 127.0.0.1:8080 by default

Point a compatible AI application or agent at the router. Clients get a Millwright key, never an upstream provider key. Every response that reached a decision carries X-Millwright-Trace-ID, X-Millwright-Model, X-Millwright-Provider, and X-Millwright-Route-Reason; millwright trace <id> shows the full decision, including every alternative the router rejected and why.

bash

export ANTHROPIC_BASE_URL=http://localhost:8080     # router root, no path
export ANTHROPIC_AUTH_TOKEN=<your workload key>     # a Millwright key

How routing works

Classification selects a role on every request from headers and the requested model alone. Message content is never inspected.

Pick the tier by precedence: policy floors (high risk and planning always get frontier) > explicit X-Millwright-Task-Type / X-Millwright-Riskheaders > the model the client asked for. Your agent's native model picker is the tier switch.
Route within the tier: the router picks the cheapest healthy model the tier allows. A tier is a list, so models across providers compete on price and cover for each other.
Stick within the role: session affinity reuses a provider/model only within the resolved role. One opaque X-Millwright-Session-ID keeps independent cheap, mid, and frontier lanes, each with its own TTL, warmed only after provider success.
Fail over: retryable provider failures trip a circuit breaker. A request makes at most two attempts, using one alternate route only for transport failures, timeouts, or HTTP 408 / 429 / 5xx.
Record: completed requests produce a ledger event with measured usage and cost provenance, plus a route trace with the reason and rejected alternatives. Ledger writes fail open by design.

Two rules are built in and not configurable: risk: high and task_type: planning always require frontier. Every client can drive the tier through the model it selects:

Tier alias	Role	Typical work
`millwright-plan`	frontier	planning and explicitly high-risk work
`millwright-code`	mid	routine code, first-pass review, verification
`millwright-fast`	cheap	extraction, summarization, per-file audits

Catalog names (frontier-planner) and upstream names (claude-sonnet-4-5) resolve to their catalog tier. Unrecognized models fall back to the documented defaults (routine_code / low → mid).

Configuration

Two JSON files govern everything, plus provider credentials from the environment. The policy says who may call the router and what each model role is allowed to do; the catalog says which models exist, who serves them, and what they cost.

Policy

API keys are stored as exact SHA-256 hashes, never plaintext. Each key carries a team and explicit scopes. model_roles contains exactly cheap, mid, and frontier. The resolved role determines the eligible model set, and the router picks its cheapest healthy candidate when no valid affinity route exists.

configs/policy.json

{
  "version": "v0.1",
  "auth": {
    "api_keys": [
      { "name": "local-demo", "team": "platform",
        "key_hash": "b672e0…c1d6",
        "scopes": ["inference", "management:read:all"] }
    ]
  },
  "model_roles": {
    "cheap":    { "models": ["cheap-general", "cheap-coder"],
                  "allowed_for": ["extraction", "summarization", "per_file_audit"] },
    "mid":      { "models": ["mid-coder"],
                  "allowed_for": ["routine_code", "first_pass_review", "verification"] },
    "frontier": { "models": ["frontier-planner"],
                  "allowed_for": ["planning"] }
  }
}

Task-type names are yours; they only need to match what clients send in X-Millwright-Task-Type. The local-demokey's plaintext (sk-millwright-local-demo) is public. Delete it from any non-local policy.

Catalog

Each entry maps a catalog name to a provider and upstream model, with prices per MTok (input, cached input, output, cache write) and prompt-cache settings. Input, cached, and output prices are required so missing data cannot silently become free usage.

configs/models.json

{
  "version": "v0.1",
  "models": {
    "frontier-planner": {
      "provider": "anthropic",
      "upstream_model": "claude-sonnet-4-5",
      "tier": "frontier",
      "input_price_per_mtok": 3.0,
      "cached_input_price_per_mtok": 0.3,
      "output_price_per_mtok": 15.0,
      "cache_write_price_per_mtok": 3.75,
      "prompt_cache": { "supported": true, "ttl_seconds": 300, "mode": "provider" }
    }
  }
}

Provider credentials

Credentials come from the environment only, never from the JSON files, and are never logged. Register OpenAI-compatible providers by ID, set native Anthropic directly, or enable Claude via AWS Bedrock.

env

MILLWRIGHT_OPENAI_COMPAT_PROVIDERS=qwen,openai
MILLWRIGHT_PROVIDER_QWEN_BASE_URL=https://dashscope-intl…/compatible-mode/v1
MILLWRIGHT_PROVIDER_QWEN_API_KEY=…
MILLWRIGHT_ANTHROPIC_API_KEY=…          # native Anthropic
MILLWRIGHT_BEDROCK_REGION=us-east-1     # Claude via AWS Bedrock

To validate hand-edited files, run millwright serve. It refuses to start on an invalid policy or catalog and says exactly why.

Client setup

One rule everywhere: clients get Millwright API keys, never upstream provider keys. OpenAI-compatible clients call <url>/v1/chat/completions; Anthropic-compatible clients (Claude Code) point at the router root.

Mode	URL	Key
Local mock demo	`http://localhost:8080`	`sk-millwright-local-demo`
Local real models	`http://localhost:8080`	workload key from millwright init
Production	your router URL	a Millwright key you issued

Claude Code

Point Claude Code at the router root. The model choice is the tier switch: set your working tier and let background chores fall to cheap automatically.

bash

export ANTHROPIC_BASE_URL=http://localhost:8080     # router root, no path
export ANTHROPIC_AUTH_TOKEN=sk-millwright-local-demo
export ANTHROPIC_MODEL=millwright-code              # your work → mid tier
export ANTHROPIC_SMALL_FAST_MODEL=millwright-fast   # background → cheap tier

Codex, OpenCode, and Pi use the OpenAI-compatible surface at <url>/v1 with model millwright-code. Full per-client configs are in docs/clients.md.

Router API

Classification headers

Both surfaces accept the same optional headers.

Header	Effect
`X-Millwright-Task-Type`	Task type for policy routing (e.g. `summarization`, `planning`); default `routine_code`.
`X-Millwright-Risk`	`low` / `medium` / `high`; default `low`. `high` always routes to frontier.
`X-Millwright-Session-ID`	Opaque affinity scope with independent cheap, mid, and frontier lanes.

Inference surfaces

Endpoint	Surface
`POST /v1/chat/completions`	OpenAI-compatible Chat Completions, SSE streaming.
`POST /v1/messages`	Anthropic Messages, SSE streaming.
`POST /v1/messages/count_tokens`	Token count; records a trace but no spend event.
`GET /v1/models`	Tier aliases first, then Claude-shaped catalog IDs.

Translation runs both directions across the supported text-and-tools subset. A field without a faithful cross-protocol representation, including images, documents, and signed thinking, is refused with an explicit 400 rather than silently flattened. Responses that reached a decision add X-Millwright-Provider, X-Millwright-Model, X-Millwright-Route-Reason, X-Millwright-Attempt-Count, and X-Millwright-Failover-Count.

Management API

The management CLI reads these JSON endpoints, so your own dashboards can read the same numbers. They require management:read:self (own team) or management:read:all (operator).

Endpoint	Returns
`GET /v1/millwright/spend/summary`	Totals, cache read rate, frontier token share.
`GET /v1/millwright/spend/top`	Spend by team.
`GET /v1/millwright/models/mix`	Requests / tokens / cost by model + provider.
`GET /v1/millwright/cache/metrics`	Fresh / cached / written input tokens.
`GET /v1/millwright/traces/{id}`	One routing decision, fully explained.
`GET /v1/millwright/ledger/export`	Every finalized ledger event as JSONL.

/healthz, /livez, and /readyz are unauthenticated for orchestration probes. Prometheus /metrics requires management:read:all. Full reference in docs/gateway-api.md.

CLI reference

Command	What it does
`millwright init`	Interactive setup: providers, roles, and workload + operator keys.
`millwright serve`	Run the router.
`millwright spend`	Totals, cache read rate, per-team breakdown.
`millwright models`	Model / provider mix.
`millwright trace <id>`	One routing decision, fully explained.
`millwright top`	Live terminal view.
`millwright analyze`	Cost-opportunity report from a live or exported ledger.

spend, models, and trace support --json for machine-readable output. analyze separates measured cost evidence from modeled candidate economics: a lower modeled price is not a quality verdict or a deployment recommendation.

millwright analyze

millwright analyze --input ledger.jsonl --input-format millwright \
  --catalog configs/models.json \
  --candidates claude-haiku-4-5,gemini-2.5-flash \
  --report analysis.html --json-out analysis.json

Quality stays not-tested until replayed. Replay a representative sample with an approved evaluation harness before changing production routing. See docs/analyze.md.

Deployment

The router is a single binary with durable request state in the ledger: SQLite locally, Postgres in production. Affinity and circuit-breaker state are bounded but process-local, so v0.1 defaults to one replica, or requires load-balancer affinity on the session ID.

Environment reference

Variable	Default	Purpose
`MILLWRIGHT_GATEWAY_ADDR`	`127.0.0.1:8080`	Listen address; non-loopback demo configs are refused.
`MILLWRIGHT_POLICY_PATH`	`configs/policy.example.json`	Policy: keys and model roles.
`MILLWRIGHT_MODEL_CATALOG_PATH`	`configs/models.example.json`	Catalog: providers and prices.
`MILLWRIGHT_LEDGER_BACKEND`	`sqlite`	`sqlite`, `postgres`, or `memory`.
`MILLWRIGHT_POSTGRES_DSN`	—	Required for the postgres backend.
`MILLWRIGHT_OPENAI_COMPAT_PROVIDERS`	—	CSV of provider IDs to register.
`MILLWRIGHT_ANTHROPIC_API_KEY`	—	Native Anthropic upstream.
`MILLWRIGHT_BEDROCK_REGION`	—	Enables the Bedrock provider (needs AWS creds).
`MILLWRIGHT_CONCURRENCY_LIMIT`	`64`	In-flight request cap; 429 beyond it.
`MILLWRIGHT_REQUEST_TIMEOUT`	`30s`	Body-read and non-streaming upstream cap.
`MILLWRIGHT_STREAM_TIMEOUT`	`10m`	Streaming request cap.
`MILLWRIGHT_MAX_WARM_SESSIONS`	`4096`	Per-process root-session cap (LRU eviction).

Full table, including Bedrock, circuit-breaker, and cache knobs, in docs/deployment.md.

Docker Compose

bash

docker compose -f deploy/compose/docker-compose.yaml up --build

This runs the router with the mock provider and a SQLite volume, published on host loopback only. A real deployment requires a private policy/catalog pair, provider environment configuration, and MILLWRIGHT_DEMO_MODE=false.

Network security

Millwright serves plain HTTP and binds to loopback by default. For any non-local deployment, keep it on private networking and terminate TLS at a trusted reverse proxy or load balancer. Do not expose the router or its management endpoints to the public internet. The management CLI refuses non-loopback http:// URLs so an operator key is never sent in plaintext.

Kubernetes: bring your own manifests (one replica, Secret-mounted policy/catalog, managed Postgres, probes on /healthz / /readyz / /livez). Ledger writes fail open, so an accounting outage degrades spend visibility before availability.

Security

Provider credentials live only in the router's environment. They are attached by the upstream adapter, never accepted from a request body, and never logged.
Inbound keys are checked against SHA-256 hashes. Workload keys carry only inference; the operator key carries management:read:all and belongs only in operator tooling. Do not give workload keys management scopes.
Logs are structured JSON with request/trace IDs and team attribution. API keys and prompt payloads are never logged.
Delete the public local-demo key from any non-local policy, and keep the router behind TLS and private networking.

Full threat model and reporting in SECURITY.md.

Back to Millwright View the repository