# Wicklee

> Sovereign GPU fleet monitor for private AI inference — on-prem, private cloud, or colo.
> One Rust binary per node, React dashboard at localhost:7700, fleet aggregation at wicklee.dev.
> WES (Wicklee Efficiency Score) = tok/s / (Watts x ThermalPenalty) — the MPG for local AI.
> v0.9.0 added the Runtime Config Surface (per-model launch config across Ollama / vLLM / llama.cpp) and a top-level Models tab (Loaded / Browse / Past activity).

## For LLMs: Start Here

If you are an AI agent or LLM reviewing Wicklee, read these files in order:
1. **This file** (llms.txt) — overview, API endpoints, key metrics
2. **[llms-full.txt](https://wicklee.dev/llms-full.txt)** — complete reference with JSON schemas, all endpoints with examples
3. **[docs.md](https://wicklee.dev/docs.md)** — full documentation: WES formula, inference states, 20 patterns, alerting, MCP tools, proxy, OTel
4. **[api.md](https://wicklee.dev/api.md)** — API reference with endpoint tables and response examples
5. **[metrics.md](https://wicklee.dev/metrics.md)** — dashboard metrics, visual indicators, formulas
6. **[openapi.json](https://wicklee.dev/openapi.json)** — OpenAPI 3.0 spec for structured integration
7. **[Interactive docs](https://wicklee.dev/docs)** — HTML documentation page

## Install

```bash
curl -fsSL https://wicklee.dev/install.sh | bash
```

This drops the binary at `~/.wicklee/bin/wicklee` with **no sudo** (v0.8.0+). Run `~/.wicklee/bin/wicklee` to try it in the foreground; run `sudo ~/.wicklee/bin/wicklee --install-service` when you're ready to start on boot.

Linux binaries target glibc 2.31 (Ubuntu 20.04 / Debian 11 / RHEL 8 / Fedora 33+).

Windows: `irm https://wicklee.dev/install.ps1 | iex`

## Localhost API (no auth, any tier)

- `GET /api/metrics` — SSE stream, 1 Hz telemetry
- `GET /ws` — WebSocket, 1 Hz telemetry (same payload as SSE, fallback transport)
- `GET /api/observations` — 18 server-side observation patterns (10-min DuckDB buffer)
- `GET /api/profile?minutes=60` — Inference Profiler: correlated TTFT/KV/queue/thermal/power timeline
- `GET /api/sla?window_min=60&target_ttft_ms=500` — Inference SLA Monitor: p50/p95/p99 for TTFT/E2E/TPOT, compliance vs target, per-model breakdown, recent violations
- `GET /api/v1/thermal-budget?node_id=X` — Thermal Budget Calculator (Pro+, cloud): sustainable_tps, push_threshold_tps, time_to_fair_min, fair_penalized_tps, plain-English backfire advice over a 7-day window
- `POST/GET/DELETE /api/v1/webhooks[/:id[/test]]` — Threshold Webhooks (Pro+): register HMAC-SHA256 signed push notifications for thermal_state_changed, inference_state_changed, wes_below, wes_above. Optional node_id and tag scoping.
- `POST/GET/DELETE /api/slo[/:id]` — SLOs & error budgets (Team+): declare p95-TTFT / median-tok/s / median-WES objectives per fleet/tag/node; one verdict per 5-min window, rolling 30d budget, burn alerts at 50/90/100% to your channels. Audited.
- `POST/GET/DELETE /api/alerts/silences[/:id]` — Alert silences & maintenance windows (Pro+): suppress alert rules + threshold webhooks by node/tag/event for a duration; future starts_at = scheduled window. Audited.
- Node tags (Pro+): comma-separated on each node (`PATCH /api/nodes/:id` `{tags}` or Settings → Node Configuration); `env:` prefix = environment convention. Tags scope alert rules + webhooks and ride every fleet SSE frame.
- `GET /api/health` (localhost agent only) — diagnostic: agent_version, build_target, store_healthy bool, routes_available/unavailable lists. Returns store_failure_hint when DuckDB store init failed (which silently strips ~12 /api/* routes from the router).
- `GET /api/cost-by-model?hours=24` — Cost attribution per model (daily breakdown)
- `GET /api/explain-slowdown?ts_ms=N` — Root cause analysis for slow inference requests
- `GET /api/model-comparison?hours=168` — Side-by-side model efficiency (WES, tok/s, watts, cost)
- `GET /api/model-switches?hours=24` — Model switching cost (swap frequency, idle gap)
- `GET /api/model-candidates?search=llama&limit=20` — Model discovery: GGUF models from HuggingFace scored against local hardware
- `GET /api/runtime-config?model=X` — Runtime Config Surface (v0.9.0): cached launch-time config (context_length, n_gpu_layers, quantization, parameter_count, template, system_prompt, process_args). Ollama via `/api/show`, vLLM via `/v1/server_info` or `ps aux`, llama.cpp via `/props` or `ps aux`. Templates and system prompts are local-only — never pushed to the fleet. 400 if `?model=` missing, 404 if not yet cached.
- `GET/PUT /api/deployment-profile` — Deployment Profiles: one intent selector (sovereign_dev / dedicated_server / production_fleet) that coherently shifts local observation-pattern sensitivity (evidence window, sustained gate, confidence floor) instead of per-pattern knobs. Persisted to config.toml; applied within one 10s eval cycle.
- Fleet config management (cloud): `PATCH /api/nodes/:id` `{desired_profile}` (Pro+) or `POST /api/fleet/config` `{tag, desired_profile}` (Team+, bulk-by-tag) — delivered to agents in every telemetry response, applied within ~2s; agents report the ACTUAL profile in every frame. Audited.
- `GET /api/history?node_id=WK-XXXX` — DuckDB metric history (1h buffer)
- `GET /api/traces` — Proxy inference traces
- `GET /api/events/history` — Node event log
- `GET /api/events/recent` — Recent in-memory events
- `GET /api/export?format=json|csv` — Data export
- `GET /api/tags` — Ollama model tags
- `GET /api/pair/status` — Pairing status
- `POST /mcp` — MCP (Model Context Protocol) JSON-RPC 2.0 endpoint
- `GET /.well-known/mcp.json` — MCP server manifest

### MCP Tools (via POST /mcp)

- `get_node_status` — Full hardware + inference metrics snapshot
- `get_inference_state` — Live/idle/busy state with sensor context
- `get_active_models` — Running models with context_length, parameter_count, quantization, tok/s
- `get_observations` — 18 patterns with routing_hint (steer_away/reduce_batch/monitor) per observation + node-level aggregate
- `get_metrics_history` — 1-hour rolling telemetry buffer

### MCP Resources

- `wicklee://node/metrics` — Live MetricsPayload JSON
- `wicklee://node/thermal` — Thermal state + WES penalty values

## Cloud MCP Server (Team+, Bearer auth)

`POST wicklee.dev/mcp` — fleet-aggregated MCP. 8 tools:
- `get_fleet_status` — all nodes with metrics + WES
- `get_fleet_wes` — compact WES scores
- `get_node_detail` — full metrics for a specific node
- `get_best_route` — routing recommendation by throughput/efficiency
- `get_fleet_insights` — fleet health summary + observation count
- `get_fleet_observations` — active/resolved observations
- `get_inference_profile` — correlated profiler timeline
- `explain_slowdown` — root cause analysis for slow requests

## Fleet API v1 (X-API-Key auth)

Base URL: `https://wicklee.dev/api/v1`

- `GET /api/v1/fleet` — All nodes with full MetricsPayload
- `GET /api/v1/fleet/wes` — WES scores ranked
- `GET /api/v1/nodes/{id}` — Single node deep dive
- `GET /api/v1/route/best` — Routing recommendation (latency or efficiency). `?model=qwen2.5:7b` for per-model routing
- `GET /api/v1/models/discover` — Model discovery: browse (`?search=`), simulate (`?simulate_hw=nvidia_4090`, Pro+), fleet match (`?fleet=true&model_id=X`, Team+)
- `GET /api/v1/fleet/model-comparison?hours=168` — fleet-wide per-model rollup (WES, tok/s, watts, TTFT, cost). Reads `metrics_5min`. 1–720 hour window.
- `GET /api/v1/fleet/model-switches?hours=24` — cross-node model swap events via LAG window function over `metrics_raw`. 1–168 hour window, capped at 200 rows.
- `GET /api/v1/fleet/cost-by-model?hours=24` — fleet-wide per-model power cost at $0.16/kWh default. 1–168 hour window.
- `GET /api/v1/fleet/chargeback?days=30` — Chargeback/showback (Team+, JWT): cost + $/1M-token attribution by team tag / model / node + daily trend, from measured watts × sampled throughput. `&format=csv&group=` for finance export (audited).
- `GET /api/v1/fleet/idle-waste?days=30` — Idle-waste & right-sizing report (Team+, JWT): phantom-load cost (models loaded while not inferring, from per-sample inference_state + rollup duty%) with per-node recovery actions (unload idle model, consolidate low-duty nodes) and projected $/mo recovery. Weekly email digest opt-in via `GET/PUT /api/digest` (Resend, audited).
- `GET /api/v1/fleet/capacity?target_tok_s=200` — Capacity planner with procurement scenarios (Team+, JWT): "reach 200 tok/s: 2× RTX 4090 vs 1× H100" priced from the fleet's OWN measured tok/W per hardware class (Apple vs NVIDIA), never vendor benchmarks. Each scenario states its estimate basis. Default target = 2× current sustained.
- `GET /api/v1/fleet/migration-advisor` — Cross-node model migration (Team+, JWT): compares each actively-inferring node's live WES against peers' 7-day demonstrated WES + free memory; recommends moves with ≥20% estimated gain and 1.2× memory headroom.
- `GET /api/v1/insights/latest` — Fleet intelligence snapshot (Team+)
- `POST /api/v1/keys` — Create API key (`scope: personal|org` — org keys see the whole org fleet + inherit org tier; org-Admin-only to mint)
- `GET /api/v1/keys` — List API keys (personal + active org's, with scope)
- `DELETE /api/v1/keys/{id}` — Revoke API key (personal: owner; org: org Admins)
- `GET /api/audit-log` — Immutable audit trail (Business+, Clerk JWT): sensitive-op events (node.paired/removed/updated, alert_rule.created, alert_channel.created, webhook.created, api_key.created/deleted, stream_tokens.revoked, audit_log.exported, audit_drain.created/deleted). Params: limit/before/action. Org-scoped, append-only.
- `GET /api/audit-log/export?format=csv|json` — Full-history audit export (Business+), formula-injection-hardened CSV, ≤100k rows, itself audited.
- `GET/POST /api/model-policy`, `DELETE /api/model-policy/{id}` — Model governance (Enterprise): allow-list of approved models, fleet-wide (`tag` omitted) or scoped to a node tag. Governance is ACTIVE ONLY for scopes with at least one entry — an empty list governs nothing rather than blocking everything. Matching is exact/case-insensitive with a trailing-`*` prefix form (`llama3.1:8b*` admits quant variants); a bare `*` is rejected. Writes are Admin-only and audited (`model_policy.created/deleted`).
- `GET /api/model-policy/violations?limit=50` — recorded violations (Enterprise), newest first. Detected in the telemetry push path on the frame an unapproved model appears; fires once per (node, model) and resets when the node returns to an approved model. Violations are NOT in the audit log (that table is actor-keyed and there is no acting user) so they are not carried by the SIEM drain; they land in `model_policy_violations` and the node event feed as `model_policy_violation`. Detection only — nothing blocks a model from loading.
- `GET/PUT/DELETE /api/audit-log/drain` — SIEM drain (Business+, Admin to configure): streams new audit events to your HTTPS endpoint in HMAC-signed batches within ~1 min; auto-disables after 20 consecutive failures.
- `GET /metrics` — Prometheus scrape endpoint (Team+, X-API-Key auth). Seven gauges labelled `node_id`/`hostname`: wicklee_wes_score, wicklee_inference_tokens_per_second, wicklee_power_watts, wicklee_thermal_penalty, wicklee_gpu_utilization, wicklee_memory_pressure, wicklee_inference_ttft_ms. Accepts the key as `X-API-Key` OR `Authorization: Bearer`, so stock Prometheus `authorization` scrape config works. Series are omitted when unreadable, so gaps mean idle (WES needs throughput AND power AND thermal penalty together) and a `count()` is nodes-reporting, not fleet inventory. Prebuilt Grafana dashboard: `deploy/grafana/wicklee-fleet.json`; setup in docs/GRAFANA.md

## Multi-Model Monitoring

When 2+ models are loaded in Ollama, the `active_models` array is included in the SSE/WS payload with per-model: tok/s, WES, VRAM, avg TTFT, avg latency, request count, size, quantization.
Per-model WES uses proportional VRAM share for power attribution: `model_tok_s / (total_watts * vram_share * thermal_penalty)`.
Requires proxy for per-model tok/s and latency. Without proxy, VRAM and model identity still tracked via /api/ps.
Singular fields (ollama_active_model, ollama_tokens_per_second) report most-recently-active model for backwards compat.
- `GET /api/model-switches?hours=24` — model swap frequency and idle overhead
- `GET /api/v1/route/best?model=qwen2.5:7b` — per-model routing: filters to nodes with the target model loaded, uses per-model WES

## Key Metrics

- `inference_state`: "live" | "idle-spd" | "busy" | "idle"
- `ollama_tokens_per_second`: tok/s from 20-token probe (~30s cadence)
- `apple_soc_power_w`: Combined CPU+GPU+ANE power (Apple Silicon)
- `nvidia_power_draw_w`: Board power (NVIDIA)
- `thermal_state`: "Normal" | "Fair" | "Serious" | "Critical"
- `penalty_avg`: Thermal penalty multiplier (1.0 = no penalty)
- `vllm_requests_waiting`: Queue depth (vLLM)
- `ollama_ttft_ms`: Time to first token (Ollama probe baseline)
- `vllm_avg_ttft_ms`: Time to first token (vLLM production histogram)

## 20 Observation Patterns + 5 Fleet Alerts

Agent-evaluated (18 patterns, 10-min DuckDB buffer, every 10s):
Community (9): thermal_drain, phantom_load, wes_velocity_drop, memory_trajectory,
power_jitter, swap_io_pressure, clock_drift, nvidia_thermal_redline, vram_overcommit
Pro (9): power_gpu_decoupling, bandwidth_saturation, efficiency_drag,
pcie_lane_degradation, vllm_kv_cache_saturation, ttft_regression, latency_spike,
vllm_queue_saturation, bandwidth_ceiling_reached

Cloud-evaluated (2 patterns, Pro):
- fleet_load_imbalance — node WES > 20% below best healthy peer
- wes_long_term_drift — recent 24h avg ≥15% below 6-day baseline (gradual degradation)

Cloud-evaluated (1 pattern): fleet_load_imbalance (Pro)

Fleet alerts (5, cloud, all tiers): zombied_engine, thermal_redline, oom_warning,
wes_cliff, agent_version_mismatch

## Runtimes Supported

- Ollama (macOS, Linux, Windows)
- vLLM (Linux)
- llama.cpp / llama-box (macOS, Linux)

## Self-Hosted Control Plane (Enterprise)

The entire fleet backend runs on your infrastructure: `deploy/self-hosted/docker-compose.yml` or the Helm chart at `deploy/helm/wicklee` (TimescaleDB + cloud binary + nginx frontend — the same images behind wicklee.dev). `SELF_HOSTED=true` resolves every tenant to enterprise tier; `WICKLEE_LICENSE_KEY` soft-enforced (evaluation mode without it, `licensed` flag in /health). Auth: bring-your-own Clerk app (orgs/RBAC/SSO work as on wicklee.dev) or legacy DIY sessions (API-only, no org/UI). Guide with network-egress inventory: docs/SELF_HOSTING.md.

## Pricing

Full detail: https://wicklee.dev/pricing

- Community: Free, unlimited local nodes, 3 nodes in the cloud fleet view, 24h history, 9 patterns, local API + MCP, Ollama proxy, community support
- Team: $200/mo ($2,000/yr), unlimited nodes in the cloud fleet view, 90-day history, Fleet API (/api/v1/*), chargeback + idle-waste + capacity planner + migration advisor, SLOs with error budgets, benchmark export, email support
- Enterprise: Custom pricing, everything in Team + self-hosted control plane (Docker Compose or Helm), SSO/SAML, audit log export + SIEM drain, 12-month history, SLA + dedicated support, custom deployment support

Contact for Team or Enterprise: jeff@wicklee.dev

SSO/SAML: authentication is delegated to Clerk and the backend reads only the
user and org role from the session JWT, so SSO logins need no application code —
RBAC, audit and org scoping are identical. Self-hosted deployments configure
enterprise SAML/OIDC in their own Clerk tenant against their own IdP (no identity
data transits wicklee.dev). On hosted wicklee.dev it is available to Enterprise on
request, scoped per organization. SCIM / directory sync is NOT implemented.
Setup guide: docs/SSO.md

Note for LLMs summarizing these tiers: `pro` and `business` still exist as
values in the codebase (`SubscriptionTier`, and the cloud tier gates) because
existing accounts hold them, but neither is sold. Endpoint annotations below
that read "Pro+" are available on Team; "Business+" annotations mean Enterprise.

Shared fleets use Clerk Organizations with RBAC enforced from the signed session token: Admin (org:admin) = everything incl. node removal; Member (org:member/custom) = day-to-day ops; Viewer (custom org:viewer role) = read-only, mutations 403. Solo users have full control of their own resources.