Models decay quietly.
Flywheel closes the loop.
A self-hosted control plane that owns the whole lifecycle: versioned data, gated training, canary releases, live drift detection, automated retraining and one-call rollback. Drive it from the console, call it from your services, scrape it with Prometheus.
drift → deploy
—
mean cycle time
requests served
—
last 48h
p95 latency
—
per inference call
pipeline success
—
last 48h
versions registered
—
immutable artifacts
connecting to the control plane
the problem
A model is not shipped when it is deployed.
Production populations move. Scores drift before accuracy does, and accuracy drifts before anyone files a ticket. Most teams find out from a business metric weeks later, then retrain by hand.
Silent decay
Covariate shift degrades a model long before an alert fires, because nothing is watching the input distribution.
Manual retraining
Notebook → pickle → someone's laptop → production. No gate, no lineage, no way to answer 'why is this version live?'
Risky releases
New weights go to 100% of traffic at once, and rollback means finding the previous artifact by hand.
the loop
Seven stages, one closed loop.
Flywheel treats the lifecycle as a control system rather than a checklist. Each stage is a persisted pipeline step with streamed logs, a machine-readable result and an audit trail.
- 01
Version & profile data
Every dataset is immutable and profiled on ingest: quantile bins, category mass, summary statistics. That profile is the reference distribution for drift.
- 02
Validate the contract
Row counts, target domain, class balance, missingness, duplicates, constant columns. A failing contract stops the run before compute is spent.
- 03
Train a candidate
Scikit-learn pipelines with imputation, scaling and one-hot encoding baked into the artifact. Threshold selected by an F1 sweep, not assumed at 0.5.
- 04
Champion vs challenger
Both the incumbent and the candidate are scored on the same freshly sampled evaluation set, so the comparison is not contaminated by the training split.
- 05
Promotion gate
Deterministic checks: absolute floor, train/test gap, calibration, and a required margin over the champion. Every check is recorded with its numbers.
- 06
Canary release
A slice of live traffic goes to the new version with sticky hashing. A controller compares realised ROC-AUC per arm and promotes or aborts on its own.
- 07
Monitor & close the loop
PSI, KS and chi-square against the baseline profile. A drifted verdict enqueues a retrain, which re-enters at stage 01. Rollback is one call away.
The measurable claim:the platform records the elapsed time from a drift verdict to the retrained model finishing its release. That single number — drift-to-deploy — is what turns “we have MLOps” into evidence.
architecture
Two planes, one deployable unit.
A control plane for lifecycle operations and a data plane for online inference, split by responsibility but shipped together so the whole thing runs on a single small container or scales out horizontally.
Runs anywhere
Zero-config local start on SQLite with disk artifacts. Point DATABASE_URL at Postgres and artifacts move into the database so a restarted container never loses a model. Add S3 and the same interface writes to object storage.
control plane · /api
data plane · /v1
background
state
The queue lives in the database, so the executor can run embedded in the API process for a single-container deploy or as N standalone workers behind the same Postgres — no broker to operate, same code path either way.
capabilities
What is actually built.
No placeholders.
Versioned data with real profiles
Generate, upload (CSV/JSON/Parquet) or capture datasets. Contract validation and statistical profiling on every version.
Model registry with lineage
Version → dataset → training run → metrics → deployment history. Artifacts live in Postgres, on disk or in S3 so ephemeral containers do not lose them.
Durable pipeline runs
The queue is a Postgres table. Workers claim runs with a conditional update, so one embedded worker or twenty containers use identical code.
Promotion gates you can audit
No silent releases. Each candidate carries the full gate decision, including the champion score it had to beat and the margin it cleared.
Canary + automatic rollback
Traffic splitting with sticky routing keys, per-arm realised AUC, automatic promotion or abort, and a one-call rollback to the archived version.
Drift detection from first principles
PSI over the baseline's own quantile bins, two-sample KS for numeric features, chi-square for categoricals, plus prediction and performance drift.
Policy, not hard-coding
Retrain thresholds, promotion metric, minimum improvement, canary share and check cadence are runtime policy the controllers read on every pass.
Observability that ops teams expect
Prometheus metrics for latency, throughput, drift and promotions. Server-sent events for live logs. Structured audit log for every mutation.
stack
Built on boring, verifiable pieces.
- Control plane
- FastAPI · SQLAlchemy 2 (async) · Pydantic v2
- Store
- Postgres (Neon-ready) or SQLite · Postgres/S3/disk artifacts
- ML
- scikit-learn · NumPy · pandas · SciPy · joblib
- Console
- Next.js 16 App Router · TypeScript · Tailwind v4 · Recharts
- Observability
- Prometheus · SSE · structured audit log
- Delivery
- Docker · docker compose · GitHub Actions · Helm/K8s manifests
Open the console and drive the loop yourself.
Shift the live population, replay traffic, watch drift get detected, then watch a retrained model clear the gate and take canary traffic — all from the browser.