Calibrated price-range forecasts for the Counter-Strike 2 skin market.
Write-up · Quickstart · How it works · Docs
CS2 skins trade across a dozen marketplaces with no consolidated price feed. CS2 Oracle aggregates seven of them into one daily price per item. For every item it forecasts a calibrated price range at 3, 7, 14 and 30 days.
It runs unattended on GitHub Actions. Most of the engineering went into the evaluation harness, because on this problem an impressive-looking number is easy to produce and usually wrong.
Stack: Python · LightGBM · split-conformal prediction · DuckDB over Parquet · PostgreSQL (Supabase, being retired for Parquet) · FastAPI · GitHub Actions
| Output | Meaning |
|---|---|
| Price band | Split-conformal interval targeting 80% coverage |
| Centre | The band's midpoint |
move_odds |
Calibrated probability that the move clears round-trip trading costs |
anomaly_p |
Calibrated probability of an abnormal move (3/7/14d only) |
Directional accuracy is tracked but not claimed. The market trends for long enough that a constant up or down call can look skilful while carrying no information.
7 market feeds ──► Aggregator ──► Parquet archive ──► Train ──► Calibrate ──► FastAPI
(CSGOTrader dumps) outlier-voted (DuckDB, 2013–) LightGBM conformal (Supabase)
median per horizon band + isotonic
│
Backtest ◄──────────────────┘
resolves every forecast against the archive
The daily chain is Aggregator → Price Forecast → Backtest. Each step triggers when the previous one succeeds. Freshness and schema-drift checks run alongside it.
| Stage | Detail |
|---|---|
| Aggregate | Source-quality-weighted median; >2σ quotes rejected, 3-source minimum |
| Train | LightGBM per horizon: return regressor, direction classifier, exceedance and anomaly heads. Inputs are price technicals only; 85 non-price features added no measurable accuracy in a 7-fold ablation |
| Calibrate | Signed split-conformal band scaled by per-item climatology; isotonic calibration of the probabilities; served-outcome feedback gated per horizon |
| Evaluate | MAE, MAPE, interval score and directional accuracy with a Pesaran–Timmermann test. Challenger centres run in shadow and are scored through the same backtest before any manual promotion |
Requires Python 3.11+, uv and a PostgreSQL database.
uv sync --extra dev # add --extra mlops for MLflow tracking
cp backend/.env.example backend/.env # set DATABASE_URL, STEAM_API_KEY, SECRET_KEY
cd backend
uv run python scripts/run_task.py migrate
uv run uvicorn main:app --port 8000 --reloadEmpty database? Create the Alembic version table first
Some revision IDs are longer than Alembic's default 32 characters:
CREATE TABLE alembic_version (
version_num VARCHAR(255) NOT NULL,
CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);| Endpoint | Returns |
|---|---|
GET /items/{id}/prediction |
An item's band, centre, move_odds and anomaly_p |
GET /items/volatility |
Items ranked by expected swing or move_odds |
GET /accuracy/summary |
Aggregate backtest metrics, filterable by price tier |
GET /monitoring/health |
Service and data-freshness status |
Set DEBUG=true in .env to serve interactive docs at /docs.
uv run pytest # fast gate; excludes tests marked slow
uv run pytest -m slow # real-trainer tests, several minutesCI also runs ruff and mypy.
backend/
api/ FastAPI routes, schemas, serving policy
collectors/ Aggregator, backfill, supply and volume scrapers
models/ Forecaster, conformal, recalibration, data quality, source weights
backtest/ Price resolution, scoring, directional test, promotion gate
db/ Parquet store, archive reader, candidate store
migrations/ Alembic revisions
scripts/ Task runner and batch entrypoints
tests/ Pytest suite
docs/ Write-up, architecture, specs and plans, research, changelog, experiment log
.github/ Daily chain, freshness, schema drift, lint
The price archive and the forecast record live in a separate private repo, cs2-oracle-data.
Only CI writes to it, and closed months are also published as checksummed releases. The local
price-archive/ directory is a gitignored, read-only copy.
Every prediction is stored with its anchor date and horizon. The backtest resolves it once the archive covers the target date.
- One code path. Both legs of every realised return resolve through
backtest.price_resolution. - Frozen outcomes. Resolved results are never overwritten. Re-resolution is a separate, explicit step.
- Coverage-bounded maturity. A forecast matures when the archive covers its target date. Permanent gaps are marked unresolvable so they can't block later runs.
- Tiered reporting. Items at $1 and above are the headline cohort, because sub-$1 items behave differently.
- Baselined direction. Directional accuracy is always reported alongside the constant-call baseline and a date-clustered Pesaran–Timmermann test.
- Preregistration. Experiments commit their hypothesis, metric, pass bars and minimum sample before the data exists, and every harness includes a placebo arm.
Live figures are served at GET /accuracy/summary. No headline accuracy is quoted here
until the served series has the ≥20 distinct forecast dates the test needs.
| Workflow | Role |
|---|---|
| Aggregator Market Update | Daily price collection (23:00 UTC) |
| Price Forecast | Train and forecast, chained from the Aggregator |
| Backtest Accuracy | Resolves matured forecasts, chained from the forecast |
| Forecast Freshness Check | Alerts when no forecast landed |
| Schema Drift Check | Pre-merge gate: ORM models vs. Alembic migrations |
| Lint & Smoke Test | ruff, mypy, fast tests |
| Model Diagnostics | On-demand model scoring and A/B arms, one job per horizon |
| Ops Sync From Postgres | On-demand repair of the monthly forecast-record partitions |
| Archive Restore Day | On-demand repair of a deleted archive day |
Enable state lives in GitHub, not git. Check it with gh workflow list. The runbook is
docs/operations.md.
docs/PORTFOLIO.md: project write-updocs/: index, architecture notes and runbookdocs/changelog/: 240+ dated decision records with measured effectsdocs/research/: preregistrations and research notesdocs/experiment_log.csv: every shipped, refuted and void experiment, negative results included
