Oh My Pi is the coding agent. NInfer keeps your session's continuation state resident on your GPU. OMP NInfer is the qualified release that binds them with one hash-pinned Qwen3.8 27B model on an RTX 5090, 4090, or 3090 you own.
Every install route in the guide was run end to end on v0.19.0 with unmodified Oh My Pi 18.20.0 — 31 steps, three client platforms, both native lanes (receipt).
Download v0.19.0 · Lanes · Facts · Compare · Benchmarks · Architecture · Security · Troubleshooting · Roadmap · Changelog
Long coding sessions reach 100K tokens of transcript, thinking, tool calls, and images. Cloud providers meter every token and receive every file; stateless local servers re-send the whole transcript each turn. Live NInfer, llama.cpp, and vLLM processes already reuse an unchanged prefix; that is prior art, not this project's claim. OMP NInfer adds continuation you can rely on:
- Explicit and durable. Oh My Pi drives NInfer through stateful OpenAI Responses
(
previous_response_id), with qualified forks, rollback, and checkpoint restore across restarts. Oh My Pi writes the transcript first, so lost state falls back to replay, never a broken session. - Private and fail-closed. Runtime listeners bind loopback only, every request is bearer-authenticated, and model fallback is off: an unreachable GPU fails the turn instead of answering from a cloud model. This is acceptance-tested, not only configured.
- Exact and verifiable. Model SHA-256, container digest and SBOM, native package and client
checksums, and every qualification receipt are bound in one manifest;
python3 scripts/verify_release.py --require-readyproves your clone is the qualified release.
Recorded on v0.14.0 (RTX 5090): LRU bug fixed, tests green, ninfer-serve restarted mid-session, follow-up answered reusing 10,375 retained prompt tokens. 32 s, idle gaps and the 36 s reload compressed · MP4 · provenance
Use OMP NInfer when you run Oh My Pi, own an RTX 5090, 4090, or 3090 on Windows 11, and want long private Qwen3.8 27B sessions that survive server restarts.
Use something else when you want a broad model catalog (Ollama), a desktop GUI (LM Studio), the widest hardware reach or Apple silicon (llama.cpp), or multi-user throughput (vLLM). Candid comparison: decision guide.
All three lanes share a 131,072-token context, DFlash2 speculative decoding with draft window 7,
JSON-schema output, and the same 20.4 GB Qwen3.8 27B artifact (81f924d4).
| Lane | Runs as | Serves | Decode | Exact retrieval, 130,048 tokens |
|---|---|---|---|---|
| RTX 5090 | Docker Desktop/WSL2 container on Windows 11; clients: Windows (local), macOS over a manual SSH tunnel (preview), Linux under WSL2 | Two requests in flight; vision, tools, and preserved thinking | 181.71 tok/s | 56.67 s |
| RTX 4090 | Native Windows 11 package; Windows client (local) | One request at a time; text and tools | 253.71 tok/s | 83.45 s |
| RTX 3090 | Native Windows 11 package; Windows client (local) | Two requests in flight over one shared 131,072-token pool, under a 350 W serving cap; text and tools | 227.19 tok/s | 188.45 s |
Each number links to its qualification receipt. RTX 5090 retrieval is from a cold start; native decode is measured with one request at a time.
Both native lanes install on one two-GPU host, each bound to its GPU UUID, and run one at a time on
port 18082. Checkpoints are best effort and restore only on the same runtime, model, profile, and
credentials. In v0.19.0, the RTX 5090 and RTX 3090 runtimes stop retrying a refused
memory-pressure checkpoint save after six retries or 10 seconds; the RTX 4090 package does not
include that bound. Exact component, source, and model identities:
facts · glossary ·
manifest.
- Choose your lane and check its prerequisites.
- Clone tag
v0.19.0and runpython3 scripts/verify_release.py --require-ready. - Install the stock Oh My Pi client with the fail-closed config and per-model fragment.
- Run the acceptance checks at the end of your lane's route: tool turn, continuation, restart survival, fail closed.
Already running an earlier release? Follow the
upgrade steps. Native owners staying on
v0.10.0 keep the
v0.10.0 guide.
Oh My Pi (coding-agent client) → OMP NInfer (qualified appliance and release layer) → NInfer (inference engine) → Qwen3.8 27B (served model).
- Oh My Pi owns the truth. A turn advances provider state only after a complete stream and a durable transcript write.
- NInfer owns inference and retained state. The C++/CUDA engine from Neroued/ninfer and its GPU ports, plus this project's explicit Responses state and checkpoints. Retained state accelerates; it is never the source of truth.
- The manifest owns identity. One manifest binds client, runtime, model, profile, and receipts;
readystatus requires end-to-end acceptance of every documented route.
Architecture · Security model · Release lifecycle
Each lane is measured on its own gate, machine, and profile; the figure is not a ranking. Method, per-lane charts, the warm-versus-cold boundary rules, model quality, and the community table: benchmarks.
python3 scripts/verify_release.py --require-ready
python3 -m unittest discover -s tests -vBoth pass on a tagged release. A ready manifest binds the upstream Oh My Pi binaries, the model
SHA-256, the compatibility matrix, the container digest and SBOM or the native package checksums,
per-lane qualification receipts, and owner-operated acceptance of every documented route.
Lifecycle and cut rules: releases.
| Document | What it answers |
|---|---|
| Quickstart | Which lane, prerequisites, install steps, acceptance checks, operation |
| Facts | Current release identities, limits, non-claims, glossary |
| Decision guide | This versus Ollama, LM Studio, llama.cpp, vLLM |
| Compatibility | Qualified client and runtime combinations (generated) |
| Architecture | Topology, state ownership, identity chain, component boundaries |
| Benchmarks | Current results, how to read them, method, past measurements |
| Performance | Kernel program: baseline, open ideas, experiment ledger |
| Security | Trust boundary, invariants, key handling, retention, verification |
| Troubleshooting | Failure symptoms, causes, and recovery actions |
| Related work | Credits, prior-art survey, NInfer lineage, direct-route rationale |
| Upstream | Upstream positions, watch procedure, fork-point rationale |
| Releases | Current release, upgrade steps, how a release is cut, release records |
| Roadmap | Scope, open work, v1.0 criteria, decision records, shipped table |
| Changelog | What changed, per release |
| Media provenance | Checksums and capture notes for every recording |
- Ran a clean install? Friction is a bug; report time-to-first-turn and every manual step: clean-install report.
- Measured your card? After the acceptance checks pass, submit the performance result form.
- Work on CUDA kernels? Pick a bottleneck from the performance program and send before/after receipts.
- Need another model or profile? File a model/profile request.
- Contributing without a GPU? Offline rehearsal needs no GPU and never qualifies a release: contributor acceptance and doctor.
Routing rules: CONTRIBUTING.md. Strip keys, hostnames, prompts, outputs, and raw
logs before filing. Support covers v0.19.0 and its exact profiles for a single local owner; no
SLA. Vulnerabilities: private reporting, never a public issue.
- Oh My Pi by can1357, built on Pi by Mario Zechner (MIT). The client is the unmodified upstream binary, and this project publishes no client.
- NInfer by Neroued (Apache-2.0). The C++/CUDA engine,
.ninferformat, MTP and DFlash2 speculative decoding, and evaluation campaigns. Public runtime source for all three lanes: alphastorm/ninfer. - The Qwen team (Apache-2.0). Qwen3.8 27B, served from the registered conversions at neroued/Qwen3.8-27B-NInfer.
- UDPSendToFailed/ninfer-4090 and Don-Chad/ninfer-3090 (Apache-2.0). The RTX 4090 and RTX 3090 ports the native lanes build on.
- Gated DeltaNet (arXiv: 2412.06464), Tri Dao's ReplaySSM note,
Z-Lab's DFlash, Unsloth's NVFP4 weights, and vendored
utf8proc,nlohmann/json, andcpp-httplib; tracked in upstream.
Community project; not affiliated with or endorsed by Oh My Pi, Qwen, or NVIDIA.
MIT. NInfer and the Qwen artifacts retain their own licenses and notices.