A C++/CUDA inference engine for Qwen3.8-27B on one RTX 4090 (24 GB, Ada, sm_89). It is a port of Cinference (RTX 5090), which is derived from NInfer. See Attribution.
Production configuration: MTP speculative decoding with a fixed 7-token draft window (K7), K8V4 KV cache (FP8 keys, NVFP4 values), 188,416-token engine context, concurrency 1.
| Workload (greedy, thinking off unless noted) | Prompt tokens | K0 tok/s | K7 tok/s | K7 draft acceptance |
|---|---|---|---|---|
| Needle recall | 8,219 | 51.7 | 275.2 | 0.99 |
| Needle recall | 130,421 | 42.9 | 226.1 | 0.99 |
| Needle recall | 189,004 | 39.7 | 209.4 | 0.99 |
| Structured JSON | 77 | 53.0 | 228.0 | 0.72 |
| Reasoning (thinking on) | 122 | 52.9 | 179.9 | 0.54 |
| Prose | 34 | 52.7 | 87.7 | 0.19 |
K0 is plain autoregressive decode (no speculation); it is the raw decode baseline. Speculative throughput depends on how often the target model accepts the drafted tokens, so it varies by roughly 3x across workloads. These 2026-09-23 figures use greedy decoding on the stock model. The 200+ tok/s figures apply to predictable output (retrieval, structured data, code), not to temperature-1 Claude Code traffic. Current-branch measurements below keep the two workloads separate.
Measured 2026-09-23 on one RTX 4090 with
tools/bench/cinf_matrix.py. The raw per-request data is in
results/rtx4090-20260923/.
Decode tok/s: completion tokens after the first token, divided by decode wall time. Values are medians of 2 repetitions for the short workloads and 1 run for the recall workloads.
| Workload | Prompt tokens | K0 | K3 | K5 | K7 | K10 |
|---|---|---|---|---|---|---|
| short answer | 24 | 78.8 | 134.1 | 115.0 | 112.2 | 97.3 |
| structured JSON | 77 | 53.0 | 156.9 | 176.2 | 228.0 | 200.1 |
| code | 69 | 52.7 | 155.2 | 177.9 | 209.1 | 206.7 |
| reasoning (thinking on) | 122 | 52.9 | 153.0 | 150.0 | 179.9 | 176.4 |
| prose | 34 | 52.7 | 98.2 | 86.5 | 87.7 | 74.6 |
| long generation (~2.5K tokens) | 55 | 52.5 | 105.7 | 95.9 | 96.6 | 90.2 |
| needle recall 8K | 8,219 | 51.7 | 170.2 | 218.2 | 275.2 | 312.5 |
| needle recall 32K | 32,686 | 49.5 | 163.8 | 209.8 | 263.1 | 295.0 |
| needle recall 64K | 65,275 | 47.2 | 154.9 | 198.0 | 247.8 | 289.3 |
| needle recall 131K | 130,421 | 42.9 | 142.1 | 181.8 | 226.1 | 241.0 |
| needle recall 189K | 189,004 | 39.7 | 132.4 | 169.1 | 209.4 | 232.5 |
Round latency (one draft plus one verify step), in ms:
| Setting | Short prompt | 189K prompt |
|---|---|---|
| K0 (one token per step) | 18.9 | 25.2 |
| K3 | 22.2 | 30.2 |
| K7 | 26.6 | 38.2 |
| K10 | 30.5 | 45.9 |
Draft acceptance at K7:
| Workload | Acceptance |
|---|---|
| recall | 0.99 |
| structured | 0.72 |
| code | 0.65 |
| reasoning | 0.54 |
| long generation | 0.23 |
| prose | 0.19 |
Prefill and time to first token (TTFT) at K7:
| Prompt tokens | TTFT | Prefill tok/s |
|---|---|---|
| 8,219 | 3.6 s | 2,285 |
| 32,686 | 15.2 s | 2,153 |
| 65,275 | 33.1 s | 1,971 |
| 130,421 | 77.7 s | 1,680 |
| 189,004 | 127.4 s | 1,485 |
Prefill speed does not depend on K.
The integrated branch samples each positive-temperature MTP proposal from its own distribution
q (top-12 support) and uses target probabilities p for acceptance and the residual
(p-q)+ after rejection. The target sampling settings are unchanged. A multiblock selector
keeps proposal selection inexpensive for the wide Q4 draft head. Greedy requests retain the
greedy proposal route. Tests cover generated q passed through target verification, rejection,
and residual correction.
The stock-model K7 greedy matrix was rerun on 2026-09-24 with the integrated binary, the same script and settings as the table above (two repetitions for short cases, one for recall). The current results are essentially the same as the 2026-09-23 K7 column; the other K columns above are historical controls and were not rerun.
| Workload | Previous K7 tok/s | Current K7 tok/s |
|---|---|---|
| short answer | 112.2 | 111.4 |
| structured JSON | 228.0 | 221.6 |
| code | 209.1 | 209.2 |
| reasoning (thinking on) | 179.9 | 179.9 |
| prose | 87.7 | 88.8 |
| long generation | 96.6 | 97.3 |
| needle recall 8K | 275.2 | 275.5 |
| needle recall 32K | 263.1 | 263.4 |
| needle recall 64K | 247.8 | 248.1 |
| needle recall 131K | 226.1 | 226.1 |
| needle recall 189K | 209.4 | 209.6 |
All six recall markers passed at every length. The code workload's generated unit tests failed, as they did in the original matrix. The current K7 summary contains the measured rates, acceptance, TTFT, and checks.
A separate replay measured the workload that motivated sampled drafting: 32 successful temperature-1 Claude Code requests on the abliterated model, in captured order with byte-identical request bodies, prefix reuse, K7, Q4 draft head, K8V4, and seed 42. Both builds used the same prompts and production flags. This is a decode-rate comparison, not the greedy matrix above:
| Captured production replay | Greedy drafts | Sampled drafts | Change |
|---|---|---|---|
| Aggregate decode tok/s | 127.4 | 141.6 | +11.1% |
| Tokens per round | 3.80 | 4.20 | +10.5% |
| Mean ms per round | 29.82 | 29.68 | -0.5% |
The sampled build was faster on 27 of 32 paired requests (paired decode-rate geometric mean +12.5%); both runs returned HTTP 200 for all 32. Output lengths and tool-call counts differed, so this replay does not assess task quality or imply matching text at a fixed seed. Exact rejection correction is intended to preserve the target sampling distribution, which is covered by the focused sampling and speculative-round tests. The private request bodies are not redistributed; aggregate replay data records the measurement.
The previous FNN deployment was the NInfer RTX 4090 port
running at K3 with its rk4v4-e8 KV cache. It was measured with the same tool and workloads
(oldprod-k3-summary.json).
| Workload | Previous (K3) | This (K7) | Change |
|---|---|---|---|
| structured JSON | 151.4 | 228.0 | +51% |
| code | 154.9 | 209.1 | +35% |
| reasoning | 147.6 | 179.9 | +22% |
| prose | 99.0 | 87.7 | −11% |
| long generation | 102.9 | 96.6 | −6% |
| needle recall 8K | 171.4 | 275.2 | +61% |
| needle recall 32K | 156.4 | 263.1 | +68% |
| needle recall 131K | 134.5 | 226.1 | +68% |
Interpretation: a fixed K7 window helps when acceptance is high and costs throughput when it is low. Each K7 round is about 4.4 ms slower than a K3 round, and low-acceptance text doesn't gain enough extra accepted tokens to pay for it. K3 remains faster for prose and long free-form generation. K7 was chosen because it leads or ties on structured output, code and reasoning, and loses less on prose than K10.
- Hardware: one RTX 4090 (24 GB), Linux, CUDA 13.x. One server process per mode, started
with
--greedy --no-prefix-reuse, so every request prefills from scratch. - Workloads:
- short factual answer
- JSON extraction (validated)
- Python with unit tests (extracted and executed)
- prose
- arithmetic reasoning with thinking on (answer checked)
- ~2.5K-token long generation
- synthetic needle retrieval with 6 markers at 8K–189K prompt tokens
- Recorded:
- TTFT, prefill tok/s and decode tok/s
- ms per round and draft acceptance, from the server's request log
- GPU memory and utilization, sampled every 0.5 s from
nvidia-smi - KV fill, output SHA-256, and the first divergence from K0
- Tool:
python3 tools/bench/cinf_matrix.py --out bench_results_matrix \
--modes k0,k3,k5,k7,k10 --recall-tokens 8192,32768,65536,131072,190000 \
--reps 2 --max-context 196608 --kv-dtype k8v4 \
--bin build-sm89/apps/ninfer-serve --model models/<artifact>.ninferThe matrix used a 196,608-token capacity to reach the 189K prompt. Production uses 188,416 tokens for VRAM headroom (see below).
Measured facts:
- Needle recall: all 6 markers are retrieved at every context and every K. The output is byte-identical to K0.
- Short answer: the output is byte-identical to K0 at every K.
- Structured JSON: passes validation at every K. It is identical to K0 at K3, K5 and K10. At K7 it differs by one token (170 vs 171).
- Prose, reasoning and long generation: outputs diverge from K0 after the first few tokens at every K > 0, but all still pass their checks, including the reasoning answer.
- Code: the generated unit tests fail at every K, including K0, so this reflects the model's output rather than the speculative path.
- Determinism: repeated runs with the same mode are byte-identical.
- KV value decode: the new NVFP4 decode is bit-identical to the previous implementation over all 256 byte values and every legal E4M3 scale code. This was checked with an exhaustive device test during development; the test is not committed.
- Attention tests:
ninfer_softmax_attention_test --k8v4-onlypasses, including the new verify widths 9–16, CUDA Graph replay, and contexts up to 32,768.
Interpretation: divergence under speculation comes from numerical differences between the batched verify path and one-token decode. Greedy selection amplifies these into different continuations. Retrieval outputs, which are strongly determined by the prompt, don't change. Perplexity was not re-measured. Prefill changed only through the bit-identical V decode.
On top of the Cinference RTX 4090 port (MTP-10, Ada fallbacks, K8V4 cache):
- Sampled MTP drafting at positive temperature: draft probabilities are carried into exact sparse target rejection correction, with a multiblock top-12 proposal selector for the 131,072-row Q4 head. Greedy decoding continues to use greedy drafts.
- Faster NVFP4 V-cache decode: E2M1 nibbles are placed directly into FP16 bits and
scaled with two
HMUL2s, replacing a per-nibble switch (src/ops/kv_cache/nvfp4_group16_codec.cuh). Measured during development at 25K keys: T1 attention went from 240 to 101 µs, and T11 from 594 to 191 µs. - Single-pass K8V4 attention for verify widths 9–16 at batch 1, replacing the chunked
two-pass path (
src/ops/softmax_attention/). - Ada small-T tensor-core kernels for Q4/Q5 projections at T = 2–16: attention input,
SwiGLU,
linear_add, and GDN value/z. The weights are read once per verify round (src/ops/linear/ada_small_t_mma.cuh). - Serving and benchmark tooling:
- a Prometheus
/metricsendpoint (llamacpp:prompt_tokens_total,llamacpp:tokens_predicted_total, decode rounds, request gauges) ninfer_qwen3_5_mtp_round_benchfixed for K ≤ 4tools/bench/cinf_matrix.py,cinf_k_bench.pyandcinf_profile.py(nsys)
- a Prometheus
Measured with K8V4 KV and MTP enabled:
| KV capacity (tokens) | Free VRAM after reservation |
|---|---|
| 131,072 | 2.42 GiB |
| 163,840 | 1.59 GiB |
| 188,416 (production) | 997 MiB |
| 196,608 | 937–965 MiB |
| 229,376 / 262,144 | fails to start |
At 196,608 tokens without speculation (K0), 2.00 GiB is left free. Peak process memory is about 22.0 GB at K0 and 23.1 GB with MTP. The native model context is 262K. 188K is the practical ceiling on 24 GB with about 1 GiB of headroom.
- Fixed K: fixed K7 is 6–11% slower than the previous engine's K3 on prose and long generation. Adaptive per-request K is not implemented; it is future work.
- Context: the ceiling is 188K on 24 GB, versus 262K native.
- Long-context TTFT: TTFT grows superlinearly, to 78 s at 131K and 127 s at 189K.
- Concurrency: 1.
- Output divergence: outputs diverge from K0 on open-ended text (see Correctness).
- Remaining cost per K10 round:
- The Q5 17,408-wide projection kernel reaches about 520 GB/s.
- Drafting (the MTP layer plus the proposal head, about 9 ms) is bandwidth-bound. Reducing it requires a smaller draft head in the artifact, not kernel work.
- Blackwell-only paths: native NVFP4 weight kernels and TMA prompt kernels fail closed on
Ada.
ninfer_softmax_attention_testin full or--nvfp4-onlymode aborts on those stubs. - Model artifact: the benchmarked artifact is not redistributed here. See Model artifact for how to recreate it.
All results use qwen3_8_27b.v3.ninfer: stock Qwen3.8-27B (not an abliterated or fine-tuned
variant). It has a Q4/Q5 groupwise-int Text body with Q8 (q8_g32_fp16) token embedding and
output head, and includes the MTP and proposal-head weights. The quantization choices are
upstream's; this port doesn't requantize anything.
- Source: neroued/Qwen3.8-27B-NInfer
at revision
3526913004b1cf552cb57b88d6a5c6f5e4a89a70. That is the v2 container,qwen3_8_27b.ninfer, 18,210,531,328 bytes, SHA-256eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e. The currentmainrevision is a different, 20.4 GB v3 artifact with DFlash2 weights, and it was not benchmarked. - Conversion: upgrade the container with
tools/upgrade_ninfer_v2_to_v3.py. This file is unchanged from upstream Cinferenceb74044f; its SHA-256 ise110228d…c8b54. The script copies weight bytes unchanged and installs the maintainedqwen3_8.jinjachat template. - Output: 18,210,749,936 bytes. The benchmarked file's SHA-256 is
5f67e9e4cfc7d17405247d37faddd7f9fd86c47810fce9c42785b259d68c274d.
hf download neroued/Qwen3.8-27B-NInfer qwen3_8_27b.ninfer \
--revision 3526913004b1cf552cb57b88d6a5c6f5e4a89a70 --local-dir models/v2
echo 'eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e models/v2/qwen3_8_27b.ninfer' | sha256sum -c
python3 tools/upgrade_ninfer_v2_to_v3.py models/v2/qwen3_8_27b.ninfer models/qwen3_8_27b.v3.ninferThe upgrade writes a random 16-byte container identity (bytes 17–32), so a rebuilt file's SHA-256 won't match. Re-running the upgrade on the original machine produced a file that differs from the benchmarked one only in those 16 bytes. To check a rebuild:
cmp -l models/qwen3_8_27b.v3.ninfer /path/to/reference.v3.ninfer | awk '$1<17||$1>32' | wc -l # 0FNN currently serves
windowsxp811203/Qwen3.8-27B-Abliterated,
revision 40519381cd229c0bd9a54470122e59d592ed534b. That model is stock Qwen3.8-27B with a
refusal direction projected out of 131 residual-writing tensors; its config.json is identical to
stock. It was converted from the BF16 weights with the same official recipe as the stock artifact:
python3 -m tools.convert --model /path/to/Qwen3.8-27B-Abliterated --recipe qwen3_8_27b \
--components text,vision,mtp \
--resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
--proposal --name qwen3.8-27b --out models/qwen3_8_27b.ablit-wxp.ninfer- Tooling: converter from commit
b40d66d, run with PyTorch 2.14 on CUDA. Conversion took 127 s. - Output: 18,210,704,128 bytes, SHA-256
b7c6693244cf7ab08554fff41c4f3c67695b6ec388c99eed6f12e0e71046fda8. - Checks (production flags, 1 repetition,
results/rtx4090-20260923-abliterated/): every check passes at K0 and K7.
| Workload | Abliterated K7 tok/s | Acceptance | Stock K7 tok/s | Acceptance |
|---|---|---|---|---|
| short answer | 111.3 | 0.71 | 112.2 | 0.71 |
| structured JSON | 236.4 | 0.75 | 228.0 | 0.72 |
| prose | 88.6 | 0.19 | 87.7 | 0.19 |
| reasoning | 164.6 | 0.49 | 179.9 | 0.54 |
| needle recall 32K | 264.2 | 0.99 | 263.1 | 0.99 |
K0 decode is 52.8–53.1 tok/s, the same as stock. The 32K recall output is byte-identical to K0. The server starts at 188,416 tokens with 997 MiB free, the same as stock. A request that stock models typically refuse was answered.
Requirements: Linux, CUDA 13.x, CMake ≥ 3.28, Ninja, and an RTX 4090 or another sm_89 GPU.
cmake -S . -B build-sm89 -G Ninja -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CUDA_ARCHITECTURES=89 -DNINFER_BUILD_APPS=ON \
-DNINFER_BUILD_BENCHMARKS=ON -DBUILD_TESTING=ON
ninja -C build-sm89 ninfer-serve ninfer_softmax_attention_test ninfer_qwen3_5_mtp_round_bench
./build-sm89/tests/ninfer_softmax_attention_test --k8v4-onlyLeave out -DBUILD_TESTING=ON, -DNINFER_BUILD_BENCHMARKS=ON and the extra targets to build
only the server. See also build system and
PORT-RTX4090.md.
The production flags:
./build-sm89/apps/ninfer-serve models/<artifact>.ninfer \
--host 127.0.0.1 --port 8080 --model-id qwen/qwen3.8-27b \
--max-context 188416 --kv-capacity 188416 --kv-dtype k8v4 \
--max-concurrency 1 --max-pending-requests 16 --pending-timeout-ms 600000 \
--prefill-chunk 1024 --spec mtp --draft-tokens 7 --lm-head-draft \
--preserve-thinking --default-max-tokens 8192The server exposes the OpenAI Chat Completions and Responses APIs, the Anthropic Messages API,
/health and /metrics. See docs/serving.md and docs/cli.md.
The deployment files are in deploy/fnn/:
- Copy
cinference-qwen38-4090.env.exampleto~/.config/cinference/qwen38.env(mode 600) and set absolute paths. The live configuration stays outside the checkout, sogit pullcan't change or remove it. - Adjust
WorkingDirectoryincinference-qwen38-4090.serviceif the checkout lives elsewhere. - Install it as a user unit:
cp deploy/fnn/cinference-qwen38-4090.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now cinference-qwen38-4090
loginctl enable-linger "$USER" # keep running without a login sessionThe unit restarts on failure after 5 s. In testing it recovered from kill -9 in 17 s. Point
CINF_BIN at a copied release binary rather than the build tree, so rebuilds don't affect the
running service.
To roll back, the unit declares Conflicts= with the previous ninfer-qwen38-4090.service:
systemctl --user disable --now cinference-qwen38-4090
systemctl --user enable --now ninfer-qwen38-4090- NInfer by Neroued and contributors: the engine, the
.ninferformat, the model cards and most of the source. - Cinference by satellitedown: MTP-10,
capture-derived CUDA Graph reuse, and the RTX 5090 results in
results/rtx5090-archive-recall.json. - ninfer-3090 by Don-Chad: the GitHub fork parent of this repository.
- RTX 4090 port and the changes listed above: Joshua Ramirez.
Apache License 2.0; see LICENSE and NOTICE. Vendored dependencies keep their own licenses. No model weights are included.