rtx-3090

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

THE FIRST DLSS 5 MANAGER

Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig

llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

run a 27b local ai on your gaming gpu, rtx 3060 and up, radeon rx 6600 and up in beta, 8gb or more, linux and windows, one line to install

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

Run Qwen3.8-27B at 140 tok/s on a single RTX 3090, up to 2x llama.cpp. Fast local LLM inference server with OpenAI- and Anthropic-compatible APIs, 256K context, native Windows and Linux.

Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.

vLLM 0.33 fork for consumer GPUs: 3 x 262K sessions on Ampere,Ada and more! Qwen 3.8 Flash-Next at 207 tok/s, up to 17 on bigger cards. Seven checkpoints incl. Qwen3.8-27B, Nemotron 3.5 Super VL and Ornith 1.5. One command: merlin serve.

Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from the committed data by a checker that fails on drift, a coverage probe reports how many of them it actually covers, and ERRATA.md lists this study's own retracted claims.

Pre-driver FMLA32 Tensor Core unlocker and PTX benchmark console for GeForce RTX 40-series, RTX 30-series and RTX 20-series on Windows 10/11 (UEFI USB) and Linux (gpu_unlock.ko), with friendly UI for Win/Linux. Restores full 2x FP16/BF16/TF32->FP32 accumulate rate with automatic hardware profiling, pre-flight ROM checks, and zero VBIOS flashing.

The bare metal in my basement

Qwen3.8-27B on a single 24 GB GPU at 1.8–2× llama.cpp's decode speed. Bit-exact MTP speculative decoding, exact conversation caching, and OpenAI + Anthropic APIs for coding agents. Full 262k context on one 3090 with a near-lossless 6/5-bit KV cache (last 2,048 tokens kept at 8-bit), decodes 77.8 tok/s at full depth.

Local LLM speed measurements from a 4x RTX 3090 rig, with exact launch commands and per-run provenance.

What I run on my inference server: Qwen3.8-Flash-Next on two RTX 3090s, and how I got here

Qwen3.8 Flash Next AWQ INT4 4x3090 v1: 262K, Vision, FP8 QSA KV and CUDA Graphs on 4x RTX 3090

Universal one-click UEFI pre-boot rate-limiter unlock for NVIDIA GeForce RTX 40 & 30 Series (Ada Lovelace & Ampere). Uncaps full tensor performance across Windows, PyTorch, CUDA, and WSL2.