Distribute and run LLMs with a single file.
High-speed Large Language Model Serving for Local Deployment
Mano-P: Open-source GUI-VLA agent for edge devices. #1 on OSWorld (specialized, 58.2%). Runs locally on Apple M4 Mac mini/MacBook — no data leaves your device.Mano-P 是一个开源 GUI-VLA 项目，支持在 Mac mini/MacBook 上或通过算力棒本地运行推理，实现纯视觉驱动的跨平台 GUI 自动化操作。数据完全本地处理，支持复杂多步骤任务规划与执行。
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
Go With Your Own Intelligence! Use Go for hardware accelerated local inference with llama.cpp, whisper.cpp, and stable-diffusion.cpp directly integrated into your Go applications. Kronk provides a high-level API and production ready model server.
Free AI coding environment: Ollama + Python
斑海豹 — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions. 小海豹，大决断。
Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig
SOTA model-agnostic quantization for GGUF, local inference and beyond. Quantize and run text, image, video and audio models with llama.cpp, vLLM, Diffusers and PyTorch.
[ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.
Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant
InferrLM - On-device AI for iOS & Android
Local LLM inference on Xbox Series S in Developer Mode: llama.cpp (GGUF) + ONNX Runtime GenAI, an OpenAI-compatible LAN endpoint (off by default), SD-Turbo image generation and experimental on-device fine-tuning. Not yet tested on Series X. Independent project, not affiliated with Microsoft.
Real browser automation with local GLiNER2 inference and open weights. Work in progress. Contributions welcome.
Local proxy that learns your app's typed LLM decisions and answers them with a Laya head. Jev and OpenAI compatible.
An open-source, coding agent harness designed specifically for local LLMs. Designed as everything a user needs to jumpstart their own harness (detailed event steams, logs, and performance data) or quickly replace closed source one (featuring out of the box plugins for web search, image gen, memory, auto research and more in dev).
Run real AI models on the GPU you already own: language, speech, image and video models from one engine and one container format, with or without PyTorch. NVIDIA GPUs, Mac GPUs through Metal, CPU; other GPUs experimental. Apache 2.0.
Local MiniMax H3 Image generation and editing on Apple Silicon with MLX, with ultra-low ram
Self-hosted Thai AI infrastructure — Mac Mini M4 Pro running TURBO-Fable 27B at 18.5 tok/s for $0.54/day. Fleet of 9 machines, GOD CONFIG v7.0, 40 IRON safety rules. Production-grade Open WebUI + LiteLLM + Ollama MLX stack.
To associate your repository with the local-inference topic, visit your repo's landing page and select "manage topics."