Model catalog

Google's Gemma 3n E4B-IT, a compact 4B-effective instruction-tuned multimodal-capable model. Optimized for efficient on-device inference with strong multilingual coverage.

LocalAI-compatible model gpt-4

LocalAI-compatible model gpt-4o

LocalAI-compatible model gpt-5.4-mini

Meta's Llama 3.2 1B Instruct, an ultra-compact instruction-tuned model designed for on-device and edge deployments. Suitable for lightweight chat, summarization, and prompt classification on CPU.

Microsoft's 3.8B parameter small language model with surprisingly strong reasoning and instruction-following for its size. Runs comfortably on CPU and outperforms many larger models on common benchmarks.

A 3B parameter instruction-tuned model from Alibaba's Qwen 2.5 series, offering a strong balance between speed and reasoning capability. Runs well on CPU and supports an 8K context window.

Alibaba's Qwen3 4B Instruct (2507 release), a compact dense model with improved reasoning and instruction following from the Qwen3 family. Runs comfortably on CPU at Q4.

Reasoning-tuned variant of Qwen3 4B (2507 release) with extended chain-of-thought style outputs. Compact enough for CPU but designed for step-by-step problem solving.

curl http://localhost:8080/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "<positive prompt>|<negative prompt>", "step": 25, "size": "512x512" }'

A tiny 1.1B parameter chat model that runs efficiently on CPU with minimal RAM. Ideal for low-resource environments, fast prototyping, and embedded use cases where response speed matters more than quality.

Google DeepMind's Gemma 2 9B instruction model featuring sliding window attention and logit soft-capping for high output quality. Strong performance on reasoning and chat tasks, optimized for GPU inference.

Google's Gemma 3 27B Instruct, the largest dense Gemma 3 variant with broad multilingual coverage and strong general assistant quality.

Z.ai's GLM-4.5-Air mixture-of-experts model. ~106B total parameters with sparse activation, providing competitive quality at lower active-parameter cost than dense 70B alternatives.

OpenAI's open-weight gpt-oss-120b model. Largest of the gpt-oss family, designed for high-quality assistant tasks. Requires very high VRAM.

OpenAI's open-weight gpt-oss-20b chat model, released as part of the OpenAI open-weights initiative. Strong general-purpose assistant in a 20B parameter footprint.

Meta's Llama 3 8B instruction-tuned model offering strong general-purpose chat and reasoning performance. Requires GPU for comfortable inference and delivers excellent quality for its size.

failspy's abliterated derivative of Meta Llama 3 8B Instruct, packaged as a Q4 GGUF for LocalAI llama.cpp chat inference. Intended for controlled research and evaluation where reduced refusal behavior is explicitly desired.

bartowski's Q4_K_S GGUF quantization of failspy Meta Llama 3 8B Instruct abliterated v3, targeting lower VRAM use than Q4_K_M while remaining suitable for LocalAI llama.cpp chat inference.

Meta's Llama 3.1 8B instruction model quantized at Q6_K for near-lossless quality. Adds 128K context support and improved multilingual capabilities over Llama 3, at the cost of higher VRAM usage.

Meta's Llama 3.3 70B Instruct. 70B-class chat quality with the improved 3.3 instruction tuning. Requires a high-end GPU at Q4.

Meta's Llama 4 Scout 17B-16E Instruct. Native mixture-of-experts model with 16 experts. Strong multimodal-capable architecture from the Llama 4 family.

LocalAI-compatible model minicpm-v-4.5-q4-k-m

Mistral AI's 7B instruction model v0.2 with a 32K token sliding-window context, strong instruction following, and high-quality output. A well-rounded GPU model balancing speed and capability.

Alibaba's Qwen 2.5 14B instruction model with near-frontier quality on reasoning, coding, and mathematics. Demands significant VRAM but delivers substantially better output than 7B-class models.

Alibaba's Qwen2.5 72B Instruct dense model. High-end chat and reasoning quality for self-hosted deployments with substantial GPU.

LocalAI-compatible model qwen2.5-7b-functioncall-q4-k-m

Alibaba's Qwen 2.5 7B instruction model excelling at reasoning, mathematics, coding, and multilingual tasks. Supports up to 128K context and delivers frontier-competitive performance for a 7B model.

Alibaba's Qwen3 14B dense model. Higher-quality alternative to the 8B version with substantially better reasoning at the cost of more VRAM.

Alibaba's Qwen3 30B-A3B mixture-of-experts model. 30B total parameters with ~3B active per token, offering 32B-class quality at near-8B inference cost.

Alibaba's Qwen3 32B dense model. Production-grade quality for chat, reasoning, and code with a sizeable VRAM footprint.

Alibaba's Qwen3 8B dense model with strong general chat, reasoning, and tool-use performance. A balanced default for GPU-backed self-hosted deployments.

Alibaba's Qwen3-Next 80B-A3B Instruct. Next-generation MoE with 80B total / ~3B active parameters per token. Highest Qwen3 quality tier accessible at near-8B serving cost.

Alibaba's QwQ-32B reasoning model. Specialized for chain-of-thought and multi-step problem solving with extended reasoning traces.

DeepSeek's 1.3B code model trained on 2 trillion tokens of code and natural language. Despite its tiny size, it delivers strong code completion performance and runs comfortably on CPU.

Alibaba's Qwen 2.5 Coder 1.5B instruction model, a compact code-completion and generation model optimized for CPU inference. Supports multiple programming languages and an 8K context window.

Alibaba's Qwen2.5-Coder 3B Instruct, an instruction-tuned coding model in a small footprint. Suitable for in-editor completion and code-aware chat on CPU-only nodes.

Stability AI's 3B code completion model trained on diverse programming languages. A lightweight CPU-friendly model well-suited for single-file code completion and generation tasks.

Meta's Code Llama 7B instruction model, fine-tuned from Llama 2 on code-heavy datasets. Supports code generation, completion, and infilling across many programming languages.

DeepSeek's 6.7B instruction-tuned code model trained on 2T tokens of code. Strong across multiple programming languages with fill-in-the-middle support. Requires GPU for practical inference speed.

DeepSeek Coder V2 Lite is a 16B MoE model (2.4B active params) with state-of-the-art coding performance. Supports 128K context, fill-in-the-middle, and outperforms GPT-4o on many coding benchmarks.

Alibaba's Qwen2.5-Coder 32B Instruct. Flagship dense coder model delivering state-of-the-art open-weight code generation, refactor, and reasoning over multi-file contexts.

Alibaba's Qwen 2.5 Coder 7B instruction model, excelling at code generation, debugging, and explanation across 92 programming languages. Supports a 16K context window with strong benchmark performance.

Alibaba's Qwen3-Coder 30B-A3B Instruct MoE coder. 30B total with ~3B active parameters per token. Balances large-model code quality with efficient serving cost.

BigCode's StarCoder2 15B, a flagship open code model trained on The Stack v2 (600+ languages). Delivers high-quality code completion and generation for professional codebases with a 16K context window.

BigCode's StarCoder2 7B trained on 600+ programming languages with permissive licensing. Strong fill-in-the-middle and code completion capabilities, especially for GitHub-style repositories.

BAAI's BGE Small English embedding model (33M params) producing 384-dimensional vectors. Lightweight and CPU-efficient with strong performance on English retrieval and semantic similarity tasks.

Nomic AI's 137M-parameter embedding model supporting both short and long documents (up to 8192 tokens). Achieves strong performance on MTEB benchmarks with fully open weights and training data.

BERT-based small English embedding model, aliased to text-embedding-ada-002 for compatibility. Lightweight and CPU-efficient with strong performance on English retrieval and semantic similarity tasks.

BAAI's BGE Base English embedding model (109M params) producing 768-dimensional vectors. A solid general-purpose English embedding model balancing size and retrieval quality, ideal for GPU-accelerated pipelines.

BAAI's BGE Large English embedding model (335M params) producing 1024-dimensional vectors. Top-tier English retrieval quality on MTEB, best used when embedding accuracy is more important than speed.

Jina AI's 137M-parameter English embedding model supporting an 8192-token context, far longer than most embedding models. Well-suited for document-level retrieval and long-context RAG pipelines.

Microsoft's multilingual E5 Base embedding model (278M params) supporting 100+ languages. Produces 768-dimensional vectors with strong cross-lingual retrieval and semantic similarity performance.

Microsoft's multilingual E5 Large embedding model (560M params) for 100+ languages. Delivers high-quality cross-lingual embeddings (1024-dim) at the cost of increased memory and compute requirements.

Mixedbread AI's 335M-parameter large English embedding model achieving state-of-the-art performance on MTEB. Produces 1024-dimensional vectors with strong retrieval, clustering, and semantic similarity results.

Stable Diffusion v1.5 Inpainting fine-tune by RunwayML, specialized for masked image editing and object removal/replacement. Accepts an image and a mask to regenerate only the selected region.

Stable Diffusion v1.5 by RunwayML, the foundational open image generation model. Generates 512px images with broad community support, extensive LoRA/fine-tune ecosystem, and reasonable CPU inference speed.

CagliostroLab's Animagine XL 4.0, an SDXL fine-tune specialized for high-quality anime-style image generation. Excels at character illustrations, scenes, and anime art with rich prompt support.

Black Forest Labs FLUX.1 Dev, a 12B-parameter guidance-distilled text-to-image model delivering top-tier image quality. Best-in-class prompt following and detail, requiring substantial VRAM for inference.

Black Forest Labs FLUX.1 Schnell, a 12B-parameter distilled text-to-image model optimized for speed, generating high-quality images in 1-4 steps. Commercially permissive Apache 2.0 license.

Playground AI's Playground v2.5, an SDXL-based aesthetic fine-tune achieving state-of-the-art human preference scores at 1024px. Excels at photorealistic and artistic generation with high visual quality.

Stable Diffusion XL Base 1.0 by Stability AI, generating high-fidelity 1024px images with improved prompt adherence. A major quality step up from SD1.5, requiring GPU for practical generation speed.

Stability AI's Stable Diffusion 3.5 Large (8B parameter MMDiT), delivering state-of-the-art text-to-image quality with excellent typography and prompt adherence at 1024px resolution.

BAAI's BGE Reranker Base, a 110M-parameter cross-encoder for re-ranking retrieval results. Significantly improves RAG pipeline precision by scoring query-document relevance pairs directly.

You can test this model with curl like this: curl http://localhost:8080/v1/rerank \ -H "Content-Type: application/json" \ -d '{ "model": "jina-reranker-v1-base-en", "query": "Organic skincare products for sensitive skin", "documents": [ "Eco-friendly kitchenware for modern homes", "Biodegradable cleaning supplies for eco-conscious consumers", "Organic cotton baby clothes for sensitive skin", "Natural organic skincare range for sensitive skin", "Tech gadgets for smart homes: 2024 edition", "Sustainable gardening tools and compost solutions", "Sensitive skin-friendly facial cleansers and toners", "Organic food wraps and storage solutions", "All-natural pet food for dogs with allergies", "Yoga mats made from recycled materials" ], "top_n": 3 }'

Jina AI's tiny 33M-parameter English reranker optimized for speed and CPU efficiency. A very fast cross-encoder for reranking that sacrifices some accuracy for minimal compute overhead.

BAAI's BGE Reranker Large, a 335M-parameter high-accuracy cross-encoder for English reranking. Delivers top-tier query-document relevance scoring for demanding retrieval pipelines.

BAAI's BGE Reranker v2-Gemma, built on a Gemma-2B backbone for state-of-the-art reranking accuracy. Substantially outperforms smaller rerankers on complex relevance tasks at the cost of higher VRAM.

BAAI's BGE Reranker v2-M3, a 570M-parameter multilingual cross-encoder built on bge-m3. Strong reranking quality across 100+ languages, suitable for multilingual RAG and search pipelines.

Jina AI's 278M-parameter multilingual reranker v2 supporting 100+ languages. High-quality cross-encoder scoring with strong multilingual retrieval performance and long-context (8192-token) support.

Kyutai's Pocket TTS, an ultra-lightweight text-to-speech model running entirely on CPU with ~200ms latency. Includes 8 built-in voices (including azelma) and supports custom voice cloning.

LocalAI-compatible model silero-vad

To test if this model works as expected, you can use the following curl command: curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{ "model":"tts-1", "input": "Hi, this is a test." }'

OpenAI's Whisper Base (74M parameters), a lightweight speech recognition model balancing speed and transcription quality. Good for CPU inference on clear audio across multiple languages.

OpenAI's Whisper Tiny (39M parameters), the smallest Whisper variant for speech-to-text. Fastest inference speed with acceptable accuracy for clear audio in English and common languages.

Alibaba's Qwen3 TTS 1.7B CustomVoice model for expressive, natural-sounding text-to-speech synthesis. Supports streaming generation, zero-shot voice cloning, and custom voice design.

Microsoft's VibeVoice Realtime 0.5B, a next-token diffusion TTS model achieving ~300ms end-to-end latency. Combines a Qwen2.5-0.5B backbone with an acoustic decoder for real-time expressive speech synthesis.

OpenAI's Whisper Large v3 (1.55B parameters), the highest-accuracy open speech recognition model. Near-human transcription quality across 99 languages, requires GPU for practical inference speed.

OpenAI's Whisper Large v3 Turbo (809M parameters), a pruned version of Large v3 with only 4 decoder layers. Near-large-v3 accuracy at roughly 3x the speed, ideal for real-time transcription on GPU.

OpenAI's Whisper Small (244M parameters) via faster-whisper for optimized GPU inference. Delivers solid multilingual transcription quality at moderate speed, suitable for production workloads.

LLaVA 1.5 based on Phi-2 (2.7B), a compact vision-language model capable of image understanding and visual question answering. CPU-friendly size with good accuracy for everyday vision tasks.

Vikhyat's Moondream2 (~1.86B parameters), a tiny but capable vision-language model designed for efficient image captioning and visual Q&A. Runs on modest hardware while delivering surprisingly good image understanding.

BakLLaVA-1, a vision-language model combining Mistral-7B with the LLaVA 1.5 architecture. Inherits Mistral's strong language capabilities while adding image understanding and visual Q&A.

OpenGVLab's InternVL2 8B, a high-performance vision-language model combining InternViT-300M with InternLM2.5-7B. Excels at OCR, charts, document understanding, and multi-image reasoning tasks.

LLaVA 1.5 7B, a vision-language model combining Vicuna-7B with a CLIP visual encoder. Strong image understanding, visual question answering, and scene description capabilities with GPU inference.

LLaVA 1.6 (LLaVA-NeXT) 13B based on Vicuna-13B, with improved image resolution (up to 2880px), stronger reasoning, and better OCR than LLaVA 1.5. High-quality visual Q&A and detailed scene analysis.

Alibaba's Qwen2-VL 2B, a compact vision-language model with dynamic resolution support and understanding of images, videos, and documents. Strong performance for its size on visual reasoning tasks.

Alibaba's Qwen2-VL 7B, a powerful vision-language model supporting dynamic image resolutions, video understanding, and long documents. State-of-the-art performance on visual reasoning and document parsing.

Alibaba's Qwen2.5-VL 72B Instruct flagship vision-language model. Top-tier open-weight image understanding and multimodal reasoning. Requires very high VRAM.

Alibaba's Qwen3-VL 30B-A3B Instruct MoE vision-language model. 32B-class image-understanding quality at ~3B active inference cost, ideal for high-volume multimodal pipelines.

Alibaba's Qwen3-VL 8B Instruct vision-language model. Strong general-purpose image understanding, OCR, and multimodal chat at a compact 8B size.