New in llama.cpp: Model Management

Source: HuggingFace Date: 2025-12-11 URL: https://huggingface.co/blog/ggml-org/model-management-in-llamacpp

Summary

Library update: llama.cpp server adds router mode — dynamic multi-model management with auto-discovery of GGUF files, on-demand loading, automatic LRU eviction when a model limit (default 4) is reached, and per-process crash isolation. Models can be requested by name in API calls; server switches without restart. Auto-discovers from ~/.cache or custom --models-dir. No benchmarks; feature-focused.

Implications

Open-weights ecosystem health. Router mode makes llama.cpp’s server viable for multi-tenant and A/B testing deployments that previously required separate server instances. The LRU eviction with up to 4 concurrent models covers most local development and small-scale production scenarios without memory management overhead.

Model release cadence — local inference. As llama.cpp gains more production-grade server features (routing, multi-model, crash isolation), it becomes a credible alternative to vLLM for local and edge deployments — particularly for the GGUF-quantized model ecosystem that is already first-class on llama.cpp. Watch whether further llama-server improvements close the gap with vLLM’s feature set on non-GPU hardware.