Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally.
This feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected.
Start the server in router mode by not specifying a model:
This auto-discovers models from your llama.cpp cache (LLAMA_CACHE or ~/.cache/llama.cpp). If you've previously downloaded models via llama-server -hf user/model, they'll be available automatically.
On the first request, the server automatically loads the model into memory (loading time depends on model size). Subsequent requests to the same model are instant since it's already loaded.
Returns all discovered models with their status (loaded, loading, or unloaded).
All loaded models will use 8192 context and full GPU offload. You can also define per-model settings using presets:
The built-in web UI also supports model switching. Just select a model from the dropdown and it loads automatically.
We hope this feature makes it easier to A/B test different model versions, run multi-tenant deployments, or simply switch models during development without restarting the server.
Have questions or feedback? Drop a comment below or open an issue on GitHub.
Supported via presets.ini, where you can specify the mmproj (and other long and short arguments) per model.
mmproj is also automatically selected for cached models, downloaded via the -hf user/model option
Awesome new feature! Can model selection be done on something other than requested model name?
Source link
