Run a local model
GGUF files, Hugging Face downloads, engine options, and llama-swap configuration.
Settings → Local models manages GGUF files, Hugging Face downloads, and llama-swap configuration. Chat models selects models and defaults for conversations. Providers manages server connections and authentication.
Settings tabs#
| Tab | Contents |
|---|---|
| Your models | Files found on disk and models read from llama-swap configuration. |
| Download models | Public Hugging Face GGUF search and download review. |
| Engines | Installed engine tools, model folders, and the target llama-swap config. |
Connect a server#
Use Settings → Providers → + → A local model to connect an existing OpenAI-compatible server. Select a detected server or enter its inference API base URL, test the connection, and save it. Then select one of its models in the chat picker.
Use the API address rather than the server's browser interface. The port and
/v1 prefix depend on the server. Text generation and structured tool calling are separate capabilities; coding tasks require a suitable model and server configuration.Configure engines and folders#
Open Local models → Engines. Check the detected llama.cpp and llama-swap tools, choose model folders, and select the llama-swap config to use for downloads and model settings.
pi-kit uses an existing llama-swap service. It does not start or stop a separate private runtime. Engine installation actions are offered where supported; the service itself runs independently.
Search Hugging Face#
Open Download models, enter a model family or file name, and choose Find models that fit. An empty query browses public GGUF files. Supported sources include ggml-org, Qwen, unsloth, and bartowski.
Results include the source repository, license, file size, and estimated memory requirements. Estimates reserve space for runtime context and other processes; they are not performance measurements.
Download a file#
- Select the target llama-swap config in Engines.
- Search and select a supported single-file GGUF.
- Review the source, license, exact file, disk requirement, and memory estimate.
- Confirm the download.
- Wait for download and checksum verification.
Downloads are pinned to an immutable revision, size, and SHA-256. Verified files are added to the selected llama-swap config. Reload the service if it does not watch configuration changes.
Gated repositories, split GGUFs, and separate vision projectors may require manual setup. The catalog does not test model quality, vision support, or tool calling.
Manage downloaded and configured models#
Your models shows files on disk and configuration entries. Configured models can remain listed while llama-swap is stopped. Refresh rereads the relevant files.
File size and deletion controls apply to discovered local files. Cleanup of incomplete downloads is separate from deletion of completed models. A visible model entry does not indicate that the server has loaded it.
Model settings#
A model's settings can include Runtime, Generation defaults, and MTP, depending on the engine and launcher.
Runtime options include context size, GPU layers, CPU threads, parallel slots, and idle unloading. Generation defaults cover sampling options. Blank fields use engine or environment defaults. Custom launchers may expose only options already declared in their command.
Saving updates the existing config. Engine changes normally apply on the next model load. An explicit reload uses the existing llama-swap instance and may interrupt active requests or unload other models. MTP requires compatible model and engine support; the interface reports unverified or unavailable states.
Common failures#
| Symptom | Check |
|---|---|
| Download cannot start | Select a valid target config and check free disk space. |
| Integrity check fails | Retry the selected file; partial files are not completed models. |
| Model is listed but unavailable | Check the running service and whether it has reloaded the config. |
| Save reports an external config change | Reload before editing again. |
| Tool calls fail | Check the model, server template, and reported capability. |
| Reload or installation is unavailable | Read the platform-specific reason shown by the control. |