Set up llama.cpp
Serve a GGUF model with llama.cpp's llama-server, add it to Borago Board as a self-hosted service, set its context window and run a first task.
llama.cpp's llama-server serves a GGUF model file over the OpenAI chat completions protocol. Borago Board runs tasks on it with no API key and no cost per run. The concepts are the same as in Run tasks on a local model with Ollama.
You need
- The desktop app and a project, see Creating a project.
- llama.cpp installed on the model machine, from its repository.
Steps
1. Start the server
--jinja turns on tool calling, which a task needs. -c sets the context window. -hf downloads a model from Hugging Face.
llama-server --jinja -c 32768 -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_MThe server listens at http://localhost:8080/v1 by default. Without -c it uses the model's full trained window, which may not fit in memory.
2. Add the service
Open Settings from the gear and choose Self-hosted services. Press Add a self-hosted service. Name it, set Server address to http://localhost:8080/v1, or the model machine's address on your network, and press Add. Fill API key (optional) only if you started the server with --api-key. The key is entered in the desktop app. In the cloud browser and on the phone the field is disabled, and a stored key stays stored.
3. Enter the context window
Press Edit on the row. Type the number you gave -c into Context window (tokens). llama-server does not report it, and a figure that is too high makes the server cut prompts short without an error. Press Verify to check the model makes tool calls, then Save. If the model can read images, turn on Image attachments yourself, since llama-server does not say.
4. Choose it for the runs
In the project, press Edit project, open AI Roles, and in Task runner choose your service from Self-hosted AI services (your hardware). Choose the model and press Save changes.
What happens. Self-hosted runs have a shell and a file editor in the project container, a code search and task breakdown. They have no web search, no page fetch and no investigator. With Tool calling on Automatic (recommended), a model whose tool calls fail is switched to a plain text format for that run.
First task to try
Create a file named hello.txt in the project's root folder containing the
single line "Hello from a local model". Then list the root folder.Related: Using a local AI model, Run tasks on a local model with Ollama, Set up LM Studio, Choosing an AI provider