Borago Board logo Borago Board

Using a local AI model

Run tasks on your own hardware · no API key · no cost per task

What this gives you

Borago Board can run tasks on a model running on a computer you own, instead of on Claude, Codex or Grok. There is no API key, nothing is sent to a provider, and tasks cost nothing to run. The trade is quality and speed: a local model does less well on complicated work than a frontier model, and needs a reasonably powerful graphics card.

The model server can be the same computer that runs Borago Board, or another one on your network. It is only an address, so nothing changes about how you use the board. Borago Board makes the request itself, so your project container needs no access to it.

What you need

Any server that speaks the OpenAI-compatible API works: Ollama (easiest), vLLM, llama.cpp (llama-server), LM Studio, SGLang or TGI. The rest of this page uses Ollama, because it is the one Borago Board can also download models for and start on your behalf.

For hardware: aim for 24 GB of video memory, or 32 GB of unified memory on an Apple silicon Mac. Below that, models become unreliable at calling tools, which is all a task really consists of. Smaller models are still selectable and Borago Board handles their mistakes gracefully, but they will not finish real work.

Setting it up

Install Ollama from ollama.com/download, then download a model. The default Borago Board suggests is a good starting point: 

ollama pull qwen3-coder:30b

That is about 19 GB to download. Then in Borago Board open a project, go to AI & Sign-in, and under Local AI services choose Add. Give it a name, leave the address as http://localhost:11434/v1 if the server is on this machine, pick your model, and press Verify.

Verify tells you what the server can actually do before you queue any work: whether it has the model, whether the model can call tools, how much context it will give you, and how fast it responds. If the model is missing, Borago Board offers to download it for you. If Ollama is installed but not running, it offers to start it.

The one setting that matters most

Ollama serves every model with a 4,096 token context window by default, no matter how much the model itself supports. That is small for real work: a task description, the project's instructions and a couple of file reads will fill it. When the prompt does not fit, Ollama quietly cuts the beginning off rather than reporting an error, so the model appears to ignore its instructions.

Borago Board detects the real figure and shows it on the Verify card - for example “4,096 of 262,144 supported” - and sizes its prompts to what the server actually grants, so nothing is silently truncated. But to get the benefit of a model with a large context window, raise it on the server. On Linux: 

sudo systemctl edit ollama.service

Add:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama

On macOS run launchctl setenv OLLAMA_CONTEXT_LENGTH 32768 and restart the Ollama app. Note that this lasts until you reboot - to make it permanent, add the same line to a login script, or use the per-model method below, which survives restarts. On Windows add OLLAMA_CONTEXT_LENGTH as a user environment variable and restart Ollama from the tray.

A larger window uses more memory, so if your card is close to full you may find the model spills onto the CPU and slows down. 32,768 is a sensible starting point. Borago Board cannot set this for you - the OpenAI-compatible API that every server shares has no way to request it.

Raising it for one model only

OLLAMA_CONTEXT_LENGTH applies to every model the server loads. If you only want the larger window for the model Borago Board uses - so a chat model you also run keeps a small, fast one - bake it into a copy of the model instead: 

printf 'FROM qwen3-coder:30b\nPARAMETER num_ctx 32768\n' > Modelfile
ollama create qwen3-coder-32k -f Modelfile

Then pick qwen3-coder-32k in Borago Board. It is not a second copy of the weights, only a new name pointing at them with a different setting. Note that num_ctx sent in an API request does not work: Ollama's OpenAI-compatible endpoint ignores it, which is why the setting has to live on the server.

Other servers

The same rule applies everywhere - the window is set when the server starts, not by Borago Board - only the flag changes:

  • llama.cpp (llama-server) - --ctx-size 32768 (or -c 32768). It defaults to the model's own trained window, so this is usually about lowering it to fit, not raising it.
  • vLLM - --max-model-len 32768. It also defaults to the model's full window and will refuse to start if that does not fit in your video memory, which at least tells you plainly.
  • LM Studio - Context Length on the model's load settings, before you start the server. Changing it means reloading the model.
  • SGLang - --context-length 32768. TGI - --max-total-tokens 32768.

Only Ollama reports its context window over the network, so for the others Borago Board cannot show you the real figure on the Verify card. Set it on the server, then set the same number in the Context window field on the service in Borago Board so it can size its prompts correctly.

Using a server on another computer

Ollama only listens to its own machine by default. To reach it from another computer, set its address to all interfaces - in the same file as above: 

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=32768"

Then use that machine's address in Borago Board, for example http://192.168.1.5:11434/v1.

Ollama has no password. Anything that can reach that port can use your graphics card and read which models you have. Restrict it to the computer running Borago Board: 

sudo ufw allow from 192.168.1.10 to any port 11434 proto tcp

Never forward this port to the internet. If you need access from outside your home, use a VPN such as WireGuard or Tailscale and point Borago Board at the VPN address.

Choosing a model

The models that work best on a single graphics card are mixture-of-experts designs: large in total, but only a small part of the model runs for each word, so they stay fast even when they do not fully fit.

  • qwen3-coder:30b - about 19 GB. The default, and the best fit for a 24 GB card. Built for exactly this kind of tool-driven work.
  • devstral:24b or qwen3.6:27b - about 14 to 17 GB. Simpler alternatives if the above does not suit your setup.
  • qwen3-coder-next:latest - about 52 GB. Best quality per gigabyte if you have 64 GB of unified memory or a large card.
  • gpt-oss:20b - about 14 GB, for a 16 GB card.

You are not limited to this list. Borago Board's model box accepts anything your server can load, including a model straight from Hugging Face such as hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M if you want a different quality setting.

Small models, and what to expect

Borago Board lets you select models well below the recommended size, after asking you to confirm you understand. This is genuinely useful for trying the feature out. What you should expect is that they often cannot call tools at all, lose track partway through a task, or announce they have finished without doing anything.

None of that breaks the board. Tasks that fail come back with an explanation and can be retried; a model that cannot call tools properly is given a simpler text format to use instead; one that repeats itself is stopped rather than left running. But a small model will not finish real work, and no amount of handling changes that.

If something goes wrong

“Nothing is listening there” - the server is not running, or the address is wrong. Check with curl http://your-address:11434/v1/models from the computer running Borago Board, not from the server itself.

“Does not have that model” - press Verify and use the download button, or run ollama pull on the server.

The model server crashed - usually the graphics card running out of memory, most often when swapping between two large models. It normally recovers by itself and the task retries. If it keeps happening, avoid keeping two large models in use on the same machine.

Tasks are very slow - the model probably does not fit in video memory and is partly running on the processor. Verify reports how long the server took to answer a trivial prompt, which is a good early warning. A smaller model, or a smaller quality setting of the same model, will usually be much faster.