vLLM is a model server for a machine with a capable NVIDIA or AMD graphics card. It speaks the OpenAI chat completions protocol, so Borago Board can run tasks on it with no API key and no cost per run. The concepts are the same as in Run tasks on a local model with Ollama.

You need

The desktop app and a project, see Creating a project.
vLLM installed on the model machine, following vLLM's documentation.

Steps

1. Start the server with tool calling on

vLLM makes tool calls only when started with a tool call parser that matches the model. vLLM's documentation lists the parser for each model family. For Qwen3-Coder it is this.

vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct --max-model-len 32768 --enable-auto-tool-choice --tool-call-parser qwen3_xml

The server listens at http://localhost:8000/v1 by default. --max-model-len is the context window. vLLM refuses to start when the window does not fit in graphics memory.

2. Add the service

Open Settings from the gear and choose Self-hosted services. Press Add a self-hosted service. Name it, set Server address to http://localhost:8000/v1, or the model machine's address on your network, and press Add. Fill API key (optional) only if you started vLLM with --api-key. The key is entered in the desktop app. In the cloud browser and on the phone the field is disabled, and a stored key stays stored.

3. Enter the context window

Press Edit on the row. Type the same number you gave --max-model-len into Context window (tokens). vLLM does not report it, and a figure that is too high makes the server cut prompts short without an error. Press Verify to check the model is there and makes tool calls, then Save.

4. Choose it for the runs

In the project, press Edit project, open AI Roles, and in Task runner choose your service from Self-hosted AI services (your hardware). Choose the model and press Save changes.

What happens. Self-hosted runs have a shell and a file editor in the project container, a code search and task breakdown. They have no web search, no page fetch and no investigator. With Tool calling on Automatic (recommended), a model whose tool calls fail is switched to a plain text format for that run.

First task to try

Create a file named hello.txt in the project's root folder containing the
single line "Hello from a local model". Then list the root folder.

Source as Markdown: /docs/tutorials/keys/vllm.md · Updated 2026-10-07 · Was this right? Tell us

Set up vLLM