# Set up vLLM

Serve a model with vLLM's OpenAI-compatible server, add it to Borago Board as a self-hosted service, set its context window and run a first task.

vLLM is a model server for a machine with a capable NVIDIA or AMD graphics card. It speaks the OpenAI chat completions protocol, so Borago Board can run tasks on it with no API key and no cost per run. The concepts are the same as in [Run tasks on a local model with Ollama](/docs/tutorials/local-model-with-ollama/).

## You need

- The desktop app and a project, see [Creating a project](/docs/guides/creating-a-project/).
- vLLM installed on the model machine, following [vLLM's documentation](https://docs.vllm.ai/).

## Steps

### 1. Start the server with tool calling on

vLLM makes tool calls only when started with a tool call parser that matches the model. vLLM's documentation lists the parser for each model family. For Qwen3-Coder it is this.

```
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct --max-model-len 32768 --enable-auto-tool-choice --tool-call-parser qwen3_xml
```

The server listens at `http://localhost:8000/v1` by default. `--max-model-len` is the context window. vLLM refuses to start when the window does not fit in graphics memory.

### 2. Add the service

Open Settings from the gear and choose **Self-hosted services**. Press **Add a self-hosted service**. Name it, set **Server address** to `http://localhost:8000/v1`, or the model machine's address on your network, and press **Add**. Fill **API key (optional)** only if you started vLLM with `--api-key`. The key is entered in the desktop app. In the cloud browser and on the phone the field is disabled, and a stored key stays stored.

### 3. Enter the context window

Press **Edit** on the row. Type the same number you gave `--max-model-len` into **Context window (tokens)**. vLLM does not report it, and a figure that is too high makes the server cut prompts short without an error. Press **Verify** to check the model is there and makes tool calls, then **Save**.

### 4. Choose it for the runs

In the project, press **Edit project**, open **AI Roles**, and in **Task runner** choose your service from **Self-hosted AI services (your hardware)**. Choose the model and press **Save changes**.

What happens. Self-hosted runs have a shell and a file editor in the project container, a code search and task breakdown. They have no web search, no page fetch and no investigator. With **Tool calling** on **Automatic (recommended)**, a model whose tool calls fail is switched to a plain text format for that run.

## First task to try

```
Create a file named hello.txt in the project's root folder containing the
single line "Hello from a local model". Then list the root folder.
```

Related: [Using a local AI model](/docs/local-models/), [Run tasks on a local model with Ollama](/docs/tutorials/local-model-with-ollama/), [Set up llama.cpp](/docs/tutorials/keys/llama-cpp/), [Choosing an AI provider](/docs/guides/choosing-an-ai-provider/)