All pages
Docs · AI features
Local LLM
Availability: macOS.
The Local LLM tab in the right sidebar manages models you run locally through Ollama (or any compatible server — LM Studio, vLLM, llama.cpp). From here you can see installed models, pull new ones, watch what's loaded in memory, and chat with a model ad-hoc — no cloud key required.
The tab loads even if Ollama isn't running; you just can't pull, delete, or chat until a server is reachable.
Endpoint
The header shows connection status (a green/red pill) and an editable
endpoint URL, defaulting to http://127.0.0.1:11434. Refresh re-queries
installed and running models.
If the endpoint is local and unreachable, a Start Ollama affordance appears:
- Ollama.app installed → Start Ollama launches the app.
- CLI only (Homebrew) → Start Ollama runs
ollama servein the background and polls for ~15s. - Nothing found → Install Ollama opens ollama.com/download.
Pointing the endpoint at a remote or OpenAI-compatible (/v1) server
disables pull/delete/unload — those are local-Ollama operations.
Sections
- Installed — downloaded models with size, parameter count, quantization, context window, and whether they can use the agent's tools (tools) or not (chat only). Each row's menu: Set as agent default (pins it to the Agent tab's model picker), Chat with this model, Delete weights…. The current default carries a Default badge.
- Pull a model — type a model tag (e.g.
qwen3-coder:30b) and Pull to download. In-flight pulls show progress and a cancel button; Clear finished clears completed rows. - Loaded in memory — what's currently in VRAM (via Ollama's
/api/ps), with VRAM usage and auto-evict time. Unload evicts a model. - Try a model — a throwaway inline chat for testing a model. Not persisted; Clear resets the transcript.
First-class coding models
Three Qwen coder models are wired into the Agent tab's model picker as local options:
- Qwen3-Coder 30B (
qwen3-coder:30b) — the local default; a MoE model that runs well on 16 GB Macs. - Qwen2.5-Coder 14B (
qwen2.5-coder:14b) and 7B (qwen2.5-coder:7b) — smaller alternatives.
Set as agent default writes the model into the project's
ollamaModelName and switches the Agent's active provider to local
Ollama. The endpoint you set here syncs to Agent settings automatically.
How the agent talks to a local model
A local model runs in far less room than a hosted one, so the agent sends it less, and checks that what it sends fits:
- A core set of tools. A local model gets 15 tools instead of the
full catalog:
bash,get_context,read_file,write_file,edit_file,list_dir,glob,search_files,git_status,git_diff,git_log,read_terminal,run_command,web_fetchandhelp_read. Every tool's description rides in every request, and the full set alone is larger than most local models' window. To send a different set, turn tools on or off under the gear menu → Project agent settings… → Tool allowlist. A project or persona allowlist is used as given, on any model. - A window sized to the request. Each request tells Ollama how much
context to load, based on the request's size and never more than the
model supports. A model already loaded with enough room keeps its window,
since changing it makes Ollama reload the model. On a model that isn't
loaded yet, the first message takes a few seconds longer. To use one size
for every request instead — say you've set a larger default in Ollama
itself, which its API doesn't report — set Context window in the
Providers sheet's local section. A model built with its own
num_ctxin its Modelfile keeps at least that much. - Shorter tool results. A local model keeps about a quarter of its window from any one tool result — about 32 KB on a 32K model — with the middle cut out and a note telling the model how to ask for the rest. One large file read would otherwise fill the whole window.
- Older context left out, rather than a dead end. When a chat grows past what the model holds, the agent leaves older context out of that request — old tool output first, then long earlier messages, then the oldest exchanges — and says so under the reply. Your saved transcript keeps everything; only what the model saw for that reply is smaller. Auto-summarize still runs at its usual threshold, measured against the model's real window (32K for the Qwen2.5-Coder models, 256K for Qwen3-Coder).
- A clear error only when nothing can be left out. If your latest message alone is bigger than the model holds, the agent stops before sending and says so; shorten it, or switch models. Sending a shorter message works straight away — the long one is then older context, and gets left out. If the project's own instructions and tools don't fit, the error says that instead: unpin files, shorten AGENT.md, narrow the tool allowlist, or use a bigger model.
- A model that can't use tools still works. Some models — Gemma, for instance — have no tool support at all. Ollama used to refuse the request outright. The agent now sends that model no tools and answers as a plain conversation, with a note in the chat and a chat only chip in the meters row. It can talk, but it can't read files or run commands.
- Tool calls written as text. Some local models write a tool call as
plain JSON instead of in the format Ollama recognises. Qwen2.5-Coder
does this. A reply that is only a tool call naming one of the tools it
was sent runs as that tool call, whether it's bare JSON, wrapped in
<tool_call>tags, Mistral's[TOOL_CALLS]or a JSON code block. A reply that starts like one (with{,[{, a tag or a JSON code block) appears when it ends instead of streaming. A call to a tool the model wasn't sent, or one cut off by the output limit, stays as text.
Pointing the agent at an OpenAI-compatible (/v1) server — LM Studio,
vLLM, Ollama's /v1 — gets the core tool set and the tool-call handling,
and drops tools if the server refuses them, but not the window sizing or
the trimming: those servers fix the context size when the model is loaded
and don't report it, so set it there.
Relationship to the agent and trmnly-code
The Local LLM tab and the Agent → Providers sheet edit the same per-project Ollama endpoint and model. Picking a local model here is the same as picking it in the Agent tab.
The headless trmnly-code CLI (installable from the app menu →
Install trmnly-code CLI…) is a terminal coding agent that shares the
in-app agent's tools and approval rules. It defaults to a local Qwen
coder model via Ollama, but --model / --provider can point it at any
provider (anthropic, openai, gemini, ollama, lmstudio, openrouter).