With just 8GB of video memory, you can deploy a 30 tk/s multimodal AI in your laptop.

Machine-translated with NiuTrans. Read the Chinese original.

My Laptop is a very ordinary gaming Laptop: a Ryzen 7 8845H, 16GB of RAM, an RTX 4060 Laptop, and only 8GB of video memory. According to many people's impression, this configuration can only be used to "play around" with local large models.

But now, when I double-click a startup file every day, the browser opens a chat page: I can view images, watch videos, switch the depth of my thinking, and call native tools. My text generation speed is approximately... 31 tokens/sThe entire process involves no internet connection and no API fees. The one running inside is Qwen3.5-9BThe reasoning framework is llama.cpp。

This article is divided into three parts: How I chose the framework and model; An installation tutorial for beginners, written for every command; And the pitfalls I've encountered and the corresponding solutions.

First, let's talk about the conclusion: What ultimately gets started?

Project	My Configuration
Operating system	Windows x64
CPU	AMD Ryzen 7 8845H
graphics card	NVIDIA GeForce RTX 4060 Laptop, 8GB Memory
memory	16GB
reasoning procedure	llama.cpp b10909, Windows CUDA 12.4 Precompiled Packages
model	Qwen3.5-9B, Q4_K_M quantization (ca. 5.75 GiB)
multimodal assembly	Compatible BF16 mmproj (approx. 0.86 GiB)
context length	4096 tokens
Access method	Browser opens http://127.0.0.1:8080, or OpenAI compatible interface

Actual measurement data (all records from my machine at the time, not lab benchmarks):

configuration	generation speed	Memory usage of the entire graphics card
Qwen3.5-9B, Plain Text, 8K Context	Approximately 31.3 tokens/s	6429 MiB / 8188 MiB
Qwen3.5-9B, Loading Multimodal Components, 4K Context	The webpage displays approximately 34.6 tokens/s during short replies.	7364 MiB / 8188 MiB
Control: Gemma 4 E4B, Multimodal, 8K Context	Approximately 51–54 tokens/s	5209 MiB / 8188 MiB

Two points to clarify. First, the "30 tk/s" in the title comes from that test in plain text mode; In multimodal mode, the 34.6 generated only 5 word elements, which is too short and only means that "the speed is on the same order of magnitude" and cannot be considered a formal result. Second,nvidia-smi It shows the usage of the entire graphics card, including the desktop and browser, and is not equivalent to the model's exclusive memory.

Why llama.cpp

There are many tools for running large models locally, but the one I ended up using was... llama.cppWhat I valued were these points:

• There is no need to compile or install a Python environment. The official release page directly provides a pre-compiled compressed package for Windows + CUDA, which can be extracted and used. I didn't install the CUDA Toolkit or compile any C++ code throughout the process.
• A program gets things done. llama-server.exe Once launched, it provides both a web chat interface and an OpenAI-compatible API, allowing other clients to connect directly.
• The GGUF quantization model was used. After quantizing the 9B model to Q4_K_M, it is less than 6 GiB, and 8GB is enough to store it.
• All the parameters are clearly displayed. The length of the context, the number of layers placed in the graphics card, and the precision of the cache are all command-line parameters, so you know which one to adjust when there is not enough graphics memory.

The price is that it doesn't "click" like some graphical tools, requiring you to type several commands. That's why I wrote this tutorial.

Why was Qwen3.5-9B ultimately chosen?

I didn't choose the right one from the beginning; I changed it three times in between:

1. The first one downloaded was Qwen3-8B. Halfway through the download I changed my mind and stopped the download; it never started running from beginning to end.
2. The first one to actually run was Google's Gemma 4 E4B.(Official QAT Q4_0 quantified version). It's fast, with 51–54 tokens/s on my machine, only about 5.2GB of video memory, and it has multimodal and can even enable 8K contexts.
3. Finally, it was replaced with Qwen3.5-9B. The speed dropped to around 31 tokens/s, and the video memory was tighter, but I wanted the one with stronger overall capabilities.

Before switching, I compared several models that could be considered with 8GB of video memory: Qwen3.5-4B, Gemma 4 E4B, Qwen3.5-9B, Gemma 4 12B, and gpt-oss-20b. To be honest, only the speed of the Gemma 4 E4B in that comparison chart was tested by me. The speed of other models was estimated based on hardware, and the capability score was also a reference value obtained by combining various publicly available benchmarks. The test conditions are not uniform. It helped me make trade-offs, but it wasn't a rigorous ranking.

What really made me decide was the data in the official Qwen model card. The image below is from Official Model Card for Qwen3.5-9BThis is a comparison of the scores of the Qwen3.5 small-sized model with other models:

Benchmark comparison chart in the official Qwen model card: Scores of Qwen3.5 small-sized models compared to other models

Image source: Qwen official model card.

The model card also contains a complete score table; I've selected a few representative lines. First, let's look at pure language capabilities. The comparison object is a model with a much larger number of parameters:

test	GPT-OSS-120B	GPT-OSS-20B	Qwen3-30B-A3B (Thinking)	Qwen3.5-9B	Qwen3.5-4B
MMLU-Pro (Knowledge)	80.8	74.8	80.9	82.5	79.1
C-Eval (Chinese knowledge)	76.2	71.4	87.4	88.2	85.1
GPQA Diamond（ Scientific reasoning)	80.1	71.5	73.4	81.7	76.2
IFEval (instruction followed)	88.9	88.2	88.9	91.5	89.8
LongBench v2 ( (Long text)	48.2	45.6	44.8	55.2	50.0
LiveCodeBench v6 ( programming)	82.7	74.6	66.0	65.6	55.8

Looking at visual understanding, the comparison objects are two lightweight cloud models and the visual model of the previous generation 30B:

test	GPT-5-Nano	Gemini-2.5-Flash-Lite	Qwen3-VL-30B-A3B	Qwen3.5-9B	Qwen3.5-4B
MMMU (Multidisciplinary Illustration and Text)	75.8	73.4	76.0	78.4	77.6
MathVision (doing math by looking at pictures)	62.2	52.1	65.7	78.9	74.6
OCRBench (character recognition)	75.3	82.5	83.9	89.2	85.0
RealWorldQA (Real-world Q&A)	71.8	72.2	77.4	80.3	79.5
VideoMME (Video with subtitles)	71.7	74.6	79.9	84.5	83.5

There are three things to keep in mind when reading these numbers:

• This is Qwen's official resultsIt tests a model with complete accuracy. I'm running the Q4_K_M quantitative version locally, and the context is only 4096, which will reduce actual performance. I didn't retest these benchmarks on my own machine.
• It is not a comprehensive lead. In programming tests (LiveCodeBench, OJBench), the 9B is significantly inferior to the GPT-OSS-120B, which is clearly visible in the table.
• For me, the key points are two other things: a 9B model can match or even surpass the previous generation 30B model in terms of knowledge, instruction compliance, and long text; Moreover, it natively supports viewing images and videos, eliminating the need to find a separate visual model.

To sum up my trade-offs in one sentence:The Gemma 4 E4B is faster and more memory-efficient, while the Qwen3.5-9B is more comprehensive. If you care more about speed, or if you only have 6GB of video memory, the former is a good choice; The following steps in this article also apply to replacing the model file.

Before you start: What you need to prepare

• One Windows 64-bit Computer, with NVIDIA graphics card, around 8GB of video memoryAnd the graphics card driver has been installed.
• about 8GB of free hard disk space(Program approx. 0.6GB, model approx. 6.6GB).
• Networks with access to GitHub and Hugging Face. The model file is over 6GB, so be patient if your internet speed is slow.

All commands in this article are in PowerShell It runs inside. How to open: Press Win Keys, input powershellPress Enter. Simply copy the entire command and press Enter. Anti-quotation marks at the end of a command ` This indicates that "this line is not over yet," and pasting the entire paragraph together will prevent errors.

All files will be placed in the "Documents" folder. llama.cpp In the directory, the command will automatically find your own "document" path, without having to manually change the username.

Step 1: Confirm the hardware

First, check if the graphics card and memory meet the requirements:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader

Normally it will output a line similar to NVIDIA GeForce RTX 4060 Laptop GPU, 8188 MiB, ... The content. Let's take another look at the CPU and memory:

Get-CimInstance Win32_Processor |
    Select-Object Name, NumberOfCores, NumberOfLogicalProcessors

Get-CimInstance Win32_ComputerSystem |
    Select-Object @{Name='RAM_GiB';Expression={
        [math]::Round($_.TotalPhysicalMemory / 1GB, 1)
    }}

Possible problems

• Cannot be found message nvidia-smiThis indicates that no NVIDIA drivers are installed, or that this computer does not have an NVIDIA graphics card. First, go to the NVIDIA official website to install the driver.
• Video memory should be nvidia-smi According to。 Built-in Windows Win32_VideoController.AdapterRAM Large memory does not display correctly on some machines. In addition, it is also called RTX 4060, but the laptop version is different from the desktop version, and the 8GB version is different from other versions. Don't just look at the model name.

Step 2: Download and extract llama.cpp

I use it regularly b10909 This version, not the "latest version". The reason is simple: I verified every step in the tutorial on this version, and the "latest version" may look different in a few months.

Download requiredTwoCompressed files are indispensable: one is the llama.cpp program itself, and the other is the CUDA runtime it depends on. The following command will create the directory, download it, and extract it:

$ErrorActionPreference = 'Stop'
$llamaRoot = Join-Path ([Environment]::GetFolderPath('MyDocuments')) 'llama.cpp'
$downloadDir = Join-Path $llamaRoot 'downloads'
$modelDir = Join-Path $llamaRoot 'models'
$binDir = Join-Path $llamaRoot 'bin'

New-Item -ItemType Directory -Force -Path $downloadDir, $modelDir, $binDir | Out-Null

curl.exe -L --fail --retry 3 --progress-bar `
    -o (Join-Path $downloadDir 'llama-b10909-bin-win-cuda-12.4-x64.zip') `
    'https://github.com/ggml-org/llama.cpp/releases/download/b10909/llama-b10909-bin-win-cuda-12.4-x64.zip'
if ($LASTEXITCODE -ne 0) { throw 'llama.cpp 下载失败' }

curl.exe -L --fail --retry 3 --progress-bar `
    -o (Join-Path $downloadDir 'cudart-llama-bin-win-cuda-12.4-x64.zip') `
    'https://github.com/ggml-org/llama.cpp/releases/download/b10909/cudart-llama-bin-win-cuda-12.4-x64.zip'
if ($LASTEXITCODE -ne 0) { throw 'CUDA 运行库下载失败' }

Expand-Archive -LiteralPath (Join-Path $downloadDir 'llama-b10909-bin-win-cuda-12.4-x64.zip') `
    -DestinationPath $binDir -Force
Expand-Archive -LiteralPath (Join-Path $downloadDir 'cudart-llama-bin-win-cuda-12.4-x64.zip') `
    -DestinationPath $binDir -Force

The two compressed files are approximately 254MB and 391MB, respectively. After extraction, confirm that the program can recognize the graphics card:

Set-Location $binDir
.\llama-cli.exe --list-devices
.\llama-server.exe --version

The first item should list your NVIDIA graphics card, and the second item should output information containing... build 10909 The version number.

Possible problems

• Must write curl.exeDon't just write curl。 In old versions of Windows PowerShell,curl It's an alias for another command with completely different parameters, and it will directly throw an error.
• Only one compressed file was downloaded.The program will not be able to use the graphics card because the CUDA DLL is missing. Both packages need to be extracted to the same bin Catalogue.
• Variables are lost when PowerShell is turned off。 If you close the window midway through, you need to reopen it with the above definitions first. $llamaRoot、$downloadDir、$modelDir、$binDir Run those lines again.
• The "CUDA 12.4" here is just the running environment that comes with this precompiled package.No needYou install the CUDA Toolkit separately.

Step 3: Download the model

Two files are required, both from Bartowski's Qwen3.5-9B GGUF Repository：

file	effect	size
Qwen_Qwen3.5-9B-Q4_K_M.gguf	Main Model	About 5.75 GiB
mmproj-Qwen_Qwen3.5-9B-bf16.gguf	Multimodal components allow models to "see" images and videos.	About 0.86 GiB

$qwenModel = Join-Path $modelDir 'Qwen_Qwen3.5-9B-Q4_K_M.gguf'
$qwenMmproj = Join-Path $modelDir 'mmproj-Qwen_Qwen3.5-9B-bf16.gguf'

curl.exe -L --fail --retry 5 --continue-at - --progress-bar `
    -o $qwenModel `
    'https://huggingface.co/bartowski/Qwen_Qwen3.5-9B-GGUF/resolve/main/Qwen_Qwen3.5-9B-Q4_K_M.gguf?download=true'
if ($LASTEXITCODE -ne 0) { throw 'Qwen 主模型下载失败' }

curl.exe -L --fail --retry 5 --continue-at - --progress-bar `
    -o $qwenMmproj `
    'https://huggingface.co/bartowski/Qwen_Qwen3.5-9B-GGUF/resolve/main/mmproj-Qwen_Qwen3.5-9B-bf16.gguf?download=true'
if ($LASTEXITCODE -ne 0) { throw 'Qwen 多模态组件下载失败' }

In the order --continue-at - It's a break-point resumption: If the download breaks midway through, running the same command again will pick up the download from where it left off, without having to start over.

Check whether the file is complete

It's not uncommon for a 6GB file to download errors, and a corrupted model file can cause all sorts of inexplicable loading failures. Therefore, be sure to verify it once after downloading:

$expectedHashes = @{
    'Qwen_Qwen3.5-9B-Q4_K_M.gguf' = 'd784ce9eda1a5a7b51e8f705a9e6310844bf4f173654d115823c775fdea56d43'
    'mmproj-Qwen_Qwen3.5-9B-bf16.gguf' = 'd89c4bc142d02ed64aeed5c0a358bdead9109f21f4ada03a6b2df17a1aa94d9e'
}

foreach ($modelName in $expectedHashes.Keys) {
    $modelPath = Join-Path $modelDir $modelName
    $actualHash = (Get-FileHash -LiteralPath $modelPath -Algorithm SHA256).Hash.ToLowerInvariant()
    if ($actualHash -ne $expectedHashes[$modelName]) {
        throw "模型哈希不匹配：$modelName"
    }
    [pscustomobject]@{ File=$modelName; Verified=$true; SHA256=$actualHash }
}

Both files display Verified = True No problem. These two hash values are the values I validated when I downloaded them.

Possible problems

• Prompt "Model hash mismatch"There are two possibilities. First, if the file is not downloaded completely or is corrupted, run the download command again to continue the transfer and verify it again. Secondly, the repository author later updated the file (in the download link). main (This points to the latest status of the repository). In this case, you can go to the repository page to verify the SHA-256 of the file.
• I don't want it after downloading half of it.Press directly Ctrl+C Interruption. The remaining half-file extension is... .ggufBut it doesn't work. Don't use it to load; delete it or change the name (I changed it to...). .partialTo avoid confusion later.
• File size does not matchIn PowerShell 1GB It is actually 1024³ bytes, so it calculates GiB. The GB listed on the webpage will be slightly different from it, which is normal.

Step 4: First Startup

First, start it directly from the command line to confirm that everything is working properly, and then double-click the file to start it.

Set-Location $binDir
.\llama-server.exe -m $qwenModel --mmproj $qwenMmproj `
    -ngl 99 -c 4096 -np 1 `
    --cache-type-k q8_0 --cache-type-v q8_0 `
    --image-min-tokens 1024 `
    --reasoning off `
    --host 127.0.0.1 --port 8080

A lot of loading information will scroll through the window. Once it has quieted down and the server has started listening, open it with your browser:

http://127.0.0.1:8080

You will see a chat page. Ask a random question, then try dragging a picture in to describe it. To stop the model, return to the PowerShell window and press Ctrl+C。

What do these parameters mean respectively:

parameter	meaning
-m	Path to the main model file
--mmproj	The path of the multimodal component. You must use the one that matches the main model; you cannot mix it with other models.
-ngl 99	Put all the layers of the model into the graphics card as much as possible. It's not that the model has 99 layers.
-c 4096	The context length is the total amount that the model can "remember" in a single conversation. This is the runtime setting, not the upper limit of the model
-np 1	Only one request is processed at a time, saving resources.
--cache-type-k/v q8_0	Let the session cache use more memory-saving precision. It is different from the quantification of the model itself.
--image-min-tokens 1024	Minimum number of lexical elements used when processing images
--reasoning off	Thinking is turned off by default. We'll explain later how to change it to allow switching on the webpage.
--host 127.0.0.1	Allows local access only, not to local area networks or the Internet
--port 8080	Ports used by web pages and interfaces

Possible problems

• Insufficient memory, loading failure, or exceptional lagAfter loading the multimodal components with 8GB of graphics memory, there is less than 1GB left (my entire graphics card occupies 7364 MiB). First, close programs that occupy video memory, such as games, other AI software, and browsers with many tabs open. If that doesn't work, then... -c 4096 Smaller, or remove --mmproj $qwenMmproj This paragraph only contains text.
• I only want to talk about words; I want a longer context.Remove --mmproj That part, put -c Change to 8192。 The 31.3 tokens/s I measured is this configuration, and the video memory usage is 6429 MiB. Text and multimodal use the same model; the only difference is whether or not visual components are loaded.
• The webpage cannot be openedThe model will take some time to load. It will refresh after the loading information in the window stops. You can also run it in another PowerShell window Invoke-RestMethod 'http://127.0.0.1:8080/health' Check if the service is ready.
• Prompt port is occupiedThis indicates that a program is already in use on the 8080, and it's very likely that the one you started earlier hasn't been closed yet. put --port 8080 Change to another number, for example 8089The website address was also changed accordingly.
• Do not open two copies at the same time.The two model services use different ports and are also competing for memory from the same graphics card; 8GB is simply not enough.
• "Multimodal" refers to pictures and videos。 In my configuration, the audio option in the interface is unavailable.

Step 5: Create a file that can be launched by double-clicking

It's too much trouble to type such a long command every time. Just create a startup file and double-click it later.

The following PowerShell command will be... llama.cpp Generate under directory start-qwen.cmd。 It's a simplified version of the launcher I use every day, removing the "custom web pages" and "Agent" options. Those two options will be discussed in the advanced section later.

$launcherPath = Join-Path $llamaRoot 'start-qwen.cmd'
$launcherText = @'
@echo off
setlocal
title Qwen3.5 9B Multimodal - llama.cpp

set "ROOT=%~dp0"
set "BIN=%ROOT%bin"
set "MODEL=%ROOT%models\Qwen_Qwen3.5-9B-Q4_K_M.gguf"
set "MMPROJ=%ROOT%models\mmproj-Qwen_Qwen3.5-9B-bf16.gguf"

if not exist "%BIN%\llama-server.exe" goto missing_files
if not exist "%MODEL%" goto missing_files
if not exist "%MMPROJ%" goto missing_files

cd /d "%BIN%"
echo Starting Qwen3.5 9B multimodal chat with NVIDIA GPU acceleration...
echo The chat page will open at http://127.0.0.1:8080
echo Close this window or press Ctrl+C to stop the model.
echo.

start "" powershell.exe -NoProfile -WindowStyle Hidden -Command "Start-Sleep -Seconds 6; Start-Process 'http://127.0.0.1:8080'"
"%BIN%\llama-server.exe" -m "%MODEL%" --mmproj "%MMPROJ%" -ngl 99 -c 4096 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --image-min-tokens 1024 --reasoning off --host 127.0.0.1 --port 8080

echo.
echo Qwen3.5 has stopped. Press any key to close this window.
pause >nul
exit /b

:missing_files
echo ERROR: One or more llama.cpp or model files are missing.
echo Installation folder: %ROOT%
echo.
pause
exit /b 1
'@

$launcherText = $launcherText -replace "`r?`n", "`r`n"
[IO.File]::WriteAllText($launcherPath, $launcherText, [Text.Encoding]::ASCII)

Then double-click in the "Documents\llama.cpp" folder. start-qwen.cmdA black window will pop up to load the model, and the browser will automatically open after 6 seconds. Close that black window, and the model stops.

This file does several things: first, it checks if the program and model files are there; if they are missing, it will stop and let you know. use cd /d Switch directories so that they can be used on other drive letters; The startup command is written as a whole line; Finally use pause Guarantee that the window does not flash when an error occurs.

Possible problems: The window flashes and disappears after double-clicking, or "no response".

This is the pit I've been in the longest. Nothing happens after double clicking on my first version of the startup file. Don't rely on guessing; first, catch the error message:

cmd.exe /d /c "call `"$launcherPath`""

What I was seeing at the time was this output:

'NVIDIA' is not recognized as an internal or external command
'-NoProfile' is not recognized as an internal or external command
The system cannot find the path specified.

In other words, CMD splits up fragments that originally belonged to the same command and executes them as independent commands. This has nothing to do with the model or graphics card driver; the problem lies with the text of the batch file itself: encoding, line breaks, and usage. ^ How to write "continue". I didn't conduct control experiments on each item, so I can't say for sure which character caused it, but the following approach has not caused any problems since:

1. Document content is in English (ASCII) only. The Chinese instructions are written in another document. The filename can be in Chinese, but do not write Chinese characters in the file.
2. Line breaks use Windows' CRLF.
3. The start command is written as a whole line, no need ^ Line break.
4. reserve pauseThis way, you can see the error message when an error occurs.

The last two lines of the command above that generates the startup file correspond to steps 1 and 2. If you edited it manually using Notepad, you can check and correct it using the following command:

$launcherText = [IO.File]::ReadAllText($launcherPath)
if ($launcherText -match '[^\x00-\x7F]') {
    throw '启动器里还有非英文字符，请先删掉'
}
$launcherText = $launcherText -replace "`r?`n", "`r`n"
[IO.File]::WriteAllText($launcherPath, $launcherText, [Text.Encoding]::ASCII)

There's another small detail: the browser automatically opens using a fixed 6-second delay, and it doesn't actually detect that the model has completed loading. If the page opens before the model is ready, just wait a few seconds to refresh.

At this point, a local AI that can read images is ready for daily use. The following content is optional.

Step 6: See how fast it really is

If you want to know the actual speed on your machine, you can have the server report the count automatically. While the model is running, open another PowerShell window:

$body = @{
    model = 'qwen3.5-9b'
    messages = @(@{ role='user'; content='请用中文列出五条提高专注力的建议，每条一句话，直接回答。' })
    temperature = 0.7
    top_p = 0.8
    max_tokens = 256
    stream = $false
    chat_template_kwargs = @{ enable_thinking=$false }
} | ConvertTo-Json -Depth 8

$resp = Invoke-RestMethod -Uri 'http://127.0.0.1:8080/v1/chat/completions' `
    -Method Post -ContentType 'application/json; charset=utf-8' `
    -Body ([Text.Encoding]::UTF8.GetBytes($body)) -TimeoutSec 120

[pscustomobject]@{
    Reply = $resp.choices[0].message.content
    CompletionTokens = $resp.usage.completion_tokens
    SpeedTokensPerSec = [math]::Round($resp.timings.predicted_per_second, 1)
} | Format-List

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader

SpeedTokensPerSec It refers to the generation speed.

Precautions when reading

• Test answers that are too short do not count. The speed fluctuates greatly when generating only a few words; it should generate at least one or two hundred words.
• Don't mistake "prompt word processing speed" for generation speed. Web pages sometimes display numbers of thousands of words per second; that's the speed at which the model reads your question, not the speed at which it writes the answer.
• Speed is affected by many factors.Whether the laptop is plugged in, power mode, temperature, context length, and whether thinking is enabled. It's normal for the two tests to be different.

Advancement 1: Enabling web pages to switch between "thinking modes"

Qwen3.5 supports "thinking": reason a paragraph before answering, which is suitable for questions such as math and coding, but it will seem slow in everyday chat. The previous startup command was used --reasoning off It's turned off by default. Ideally, you can switch between them at any time within the webpage.

The problem is, the webpage in version b10909 is...When only one model is loaded on the desktopThe Reasoning submenu is omitted from the "+" menu. You need to distinguish between three things: whether the model supports switching thinking, whether the server is on or off by default, and whether the webpage gives you the toggle. The first two items are fine; the only thing missing is the menu on the webpage. This conclusion only applies to b10909 and does not mean that subsequent versions will have the same problem; The repair method referenced upstream... PR #27985。

First, confirm that the interface itself can indeed close thinking on request (executed in another window while the model is running):

$chatBody = @{
    model = 'Qwen_Qwen3.5'
    messages = @(@{ role='user'; content='Reply with OK only.' })
    stream = $false
    max_tokens = 16
    reasoning_format = 'auto'
    chat_template_kwargs = @{ enable_thinking=$false }
} | ConvertTo-Json -Depth 8

$chatReply = Invoke-RestMethod `
    -Method Post -Uri 'http://127.0.0.1:8080/v1/chat/completions' `
    -ContentType 'application/json; charset=utf-8' `
    -Body ([Text.Encoding]::UTF8.GetBytes($chatBody)) -TimeoutSec 60

[pscustomobject]@{
    Content = $chatReply.choices[0].message.content
    Reasoning = $chatReply.choices[0].message.reasoning_content
    Finish = $chatReply.choices[0].finish_reason
}

What I get here is Content = OK，Reasoning It is empty. This means that as long as the webpage sends the selection to the server, that's fine.

Next, you need to build your own webpage. This step requires installation first. Git and Node.js(I'm using Node v22.15.0, npm 10.9.2). Build only the web front-end, no C++ compilation required.

1. Download the corresponding version of source code

$sourceRoot = Join-Path ([Environment]::GetFolderPath('MyDocuments')) 'llama-source-b10909'
if (Test-Path -LiteralPath $sourceRoot) { throw '源码目录已存在，请换一个目录' }
git clone --depth 1 --branch b10909 https://github.com/ggml-org/llama.cpp.git $sourceRoot

2. Change a file

Open with any text editor:

tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionAdd/ChatFormActionAddDropdown.svelte

Change the import statement to introduce the thinking submenu component at the same time:

- import { ChatFormActionAddToolsSubmenu, McpLogo } from '$lib/components/app';
+ import {
+   ChatFormActionAddReasoningSubmenu,
+   ChatFormActionAddToolsSubmenu,
+   McpLogo
+ } from '$lib/components/app';

Then render it in the additional menu on the desktop, adding a separator line:

<ChatFormActionAddReasoningSubmenu />
<DropdownMenu.Separator />

This component is already in the source code, but this menu doesn't use it. So this is just taking back existing things, not inventing a new mechanism.

3. Install dependencies, check, and build.

Set-Location (Join-Path $sourceRoot 'tools\ui')
npm ci
npm run check
npm run build

npm run check The result should be 0 errors and 0 warnings. You may get a "File size exceeds warning threshold" message during build; that's just a warning, not an error.

4. Place the build results in the installation directory.

$uiDist = Join-Path $sourceRoot 'tools\ui\dist'
$webuiDir = Join-Path $llamaRoot 'webui'
New-Item -ItemType Directory -Force -Path $webuiDir | Out-Null
Get-ChildItem -LiteralPath $uiDist -Force |
    Copy-Item -Destination $webuiDir -Recurse -Force

5. Modify the startup command

Put the startup command --reasoning off Replace with:

--reasoning auto --path "%WEBUI%"

And add a line to the variable section before the startup file. set "WEBUI=%ROOT%webui"。 If you start it directly in PowerShell, write it as --reasoning auto --path (Join-Path $llamaRoot 'webui')。

After restarting, click the "+" in the lower left corner of the input box to see the Reasoning menu, where you can switch between off, low, medium, high, and maximum.

Possible problems

• --path You must add it. If you don't add it, the program will still open the same built-in webpage without a menu, and you might think that the changes haven't taken effect.
• The webpage remains unchanged after modifications.This webpage will be cached by your browser. Press Ctrl+F5 Force refresh; If the page prompts "Available updates", click "Refresh".
• Used when copying files Copy-Item -LiteralPath (Join-Path $uiDist '*') Nothing was copied.：-LiteralPath It means "take the path literally" and will not expand. *。 You need to use it as above first. Get-ChildItem List them out before copying them.
• The higher the level of thinking, the better. The thinking budgets for low, medium, and high are 512, 2048, and 8192 vocabulary elements, respectively. But my total context is only 4096, and the "high" 8192 can't fit at all. Select "Off" for regular chat, and "Medium" for math, coding, or complex reasoning.

Advanced 2: Open the Agent tool

This version of llama.cpp allows the model to call native tools. The approach isThe original complete startup commandPlus --agentIt's not about starting another one that only brings... --agent The process:

- --reasoning auto --path "%WEBUI%" --host 127.0.0.1 --port 8080
+ --reasoning auto --agent --path "%WEBUI%" --host 127.0.0.1 --port 8080

The parameters may differ between different versions. First, check if it's included in the help for your version:

& (Join-Path $binDir 'llama-server.exe') --help 2>&1 |
    Select-String -Pattern '^--agent|\s--agent|--tools' -Context 0,3

After starting, you can check if the tool is ready:

$serverProps = Invoke-RestMethod 'http://127.0.0.1:8080/props'
$serverTools = Invoke-RestMethod 'http://127.0.0.1:8080/tools'
[pscustomobject]@{
    Vision = $serverProps.modalities.vision
    ToolCount = @($serverTools).Count
    Tools = (@($serverTools) | ForEach-Object { $_.name }) -join ', '
}

My server has seven tools: reading files, searching files, searching file contents, executing commands, writing files, editing files, and viewing runtime information. Two more browser-side tools will appear on the webpage, so the total number displayed on the interface is 9, which is not contradictory to the 7 returned by the interface.

Security issues must be addressed here. Once the Agent is enabled, the model can read and write files on your computer and execute commands; it is no longer just a chat window. So:

• keep --host 127.0.0.1，don'tChange to 0.0.0.0Do not expose it to local area networks or public networks.
• Monitoring only the local machine does not mean absolute security. When the model is asked to process a web page or document of unknown origin, it may contain text that tricks it into performing an action.
• Don't add tools when you don't need them. --agent。

I only verified that the tool loaded properly and the menu displayed; I did not make the model execute write files or dangerous commands for testing purposes.

Step 3: Change the webpage interface to Chinese

The b10909's web interface is in English, and much of the text is hard-coded into the source code, with no readily available "language" option to switch between. My approach is to translate at the source code level and then rebuild it, using the same process as fixing the thinking menu above.

Specifically, I wrote a script that uses the Svelte compiler and TypeScript syntax analysis to read the source code, replaces the text on the interface with a Chinese-English bilingual dictionary, and then manually adds dynamic text that the script cannot handle (such as spelled-out sentences like "maximum 2,048 vocabulary"). The translation scope includes chat pages, sidebars, settings pages, think menus, tools menus, build progress, and statistics.

There are hundreds of entries in the dictionary, but I won't post them here. I'll just share a few useful tips for those who want to do it themselves:

• Translate only the interface text; do not modify the strings used within the program. Like thinking hierarchy off、mediumThe tool's function names and interface parameter names are agreed upon between programs, and translation will cause errors. I mistakenly translated an internal prefix string halfway through, and only discovered it during checking and changed it back.
• You can't just rely on "search for English and then replace". Even if you use grammatical analysis, you still have to look at the semantics one by one.
• Back up before translating. I keep a repaired English version of the thinking menu, which I can switch back to at any time.
• After the revisions are completed, inspections and tests are required. All 674 unit tests included in the project passed after translation; After that, I added a few more words and redid the checking, building, and browser verification.
• The model's answers, your input, and the results returned by the tool will not be translated; only the interface will be translated into Chinese.

The final result is that chat, settings, thinking patterns, tool menus, and generation progress are all in Chinese, while the model and startup parameters remain completely unchanged.

Access to other software

llama-server It provides an OpenAI-compatible interface, so many clients that support custom interface addresses can connect directly:

• interface address (Base URL):http://127.0.0.1:8080/v1
• Model name: Use the following command to check; don't guess based on the filename.
• API Key: The server has not set a Key. If the client requires it to be filled in, whether any placeholder value will work depends on the client.

Invoke-RestMethod 'http://127.0.0.1:8080/v1/models'
Invoke-RestMethod 'http://127.0.0.1:8080/health'

Example of direct call using PowerShell (note that Chinese characters must be sent in UTF-8, otherwise the characters will be garbled):

$modelList = Invoke-RestMethod 'http://127.0.0.1:8080/v1/models'
$actualModelId = $modelList.data[0].id
$requestJson = @{
    model = $actualModelId
    messages = @(@{ role='user'; content='请用中文介绍一下你自己。' })
    stream = $false
    max_tokens = 256
    chat_template_kwargs = @{ enable_thinking=$false }
} | ConvertTo-Json -Depth 8

Invoke-RestMethod -Method Post `
    -Uri 'http://127.0.0.1:8080/v1/chat/completions' `
    -ContentType 'application/json; charset=utf-8' `
    -Body ([Text.Encoding]::UTF8.GetBytes($requestJson))

These commands will report connection errors when the model is not started, which is normal; they will not help you start the model.

I want to switch to another model in the future.

The process is the same:

1. Verify that your llama.cpp version supports that model.
2. Quantizes version and context length by memory size.
3. Download and verify the complete GGUF file from a trusted repository.
4. If it is a multimodal model, downloadMatching itmmproj; A plain text model is not required.
5. First, stop the running model, then change the startup command. -m and --mmproj。 It is recommended to test run on a different port first, confirm that there are no problems, and then modify the startup files used daily.

& (Join-Path $binDir 'llama-server.exe') `
    -m (Join-Path $modelDir 'YOUR-MODEL.gguf') `
    -ngl 99 -c 4096 -np 1 `
    --host 127.0.0.1 --port 8089

YOUR-MODEL.gguf It's a placeholder name; you need to replace it with a real filename. In addition, the "Think Switch" is a feature supported by Qwen's chat templates, but not all models have it. You need to confirm again after changing models.

FAQ Quick Check

phenomenon	Causes and Treatment
Cannot be found message nvidia-smi	No NVIDIA driver installed, install the driver first.
curl Report parameter error	Write curl.exe
The program does not require a graphics card and is extremely slow.	Check if the compressed file in the CUDA runtime has been extracted. bin； use llama-cli.exe --list-devices Confirm that the graphics card can be
Model loading failed	Verify SHA-256 first to rule out file corruption
Insufficient video memory	Close other programs that occupy video memory; reduce -c； remove --mmproj Only run text
Double-clicking the startup file flashes by.	use cmd.exe /d /c "call 文件路径" Catch and report errors; The file content is written only in English, CRLF line breaks, and the startup command is written on one line.
The browser opens blank or cannot connect.	The model is still loading, please wait a few seconds to refresh; Or inquire /health
Port occupied	The previous one wasn't closed, or you could switch to another one. --port
I always have to "think" for a long time before answering.	Startup parameters --reasoning offAlternatively, select "Close" from the Think menu on the webpage.
I changed the webpage but I don't see any changes.	Confirm that the startup parameters --path； Press Ctrl+F5 Force refresh
When copying files * Not effective	-LiteralPath Instead of expanding wildcards, use Get-ChildItem ... | Copy-Item
I want to delete the old model	Confirm that the new model has been downloaded, verified, and can be loaded before deleting. I'm using move to recycle bin instead of permanently deleting it directly.

At the end

Looking back, this didn't require particularly powerful hardware or knowledge of deep learning. There are three areas where time is really spent: trading off between "fast" and "comprehensive" when choosing a model, getting stuck on the encoding problem of a batch file, and figuring out whether the "think switch is missing" is a problem with the model, the server, or the webpage. Explain these three points clearly, and the rest is just to type the commands in order.

A few reminders:

• The speed and memory figures in this article are records of my machine at the time; they will be different with different computers and power modes.
• The model's benchmark score comes from Qwen official website and measures a model with complete accuracy; I'm running the quantified version locally; there are no retests.
• Local models can also make serious mistakes, and you have to verify important information yourself.

References

• call.cpp b10909 Release page
• llama.cpp b10909 Server Documentation
• call.cpp b10909 web page source code
• Think menu fix: PR #27985 with issue #27981
• Qwen3.5-9B Official Model Card(Source of benchmark data and comparison chart)
• Qwen3.5-9B GGUF (Bartowski quantification)
• Gemma 4 E4B Official QAT GGUF