LLM on kenji.blog

A Complete Guide to llama.cpp and Customization with C++

Fri, 11 Sep 2026 11:00:00 +0900

In recent years, the evolution of Large Language Models (LLMs) has been tremendous, and their scope of application is expanding daily. However, running models with billions or tens of billions of parameters locally typically requires a high-end GPU with an enormous amount of VRAM. Breaking through this “hardware barrier” and making practical LLM inference possible on everyday PCs, Macs, and even devices like the Raspberry Pi is llama.cpp.

This article goes beyond just explaining how to use the command-line tool. It provides an extremely detailed explanation for engineers, covering the architecture of its underlying technology ggml, the mathematical background of Transformers and quantization, and how to use the C++ API to integrate and customize LLMs within your own applications.

1. Overview of llama.cpp and ggml

llama.cpp is a lightweight LLM inference engine written in C/C++, developed by Georgi Gerganov. Originally created with the goal of running Meta’s LLaMA model quickly on Apple Silicon (M1/M2 Macs), it now supports a variety of architectures and models.

Its biggest feature is that it is a pure C/C++ implementation with no external dependencies. Because it doesn’t require a massive ecosystem like Python or PyTorch and can be compiled as a single executable, deployment is incredibly easy.

The heart of llama.cpp is the tensor math library ggml. ggml was designed from the ground up to maximally optimize matrix operations in machine learning on CPUs (and some GPUs).

1.1 Why is llama.cpp so fast?

Leveraging Memory Mapping (mmap): When loading model weights into memory, it uses the OS’s mmap to avoid loading everything into RAM, enabling fast startups and saving memory.
Thorough Optimization of SIMD Instructions: It utilizes CPU-specific instruction sets like AVX2, AVX-512, ARM NEON, and Apple AMX to perform ultra-fast matrix multiplications.
Quantization: It compresses 16-bit floating-point (FP16) weights into 4-bit, 5-bit, or 8-bit integers, resolving the memory bandwidth bottleneck (more on this later).

2. Mathematical Background: Transformers and Quantization

To deeply understand llama.cpp, you need to know the mathematical formulas it calculates and how it approximates these calculations.

2.1 The Inference Process of a Transformer

Models like LLaMA adopt an auto-regressive Transformer decoder architecture. The core of text generation is the Self-Attention mechanism.

For an input hidden state matrix $X \in \mathbb{R}^{N \times d}$, the Query $Q$, Key $K$, and Value $V$ are calculated by multiplying with weight matrices:

$$ Q = X W_Q, \quad K = X W_K, \quad V = X W_V $$

Here, the output of Attention is defined as follows:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

In the inference loop of llama.cpp, the bottleneck is the multiplication of these massive matrices $W_Q, W_K, W_V$ and the feed-forward network (FFN) weight matrices with the vector $X$ ($N=1$ in the generation phase because it processes one token at a time), which is GEMV (General Matrix-Vector Multiplication).

2.2 The Mathematical Foundation of Quantization

In inference where memory access bandwidth becomes a bottleneck, quantization—representing weight parameters with a small number of bits—is essential. Here, we’ll explain the basic principles of the block-wise quantization widely used in llama.cpp (e.g., Q4_K or Q4_0).

For example, consider a block $w = [w_1, w_2, \dots, w_B]$ of length $B$ (usually 32 or 64) that is part of an FP16 weight matrix $W$. This block is approximated using 4-bit integers $q_i \in [-8, 7]$ and a single scaling factor $\Delta$ (FP16 or FP32):

$$ w_i \approx \Delta \times q_i $$

$\Delta$ is determined based on the maximum absolute value within the block:

$$ \Delta = \frac{\max_i |w_i|}{7} $$

When calculating the dot product $y = w \cdot x$ using the quantized weights, if the input vector $x$ is similarly quantized to $x_i \approx \Delta_x \times q_{x, i}$, we get:

$$ y = \sum_{i=1}^{B} w_i x_i \approx \Delta \Delta_x \sum_{i=1}^{B} q_i q_{x, i} $$

The $\sum q_i q_{x, i}$ part becomes a pure integer operation, which can be computed in parallel very quickly using SIMD instructions. This is the mathematical trick behind the astonishing speed of llama.cpp on CPUs.

3. Architecture and Inference Flow

To understand the internal workings of llama.cpp, the following Mermaid diagram shows the overall system architecture and data flow.

graph TD A["User Input (String)"] --> B["llama.cpp Tokenizer"] B --> C["Token IDs (int32 array)"] C --> D["Context Buffer (KV Cache)"] D --> E["ggml Compute Graph"] E --> F["Transformer Layers"] subgraph "ggml Engine" F --> G["Self-Attention (RoPE)"] G --> H["Feed Forward Network"] H --> F end F --> I["Logits (Vocabulary Size)"] I --> J["Sampler (Temperature, Top-K, Top-P)"] J --> K["Selected Token ID"] K --> L["llama.cpp Detokenizer"] L --> M["Output String"] K -. "Auto-regressive loop" .-> D

Text generation is an auto-regressive loop where, each time a single token is output, it is added to the KV Cache as the next input and passes through the compute graph again.

4. Environment Setup and Build Guide

Before embedding llama.cpp into a C++ project, let’s first build the source code.

4.1 Cloning the Repository

1
2


git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

4.2 Building with CMake

When embedding it into other applications as a C++ project, using CMake is the most standard approach. By enabling accelerators (backends) for your specific platform, you can speed up computations.

CPU Only (Basic Build):

1
2
3


mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

Using NVIDIA GPU (CUDA):

1
2
3


mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j 8

Using Apple Silicon (Metal):

1
2
3


mkdir build && cd build
cmake .. -DGGML_METAL=ON
cmake --build . --config Release -j 8

Upon a successful build, executable files like llama-cli and the llama library (along with the ggml library) for linking via the C++ API discussed later will be generated in the build/bin/ directory.

5. Introduction to C++ Customization: Using the llama.cpp API

From here on, we will discuss the main topic: controlling llama.cpp from C++ code. To embed an LLM into your own application (e.g., a game engine, desktop app, or embedded system) rather than just using command-line tools, you need to hit the C++ API directly.

llama.cpp primarily provides a C language interface through a header file called llama.h. We use this interface even when calling from C++.

5.1 Minimal Necessary Includes and Setup

When using llama.cpp in your project, include the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


#include "llama.h"
#include <iostream>
#include <vector>
#include <string>
#include <stdexcept>

// Macro for error handling
#define LLAMA_ASSERT(x) \
 do { \
 if (!(x)) { \
 std::cerr << "Assertion failed: " << #x << std::endl; \
 std::terminate(); \
 } \
 } while (0)

5.2 Loading the Model and Initializing the Context

First, load a .gguf format model file and allocate the context (memory space and KV cache) for inference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


int main(int argc, char ** argv) {
 if (argc < 2) {
 std::cerr << "Usage: " << argv[0] << " <model.gguf>" << std::endl;
 return 1;
 }
 std::string model_path = argv[1];

 // 1. Initialize the backend (set up environment like CPU/GPU)
 llama_backend_init();

 // 2. Get default settings for model parameters
 llama_model_params model_params = llama_model_default_params();
 model_params.n_gpu_layers = 35; // Number of layers to offload to GPU

 // 3. Load the model
 llama_model * model = llama_load_model_from_file(model_path.c_str(), model_params);
 if (model == nullptr) {
 std::cerr << "Failed to load model" << std::endl;
 return 1;
 }

 // 4. Set context parameters
 llama_context_params ctx_params = llama_context_default_params();
 ctx_params.n_ctx = 2048; // Maximum context size (number of tokens)
 ctx_params.n_threads = 8; // Number of CPU threads used for inference

 // 5. Create the context
 llama_context * ctx = llama_new_context_with_model(model, ctx_params);
 if (ctx == nullptr) {
 std::cerr << "Failed to create context" << std::endl;
 llama_free_model(model);
 return 1;
 }

 std::cout << "Model and context loaded successfully!" << std::endl;
 // ... subsequent processing

5.3 Tokenization of the Prompt

An LLM does not understand text directly; it processes strings as sequences of integer IDs (tokens). Therefore, the input string must be converted into tokens.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 std::string prompt = "Q: What is the capital of Japan?\nA:";
 std::vector<llama_token> tokens_list;
 tokens_list.resize(prompt.length() + 4); // Buffer size with some headroom

 // Whether to prepend a special token (like BOS: Begin of Sequence)
 bool add_special = true;
 // Convert string to an array of token IDs
 int n_tokens = llama_tokenize(
 model,
 prompt.c_str(),
 prompt.length(),
 tokens_list.data(),
 tokens_list.size(),
 add_special,
 false // parse_special
 );

 if (n_tokens < 0) {
 // Handling for when the buffer is insufficient, like reallocating and retrying, is necessary (omitted for brevity)
 std::cerr << "Failed to tokenize prompt" << std::endl;
 return 1;
 }
 tokens_list.resize(n_tokens);

5.4 Inference Loop and Sampling

Build a loop that feeds tokens into the model, obtains the probability distribution (Logits) of the next token, and samples from it to determine the next token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72


 // Maximum number of tokens to generate
 const int max_gen_tokens = 100;

 // Initialize structure for batch evaluation
 llama_batch batch = llama_batch_init(512, 0, 1);

 // Add prompt tokens to the batch
 for (size_t i = 0; i < tokens_list.size(); i++) {
 llama_batch_add(batch, tokens_list[i], i, { 0 }, false);
 }
 // Set to output logits (predictions) only for the very last token of the prompt
 batch.logits[batch.n_tokens - 1] = true;

 // Initial evaluation (feeding the prompt to the model)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "llama_decode() failed" << std::endl;
 return 1;
 }

 int n_cur = batch.n_tokens; // Current context length
 int n_decode = 0;

 std::cout << "\nOutput: ";

 // Initialize sampler context (settings for Temperature, Top-K, Top-P, etc.)
 llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
 llama_sampler_chain_add_top_k(smpl, 40);
 llama_sampler_chain_add_top_p(smpl, 0.9f, 1);
 llama_sampler_chain_add_temp(smpl, 0.7f);
 llama_sampler_chain_add_dist(smpl, 1234); // Seed value

 while (n_decode < max_gen_tokens) {
 // 1. Sampling: Predict the next token based on current context
 llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);

 // 2. If token is EOS (End of Sequence), break the loop
 if (llama_token_is_eog(model, new_token_id)) {
 break;
 }

 // 3. Decode token to string (text) and print
 char buf[128];
 int n_chars = llama_token_to_piece(model, new_token_id, buf, sizeof(buf), 0, false);
 if (n_chars > 0) {
 std::cout << std::string(buf, n_chars) << std::flush;
 }

 // 4. Prepare the newly generated token as the next batch
 llama_batch_clear(batch);
 llama_batch_add(batch, new_token_id, n_cur, { 0 }, true);

 // 5. Evaluate the model (update KV cache and predict next)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "Failed to evaluate" << std::endl;
 break;
 }

 n_cur += 1;
 n_decode += 1;
 }

 std::cout << std::endl;

 // Cleanup
 llama_sampler_free(smpl);
 llama_batch_free(batch);
 llama_free(ctx);
 llama_free_model(model);
 llama_backend_free();

 return 0;
}

This code implements a custom inference loop using the basic API of llama.cpp. It uses the llama_batch struct to manage token groups and executes the forward pass of the neural network using llama_decode.

6. Advanced Customization Examples: Logit Manipulation and Penalty Control in C++

When you want to go beyond simple text generation—such as forcing output in a specific format (e.g., JSON only) or suppressing specific forbidden words—you directly manipulate the Logits prior to sampling from the C++ side.

You can retrieve the array of raw scores (values before being converted to probabilities) right before the model outputs each token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Retrieve the raw logits array immediately after inference, prior to sampling
float * logits = llama_get_logits_ith(ctx, batch.n_tokens - 1);
int n_vocab = llama_n_vocab(model);

// List of forbidden token IDs (for example: 1234, 5678)
std::vector<llama_token> forbidden_tokens = { 1234, 5678 };

// Set the occurrence probability of forbidden tokens to 0 (make the Logit negative infinity)
for (llama_token bad_tok : forbidden_tokens) {
 logits[bad_tok] = -INFINITY;
}

In this way, directly interacting with the C++ API allows for “micro-millisecond interventions per inference cycle” that would be difficult or incur high overhead if done via LangChain or Python.

7. The Secrets of Performance Tuning

After finishing your C++ implementation, here are a few checkpoints for maximizing speed for actual deployment.

Optimizing Batch Processing: When handling requests from multiple users concurrently, include multiple sequences in llama_batch and call llama_decode at once (Continuous Batching). This drastically improves throughput by coalescing memory access.
Enabling Flash Attention: By setting ctx_params.flash_attn = true; in the context parameters, you can speed up Attention calculations while reducing memory usage. This setting is essential when dealing with long contexts (tens of thousands of tokens).
NUMA Support: In multi-socket server environments, properly configuring NUMA before llama_backend_init() can reduce memory access latency.

8. Conclusion

In this article, we covered everything in detail, starting from the mathematical background of llama.cpp to an explanation of its architecture, and finally, how to build a custom inference engine fully utilizing the C++ API.

While the Python ecosystem is highly convenient for prototyping, the direct control offered by C/C++ based llama.cpp demonstrates overwhelming power in production environments that demand edge device deployment, game integration, and real-time processing.

By all means, try writing C++ code yourself and experience the joy of freely manipulating LLMs in a local environment.

Reference Links

llama.cpp Official Repository

ggml - Tensor Library

Attention Is All You Need (Vaswani et al., 2017)

【2026 Edition】 The Complete Guide to Running Local LLMs in a Windows Environment

Fri, 11 Sep 2026 10:00:00 +0900

1. Introduction: Why Local LLMs on Windows Now?

As of 2026, the evolution of generative AI and Large Language Models (LLMs) shows a major paradigm shift from gigantic cloud-based API services to “local LLMs” running on personal PCs and on-premise environments. While cloud AIs like OpenAI’s GPT-5 and Anthropic’s Claude 3.5 are incredibly powerful, not all companies and individuals can send all their data to the cloud. From the perspectives of privacy, security, latency, and long-term sustainable costs, the demand for local LLMs is exploding like never before.

The evolution of the local LLM ecosystem, especially in the Windows environment, is remarkable. Until a few years ago, “Linux for AI development and execution” was common sense, but as of 2026, Windows has transformed into an extremely powerful and accessible AI platform.

In this article, based on the latest technology trends of 2026, we provide a complete guide to building, operating, and optimizing local LLMs in a Windows environment. From easy setup using Ollama for beginners to extreme optimization using llama.cpp for advanced users, and further deep dives into the mathematical approach of VRAM calculation, deep understanding of the architecture, and local fine-tuning, we will explain everything thoroughly with an overwhelming volume.

1.1 Technology Trends Surrounding Local LLMs in 2026

The major trends shaping the current local LLM ecosystem are as follows:

Complete Popularization of the GGUF Format: GGUF (GPT-Generated Unified Format), which integrates metadata and tensors into a single file, has completely become the de facto standard. With this, simply downloading a single file from Hugging Face makes it executable in any environment.
Democratization of the MoE (Mixture of Experts) Architecture: Many small but high-performance MoE models have been released. By activating only a portion of the experts during inference, they achieve performance comparable to giant models while keeping the computational load on consumer PCs low.
Advanced Abstraction and Optimization of Inference Engines: Tools like Ollama, LM Studio, and AnythingLLM have been refined so that users no longer need to be aware of complex dependencies like CUDA driver installations. Also, the native Windows support for FlashAttention 3 has dramatically improved inference speed.
Utilization of NPUs and the Rise of Windows Copilot+ PCs: Even on laptops without GPUs, the technology to run small LLMs (SLM: Small Language Models) with low power consumption using the built-in NPU (Neural Processing Unit) has entered the practical stage.

2. Hardware Requirements and OS Preparation

To run local LLMs at practical speeds (15-30 tokens per second or more), selecting the right hardware is the most important factor.

2.1 Recommended Hardware Configuration

With the evolution of AI PCs, required specs are also changing.

OS: Windows 11 Pro (24H2 or later). Essential for fully utilizing WSL2’s features, advanced memory management, and the latest DirectML APIs.
CPU: Intel Core Ultra 200 series or higher, or AMD Ryzen 9000 series or higher. When using CPU inference alongside, broad-bandwidth memory communication is indispensable.
RAM: Minimum 32GB, recommended 64GB or more. Main memory bandwidth (MB/s) becomes a crucial bottleneck during CPU inference or offloading. High-speed memory of DDR5-6000 or above is ideal.
GPU: NVIDIA RTX 4000/5000 series. The most important thing for local LLMs is not computing performance but “VRAM capacity”.
- Entry: RTX 4060 Ti (16GB version) - Best cost performance. Ideal for 8B-14B class models.
- Mid-range: RTX 4070 Ti SUPER (16GB) / RTX 4080 SUPER (16GB)
- High-end: RTX 4090 (24GB) / RTX 5090 (32GB) - Necessary to run 30B-70B class quantized models.
Storage: PCIe Gen4 or Gen5 NVMe SSD. Dramatically reduces the load times of models that are tens of gigabytes in size.

2.2 Setting up WSL2 (Windows Subsystem for Linux 2)

While many GUI tools work natively on Windows, WSL2 is extremely useful for Python development, compiling the latest tools, and the LoRA fine-tuning mentioned later. In the latest Windows 11 environment, just by installing the NVIDIA driver on the host side, the GPU (CUDA) can be transparently used from WSL2.

Open PowerShell with administrator privileges and execute the following:

1
2
3
4
5


# Install WSL2 and the latest Ubuntu
wsl --install -d Ubuntu-24.04

# Update the kernel
wsl --update

After installation, run nvidia-smi inside the WSL2 terminal, and if the GPU is recognized correctly, it is a success.

3. Local LLM Architecture and Inference Mechanism

Understanding how models generate text in a local environment and their internal structure is very useful for troubleshooting and optimization.

The following Mermaid diagram shows a typical local LLM inference pipeline.

graph TD User["User Input (Prompt)"] --> Tokenizer["Tokenizer"] Tokenizer --> Embedding["Embedding Layer"] subgraph "Transformer Block (x Layers)" Embedding --> Attn["Self-Attention"] Attn --> KVCache["KV Cache (Key/Value Storage)"] Attn --> FFN["Feed-Forward Network (FFN)"] end FFN --> Logits["Logits Calculation"] Logits --> Sampler["Sampler (Temperature, Top-K, Top-P)"] Sampler --> OutputToken["Output Token"] OutputToken --> |"Autoregressive Generation"| Tokenizer OutputToken --> Decoder["Detokenizer"] Decoder --> FinalOutput["Final Output Text"]

3.1 Two Phases: Prefill and Decode

LLM text generation is divided into two phases with different computational characteristics.

Prefill Phase (Prompt Processing): The phase that processes and understands the entire input prompt at once. Since parallel computing is possible, the computational power of the GPU (FLOPS) directly links to speed. If the prompt is long, this phase can take several seconds.
Decode Phase (Token Generation): The phase that predicts one token at a time and feeds it to the next input (autoregressive). Since parallel computing is restricted in this phase, GPU VRAM bandwidth (Memory Bandwidth) becomes the definitive bottleneck.

4. Mathematical Understanding of VRAM Consumption and Model Size

To correctly determine “which model will run on my PC?”, you need to understand the VRAM calculation formula. When VRAM shortages cause a fallback to system memory (RAM), inference speeds drop by 10x to 100x.

4.1 Base VRAM Based on Parameter Size

This is the amount of memory needed to load the model’s weights into VRAM. Calculate it using the model size $P$ (number of parameters, unit: 1 billion = 1B) and the number of bytes per parameter $B$.

$$ V_{base} = P \times B \quad \text{(GB)} $$

For example, when loading an 8B (8 billion) parameter model in FP16 (half-precision floating point, 16 bits = 2 bytes):

$$ V_{base} = 8 \times 2 = 16 \text{ GB} $$

In other words, even a GPU with 16GB of VRAM will reach its limit just by loading the model.

4.2 The Magic of Quantization

This is where “quantization” comes in. By reducing the precision of the parameters, the model size is drastically shrunk. In the case of the most common 4-bit quantization (e.g., Q4_K_M), it averages to about 0.55 bytes per parameter.

$$ V_{base\_4bit} = 8 \times 0.55 = 4.4 \text{ GB} $$

With this, if you have 16GB of VRAM, you can run an 8B model with plenty of headroom.

4.3 KV Cache Calculation (GQA Supported Version)

During inference, the “KV cache” needed to retain past context consumes VRAM. The latest models, such as Llama 3, use GQA (Grouped Query Attention) to save memory.

The KV cache consumption $V_{kv}$ (in gigabytes) is expressed by the following formula:

$$ V_{kv} = 2 \times b \times s \times l \times \left( \frac{h_{kv}}{h_q} \right) \times h_q \times d \times B_{kv} \div 10^9 $$

Simplifying this using the number of key/value heads $h_{kv}$:

$$ V_{kv} = 2 \times b \times s \times l \times h_{kv} \times d \times B_{kv} \div 10^9 $$

Where:

$b$: Batch size (usually 1 for individual local use)
$s$: Sequence length (context length, e.g., 8192)
$l$: Number of layers (e.g., 32)
$h_{kv}$: Number of KV heads (e.g., 8)
$d$: Number of dimensions per head (e.g., 128)
$B_{kv}$: Number of bytes for KV cache (2 for FP16)

Calculation example (Llama 3 8B, context 8192, FP16 cache): $V_{kv} = 2 \times 1 \times 8192 \times 32 \times 8 \times 128 \times 2 \div 10^9 \approx 1.07 \text{ GB}$

Note that the longer the context length $s$ is, the needed VRAM increases linearly.

5. Practice 1: Fastest and Shortest Setup using Ollama

Now that you understand the theory, let’s actually run an LLM in a Windows environment. As of 2026, the most user-friendly tool is “Ollama”. It provides an intuitive Docker-like CLI.

5.1 Installation and Execution

Download the Windows installer from the Ollama Official Website and run it.
Open PowerShell and enter the following command. Here, we’ll use the Japanese-compatible llama3:8b.

1

ollama run llama3:8b

The model will be downloaded on the first run. Once complete, you can interact with it directly in the terminal.

5.2 Creating a Custom AI with a Modelfile

You can easily create an AI with a specific persona. Create a Modelfile in an arbitrary location.

1
2
3
4
5
6
7
8
9


FROM llama3:8b

SYSTEM """
You are an exceptionally talented senior software engineer.
For user questions, always provide code examples and answer logically and concisely.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Build and run your custom model with the following commands:

1
2


ollama create SeniorDev -f ./Modelfile
ollama run SeniorDev

5.3 Usage from External Apps (AI Editors)

Ollama exposes an OpenAI-compatible API endpoint at http://localhost:11434. By simply setting this URL in the backend settings of VS Code extensions like Cursor or Continue.dev, and specifying the model name such as SeniorDev, a powerful local coding assistant is realized for free.

6. Practice 2: Extreme Performance Tuning with llama.cpp

If you want fine-grained memory management or want to be the first to try out the latest formats (like EXL2 or IQ quantization), you manipulate the core engine llama.cpp directly.

6.1 llama.cpp Build Steps

In a Windows environment, the best approach is building from source using CUDA Toolkit and CMake.

1
2
3
4
5
6
7
8


git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

# Configure for CUDA support and compile
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=OFF
cmake --build . --config Release -j 16

6.2 Advanced Launching in Server Mode

Host the model using the built llama-server.exe.

1
2
3
4
5
6
7


.\bin\Release\llama-server.exe `
 --model "C:\models\Llama-3-8B-Instruct.Q4_K_M.gguf" `
 --ctx-size 8192 `
 --n-gpu-layers 99 `
 --threads 8 `
 --flash-attn `
 --port 8080

--n-gpu-layers 99: Offloads all possible layers to GPU VRAM.
--flash-attn: Enables FlashAttention 3, achieving improved inference speed and reduced VRAM consumption for the KV cache.

7. GUI Frontend: LM Studio and Building Local RAG

If you’re resistant to the command line, or intuitively want to perform RAG (Retrieval-Augmented Generation), you can use a GUI.

7.1 LM Studio

LM Studio is a brilliant application that bundles model search, downloading, system requirement pre-checks, and a chat UI all into one. Just by pressing the “Local Server” button in the app, an OpenAI-compatible API starts up.

7.2 RAG Architecture using AnythingLLM

Here is the architecture diagram of a RAG environment for reading internal documents and personal notes.

graph LR Document["Document (PDF, MD)"] --> Chunking["Chunking"] Chunking --> EmbedModel["Embedding Model"] EmbedModel --> VectorDB["Vector Database"] UserQuery["User Query"] --> EmbedQuery["Query Embedding"] EmbedQuery --> VectorDB VectorDB --> |"Similarity Search"| RetrievedDocs["Extract Relevant Docs"] UserQuery --> PromptBuilder["Prompt Generation"] RetrievedDocs --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer"]

Using the AnythingLLM desktop version (Windows), just specify Ollama (LLM and Embedding) from the settings screen and set it up to use a local VectorDB (LanceDB). This architecture can be completed in minutes. A private AI is born that does not send any data externally.

8. Fine-Tuning (LoRA) on Windows WSL2

If you want to not just run locally but make the model smarter with your own data, fine-tuning using LoRA (Low-Rank Adaptation) is possible. As of 2026, by using a library called “Unsloth”, an 8B model can finish training in a few hours on a Windows WSL2 environment even with 16GB of VRAM.

Execute the following within WSL2’s Ubuntu to build the environment.

1
2
3
4


conda create --name unsloth_env python=3.11
conda activate unsloth_env
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Unsloth optimizes CUDA kernels to the extreme, providing about twice the training speed and half the VRAM consumption compared to the standard Hugging Face libraries. Just spin up a Jupyter Notebook and load your dataset (JSONL format), and training for several epochs is possible even on an RTX 4060 Ti with 12GB to 16GB of VRAM.

9. Performance Troubleshooting

Common problems faced and their solutions.

1. Inference speed is extremely slow (1-2 tokens/s)

Cause: The model doesn’t fit entirely into VRAM and is being offloaded to system memory (RAM). Solution: Check “Dedicated GPU memory” in the Task Manager. If it’s hitting the limit, decrease the context size (-c), or use a model with lower bit quantization (like Q4_K_M).

2. “CUDA out of memory” error

Cause: VRAM has been completely exhausted. This occurs especially when the context is prolonged and the KV cache becomes bloated. Solution: Intentionally restrict the values to smaller ones using num_ctx for Ollama, or -c for llama.cpp.

3. Strange Japanese Generation

Cause: Mismatch in prompt templates, or an unsupported model. Solution: Use models that include Instruct in their name, and ensure that the tool is selecting the correct template specified by the model author, such as the ChatML or Llama3 format.

10. Conclusion and Future Prospects

In 2026, building a local LLM in a Windows environment is no longer the privilege of a limited number of engineers. With the de facto standardization of the GGUF format, the emergence of refined ecosystems like Ollama and LM Studio, and hardware optimizations led by FlashAttention, anyone can easily obtain an enterprise-grade AI environment.

Please make use of the following points explained in this article:

Use mathematical VRAM calculations to logically select the optimal model size and quantization level for your PC specs.
Build your environment at maximum speed using Ollama, and dramatically improve productivity by integrating it with AI editors.
Bring out the ultimate performance of your hardware with the advanced parameter control of llama.cpp.
Build a secure local RAG system to handle confidential data with AnythingLLM.
Nurture a custom AI with your own specialized knowledge by utilizing Unsloth (WSL2).

The “democratization” of AI is no longer a buzzword, but a real system running on your Windows desktop. Free yourself from the usage costs of cloud APIs and information leak risks, and step into the world of free and powerful private AI right now.