Local AI on kenji.blog

How to Tune TinyLLaMA the Fastest in an On-Premises Environment

Fri, 11 Sep 2026 16:00:00 +0900

1. Introduction: Why TinyLLaMA and On-Premises Now?

The evolution of Large Language Models (LLMs) is proceeding at an incredible speed, but along with it, the number of model parameters continues to inflate to the scale of hundreds of billions. While ultra-giant models like GPT-4 and Claude 3 boast unparalleled performance, the computational costs required for inference and training, as well as security and data privacy concerns when using external APIs, present significant hurdles for companies. In business operations that handle highly sensitive internal data or personal information in particular, sending data to public LLM APIs on the cloud is often unacceptable from a compliance perspective (such as GDPR or APPI).

This is where Small Language Models (SLMs) and local operation in an on-premises environment are stepping into the spotlight. Among them, “TinyLLaMA” is a compact model with only 1.1B (1.1 billion) parameters, yet it has been pre-trained on a massive dataset of approximately 3 trillion tokens, demonstrating astonishing performance compared to models in the same class.

In this article, we provide a complete guide to fine-tuning TinyLLaMA “fastest and highly efficiently” for your company’s specific tasks in an on-premises environment (local servers or workstations). We will comprehensively explain everything from the mathematical background to the latest optimization technologies and specific PyTorch implementation code.

2. TinyLLaMA Architecture and Features

TinyLLaMA follows the LLaMA (Large Language Model Meta AI) architecture developed by Meta. While keeping the number of parameters down to 1.1B, it uses the same technology stack as LLaMA 2, which features highly compatible ecosystem integration.

Major Architectural Components

RMSNorm (Root Mean Square Normalization): A normalization technique that improves computational efficiency by omitting mean subtraction from conventional LayerNorm calculations. It improves throughput while maintaining training stability.
SwiGLU Activation Function: In the Feed Forward Network (FFN), SwiGLU is adopted instead of the conventional ReLU or GELU. Mathematically, it is expressed as follows: $$ \text{SwiGLU}(x, W, V) = \text{Swish}(xW) \otimes (xV) $$ Here, $\otimes$ represents the element-wise product (Hadamard product), and the Swish function is $\text{Swish}(z) = z \cdot \sigma(\beta z)$. This significantly improves expressive power.
RoPE (Rotary Position Embedding): A method that combines the advantages of absolute position encoding and relative position encoding. It has high generalization performance even when the sequence length is extended.
Grouped Query Attention (GQA): An intermediate approach between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), grouping key and value heads to save memory bandwidth and dramatically improve inference speed.

The following Mermaid diagram illustrates the overall data flow and Transformer block structure of TinyLLaMA.

graph TD A["Input Text"] --> B["Tokenizer (BPE)"] B --> C["Embedding Layer"] C --> D["Transformer Blocks (x22 Layers for TinyLLaMA)"] D --> E["RMSNorm (Final)"] E --> F["Linear Projection (Vocab Size)"] F --> G["Output Probabilities (Softmax)"] subgraph "Transformer Block Anatomy" D1["Input Hidden State"] --> D2["RMSNorm"] D2 --> D3["Grouped Query Attention (GQA)"] D3 --> D4["Residual Add"] D4 --> D5["RMSNorm"] D5 --> D6["SwiGLU FFN"] D6 --> D7["Residual Add"] D7 --> D8["Output to Next Layer"] D1 -.-> D4 D4 -.-> D7 end

3. Breakthrough in Fine-Tuning: LoRA and QLoRA

To perform full-parameter fine-tuning in an on-premises environment, even a 1.1B model consumes tens of GB of VRAM (Video RAM) to maintain optimizer states and gradients. To learn efficiently with limited resources, PEFT (Parameter-Efficient Fine-Tuning) techniques such as “LoRA” and its quantized extension “QLoRA” are essential.

3.1 Mathematical Background of LoRA (Low-Rank Adaptation)

LoRA is a method that fixes (freezes) pre-trained weight matrices and approximates the weight updates ($\Delta W$) as the product of two small, low-rank matrices.

Let the pre-trained weights be $W_0 \in \mathbb{R}^{d \times k}$. In full fine-tuning, $W_0$ itself is updated to $W_0 + \Delta W$, but in LoRA, the update matrix $\Delta W$ is decomposed as follows:

$$ \Delta W = B \times A $$

Here, $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, where $r$ is a hyperparameter called Rank, and it is a very small value (usually 8, 16, 32, etc.) that satisfies $r \ll \min(d, k)$.

The forward pass calculation is as follows:

$$ h = W_0 x + \Delta W x = W_0 x + B A x $$

In the initial state, matrix $A$ is randomly initialized with a normal distribution (Gaussian distribution), and matrix $B$ is initialized as a zero matrix. As a result, $\Delta W$ at the start of training is zero, allowing training to start with the base model’s output perfectly preserved.

graph LR X["Input Vector x"] --> W0["Frozen Pre-trained Weight (W_0)"] X --> A["Trainable LoRA Matrix A (r x k)"] A --> B["Trainable LoRA Matrix B (d x r)"] W0 --> Add["Vector Addition"] B --> Add Add --> Y["Output Vector h"]

3.2 The Innovativeness of QLoRA (Quantized LoRA)

QLoRA pushes the LoRA approach further by quantizing the base model $W_0$ into 4-bit precision (NormalFloat 4, NF4) before loading it into memory. This drastically reduces VRAM consumption.

QLoRA incorporates three important technologies:

4-bit NormalFloat (NF4) Quantization: A theoretically optimal data type optimized for normally distributed weights.
Double Quantization: By quantizing the quantization constants (scale factors) themselves, it saves even more memory.
Paged Optimizers: Uses NVIDIA’s unified memory feature to temporarily evict optimizer states to CPU RAM when VRAM is insufficient.

With this, fine-tuning that typically requires 16GB to 24GB of VRAM can comfortably run even on consumer-grade GPUs (like RTX 3060 12GB or RTX 4070).

4. Hardware Requirements and Setup in an On-Premises Environment

When tuning TinyLLaMA (1.1B) with QLoRA, hardware requirements are kept extremely low.

Recommended Hardware Specifications

GPU: NVIDIA RTX 3060 (12GB), RTX 3090/4090 (24GB), or NVIDIA A10G/A100, etc. It can run with a minimum of 8GB of VRAM, but 12GB or more is recommended to increase the batch size.
CPU: A modern CPU with 8 or more cores (Intel Core i7/i9, AMD Ryzen 7/9)
RAM: 32GB or more (crucial as a swap destination from VRAM when using Paged Optimizers)
Storage: NVMe SSD (to speed up dataset loading and model saving)

Software Environment Setup

Here is the setup procedure assuming an Ubuntu 22.04 LTS environment. We will use Python 3.10 or later.

1
2
3
4
5
6
7
8
9


# Create and activate a virtual environment
python3 -m venv tinyllama_env
source tinyllama_env/bin/activate

# Install PyTorch (for CUDA 12.1)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install transformer-related libraries
pip install transformers datasets peft trl accelerate bitsandbytes

5. Optimization Techniques for the Fastest Tuning

To complete tuning not just by running scripts but doing so at the “fastest” speed, you need to combine the following optimization techniques.

5.1 Flash Attention 2

The standard Attention mechanism has time and space complexity of $O(N^2)$ for sequence length $N$. Flash Attention 2 optimizes memory access between the GPU’s SRAM and HBM (High Bandwidth Memory), eliminating the IO bottleneck without reducing the computational complexity. This boosts the training speed by several times and drastically reduces memory consumption.

5.2 Gradient Checkpointing

Instead of storing all intermediate activations computed during the forward pass in VRAM, this technique saves only a portion of them and recomputes them when needed during the backward pass. The computation time increases by about 20%, but memory consumption is drastically reduced, allowing for a larger batch size and resulting in improved overall throughput.

5.3 Mixed Precision Training and Bfloat16

To maximize the use of GPU Tensor Cores, calculations during training are performed in bfloat16 (Brain Floating Point). Since the bit length of the exponent is the same as float32 compared to float16, the risk of overflow and underflow is extremely low, stabilizing the training process.

6. Practice: QLoRA Fine-Tuning Code for TinyLLaMA

Let’s break down the PyTorch script for the fastest tuning incorporating all the optimizations mentioned above. Here, we will use the SFTTrainer from Hugging Face’s trl (Transformer Reinforcement Learning) library.

6.1 Dataset Preparation and Model Loading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


import torch
from datasets import load_dataset
from transformers import (
 AutoModelForCausalLM,
 AutoTokenizer,
 BitsAndBytesConfig,
 TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# 1. Specify model and tokenizer
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

# 2. 4-bit quantization settings for QLoRA
bnb_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_use_double_quant=True,
 bnb_4bit_quant_type="nf4",
 bnb_4bit_compute_dtype=torch.bfloat16 # Perform calculations in bfloat16
)

# 3. Load model (Enable Flash Attention 2)
print("Loading model...")
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 quantization_config=bnb_config,
 device_map="auto",
 use_flash_attention_2=True # Key to maximum speed
)

# 4. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right" # Set to right to avoid bugs during fp16/bf16 training

6.2 Applying the LoRA Adapter and Formatting the Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# 5. Prepare for k-bit training and enable gradient checkpointing
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

# 6. LoRA configuration
peft_config = LoraConfig(
 r=16, # Rank
 lora_alpha=32, # Scaling factor
 lora_dropout=0.05,
 bias="none",
 task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"] # Targeting all Linear layers improves performance
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# Example output: trainable params: 14,286,848 || all params: 1,114,335,232 || trainable%: 1.282%

# 7. Load dataset (Using Japanese Instruction dataset as an example here)
# In practice, you would load an on-premises private JSONL file, etc.
dataset = load_dataset("kunishou/databricks-dolly-15k-ja", split="train")

def format_instruction(sample):
 """
 Formats the string according to ChatML format or prompt template
 """
 prompt = f"<|im_start|>user\n{sample['instruction']}"
 if sample.get("input", "") != "":
 prompt += f"\n{sample['input']}"
 prompt += f"<|im_end|>\n<|im_start|>assistant\n{sample['output']}<|im_end|>"
 return {"text": prompt}

dataset = dataset.map(format_instruction)

6.3 Executing the Training

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


# 8. Set training arguments
training_args = TrainingArguments(
 output_dir="./tinyllama-lora-output",
 per_device_train_batch_size=8, # Increase if you have spare VRAM
 gradient_accumulation_steps=2, # Effective batch size = 8 * 2 = 16
 optim="paged_adamw_32bit", # VRAM savings with Paged Optimizer
 save_steps=100,
 logging_steps=10,
 learning_rate=2e-4,
 fp16=False,
 bf16=True, # Mixed precision training (bfloat16)
 max_grad_norm=0.3,
 max_steps=500, # 500 steps for testing. Specify by epochs in production
 warmup_ratio=0.03,
 group_by_length=True,
 lr_scheduler_type="cosine",
)

# 9. Start training with SFTTrainer
trainer = SFTTrainer(
 model=model,
 train_dataset=dataset,
 peft_config=peft_config,
 dataset_text_field="text",
 max_seq_length=1024, # Adjust according to the expected input length
 tokenizer=tokenizer,
 args=training_args,
)

print("Starting training...")
trainer.train()

# 10. Save the LoRA adapter
trainer.model.save_pretrained("./tinyllama-lora-final")
tokenizer.save_pretrained("./tinyllama-lora-final")
print("Training complete and model saved.")

7. Performance Evaluation and Troubleshooting

Here are common issues encountered when running training in an on-premises environment and their solutions.

OOM (Out Of Memory) occurs:
- Lower per_device_train_batch_size to 1.
- Maintain the effective batch size by increasing gradient_accumulation_steps.
- Shorten max_seq_length from 2048 to 1024 or 512.
Loss doesn’t drop or diverges:
- The learning rate (learning_rate) might be too high. Try lowering it from 2e-4 to around 5e-5.
- If using Float16 instead of Bfloat16, gradient underflow might be occurring. Ensure bf16=True is set.
Mysterious strings are generated during inference:
- Ensure padding_side="right" is correctly set. Also, verify that the dataset format (<|im_start|> and other special tokens) is consistent with the base model’s pre-training.

8. Model Deployment After Tuning

Once tuning is complete, what gets saved is not the “entire base model,” but only the “LoRA adapter (delta weights),” which is a few MB to tens of MB in size. To perform fast inference, you need to merge (integrate) these LoRA weights back into the original base model and export it as a single model.

Model Merging Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

output_dir = "./tinyllama-lora-final"

# Load model and adapter in FP16/BF16
model = AutoPeftModelForCausalLM.from_pretrained(
 output_dir,
 device_map="auto",
 torch_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(output_dir)

# Merge weights and save
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./tinyllama-merged", safe_serialization=True)
tokenizer.save_pretrained("./tinyllama-merged")
print("Model merged and saved successfully!")

Launching an Ultra-Fast Inference Server with vLLM

For deployment in an on-premises environment, we strongly recommend using vLLM or TGI (Text Generation Inference) instead of Hugging Face’s standard pipeline to maximize inference speed (Tokens per second). vLLM uses PagedAttention technology to prevent GPU memory fragmentation and dramatically improves the processing capability for concurrent requests.

The following Mermaid diagram shows the pipeline from training to deploying an inference server.

graph TD A["Raw Private Data"] --> B["Preprocessing & Formatting (JSONL)"] B --> C["QLoRA Fine-Tuning (SFTTrainer)"] C --> D["LoRA Adapter Weights (.safetensors)"] D --> E["Merge with Base TinyLLaMA 1.1B"] E --> F["Merged Model"] F --> G["Deploy via vLLM Server"] G --> H["API Endpoint / UI (e.g. Chatbot)"]

Starting an API server using vLLM can be completed with the following single command:

1
2
3
4
5
6


python -m vllm.entrypoints.openai.api_server \
 --model ./tinyllama-merged \
 --host 0.0.0.0 \
 --port 8000 \
 --max-model-len 2048 \
 --dtype bfloat16

Now, an OpenAI API-compatible endpoint is built in your on-premises environment, allowing you to utilize local AI securely and quickly.

9. Conclusion

In this article, we explained the methods for performing the fastest and most memory-efficient fine-tuning in an on-premises environment for “TinyLLaMA”, a model that is highly performant despite being lightweight with 1.1B parameters.

LoRA / QLoRA enables full-fledged LLM tuning even on consumer GPUs.
Leveraging Flash Attention 2 and Gradient Checkpointing optimizes training time and VRAM consumption to the limit.
Deployment using vLLM achieves high throughput even in production environments.

Operating a local LLM on-premises is not only for protecting data confidentiality but also serves as the ultimate weapon for building specialized AI at low cost for specific domains (legal, medical, internal regulations, etc.). We encourage you to refer to this guide and nurture your own TinyLLaMA.

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

Fri, 11 Sep 2026 14:00:00 +0900

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

In recent years, interest in running Large Language Models (LLMs) in local environments has grown rapidly. In particular, small-scale models like TinyLLaMA (1.1B parameters) can perform inference at practical speeds even on limited-resource edge devices and typical laptops (including Windows environments). While development using Python and PyTorch is mainstream, when pursuing ultimate performance and memory efficiency, the combination of C++ and the C-based tensor library “ggml” has become the de facto standard.

This article provides an extremely detailed development guide for building an inference engine from scratch (or deeply understanding the internal structure of the existing llama.cpp) to load TinyLLaMA and generate text using C++.

1. Why C++ and ggml?

In the AI training phase, Python has an overwhelming advantage due to its flexibility and rich ecosystem. However, in the deployment or “Inference” phase, C++ becomes a powerful choice for the following reasons:

Overhead Reduction: The Python Global Interpreter Lock (GIL) and runtime overhead can be completely eliminated.
Memory Efficiency and Arena Allocation: By manually controlling memory allocation and deallocation, you can prevent unpredictable spikes caused by garbage collection.
Direct Hardware Access: By directly calling SIMD intrinsics such as AVX-512, AVX2, and ARM NEON, the CPU’s computational power can be maximized.
Zero Dependencies: ggml is a zero-dependency C/C++ library. As long as you have a compiler, it can be easily built even in an MSVC environment on Windows.

2. Overall Architecture

The flow of the entire inference pipeline is shown in the Mermaid diagram below. This is a series of processes starting from the user’s input text until the final next token is generated.

graph TD A["User Input Text"] --> B["BPE Tokenizer"] B --> C["Token IDs Array"] C --> D["Embedding Layer Lookup"] D --> E["Transformer Blocks"] E --> F["RMSNorm"] F --> G["LM Head Layer"] G --> H["Logits Array"] H --> I["Sampler Module"] I --> J["Next Token ID"] J --> K["Detokenizer"] K --> L["Output Text Chunk"] J -.-> |"Append to Context"| C

Since it is an autoregressive model, the output token is added back to the context and circulates as input for predicting the next token (the dotted line in the diagram).

3. Model Format and Memory Mapping (mmap)

The biggest hurdle in handling the weights of massive neural networks is disk I/O and memory consumption. In a C++ implementation, this is resolved with memory mapping (mmap).

3.1 How Memory Mapping Works and its Windows Implementation

Using mmap allows you to map the contents of a file directly into the virtual memory space of the process.

Zero-copy: Data is loaded directly from the disk into the kernel’s page cache, preventing extra copies into user space.
On-Demand Loading (Page Fault): The moment the CPU actually accesses that memory address, a page fault occurs, and only the required chunk (usually 4KB) is loaded into physical memory.

In a Windows environment, the Win32 APIs CreateFileMapping and MapViewOfFile are used instead of POSIX mmap.

sequenceDiagram participant OS["Windows OS"] participant RAM["Physical Memory"] participant App["C++ Application"] App->>OS: "CreateFileMapping / MapViewOfFile" OS-->>App: "Virtual Memory Address Pointer" App->>App: "Read Tensor Data at Pointer" OS->>RAM: "Page Fault / Load page from Disk" RAM-->>App: "Data ready for SIMD Compute"

3.2 Binary Structure of the GGUF Format

Converted from formats like Hugging Face’s .safetensors, the GGUF (GPT-Generated Unified Format) is the ultimate format for inference. It has the following strict binary layout:

Magic Bytes: 0x46554747 (GGUF).
Version: Format version number.
Tensor Count & Metadata Count: The number of tensors and key-value metadata pairs.
Metadata (Key-Value Pairs): Typed values and keys with string length prefixes.
Tensor Info: The name, number of dimensions, data type (FP16, Q4_K, etc.), and offset position in the file for each tensor.
Padding: Padding inserted so that tensor data is aligned to specific boundaries (usually 32 bytes or 64 bytes). This is essential for fast memory access with SIMD instructions (especially AVX).
Tensor Data: The actual aligned weight data array.

4. Mathematical Foundations of TinyLLaMA and C++ Algorithms

TinyLLaMA incorporates several advanced architectural ingenuities for efficiency. We explain the mathematical expressions to correctly implement these in C++.

4.1 RMSNorm (Root Mean Square Normalization)

It reduces computational cost by omitting mean centering from LayerNorm and only performing variance scaling.

$$ \text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \gamma $$

$d$ is the number of dimensions, and $\gamma$ is the learned scaling tensor. When implementing in C++, it is optimized by first rapidly calculating the sum of squares of the array using AVX2’s _mm256_fmadd_ps or similar, and then multiplying by the inverse square root (e.g., the _mm256_rsqrt_ps instruction).

4.2 RoPE (Rotary Position Embedding)

A technique that applies token position information as a rotation in tensor space. It can be seen as a rotation on a complex plane, applying the following rotation to adjacent dimension pairs $(x_1, x_2)$ of vector $x$:

$$ \text{RoPE}(x, m) = \begin{pmatrix} x_{1} \cos(m\theta) - x_{2} \sin(m\theta) \\ x_{1} \sin(m\theta) + x_{2} \cos(m\theta) \end{pmatrix} $$

Here, $m$ is the absolute position index of the token, and $\theta$ is a pre-calculated base frequency. In ggml, it is executed in parallel simply by adding a ggml_rope operator during inference graph construction.

4.3 Grouped-Query Attention (GQA)

In standard Multi-Head Attention (MHA), Query, Key, and Value each have the same number of heads. However, TinyLLaMA adopts Grouped-Query Attention (GQA) to drastically reduce memory bandwidth and KV cache consumption.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$

In GQA, multiple Query heads share a single Key/Value head. In the C++ implementation, before executing the matrix multiplication ggml_mul_mat, it is necessary to broadcast the KV tensors to match the number of Queries.

4.4 SwiGLU Activation Function

In the Feed-Forward Network (FFN) layer, SwiGLU is used instead of GELU.

$$ \text{SwiGLU}(x) = \text{Swish}(x W_{\text{gate}}) \otimes (x W_{\text{up}}) $$ $$ \text{Swish}(z) = z \cdot \sigma(z) = z \cdot \frac{1}{1 + e^{-z}} $$

In the computation graph, it is represented by combining the ggml_silu operator and ggml_mul.

5. Computation Graph Construction and Memory Management with ggml

ggml uses a “Define-and-Run” approach, constructing a static computation graph for inference and evaluating it later.

5.1 ggml_context and Arena Allocator

The most unique aspect of ggml is “arena allocation,” which avoids dynamic memory allocation (malloc or new) entirely within the inference loop. Upon initialization, a huge contiguous memory region (arena) is allocated, and the pointer to this region is incremented every time ggml_new_tensor or similar is called. Once one inference step is completed, simply resetting the allocation pointer to its initial position immediately finishes memory allocation for the next inference step.

5.2 Specific Example of Graph Construction

For each inference step, a computation graph like the following is assembled in memory:

graph TD A["Tokens Input ID"] --> B["Embed Lookup"] B --> C["ggml_rms_norm"] C --> D["Q / K / V Projections"] D --> E["ggml_rope Positional"] E --> F["KV Cache Store"] E --> G["KV Cache Load"] G --> H["Self Attention"] H --> I["Scale & Softmax"] I --> J["Attention Output"] J --> K["Out Projection"] K --> L["Add Residual"]

6. Quantization and Windows / SIMD Optimization

Handling TinyLLaMA (1.1B) in FP16 requires approximately 2.2GB of memory, but it can be dramatically compressed to around 600MB through 4-bit quantization (such as Q4_K).

6.1 Block Quantization Architecture

ggml does not quantize the entire tensor uniformly; instead, it does so in “block” units. In the Q4_0 format, 32 FP16 values are grouped into a single block.

Scale Factor: One FP16 value (2 bytes)
Quantized Data: 32 4-bit values (16 bytes) This minimizes the impact of local outliers.

6.2 Dot Product Acceleration with AVX2

When building for the latest x86 CPUs in a Windows environment, utilizing compiler flags like /arch:AVX2 allows SIMD processing to be performed in the following flow:

Load: Load 4-bit quantized data from memory into 256-bit AVX registers.
Expansion and Unpacking: Expand 4-bit values into Int8 or Int16 using bit masks and shift operations.
Dequantization: Multiply by the scale factor to convert to floating-point numbers.
FMA Operations: Execute multiply-add operations in parallel using activation values and _mm256_fmadd_ps (Fused Multiply-Add).

7. KV Cache Implementation Details

In autoregressive generation, the “KV cache” is an essential feature to skip the computation of Keys and Values for past tokens.

The key points for a C++ implementation are as follows:

Tensor Pre-allocation: Initialize a massive tensor for the KV cache (FP16 recommended) corresponding to the maximum context length (e.g., 2048 tokens).
Offset Copying: When calculations for token position $N$ are performed, the K and V vectors obtained in that step are stored into the $N$-th row of the KV cache tensor using ggml_cpy or similar.
Creating a View during Attention: When calculating attention, create a “view” that points only to the token portion from 0 to the $N$-th token and pass it to the matrix multiplication.

8. BPE Tokenizer and Decoding

The input string is treated as a UTF-8 byte sequence and matched against a predefined vocabulary. In C++, to speed up the vocabulary search, algorithms using a Trie (prefix tree) or a priority queue are implemented.

From the logits output by the LM Head, probabilities are scaled using the Temperature parameter, candidates are narrowed down using Top-K extraction or Top-P (Nucleus Sampling) methods, and the final next token is determined using random numbers.

9. Setting Up the C++ Project (Windows / PowerShell Environment)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cmake_minimum_required(VERSION 3.14)
project(TinyLLaMACpp)

set(CMAKE_CXX_STANDARD 17)

# Windows (MSVC) optimization and AVX2 flag settings
if(MSVC)
 add_compile_options(/O2 /arch:AVX2 /fp:fast)
 add_link_options(/STACK:8388608)
else()
 add_compile_options(-O3 -march=native -ffast-math)
endif()

add_library(ggml OBJECT ggml/ggml.c ggml/ggml-alloc.c)
target_compile_definitions(ggml PRIVATE GGML_USE_AVX2 GGML_USE_F16C GGML_USE_FMA)

add_executable(main main.cpp)
target_link_libraries(main ggml)

Example build commands in PowerShell:

1
2
3
4


mkdir build
cd build
cmake .. -G "Visual Studio 17 2022" -A x64
cmake --build . --config Release

10. Conclusion

Implementing an inference engine from scratch for small AI models like TinyLLaMA using C++ and ggml is a perfect opportunity to demystify the black box of deep learning and learn the beauty of low-level hardware control. Let’s pave the way for the future of edge AI while fully savoring the essence of systems programming, such as zero-copy loading using memory mapping, SIMD optimization, and KV cache construction.

[RAG Implementation Guide] How to Load Your Own Documents into Local AI

Fri, 11 Sep 2026 13:00:00 +0900

Introduction

In recent years, the evolution of Large Language Models (LLMs) has been remarkable, and many AIs, led by ChatGPT and Claude, have permeated our daily lives and business operations. However, general LLMs have a distinct weakness. That is, they only know “public information at the time of their training”. Naturally, they cannot answer questions about “private documents” such as internal company regulations, personal notes, and unpublished project materials. Forcing them to answer increases the risk of generating plausible lies (hallucinations) that differ from the facts.

Therefore, the technological architecture known as RAG (Retrieval-Augmented Generation) is currently spreading explosively worldwide. By using RAG, it becomes possible to dynamically provide unique knowledge to the LLM from an external database and have it generate accurate and well-founded answers based on it.

Furthermore, when handling enterprise domains or personal confidential information, sending data to cloud-based APIs like OpenAI is often unacceptable under security policies. What is required there is the construction of “Local RAG” combined with Local AI (an LLM that operates entirely on your own PC or on-premise server).

In this article, we will thoroughly explain everything from the fundamental theory of RAG, specific implementation methods of Local RAG using Python, mathematical background (how vector search works), to advanced techniques for running the system in production.

1. Overall Architecture of RAG

RAG is not a single AI model, but a system architecture where multiple components work together. It broadly consists of two phases: the “Ingestion (Data Loading) Phase” and the “Retrieval & Generation Phase”.

The Mermaid diagram below shows the overall picture of a RAG system.

graph TD subgraph "Ingestion Phase (Preparation)" Doc["Custom Documents (PDF, TXT, etc.)"] --> Loader["Document Loader"] Loader --> Splitter["Text Splitting (Chunking)"] Splitter --> EmbedModel1["Embedding Model"] EmbedModel1 --> VectorDB["Vector Database"] end subgraph "Inference Phase (User Query)" User["User Question (Query)"] --> EmbedModel2["Embedding Model"] EmbedModel2 --> QueryVector["Query Vector"] QueryVector --> Search["Similarity Search (Vector Search)"] VectorDB --> Search Search --> Context["Relevant Chunk Extraction (Context)"] User --> PromptBuilder["Prompt Builder"] Context --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer Generation"] end

Ingestion Phase (Preparation)

Document Loading: Loads unstructured data such as PDFs, Word documents, and text files.
Chunking (Text Splitting): Splits long texts into meaningful chunks to fit within the LLM’s input limit (context window) and to improve search accuracy.
Embedding (Vectorization): Inputs the split chunks into an Embedding Model and converts them into an array of numerical values (vectors) with hundreds to thousands of dimensions.
Saving to Database: Saves the converted vectors and their associated original text data into a Vector Database (Vector DB).

Inference Phase (Runtime)

Query Vectorization: Vectorizes the user’s question text using the same embedding model used in the preparation phase.
Similarity Search: Performs a similarity calculation between the query vector and the document vectors in the database, and retrieves the top few semantic (highly relevant) text chunks.
Prompt Building: Combines the retrieved relevant text as “context (background knowledge)” with the user’s question text to create an input prompt for the LLM.
Answer Generation: The LLM, receiving the augmented prompt, generates an answer based on the provided context information.

2. Deep Understanding of Vector Search and Embeddings

At the core of RAG is “Vector Search (Semantic Search)”. While traditional keyword search (like BM25) is based on exact word matching and frequency, vector search is based on “semantic similarity”. For example, even if the words are different, like “dog” and “puppy”, or “PC” and “computer”, they will be hit in the search if their meanings are close.

What is an Embedding Model?

An embedding model is a neural network that takes natural language text as input and outputs a fixed-length dense vector. Common models (e.g., text-embedding-3-small or the open-source multilingual-e5-large) map text into a vector of real numbers with 384 or 1024 dimensions.

In this multi-dimensional space (latent space), the model is trained so that sentences with similar meanings are closer in distance in the coordinate space.

Mathematical Background of Similarity Calculation: Cosine Similarity

When a vector database searches for relevant documents, the most commonly used distance metric is Cosine Similarity. Unlike Euclidean distance (absolute spatial distance), cosine similarity focuses on the “angle between two vectors”. Since it is less affected by the length of the sentence (the norm of the vector), it is highly suitable for calculating text similarity.

Expressed mathematically, the cosine similarity between vectors $\mathbf{A}$ and $\mathbf{B}$ is as follows:

$$ \text{Cosine Similarity}(\mathbf{A}, \mathbf{B}) = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}} $$

$\mathbf{A} \cdot \mathbf{B}$ represents the Dot Product.
$\|\mathbf{A}\|$ represents the L2 norm (length) of vector $\mathbf{A}$.
$n$ is the number of dimensions of the vector.

Cosine similarity takes a value from -1 to 1.

Close to 1: The directions of the two vectors are almost the same (meanings are very similar).
Close to 0: The two vectors are orthogonal (unrelated).
Close to -1: The two vectors are in opposite directions (meanings are opposite).

Recent Vector DBs (Chroma, FAISS, Qdrant, etc.) employ an Approximate Nearest Neighbor (ANN) algorithm called HNSW (Hierarchical Navigable Small World), optimizing them to search for documents with high cosine similarity in milliseconds even from millions of vector data.

3. Technology Stack for Building Local RAG

To build a fully local RAG that does not rely on the cloud, we leverage the open-source ecosystem. The recommended technology stack is introduced below.

Large Language Model (LLM)
- Tools: Ollama or Llama.cpp
- Models: Lightweight, high-performance open models like Llama-3-8B-Instruct, Gemma-2-9B-It, Qwen2-7B-Instruct. For Japanese tasks, Japanese-tuned models like Llama-3-ELYZA-JP-8B are suitable.
Embedding Model (Embedding)
- Models: intfloat/multilingual-e5-large or BAAI/bge-m3. When running locally, it is common to download them from Hugging Face and run them with Sentence-Transformers.
Vector Database (Vector DB)
- ChromaDB: Python-based and extremely easy to set up. Ideal for local development.
- FAISS: A fast vector search library developed by Meta.
- Qdrant / Milvus: For larger scale and production environments.
Orchestration Framework
- LangChain: The de facto standard for chaining components together.
- LlamaIndex: A data connection framework specifically specialized for RAG.

This time, we will implement it using the easiest combination to introduce: LangChain + ChromaDB + Ollama + HuggingFaceEmbeddings.

4. Implementation Tutorial: Building a Full Local RAG with Python

From here, we will build a local RAG while actually writing Python code. Please install Ollama on your PC in advance and have it running in the background. Also, pull a model on Ollama (e.g., ollama run llama3).

Step 1: Installing Required Libraries

1
2


pip install langchain langchain-community langchain-huggingface
pip install chromadb sentence-transformers pypdf

Step 2: Complete Implementation Code

Below is the complete Python script to load a PDF file, vectorize it, and have the local LLM answer questions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93


import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain_core.prompts import PromptTemplate
from langchain.chains import RetrievalQA

def main():
 # 1. Document Loading
 print("Loading document...")
 # Specify the path of the PDF you want to load
 file_path = "sample_company_policy.pdf"
 loader = PyPDFLoader(file_path)
 documents = loader.load()

 # 2. Chunking (Text Splitting)
 # Split into appropriate sizes without breaking the meaning of the sentences
 text_splitter = RecursiveCharacterTextSplitter(
 chunk_size=500, # Maximum number of characters per chunk
 chunk_overlap=50, # Overlap characters between previous and next chunks (prevents context disconnection)
 separators=["\n\n", "\n", "。", "、", " ", ""]
 )
 chunks = text_splitter.split_documents(documents)
 print(f"Split into {len(chunks)} chunks.")

 # 3. Initialization of Embedding Model (Local HuggingFace Model)
 # Using a multilingual model with strong Japanese support
 print("Loading embedding model...")
 embeddings = HuggingFaceEmbeddings(
 model_name="intfloat/multilingual-e5-large",
 model_kwargs={'device': 'cpu'} # 'cuda' or 'mps' if you have a GPU
 )

 # 4. Building Vector Database (Chroma)
 print("Building vector database...")
 persist_directory = "./chroma_db"
 vectorstore = Chroma.from_documents(
 documents=chunks,
 embedding=embeddings,
 persist_directory=persist_directory
 )
 # Creating a Retriever. Configured to get the top 3 relevant documents
 retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

 # 5. Initialization of Local LLM (Ollama)
 print("Connecting to Local LLM...")
 # Make sure to get the model in advance with 'ollama pull llama3' etc.
 llm = Ollama(model="llama3")

 # 6. Definition of Prompt Template
 prompt_template = """You are an excellent assistant familiar with company regulations and internal information.
Please answer the user's question in detail based ONLY on the following context (background information).
If you cannot find the answer from the context, please do not guess and honestly answer "I don't know from the provided information."

[Context]
{context}

[Question]
{question}

[Answer]:
"""
 PROMPT = PromptTemplate(
 template=prompt_template,
 input_variables=["context", "question"]
 )

 # 7. Building RAG Chain
 qa_chain = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff",
 retriever=retriever,
 return_source_documents=True, # Set whether to return source documents
 chain_type_kwargs={"prompt": PROMPT}
 )

 # 8. Executing a Question
 query = "Please tell me about the conditions for transportation expenses payment regarding remote work."
 print(f"\nQuestion: {query}\n")

 result = qa_chain.invoke({"query": query})

 print("[Answer]")
 print(result['result'])
 print("\n---")
 print("[Reference Sources]")
 for doc in result['source_documents']:
 print(f"- Page {doc.metadata.get('page', 'Unknown')}: {doc.page_content[:50]}...")

if __name__ == "__main__":
 main()

Explanation of Key Code Points

RecursiveCharacterTextSplitter: This is the most recommended splitter for dividing natural language. It attempts to split in the order of paragraphs (\n\n), lines (\n), and periods (。), keeping semantic blocks together as much as possible while fitting within the specified chunk_size. By setting chunk_overlap, you prevent context boundaries from being cut off and information from being lost.
HuggingFaceEmbeddings: intfloat/multilingual-e5-large is a very powerful open-source embedding model that supports multiple languages. You can vectorize text locally on your memory offline without using a cloud API (like OpenAI’s text-embedding-ada-002).
ChromaDB: Since it runs in-memory or on local storage (SQLite-based), there is no need to spin up a complex database server. By specifying persist_directory, you can skip the vectorization process on subsequent runs and load the DB from disk.

5. Advanced RAG Techniques

The basic RAG system (Naive RAG) built in the above tutorial works, but if high answer accuracy is required in a production environment, the introduction of advanced techniques like the following becomes necessary.

5.1 Hybrid Search

While vector search is good at capturing “meaning”, it can be poor at strict keyword searches like “specific proper nouns”, “product model numbers”, and “employee IDs”. Therefore, by running semantic search via vector search and keyword search using algorithms like BM25 in parallel, and integrating both results by scoring them (using techniques like Reciprocal Rank Fusion; RRF), you can drastically reduce search misses.

5.2 Re-ranking

Vector search is fast, but it does not necessarily evaluate the exact contextual relevance of the context. A general pipeline for improving search accuracy is as follows:

First-stage Retrieval: Retrieve a broad and shallow range of relevant chunks (about 20-30) from the Vector DB.
Re-ranking: Use another heavier machine learning model called a Cross-Encoder (e.g., bge-reranker) to input pairs of the user’s query and the retrieved chunks, and recalculate their semantic relevance scores.
Selection: Pass only the top 3-5 with the highest scores as the final context to the LLM prompt.

This technique prevents irrelevant noise information from being passed to the LLM, significantly increasing the precision of the answers.

graph LR Query["Query"] --> VSearch["Vector Search (Top 20)"] VSearch --> Reranker["Re-ranker Model (Cross-Encoder)"] Query --> Reranker Reranker --> TopK["High Precision Top 3"] TopK --> LLM["LLM Generation"]

5.3 Semantic Chunking and Parent Document Retrieval

Instead of mechanically splitting text by a fixed number of characters, there is a technique called “Semantic Chunking” that uses AI to detect shifts in meaning and splits the text accordingly. Also, in the “Parent Document Retriever” technique, you vectorize in very small units (like sentences) for high-precision search, but when passing it to the LLM, you provide the “original large paragraph (parent document)” containing that sentence, thus providing sufficient context to the LLM.

6. Challenges and Countermeasures when Operating Local RAG

There are unique hurdles when building and operating RAG in a local environment.

VRAM (Video Memory) Exhaustion: To run a Local LLM at a practical speed (dozens of tokens per second), you need to load the model into the GPU’s VRAM. Running an 8B class model in fp16 (16-bit floating point) requires about 16GB of VRAM, but by using Quantization technologies (compressing to 4-bit or 8-bit, such as GGUF or AWQ formats), it is possible to run it fast enough even with 8GB of VRAM (like a standard gaming PC). Llama.cpp and Ollama support these quantization formats by default.
Context Window Limits: If the amount of retrieved context is too large, it may exceed the LLM’s input limit (token limit), or the model might forget the middle part of the information (Lost in the middle phenomenon). Adjusting the number of chunks to extract and carefully selecting them through the aforementioned re-ranking techniques are essential.
Data Freshness Management: When a source document is updated, the corresponding document’s vector in the vector database also needs to be updated or deleted (CRUD operations). Since ChromaDB supports updates based on document IDs, it is practical to manage file hash values and set up a batch process to sync only the differences.

Conclusion

RAG (Retrieval-Augmented Generation) is a powerful paradigm that evolves AI from a general-purpose assistant into your “exclusive expert” or an “expert specialized in internal business”.

We found that even with highly confidential requirements where cloud services cannot be used, a complete “Local RAG” environment can be relatively easily built by combining the open-source ecosystem such as Ollama, LangChain, and ChromaDB.

Based on the advanced approaches such as mathematical understanding of vector space, text splitting, and re-ranking explained in this article, please try developing an original AI system using your own data. The speed of evolution in local AI is astounding, and the system you build today can instantly update its performance tomorrow simply by swapping in a smarter lightweight model that appears.

This blog will continue to publish deep-dive articles on AI technology and RAG. If you have any questions or feedback, please share them in the comments section.

【2026 Edition】 The Complete Guide to Running Local LLMs in a Windows Environment

Fri, 11 Sep 2026 10:00:00 +0900

1. Introduction: Why Local LLMs on Windows Now?

As of 2026, the evolution of generative AI and Large Language Models (LLMs) shows a major paradigm shift from gigantic cloud-based API services to “local LLMs” running on personal PCs and on-premise environments. While cloud AIs like OpenAI’s GPT-5 and Anthropic’s Claude 3.5 are incredibly powerful, not all companies and individuals can send all their data to the cloud. From the perspectives of privacy, security, latency, and long-term sustainable costs, the demand for local LLMs is exploding like never before.

The evolution of the local LLM ecosystem, especially in the Windows environment, is remarkable. Until a few years ago, “Linux for AI development and execution” was common sense, but as of 2026, Windows has transformed into an extremely powerful and accessible AI platform.

In this article, based on the latest technology trends of 2026, we provide a complete guide to building, operating, and optimizing local LLMs in a Windows environment. From easy setup using Ollama for beginners to extreme optimization using llama.cpp for advanced users, and further deep dives into the mathematical approach of VRAM calculation, deep understanding of the architecture, and local fine-tuning, we will explain everything thoroughly with an overwhelming volume.

1.1 Technology Trends Surrounding Local LLMs in 2026

The major trends shaping the current local LLM ecosystem are as follows:

Complete Popularization of the GGUF Format: GGUF (GPT-Generated Unified Format), which integrates metadata and tensors into a single file, has completely become the de facto standard. With this, simply downloading a single file from Hugging Face makes it executable in any environment.
Democratization of the MoE (Mixture of Experts) Architecture: Many small but high-performance MoE models have been released. By activating only a portion of the experts during inference, they achieve performance comparable to giant models while keeping the computational load on consumer PCs low.
Advanced Abstraction and Optimization of Inference Engines: Tools like Ollama, LM Studio, and AnythingLLM have been refined so that users no longer need to be aware of complex dependencies like CUDA driver installations. Also, the native Windows support for FlashAttention 3 has dramatically improved inference speed.
Utilization of NPUs and the Rise of Windows Copilot+ PCs: Even on laptops without GPUs, the technology to run small LLMs (SLM: Small Language Models) with low power consumption using the built-in NPU (Neural Processing Unit) has entered the practical stage.

2. Hardware Requirements and OS Preparation

To run local LLMs at practical speeds (15-30 tokens per second or more), selecting the right hardware is the most important factor.

2.1 Recommended Hardware Configuration

With the evolution of AI PCs, required specs are also changing.

OS: Windows 11 Pro (24H2 or later). Essential for fully utilizing WSL2’s features, advanced memory management, and the latest DirectML APIs.
CPU: Intel Core Ultra 200 series or higher, or AMD Ryzen 9000 series or higher. When using CPU inference alongside, broad-bandwidth memory communication is indispensable.
RAM: Minimum 32GB, recommended 64GB or more. Main memory bandwidth (MB/s) becomes a crucial bottleneck during CPU inference or offloading. High-speed memory of DDR5-6000 or above is ideal.
GPU: NVIDIA RTX 4000/5000 series. The most important thing for local LLMs is not computing performance but “VRAM capacity”.
- Entry: RTX 4060 Ti (16GB version) - Best cost performance. Ideal for 8B-14B class models.
- Mid-range: RTX 4070 Ti SUPER (16GB) / RTX 4080 SUPER (16GB)
- High-end: RTX 4090 (24GB) / RTX 5090 (32GB) - Necessary to run 30B-70B class quantized models.
Storage: PCIe Gen4 or Gen5 NVMe SSD. Dramatically reduces the load times of models that are tens of gigabytes in size.

2.2 Setting up WSL2 (Windows Subsystem for Linux 2)

While many GUI tools work natively on Windows, WSL2 is extremely useful for Python development, compiling the latest tools, and the LoRA fine-tuning mentioned later. In the latest Windows 11 environment, just by installing the NVIDIA driver on the host side, the GPU (CUDA) can be transparently used from WSL2.

Open PowerShell with administrator privileges and execute the following:

1
2
3
4
5


# Install WSL2 and the latest Ubuntu
wsl --install -d Ubuntu-24.04

# Update the kernel
wsl --update

After installation, run nvidia-smi inside the WSL2 terminal, and if the GPU is recognized correctly, it is a success.

3. Local LLM Architecture and Inference Mechanism

Understanding how models generate text in a local environment and their internal structure is very useful for troubleshooting and optimization.

The following Mermaid diagram shows a typical local LLM inference pipeline.

graph TD User["User Input (Prompt)"] --> Tokenizer["Tokenizer"] Tokenizer --> Embedding["Embedding Layer"] subgraph "Transformer Block (x Layers)" Embedding --> Attn["Self-Attention"] Attn --> KVCache["KV Cache (Key/Value Storage)"] Attn --> FFN["Feed-Forward Network (FFN)"] end FFN --> Logits["Logits Calculation"] Logits --> Sampler["Sampler (Temperature, Top-K, Top-P)"] Sampler --> OutputToken["Output Token"] OutputToken --> |"Autoregressive Generation"| Tokenizer OutputToken --> Decoder["Detokenizer"] Decoder --> FinalOutput["Final Output Text"]

3.1 Two Phases: Prefill and Decode

LLM text generation is divided into two phases with different computational characteristics.

Prefill Phase (Prompt Processing): The phase that processes and understands the entire input prompt at once. Since parallel computing is possible, the computational power of the GPU (FLOPS) directly links to speed. If the prompt is long, this phase can take several seconds.
Decode Phase (Token Generation): The phase that predicts one token at a time and feeds it to the next input (autoregressive). Since parallel computing is restricted in this phase, GPU VRAM bandwidth (Memory Bandwidth) becomes the definitive bottleneck.

4. Mathematical Understanding of VRAM Consumption and Model Size

To correctly determine “which model will run on my PC?”, you need to understand the VRAM calculation formula. When VRAM shortages cause a fallback to system memory (RAM), inference speeds drop by 10x to 100x.

4.1 Base VRAM Based on Parameter Size

This is the amount of memory needed to load the model’s weights into VRAM. Calculate it using the model size $P$ (number of parameters, unit: 1 billion = 1B) and the number of bytes per parameter $B$.

$$ V_{base} = P \times B \quad \text{(GB)} $$

For example, when loading an 8B (8 billion) parameter model in FP16 (half-precision floating point, 16 bits = 2 bytes):

$$ V_{base} = 8 \times 2 = 16 \text{ GB} $$

In other words, even a GPU with 16GB of VRAM will reach its limit just by loading the model.

4.2 The Magic of Quantization

This is where “quantization” comes in. By reducing the precision of the parameters, the model size is drastically shrunk. In the case of the most common 4-bit quantization (e.g., Q4_K_M), it averages to about 0.55 bytes per parameter.

$$ V_{base\_4bit} = 8 \times 0.55 = 4.4 \text{ GB} $$

With this, if you have 16GB of VRAM, you can run an 8B model with plenty of headroom.

4.3 KV Cache Calculation (GQA Supported Version)

During inference, the “KV cache” needed to retain past context consumes VRAM. The latest models, such as Llama 3, use GQA (Grouped Query Attention) to save memory.

The KV cache consumption $V_{kv}$ (in gigabytes) is expressed by the following formula:

$$ V_{kv} = 2 \times b \times s \times l \times \left( \frac{h_{kv}}{h_q} \right) \times h_q \times d \times B_{kv} \div 10^9 $$

Simplifying this using the number of key/value heads $h_{kv}$:

$$ V_{kv} = 2 \times b \times s \times l \times h_{kv} \times d \times B_{kv} \div 10^9 $$

Where:

$b$: Batch size (usually 1 for individual local use)
$s$: Sequence length (context length, e.g., 8192)
$l$: Number of layers (e.g., 32)
$h_{kv}$: Number of KV heads (e.g., 8)
$d$: Number of dimensions per head (e.g., 128)
$B_{kv}$: Number of bytes for KV cache (2 for FP16)

Calculation example (Llama 3 8B, context 8192, FP16 cache): $V_{kv} = 2 \times 1 \times 8192 \times 32 \times 8 \times 128 \times 2 \div 10^9 \approx 1.07 \text{ GB}$

Note that the longer the context length $s$ is, the needed VRAM increases linearly.

5. Practice 1: Fastest and Shortest Setup using Ollama

Now that you understand the theory, let’s actually run an LLM in a Windows environment. As of 2026, the most user-friendly tool is “Ollama”. It provides an intuitive Docker-like CLI.

5.1 Installation and Execution

Download the Windows installer from the Ollama Official Website and run it.
Open PowerShell and enter the following command. Here, we’ll use the Japanese-compatible llama3:8b.

1

ollama run llama3:8b

The model will be downloaded on the first run. Once complete, you can interact with it directly in the terminal.

5.2 Creating a Custom AI with a Modelfile

You can easily create an AI with a specific persona. Create a Modelfile in an arbitrary location.

1
2
3
4
5
6
7
8
9


FROM llama3:8b

SYSTEM """
You are an exceptionally talented senior software engineer.
For user questions, always provide code examples and answer logically and concisely.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Build and run your custom model with the following commands:

1
2


ollama create SeniorDev -f ./Modelfile
ollama run SeniorDev

5.3 Usage from External Apps (AI Editors)

Ollama exposes an OpenAI-compatible API endpoint at http://localhost:11434. By simply setting this URL in the backend settings of VS Code extensions like Cursor or Continue.dev, and specifying the model name such as SeniorDev, a powerful local coding assistant is realized for free.

6. Practice 2: Extreme Performance Tuning with llama.cpp

If you want fine-grained memory management or want to be the first to try out the latest formats (like EXL2 or IQ quantization), you manipulate the core engine llama.cpp directly.

6.1 llama.cpp Build Steps

In a Windows environment, the best approach is building from source using CUDA Toolkit and CMake.

1
2
3
4
5
6
7
8


git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

# Configure for CUDA support and compile
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=OFF
cmake --build . --config Release -j 16

6.2 Advanced Launching in Server Mode

Host the model using the built llama-server.exe.

1
2
3
4
5
6
7


.\bin\Release\llama-server.exe `
 --model "C:\models\Llama-3-8B-Instruct.Q4_K_M.gguf" `
 --ctx-size 8192 `
 --n-gpu-layers 99 `
 --threads 8 `
 --flash-attn `
 --port 8080

--n-gpu-layers 99: Offloads all possible layers to GPU VRAM.
--flash-attn: Enables FlashAttention 3, achieving improved inference speed and reduced VRAM consumption for the KV cache.

7. GUI Frontend: LM Studio and Building Local RAG

If you’re resistant to the command line, or intuitively want to perform RAG (Retrieval-Augmented Generation), you can use a GUI.

7.1 LM Studio

LM Studio is a brilliant application that bundles model search, downloading, system requirement pre-checks, and a chat UI all into one. Just by pressing the “Local Server” button in the app, an OpenAI-compatible API starts up.

7.2 RAG Architecture using AnythingLLM

Here is the architecture diagram of a RAG environment for reading internal documents and personal notes.

graph LR Document["Document (PDF, MD)"] --> Chunking["Chunking"] Chunking --> EmbedModel["Embedding Model"] EmbedModel --> VectorDB["Vector Database"] UserQuery["User Query"] --> EmbedQuery["Query Embedding"] EmbedQuery --> VectorDB VectorDB --> |"Similarity Search"| RetrievedDocs["Extract Relevant Docs"] UserQuery --> PromptBuilder["Prompt Generation"] RetrievedDocs --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer"]

Using the AnythingLLM desktop version (Windows), just specify Ollama (LLM and Embedding) from the settings screen and set it up to use a local VectorDB (LanceDB). This architecture can be completed in minutes. A private AI is born that does not send any data externally.

8. Fine-Tuning (LoRA) on Windows WSL2

If you want to not just run locally but make the model smarter with your own data, fine-tuning using LoRA (Low-Rank Adaptation) is possible. As of 2026, by using a library called “Unsloth”, an 8B model can finish training in a few hours on a Windows WSL2 environment even with 16GB of VRAM.

Execute the following within WSL2’s Ubuntu to build the environment.

1
2
3
4


conda create --name unsloth_env python=3.11
conda activate unsloth_env
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Unsloth optimizes CUDA kernels to the extreme, providing about twice the training speed and half the VRAM consumption compared to the standard Hugging Face libraries. Just spin up a Jupyter Notebook and load your dataset (JSONL format), and training for several epochs is possible even on an RTX 4060 Ti with 12GB to 16GB of VRAM.

9. Performance Troubleshooting

Common problems faced and their solutions.

1. Inference speed is extremely slow (1-2 tokens/s)

Cause: The model doesn’t fit entirely into VRAM and is being offloaded to system memory (RAM). Solution: Check “Dedicated GPU memory” in the Task Manager. If it’s hitting the limit, decrease the context size (-c), or use a model with lower bit quantization (like Q4_K_M).

2. “CUDA out of memory” error

Cause: VRAM has been completely exhausted. This occurs especially when the context is prolonged and the KV cache becomes bloated. Solution: Intentionally restrict the values to smaller ones using num_ctx for Ollama, or -c for llama.cpp.

3. Strange Japanese Generation

Cause: Mismatch in prompt templates, or an unsupported model. Solution: Use models that include Instruct in their name, and ensure that the tool is selecting the correct template specified by the model author, such as the ChatML or Llama3 format.

10. Conclusion and Future Prospects

In 2026, building a local LLM in a Windows environment is no longer the privilege of a limited number of engineers. With the de facto standardization of the GGUF format, the emergence of refined ecosystems like Ollama and LM Studio, and hardware optimizations led by FlashAttention, anyone can easily obtain an enterprise-grade AI environment.

Please make use of the following points explained in this article:

Use mathematical VRAM calculations to logically select the optimal model size and quantization level for your PC specs.
Build your environment at maximum speed using Ollama, and dramatically improve productivity by integrating it with AI editors.
Bring out the ultimate performance of your hardware with the advanced parameter control of llama.cpp.
Build a secure local RAG system to handle confidential data with AnythingLLM.
Nurture a custom AI with your own specialized knowledge by utilizing Unsloth (WSL2).

The “democratization” of AI is no longer a buzzword, but a real system running on your Windows desktop. Free yourself from the usage costs of cloud APIs and information leak risks, and step into the world of free and powerful private AI right now.