LLM on kenji.blog

How to Tune TinyLLaMA the Fastest in an On-Premises Environment

Fri, 11 Sep 2026 16:00:00 +0900

1. Introduction: Why TinyLLaMA and On-Premises Now?

The evolution of Large Language Models (LLMs) is proceeding at an incredible speed, but along with it, the number of model parameters continues to inflate to the scale of hundreds of billions. While ultra-giant models like GPT-4 and Claude 3 boast unparalleled performance, the computational costs required for inference and training, as well as security and data privacy concerns when using external APIs, present significant hurdles for companies. In business operations that handle highly sensitive internal data or personal information in particular, sending data to public LLM APIs on the cloud is often unacceptable from a compliance perspective (such as GDPR or APPI).

This is where Small Language Models (SLMs) and local operation in an on-premises environment are stepping into the spotlight. Among them, “TinyLLaMA” is a compact model with only 1.1B (1.1 billion) parameters, yet it has been pre-trained on a massive dataset of approximately 3 trillion tokens, demonstrating astonishing performance compared to models in the same class.

In this article, we provide a complete guide to fine-tuning TinyLLaMA “fastest and highly efficiently” for your company’s specific tasks in an on-premises environment (local servers or workstations). We will comprehensively explain everything from the mathematical background to the latest optimization technologies and specific PyTorch implementation code.

2. TinyLLaMA Architecture and Features

TinyLLaMA follows the LLaMA (Large Language Model Meta AI) architecture developed by Meta. While keeping the number of parameters down to 1.1B, it uses the same technology stack as LLaMA 2, which features highly compatible ecosystem integration.

Major Architectural Components

RMSNorm (Root Mean Square Normalization): A normalization technique that improves computational efficiency by omitting mean subtraction from conventional LayerNorm calculations. It improves throughput while maintaining training stability.
SwiGLU Activation Function: In the Feed Forward Network (FFN), SwiGLU is adopted instead of the conventional ReLU or GELU. Mathematically, it is expressed as follows: $$ \text{SwiGLU}(x, W, V) = \text{Swish}(xW) \otimes (xV) $$ Here, $\otimes$ represents the element-wise product (Hadamard product), and the Swish function is $\text{Swish}(z) = z \cdot \sigma(\beta z)$. This significantly improves expressive power.
RoPE (Rotary Position Embedding): A method that combines the advantages of absolute position encoding and relative position encoding. It has high generalization performance even when the sequence length is extended.
Grouped Query Attention (GQA): An intermediate approach between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), grouping key and value heads to save memory bandwidth and dramatically improve inference speed.

The following Mermaid diagram illustrates the overall data flow and Transformer block structure of TinyLLaMA.

graph TD A["Input Text"] --> B["Tokenizer (BPE)"] B --> C["Embedding Layer"] C --> D["Transformer Blocks (x22 Layers for TinyLLaMA)"] D --> E["RMSNorm (Final)"] E --> F["Linear Projection (Vocab Size)"] F --> G["Output Probabilities (Softmax)"] subgraph "Transformer Block Anatomy" D1["Input Hidden State"] --> D2["RMSNorm"] D2 --> D3["Grouped Query Attention (GQA)"] D3 --> D4["Residual Add"] D4 --> D5["RMSNorm"] D5 --> D6["SwiGLU FFN"] D6 --> D7["Residual Add"] D7 --> D8["Output to Next Layer"] D1 -.-> D4 D4 -.-> D7 end

3. Breakthrough in Fine-Tuning: LoRA and QLoRA

To perform full-parameter fine-tuning in an on-premises environment, even a 1.1B model consumes tens of GB of VRAM (Video RAM) to maintain optimizer states and gradients. To learn efficiently with limited resources, PEFT (Parameter-Efficient Fine-Tuning) techniques such as “LoRA” and its quantized extension “QLoRA” are essential.

3.1 Mathematical Background of LoRA (Low-Rank Adaptation)

LoRA is a method that fixes (freezes) pre-trained weight matrices and approximates the weight updates ($\Delta W$) as the product of two small, low-rank matrices.

Let the pre-trained weights be $W_0 \in \mathbb{R}^{d \times k}$. In full fine-tuning, $W_0$ itself is updated to $W_0 + \Delta W$, but in LoRA, the update matrix $\Delta W$ is decomposed as follows:

$$ \Delta W = B \times A $$

Here, $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, where $r$ is a hyperparameter called Rank, and it is a very small value (usually 8, 16, 32, etc.) that satisfies $r \ll \min(d, k)$.

The forward pass calculation is as follows:

$$ h = W_0 x + \Delta W x = W_0 x + B A x $$

In the initial state, matrix $A$ is randomly initialized with a normal distribution (Gaussian distribution), and matrix $B$ is initialized as a zero matrix. As a result, $\Delta W$ at the start of training is zero, allowing training to start with the base model’s output perfectly preserved.

graph LR X["Input Vector x"] --> W0["Frozen Pre-trained Weight (W_0)"] X --> A["Trainable LoRA Matrix A (r x k)"] A --> B["Trainable LoRA Matrix B (d x r)"] W0 --> Add["Vector Addition"] B --> Add Add --> Y["Output Vector h"]

3.2 The Innovativeness of QLoRA (Quantized LoRA)

QLoRA pushes the LoRA approach further by quantizing the base model $W_0$ into 4-bit precision (NormalFloat 4, NF4) before loading it into memory. This drastically reduces VRAM consumption.

QLoRA incorporates three important technologies:

4-bit NormalFloat (NF4) Quantization: A theoretically optimal data type optimized for normally distributed weights.
Double Quantization: By quantizing the quantization constants (scale factors) themselves, it saves even more memory.
Paged Optimizers: Uses NVIDIA’s unified memory feature to temporarily evict optimizer states to CPU RAM when VRAM is insufficient.

With this, fine-tuning that typically requires 16GB to 24GB of VRAM can comfortably run even on consumer-grade GPUs (like RTX 3060 12GB or RTX 4070).

4. Hardware Requirements and Setup in an On-Premises Environment

When tuning TinyLLaMA (1.1B) with QLoRA, hardware requirements are kept extremely low.

Recommended Hardware Specifications

GPU: NVIDIA RTX 3060 (12GB), RTX 3090/4090 (24GB), or NVIDIA A10G/A100, etc. It can run with a minimum of 8GB of VRAM, but 12GB or more is recommended to increase the batch size.
CPU: A modern CPU with 8 or more cores (Intel Core i7/i9, AMD Ryzen 7/9)
RAM: 32GB or more (crucial as a swap destination from VRAM when using Paged Optimizers)
Storage: NVMe SSD (to speed up dataset loading and model saving)

Software Environment Setup

Here is the setup procedure assuming an Ubuntu 22.04 LTS environment. We will use Python 3.10 or later.

1
2
3
4
5
6
7
8
9


# Create and activate a virtual environment
python3 -m venv tinyllama_env
source tinyllama_env/bin/activate

# Install PyTorch (for CUDA 12.1)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install transformer-related libraries
pip install transformers datasets peft trl accelerate bitsandbytes

5. Optimization Techniques for the Fastest Tuning

To complete tuning not just by running scripts but doing so at the “fastest” speed, you need to combine the following optimization techniques.

5.1 Flash Attention 2

The standard Attention mechanism has time and space complexity of $O(N^2)$ for sequence length $N$. Flash Attention 2 optimizes memory access between the GPU’s SRAM and HBM (High Bandwidth Memory), eliminating the IO bottleneck without reducing the computational complexity. This boosts the training speed by several times and drastically reduces memory consumption.

5.2 Gradient Checkpointing

Instead of storing all intermediate activations computed during the forward pass in VRAM, this technique saves only a portion of them and recomputes them when needed during the backward pass. The computation time increases by about 20%, but memory consumption is drastically reduced, allowing for a larger batch size and resulting in improved overall throughput.

5.3 Mixed Precision Training and Bfloat16

To maximize the use of GPU Tensor Cores, calculations during training are performed in bfloat16 (Brain Floating Point). Since the bit length of the exponent is the same as float32 compared to float16, the risk of overflow and underflow is extremely low, stabilizing the training process.

6. Practice: QLoRA Fine-Tuning Code for TinyLLaMA

Let’s break down the PyTorch script for the fastest tuning incorporating all the optimizations mentioned above. Here, we will use the SFTTrainer from Hugging Face’s trl (Transformer Reinforcement Learning) library.

6.1 Dataset Preparation and Model Loading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


import torch
from datasets import load_dataset
from transformers import (
 AutoModelForCausalLM,
 AutoTokenizer,
 BitsAndBytesConfig,
 TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# 1. Specify model and tokenizer
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

# 2. 4-bit quantization settings for QLoRA
bnb_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_use_double_quant=True,
 bnb_4bit_quant_type="nf4",
 bnb_4bit_compute_dtype=torch.bfloat16 # Perform calculations in bfloat16
)

# 3. Load model (Enable Flash Attention 2)
print("Loading model...")
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 quantization_config=bnb_config,
 device_map="auto",
 use_flash_attention_2=True # Key to maximum speed
)

# 4. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right" # Set to right to avoid bugs during fp16/bf16 training

6.2 Applying the LoRA Adapter and Formatting the Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# 5. Prepare for k-bit training and enable gradient checkpointing
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

# 6. LoRA configuration
peft_config = LoraConfig(
 r=16, # Rank
 lora_alpha=32, # Scaling factor
 lora_dropout=0.05,
 bias="none",
 task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"] # Targeting all Linear layers improves performance
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# Example output: trainable params: 14,286,848 || all params: 1,114,335,232 || trainable%: 1.282%

# 7. Load dataset (Using Japanese Instruction dataset as an example here)
# In practice, you would load an on-premises private JSONL file, etc.
dataset = load_dataset("kunishou/databricks-dolly-15k-ja", split="train")

def format_instruction(sample):
 """
 Formats the string according to ChatML format or prompt template
 """
 prompt = f"<|im_start|>user\n{sample['instruction']}"
 if sample.get("input", "") != "":
 prompt += f"\n{sample['input']}"
 prompt += f"<|im_end|>\n<|im_start|>assistant\n{sample['output']}<|im_end|>"
 return {"text": prompt}

dataset = dataset.map(format_instruction)

6.3 Executing the Training

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


# 8. Set training arguments
training_args = TrainingArguments(
 output_dir="./tinyllama-lora-output",
 per_device_train_batch_size=8, # Increase if you have spare VRAM
 gradient_accumulation_steps=2, # Effective batch size = 8 * 2 = 16
 optim="paged_adamw_32bit", # VRAM savings with Paged Optimizer
 save_steps=100,
 logging_steps=10,
 learning_rate=2e-4,
 fp16=False,
 bf16=True, # Mixed precision training (bfloat16)
 max_grad_norm=0.3,
 max_steps=500, # 500 steps for testing. Specify by epochs in production
 warmup_ratio=0.03,
 group_by_length=True,
 lr_scheduler_type="cosine",
)

# 9. Start training with SFTTrainer
trainer = SFTTrainer(
 model=model,
 train_dataset=dataset,
 peft_config=peft_config,
 dataset_text_field="text",
 max_seq_length=1024, # Adjust according to the expected input length
 tokenizer=tokenizer,
 args=training_args,
)

print("Starting training...")
trainer.train()

# 10. Save the LoRA adapter
trainer.model.save_pretrained("./tinyllama-lora-final")
tokenizer.save_pretrained("./tinyllama-lora-final")
print("Training complete and model saved.")

7. Performance Evaluation and Troubleshooting

Here are common issues encountered when running training in an on-premises environment and their solutions.

OOM (Out Of Memory) occurs:
- Lower per_device_train_batch_size to 1.
- Maintain the effective batch size by increasing gradient_accumulation_steps.
- Shorten max_seq_length from 2048 to 1024 or 512.
Loss doesn’t drop or diverges:
- The learning rate (learning_rate) might be too high. Try lowering it from 2e-4 to around 5e-5.
- If using Float16 instead of Bfloat16, gradient underflow might be occurring. Ensure bf16=True is set.
Mysterious strings are generated during inference:
- Ensure padding_side="right" is correctly set. Also, verify that the dataset format (<|im_start|> and other special tokens) is consistent with the base model’s pre-training.

8. Model Deployment After Tuning

Once tuning is complete, what gets saved is not the “entire base model,” but only the “LoRA adapter (delta weights),” which is a few MB to tens of MB in size. To perform fast inference, you need to merge (integrate) these LoRA weights back into the original base model and export it as a single model.

Model Merging Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

output_dir = "./tinyllama-lora-final"

# Load model and adapter in FP16/BF16
model = AutoPeftModelForCausalLM.from_pretrained(
 output_dir,
 device_map="auto",
 torch_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(output_dir)

# Merge weights and save
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./tinyllama-merged", safe_serialization=True)
tokenizer.save_pretrained("./tinyllama-merged")
print("Model merged and saved successfully!")

Launching an Ultra-Fast Inference Server with vLLM

For deployment in an on-premises environment, we strongly recommend using vLLM or TGI (Text Generation Inference) instead of Hugging Face’s standard pipeline to maximize inference speed (Tokens per second). vLLM uses PagedAttention technology to prevent GPU memory fragmentation and dramatically improves the processing capability for concurrent requests.

The following Mermaid diagram shows the pipeline from training to deploying an inference server.

graph TD A["Raw Private Data"] --> B["Preprocessing & Formatting (JSONL)"] B --> C["QLoRA Fine-Tuning (SFTTrainer)"] C --> D["LoRA Adapter Weights (.safetensors)"] D --> E["Merge with Base TinyLLaMA 1.1B"] E --> F["Merged Model"] F --> G["Deploy via vLLM Server"] G --> H["API Endpoint / UI (e.g. Chatbot)"]

Starting an API server using vLLM can be completed with the following single command:

1
2
3
4
5
6


python -m vllm.entrypoints.openai.api_server \
 --model ./tinyllama-merged \
 --host 0.0.0.0 \
 --port 8000 \
 --max-model-len 2048 \
 --dtype bfloat16

Now, an OpenAI API-compatible endpoint is built in your on-premises environment, allowing you to utilize local AI securely and quickly.

9. Conclusion

In this article, we explained the methods for performing the fastest and most memory-efficient fine-tuning in an on-premises environment for “TinyLLaMA”, a model that is highly performant despite being lightweight with 1.1B parameters.

LoRA / QLoRA enables full-fledged LLM tuning even on consumer GPUs.
Leveraging Flash Attention 2 and Gradient Checkpointing optimizes training time and VRAM consumption to the limit.
Deployment using vLLM achieves high throughput even in production environments.

Operating a local LLM on-premises is not only for protecting data confidentiality but also serves as the ultimate weapon for building specialized AI at low cost for specific domains (legal, medical, internal regulations, etc.). We encourage you to refer to this guide and nurture your own TinyLLaMA.

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

Fri, 11 Sep 2026 14:00:00 +0900

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

In recent years, interest in running Large Language Models (LLMs) in local environments has grown rapidly. In particular, small-scale models like TinyLLaMA (1.1B parameters) can perform inference at practical speeds even on limited-resource edge devices and typical laptops (including Windows environments). While development using Python and PyTorch is mainstream, when pursuing ultimate performance and memory efficiency, the combination of C++ and the C-based tensor library “ggml” has become the de facto standard.

This article provides an extremely detailed development guide for building an inference engine from scratch (or deeply understanding the internal structure of the existing llama.cpp) to load TinyLLaMA and generate text using C++.

1. Why C++ and ggml?

In the AI training phase, Python has an overwhelming advantage due to its flexibility and rich ecosystem. However, in the deployment or “Inference” phase, C++ becomes a powerful choice for the following reasons:

Overhead Reduction: The Python Global Interpreter Lock (GIL) and runtime overhead can be completely eliminated.
Memory Efficiency and Arena Allocation: By manually controlling memory allocation and deallocation, you can prevent unpredictable spikes caused by garbage collection.
Direct Hardware Access: By directly calling SIMD intrinsics such as AVX-512, AVX2, and ARM NEON, the CPU’s computational power can be maximized.
Zero Dependencies: ggml is a zero-dependency C/C++ library. As long as you have a compiler, it can be easily built even in an MSVC environment on Windows.

2. Overall Architecture

The flow of the entire inference pipeline is shown in the Mermaid diagram below. This is a series of processes starting from the user’s input text until the final next token is generated.

graph TD A["User Input Text"] --> B["BPE Tokenizer"] B --> C["Token IDs Array"] C --> D["Embedding Layer Lookup"] D --> E["Transformer Blocks"] E --> F["RMSNorm"] F --> G["LM Head Layer"] G --> H["Logits Array"] H --> I["Sampler Module"] I --> J["Next Token ID"] J --> K["Detokenizer"] K --> L["Output Text Chunk"] J -.-> |"Append to Context"| C

Since it is an autoregressive model, the output token is added back to the context and circulates as input for predicting the next token (the dotted line in the diagram).

3. Model Format and Memory Mapping (mmap)

The biggest hurdle in handling the weights of massive neural networks is disk I/O and memory consumption. In a C++ implementation, this is resolved with memory mapping (mmap).

3.1 How Memory Mapping Works and its Windows Implementation

Using mmap allows you to map the contents of a file directly into the virtual memory space of the process.

Zero-copy: Data is loaded directly from the disk into the kernel’s page cache, preventing extra copies into user space.
On-Demand Loading (Page Fault): The moment the CPU actually accesses that memory address, a page fault occurs, and only the required chunk (usually 4KB) is loaded into physical memory.

In a Windows environment, the Win32 APIs CreateFileMapping and MapViewOfFile are used instead of POSIX mmap.

sequenceDiagram participant OS["Windows OS"] participant RAM["Physical Memory"] participant App["C++ Application"] App->>OS: "CreateFileMapping / MapViewOfFile" OS-->>App: "Virtual Memory Address Pointer" App->>App: "Read Tensor Data at Pointer" OS->>RAM: "Page Fault / Load page from Disk" RAM-->>App: "Data ready for SIMD Compute"

3.2 Binary Structure of the GGUF Format

Converted from formats like Hugging Face’s .safetensors, the GGUF (GPT-Generated Unified Format) is the ultimate format for inference. It has the following strict binary layout:

Magic Bytes: 0x46554747 (GGUF).
Version: Format version number.
Tensor Count & Metadata Count: The number of tensors and key-value metadata pairs.
Metadata (Key-Value Pairs): Typed values and keys with string length prefixes.
Tensor Info: The name, number of dimensions, data type (FP16, Q4_K, etc.), and offset position in the file for each tensor.
Padding: Padding inserted so that tensor data is aligned to specific boundaries (usually 32 bytes or 64 bytes). This is essential for fast memory access with SIMD instructions (especially AVX).
Tensor Data: The actual aligned weight data array.

4. Mathematical Foundations of TinyLLaMA and C++ Algorithms

TinyLLaMA incorporates several advanced architectural ingenuities for efficiency. We explain the mathematical expressions to correctly implement these in C++.

4.1 RMSNorm (Root Mean Square Normalization)

It reduces computational cost by omitting mean centering from LayerNorm and only performing variance scaling.

$$ \text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \gamma $$

$d$ is the number of dimensions, and $\gamma$ is the learned scaling tensor. When implementing in C++, it is optimized by first rapidly calculating the sum of squares of the array using AVX2’s _mm256_fmadd_ps or similar, and then multiplying by the inverse square root (e.g., the _mm256_rsqrt_ps instruction).

4.2 RoPE (Rotary Position Embedding)

A technique that applies token position information as a rotation in tensor space. It can be seen as a rotation on a complex plane, applying the following rotation to adjacent dimension pairs $(x_1, x_2)$ of vector $x$:

$$ \text{RoPE}(x, m) = \begin{pmatrix} x_{1} \cos(m\theta) - x_{2} \sin(m\theta) \\ x_{1} \sin(m\theta) + x_{2} \cos(m\theta) \end{pmatrix} $$

Here, $m$ is the absolute position index of the token, and $\theta$ is a pre-calculated base frequency. In ggml, it is executed in parallel simply by adding a ggml_rope operator during inference graph construction.

4.3 Grouped-Query Attention (GQA)

In standard Multi-Head Attention (MHA), Query, Key, and Value each have the same number of heads. However, TinyLLaMA adopts Grouped-Query Attention (GQA) to drastically reduce memory bandwidth and KV cache consumption.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$

In GQA, multiple Query heads share a single Key/Value head. In the C++ implementation, before executing the matrix multiplication ggml_mul_mat, it is necessary to broadcast the KV tensors to match the number of Queries.

4.4 SwiGLU Activation Function

In the Feed-Forward Network (FFN) layer, SwiGLU is used instead of GELU.

$$ \text{SwiGLU}(x) = \text{Swish}(x W_{\text{gate}}) \otimes (x W_{\text{up}}) $$ $$ \text{Swish}(z) = z \cdot \sigma(z) = z \cdot \frac{1}{1 + e^{-z}} $$

In the computation graph, it is represented by combining the ggml_silu operator and ggml_mul.

5. Computation Graph Construction and Memory Management with ggml

ggml uses a “Define-and-Run” approach, constructing a static computation graph for inference and evaluating it later.

5.1 ggml_context and Arena Allocator

The most unique aspect of ggml is “arena allocation,” which avoids dynamic memory allocation (malloc or new) entirely within the inference loop. Upon initialization, a huge contiguous memory region (arena) is allocated, and the pointer to this region is incremented every time ggml_new_tensor or similar is called. Once one inference step is completed, simply resetting the allocation pointer to its initial position immediately finishes memory allocation for the next inference step.

5.2 Specific Example of Graph Construction

For each inference step, a computation graph like the following is assembled in memory:

graph TD A["Tokens Input ID"] --> B["Embed Lookup"] B --> C["ggml_rms_norm"] C --> D["Q / K / V Projections"] D --> E["ggml_rope Positional"] E --> F["KV Cache Store"] E --> G["KV Cache Load"] G --> H["Self Attention"] H --> I["Scale & Softmax"] I --> J["Attention Output"] J --> K["Out Projection"] K --> L["Add Residual"]

6. Quantization and Windows / SIMD Optimization

Handling TinyLLaMA (1.1B) in FP16 requires approximately 2.2GB of memory, but it can be dramatically compressed to around 600MB through 4-bit quantization (such as Q4_K).

6.1 Block Quantization Architecture

ggml does not quantize the entire tensor uniformly; instead, it does so in “block” units. In the Q4_0 format, 32 FP16 values are grouped into a single block.

Scale Factor: One FP16 value (2 bytes)
Quantized Data: 32 4-bit values (16 bytes) This minimizes the impact of local outliers.

6.2 Dot Product Acceleration with AVX2

When building for the latest x86 CPUs in a Windows environment, utilizing compiler flags like /arch:AVX2 allows SIMD processing to be performed in the following flow:

Load: Load 4-bit quantized data from memory into 256-bit AVX registers.
Expansion and Unpacking: Expand 4-bit values into Int8 or Int16 using bit masks and shift operations.
Dequantization: Multiply by the scale factor to convert to floating-point numbers.
FMA Operations: Execute multiply-add operations in parallel using activation values and _mm256_fmadd_ps (Fused Multiply-Add).

7. KV Cache Implementation Details

In autoregressive generation, the “KV cache” is an essential feature to skip the computation of Keys and Values for past tokens.

The key points for a C++ implementation are as follows:

Tensor Pre-allocation: Initialize a massive tensor for the KV cache (FP16 recommended) corresponding to the maximum context length (e.g., 2048 tokens).
Offset Copying: When calculations for token position $N$ are performed, the K and V vectors obtained in that step are stored into the $N$-th row of the KV cache tensor using ggml_cpy or similar.
Creating a View during Attention: When calculating attention, create a “view” that points only to the token portion from 0 to the $N$-th token and pass it to the matrix multiplication.

8. BPE Tokenizer and Decoding

The input string is treated as a UTF-8 byte sequence and matched against a predefined vocabulary. In C++, to speed up the vocabulary search, algorithms using a Trie (prefix tree) or a priority queue are implemented.

From the logits output by the LM Head, probabilities are scaled using the Temperature parameter, candidates are narrowed down using Top-K extraction or Top-P (Nucleus Sampling) methods, and the final next token is determined using random numbers.

9. Setting Up the C++ Project (Windows / PowerShell Environment)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cmake_minimum_required(VERSION 3.14)
project(TinyLLaMACpp)

set(CMAKE_CXX_STANDARD 17)

# Windows (MSVC) optimization and AVX2 flag settings
if(MSVC)
 add_compile_options(/O2 /arch:AVX2 /fp:fast)
 add_link_options(/STACK:8388608)
else()
 add_compile_options(-O3 -march=native -ffast-math)
endif()

add_library(ggml OBJECT ggml/ggml.c ggml/ggml-alloc.c)
target_compile_definitions(ggml PRIVATE GGML_USE_AVX2 GGML_USE_F16C GGML_USE_FMA)

add_executable(main main.cpp)
target_link_libraries(main ggml)

Example build commands in PowerShell:

1
2
3
4


mkdir build
cd build
cmake .. -G "Visual Studio 17 2022" -A x64
cmake --build . --config Release

10. Conclusion

Implementing an inference engine from scratch for small AI models like TinyLLaMA using C++ and ggml is a perfect opportunity to demystify the black box of deep learning and learn the beauty of low-level hardware control. Let’s pave the way for the future of edge AI while fully savoring the essence of systems programming, such as zero-copy loading using memory mapping, SIMD optimization, and KV cache construction.

[RAG Implementation Guide] How to Load Your Own Documents into Local AI

Fri, 11 Sep 2026 13:00:00 +0900

Introduction

In recent years, the evolution of Large Language Models (LLMs) has been remarkable, and many AIs, led by ChatGPT and Claude, have permeated our daily lives and business operations. However, general LLMs have a distinct weakness. That is, they only know “public information at the time of their training”. Naturally, they cannot answer questions about “private documents” such as internal company regulations, personal notes, and unpublished project materials. Forcing them to answer increases the risk of generating plausible lies (hallucinations) that differ from the facts.

Therefore, the technological architecture known as RAG (Retrieval-Augmented Generation) is currently spreading explosively worldwide. By using RAG, it becomes possible to dynamically provide unique knowledge to the LLM from an external database and have it generate accurate and well-founded answers based on it.

Furthermore, when handling enterprise domains or personal confidential information, sending data to cloud-based APIs like OpenAI is often unacceptable under security policies. What is required there is the construction of “Local RAG” combined with Local AI (an LLM that operates entirely on your own PC or on-premise server).

In this article, we will thoroughly explain everything from the fundamental theory of RAG, specific implementation methods of Local RAG using Python, mathematical background (how vector search works), to advanced techniques for running the system in production.

1. Overall Architecture of RAG

RAG is not a single AI model, but a system architecture where multiple components work together. It broadly consists of two phases: the “Ingestion (Data Loading) Phase” and the “Retrieval & Generation Phase”.

The Mermaid diagram below shows the overall picture of a RAG system.

graph TD subgraph "Ingestion Phase (Preparation)" Doc["Custom Documents (PDF, TXT, etc.)"] --> Loader["Document Loader"] Loader --> Splitter["Text Splitting (Chunking)"] Splitter --> EmbedModel1["Embedding Model"] EmbedModel1 --> VectorDB["Vector Database"] end subgraph "Inference Phase (User Query)" User["User Question (Query)"] --> EmbedModel2["Embedding Model"] EmbedModel2 --> QueryVector["Query Vector"] QueryVector --> Search["Similarity Search (Vector Search)"] VectorDB --> Search Search --> Context["Relevant Chunk Extraction (Context)"] User --> PromptBuilder["Prompt Builder"] Context --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer Generation"] end

Ingestion Phase (Preparation)

Document Loading: Loads unstructured data such as PDFs, Word documents, and text files.
Chunking (Text Splitting): Splits long texts into meaningful chunks to fit within the LLM’s input limit (context window) and to improve search accuracy.
Embedding (Vectorization): Inputs the split chunks into an Embedding Model and converts them into an array of numerical values (vectors) with hundreds to thousands of dimensions.
Saving to Database: Saves the converted vectors and their associated original text data into a Vector Database (Vector DB).

Inference Phase (Runtime)

Query Vectorization: Vectorizes the user’s question text using the same embedding model used in the preparation phase.
Similarity Search: Performs a similarity calculation between the query vector and the document vectors in the database, and retrieves the top few semantic (highly relevant) text chunks.
Prompt Building: Combines the retrieved relevant text as “context (background knowledge)” with the user’s question text to create an input prompt for the LLM.
Answer Generation: The LLM, receiving the augmented prompt, generates an answer based on the provided context information.

2. Deep Understanding of Vector Search and Embeddings

At the core of RAG is “Vector Search (Semantic Search)”. While traditional keyword search (like BM25) is based on exact word matching and frequency, vector search is based on “semantic similarity”. For example, even if the words are different, like “dog” and “puppy”, or “PC” and “computer”, they will be hit in the search if their meanings are close.

What is an Embedding Model?

An embedding model is a neural network that takes natural language text as input and outputs a fixed-length dense vector. Common models (e.g., text-embedding-3-small or the open-source multilingual-e5-large) map text into a vector of real numbers with 384 or 1024 dimensions.

In this multi-dimensional space (latent space), the model is trained so that sentences with similar meanings are closer in distance in the coordinate space.

Mathematical Background of Similarity Calculation: Cosine Similarity

When a vector database searches for relevant documents, the most commonly used distance metric is Cosine Similarity. Unlike Euclidean distance (absolute spatial distance), cosine similarity focuses on the “angle between two vectors”. Since it is less affected by the length of the sentence (the norm of the vector), it is highly suitable for calculating text similarity.

Expressed mathematically, the cosine similarity between vectors $\mathbf{A}$ and $\mathbf{B}$ is as follows:

$$ \text{Cosine Similarity}(\mathbf{A}, \mathbf{B}) = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}} $$

$\mathbf{A} \cdot \mathbf{B}$ represents the Dot Product.
$\|\mathbf{A}\|$ represents the L2 norm (length) of vector $\mathbf{A}$.
$n$ is the number of dimensions of the vector.

Cosine similarity takes a value from -1 to 1.

Close to 1: The directions of the two vectors are almost the same (meanings are very similar).
Close to 0: The two vectors are orthogonal (unrelated).
Close to -1: The two vectors are in opposite directions (meanings are opposite).

Recent Vector DBs (Chroma, FAISS, Qdrant, etc.) employ an Approximate Nearest Neighbor (ANN) algorithm called HNSW (Hierarchical Navigable Small World), optimizing them to search for documents with high cosine similarity in milliseconds even from millions of vector data.

3. Technology Stack for Building Local RAG

To build a fully local RAG that does not rely on the cloud, we leverage the open-source ecosystem. The recommended technology stack is introduced below.

Large Language Model (LLM)
- Tools: Ollama or Llama.cpp
- Models: Lightweight, high-performance open models like Llama-3-8B-Instruct, Gemma-2-9B-It, Qwen2-7B-Instruct. For Japanese tasks, Japanese-tuned models like Llama-3-ELYZA-JP-8B are suitable.
Embedding Model (Embedding)
- Models: intfloat/multilingual-e5-large or BAAI/bge-m3. When running locally, it is common to download them from Hugging Face and run them with Sentence-Transformers.
Vector Database (Vector DB)
- ChromaDB: Python-based and extremely easy to set up. Ideal for local development.
- FAISS: A fast vector search library developed by Meta.
- Qdrant / Milvus: For larger scale and production environments.
Orchestration Framework
- LangChain: The de facto standard for chaining components together.
- LlamaIndex: A data connection framework specifically specialized for RAG.

This time, we will implement it using the easiest combination to introduce: LangChain + ChromaDB + Ollama + HuggingFaceEmbeddings.

4. Implementation Tutorial: Building a Full Local RAG with Python

From here, we will build a local RAG while actually writing Python code. Please install Ollama on your PC in advance and have it running in the background. Also, pull a model on Ollama (e.g., ollama run llama3).

Step 1: Installing Required Libraries

1
2


pip install langchain langchain-community langchain-huggingface
pip install chromadb sentence-transformers pypdf

Step 2: Complete Implementation Code

Below is the complete Python script to load a PDF file, vectorize it, and have the local LLM answer questions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93


import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain_core.prompts import PromptTemplate
from langchain.chains import RetrievalQA

def main():
 # 1. Document Loading
 print("Loading document...")
 # Specify the path of the PDF you want to load
 file_path = "sample_company_policy.pdf"
 loader = PyPDFLoader(file_path)
 documents = loader.load()

 # 2. Chunking (Text Splitting)
 # Split into appropriate sizes without breaking the meaning of the sentences
 text_splitter = RecursiveCharacterTextSplitter(
 chunk_size=500, # Maximum number of characters per chunk
 chunk_overlap=50, # Overlap characters between previous and next chunks (prevents context disconnection)
 separators=["\n\n", "\n", "。", "、", " ", ""]
 )
 chunks = text_splitter.split_documents(documents)
 print(f"Split into {len(chunks)} chunks.")

 # 3. Initialization of Embedding Model (Local HuggingFace Model)
 # Using a multilingual model with strong Japanese support
 print("Loading embedding model...")
 embeddings = HuggingFaceEmbeddings(
 model_name="intfloat/multilingual-e5-large",
 model_kwargs={'device': 'cpu'} # 'cuda' or 'mps' if you have a GPU
 )

 # 4. Building Vector Database (Chroma)
 print("Building vector database...")
 persist_directory = "./chroma_db"
 vectorstore = Chroma.from_documents(
 documents=chunks,
 embedding=embeddings,
 persist_directory=persist_directory
 )
 # Creating a Retriever. Configured to get the top 3 relevant documents
 retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

 # 5. Initialization of Local LLM (Ollama)
 print("Connecting to Local LLM...")
 # Make sure to get the model in advance with 'ollama pull llama3' etc.
 llm = Ollama(model="llama3")

 # 6. Definition of Prompt Template
 prompt_template = """You are an excellent assistant familiar with company regulations and internal information.
Please answer the user's question in detail based ONLY on the following context (background information).
If you cannot find the answer from the context, please do not guess and honestly answer "I don't know from the provided information."

[Context]
{context}

[Question]
{question}

[Answer]:
"""
 PROMPT = PromptTemplate(
 template=prompt_template,
 input_variables=["context", "question"]
 )

 # 7. Building RAG Chain
 qa_chain = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff",
 retriever=retriever,
 return_source_documents=True, # Set whether to return source documents
 chain_type_kwargs={"prompt": PROMPT}
 )

 # 8. Executing a Question
 query = "Please tell me about the conditions for transportation expenses payment regarding remote work."
 print(f"\nQuestion: {query}\n")

 result = qa_chain.invoke({"query": query})

 print("[Answer]")
 print(result['result'])
 print("\n---")
 print("[Reference Sources]")
 for doc in result['source_documents']:
 print(f"- Page {doc.metadata.get('page', 'Unknown')}: {doc.page_content[:50]}...")

if __name__ == "__main__":
 main()

Explanation of Key Code Points

RecursiveCharacterTextSplitter: This is the most recommended splitter for dividing natural language. It attempts to split in the order of paragraphs (\n\n), lines (\n), and periods (。), keeping semantic blocks together as much as possible while fitting within the specified chunk_size. By setting chunk_overlap, you prevent context boundaries from being cut off and information from being lost.
HuggingFaceEmbeddings: intfloat/multilingual-e5-large is a very powerful open-source embedding model that supports multiple languages. You can vectorize text locally on your memory offline without using a cloud API (like OpenAI’s text-embedding-ada-002).
ChromaDB: Since it runs in-memory or on local storage (SQLite-based), there is no need to spin up a complex database server. By specifying persist_directory, you can skip the vectorization process on subsequent runs and load the DB from disk.

5. Advanced RAG Techniques

The basic RAG system (Naive RAG) built in the above tutorial works, but if high answer accuracy is required in a production environment, the introduction of advanced techniques like the following becomes necessary.

5.1 Hybrid Search

While vector search is good at capturing “meaning”, it can be poor at strict keyword searches like “specific proper nouns”, “product model numbers”, and “employee IDs”. Therefore, by running semantic search via vector search and keyword search using algorithms like BM25 in parallel, and integrating both results by scoring them (using techniques like Reciprocal Rank Fusion; RRF), you can drastically reduce search misses.

5.2 Re-ranking

Vector search is fast, but it does not necessarily evaluate the exact contextual relevance of the context. A general pipeline for improving search accuracy is as follows:

First-stage Retrieval: Retrieve a broad and shallow range of relevant chunks (about 20-30) from the Vector DB.
Re-ranking: Use another heavier machine learning model called a Cross-Encoder (e.g., bge-reranker) to input pairs of the user’s query and the retrieved chunks, and recalculate their semantic relevance scores.
Selection: Pass only the top 3-5 with the highest scores as the final context to the LLM prompt.

This technique prevents irrelevant noise information from being passed to the LLM, significantly increasing the precision of the answers.

graph LR Query["Query"] --> VSearch["Vector Search (Top 20)"] VSearch --> Reranker["Re-ranker Model (Cross-Encoder)"] Query --> Reranker Reranker --> TopK["High Precision Top 3"] TopK --> LLM["LLM Generation"]

5.3 Semantic Chunking and Parent Document Retrieval

Instead of mechanically splitting text by a fixed number of characters, there is a technique called “Semantic Chunking” that uses AI to detect shifts in meaning and splits the text accordingly. Also, in the “Parent Document Retriever” technique, you vectorize in very small units (like sentences) for high-precision search, but when passing it to the LLM, you provide the “original large paragraph (parent document)” containing that sentence, thus providing sufficient context to the LLM.

6. Challenges and Countermeasures when Operating Local RAG

There are unique hurdles when building and operating RAG in a local environment.

VRAM (Video Memory) Exhaustion: To run a Local LLM at a practical speed (dozens of tokens per second), you need to load the model into the GPU’s VRAM. Running an 8B class model in fp16 (16-bit floating point) requires about 16GB of VRAM, but by using Quantization technologies (compressing to 4-bit or 8-bit, such as GGUF or AWQ formats), it is possible to run it fast enough even with 8GB of VRAM (like a standard gaming PC). Llama.cpp and Ollama support these quantization formats by default.
Context Window Limits: If the amount of retrieved context is too large, it may exceed the LLM’s input limit (token limit), or the model might forget the middle part of the information (Lost in the middle phenomenon). Adjusting the number of chunks to extract and carefully selecting them through the aforementioned re-ranking techniques are essential.
Data Freshness Management: When a source document is updated, the corresponding document’s vector in the vector database also needs to be updated or deleted (CRUD operations). Since ChromaDB supports updates based on document IDs, it is practical to manage file hash values and set up a batch process to sync only the differences.

Conclusion

RAG (Retrieval-Augmented Generation) is a powerful paradigm that evolves AI from a general-purpose assistant into your “exclusive expert” or an “expert specialized in internal business”.

We found that even with highly confidential requirements where cloud services cannot be used, a complete “Local RAG” environment can be relatively easily built by combining the open-source ecosystem such as Ollama, LangChain, and ChromaDB.

Based on the advanced approaches such as mathematical understanding of vector space, text splitting, and re-ranking explained in this article, please try developing an original AI system using your own data. The speed of evolution in local AI is astounding, and the system you build today can instantly update its performance tomorrow simply by swapping in a smarter lightweight model that appears.

This blog will continue to publish deep-dive articles on AI technology and RAG. If you have any questions or feedback, please share them in the comments section.

ChatGPT, Gemini, and Claude APIs Thoroughly Compared! Which Should You Choose?

Fri, 11 Sep 2026 12:00:00 +0900

ChatGPT, Gemini, and Claude APIs Thoroughly Compared! Which Should You Choose?

AI technology is evolving remarkably, especially in the field of Large Language Models (LLM), where OpenAI’s ChatGPT (GPT series), Google’s Gemini, and Anthropic’s Claude are engaged in a fierce three-way battle for supremacy. As of 2026, each company is releasing new models and API features on a scale of months, or even weeks. For developers and corporate IT architects, the question of “which API to integrate into products” has become a highly critical decision that dictates the success of a project.

In this article, we will thoroughly compare and explain these top 3 AI providers’ APIs from a developer’s perspective. We go beyond just listing specifications, and delve into architecture design, detailed pricing structures, mathematical analysis of latency, concrete implementation examples using Python and Node.js, and the latest cost-optimization techniques such as prompt caching.

Our goal is for this to serve as a complete guide for readers to select the most optimal LLM API for their own use cases and build scalable, cost-effective AI applications.

1. Philosophy and Design Concepts of Each LLM API

When choosing technology, it is very important to first understand the philosophies each company uses to build their models and APIs.

1.1 OpenAI (ChatGPT)

OpenAI has set its mission to “achieve Artificial General Intelligence (AGI)” and constantly drives the de facto standard of the industry. They offer a diverse range of models tailored to use cases, such as GPT-4o, GPT-4o-mini, and the inference-specialized o1 model. Their ecosystem is the most mature, and they have the most abundant libraries and documentation, both official and unofficial.

1.2 Google (Gemini)

Google advocates “AI First” and uses the scalability of its own infrastructure (TPU networks) to its maximum advantage. Gemini 1.5 Pro/Flash’s greatest feature is its overwhelming context window of up to 2 million tokens, allowing it to process lengthy documents or hours of video/audio all at once. Its strong integration with Google Cloud (Vertex AI) is also attractive for enterprises.

1.3 Anthropic (Claude)

Anthropic is a company founded by former OpenAI members, adopting a unique safety approach called “Constitutional AI”. Claude 3.5 Sonnet and Opus have garnered enthusiastic support from many developers for their high reasoning capabilities, code generation skills, and above all, their “human-like natural dialogue” and “low hallucinations”.

2. Thorough Comparison of Model Family Specifications

We compare the specifications of the flagship models as of 2026.

Provider	Flagship Model	Max Context Length	Key Strengths	Recommended Use Cases
OpenAI	GPT-4o	128K	Speed, vision, audio support	Interactive apps, general tasks
OpenAI	o1-preview	128K	Advanced logical reasoning, math, coding	Complex algorithm generation, research
Google	Gemini 1.5 Pro	2,000K	Ultra-long text processing, multimodal (video/audio)	Analyzing massive codebases, video summarization
Google	Gemini 1.5 Flash	2,000K	Low latency, high throughput, overwhelmingly low cost	Real-time processing, batch processing of large data
Anthropic	Claude 3.5 Sonnet	200K	Coding capabilities, natural text generation	Software development assistance, advanced customer support
Anthropic	Claude 3.5 Haiku	200K	Ultra-fast response, cost performance	Edge AI, real-time chatbots

3. Deep Dive into Architecture: Behind the API Requests

When an LLM API is called, what kind of processing takes place on the backend? To optimize performance, we need to understand this architecture.

The following Mermaid diagram shows the overall picture from when an API request is sent by the client to when tokens are returned in a stream.

graph TD A["Client Application"] -->|HTTP/REST or gRPC| B["API Gateway"] B --> C["Load Balancer"] C --> D["Inference Cluster"] D --> E["Tokenizer (BPE / SentencePiece)"] E --> F["KV Cache & Attention Mechanism"] F --> G["Transformer Blocks (Forward Pass)"] G --> H["Output Layer (Logits)"] H --> I["Sampler (Temperature, Top-p, Top-k)"] I --> J["Detokenizer"] J -->|Streaming Response (Chunk)| A

3.1 Tokenization Algorithms

The text input into the API is internally divided into units called “tokens”.

OpenAI (tiktoken): Uses Byte-Pair Encoding (BPE). It compresses English extremely efficiently, but for non-alphabet languages like Japanese, the number of tokens tends to inflate.
Google (Gemini): Uses SentencePiece (Unigram Language Model). It handles multiple languages well, and tends to express even Japanese text with relatively few tokens.
Anthropic (Claude): Uses a customized version of BPE. Multilingual support has been strengthened, and since Claude 3, token efficiency for Japanese has also significantly improved.

4. Mathematical Analysis of Latency and Performance

In real-time applications, latency is directly tied to the User Experience (UX). The latency of an LLM API, $T_{total}$, can be mathematically modeled as follows:

$$ T_{total} = T_{network} + T_{TTFT} + (N \times T_{TPOT}) $$

Here, each variable has the following meaning:

$T_{network}$: Network Round Trip Time (RTT).
$T_{TTFT}$ (Time To First Token): The time until the first character is generated. It depends heavily on the cost of attention calculation, which is proportional to the square of the prompt length (number of input tokens).
$N$: The total number of tokens output.
$T_{TPOT}$ (Time Per Output Token): Generation time per token. Because it is an autoregressive model, it is calculated serially depending on the previous output.

4.1 Computational Complexity of the Self-Attention Mechanism

The computational complexity of Self-Attention in the Transformer architecture increases quadratically with respect to the input sequence length $L$.

$$ \text{Complexity} = O(L^2 \cdot d) $$

Here, $d$ is the dimensionality of the embedding vector. Because of this constraint, $T_{TTFT}$ usually worsens dramatically as the prompt gets longer. However, Google’s Gemini 1.5 employs innovative optimization architectures like “Ring Attention” and “Block-wise Compute,” successfully generating the first token in a realistic amount of time (a few seconds to tens of seconds) even when a long text of 2 million tokens is input.

5. Pricing Systems and Cost Optimization Strategies

API costs are fundamentally calculated based on the number of input and output tokens.

$$ Cost = (Tokens_{in} \times Rate_{in}) + (Tokens_{out} \times Rate_{out}) $$

However, the latest APIs have introduced new mechanisms to drastically reduce costs.

5.1 Prompt Caching

Sending a massive system prompt or a large amount of documents retrieved by RAG every time incurs an enormous cost. To address this, each company provides caching features.

In Anthropic (Claude) and Google (Gemini), caching specific text blocks can significantly reduce input costs (up to 90%).

The cost model when using a cache is as follows:

$$ Cost_{cached} = (Tokens_{cache\_write} \times Rate_{cache\_write}) + (Tokens_{cache\_read} \times Rate_{cache\_read}) + (Tokens_{out} \times Rate_{out}) $$

Here, $Rate_{cache\_read}$ is set to about 10% to 25% of the regular $Rate_{in}$. This makes it possible to run a chatbot cheaply while constantly keeping tens of thousands of lines of a codebase as background knowledge.

5.2 Batch API

For tasks that do not require real-time processing (log analysis, bulk data classification, etc.), OpenAI and Anthropic offer a Batch API. This is a powerful mechanism where you can send requests in bulk and receive results within 24 hours at half the regular API price (50% off).

6. Developer Experience (DX) and SDK Comparison

We compare the SDKs (Software Development Kits) provided by each company from the perspective of development efficiency.

6.1 OpenAI API

The most widely used, with the fastest support for third-party libraries (LangChain, LlamaIndex, etc.). Furthermore, the Structured Outputs feature guarantees a response that complies with a JSON schema with 100% accuracy, making system integration extremely easy.

6.2 Anthropic API (Claude)

The SDK interface is refined, and its TypeScript type definitions are highly regarded for being very easy to handle. The structure of the Message API is particularly intuitive, allowing simple coding for multimodal requests containing multiple images.

6.3 Google Gemini API

There are two access methods: via Google Cloud Vertex AI and via AI Studio (Google Gen AI SDK), which might slightly confuse beginners. However, the Vertex AI SDK for enterprises is fully integrated with GCP’s IAM (Identity and Access Management) system, enabling the construction of secure development environments.

7. Practical! Integrated Testing Implementation of Multiple APIs with Python

Here, we will use Python to implement a script that sends asynchronous requests simultaneously to three APIs—OpenAI, Anthropic, and Gemini—and compares their latency.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


import asyncio
import time
import os
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
import google.generativeai as genai

# Initialize clients
openai_client = AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
genai.configure(api_key=os.environ.get("GEMINI_API_KEY"))

prompt = "Please explain the basics of quantum computing and its impact on current cryptographic technologies in an easy-to-understand way for beginners."

async def fetch_openai():
 start_time = time.time()
 response = await openai_client.chat.completions.create(
 model="gpt-4o",
 messages=[{"role": "user", "content": prompt}],
 max_tokens=1024
 )
 elapsed = time.time() - start_time
 return "OpenAI (GPT-4o)", elapsed, response.choices[0].message.content

async def fetch_anthropic():
 start_time = time.time()
 response = await anthropic_client.messages.create(
 model="claude-3-5-sonnet-20240620",
 messages=[{"role": "user", "content": prompt}],
 max_tokens=1024
 )
 elapsed = time.time() - start_time
 return "Anthropic (Claude 3.5 Sonnet)", elapsed, response.content[0].text

async def fetch_gemini():
 start_time = time.time()
 model = genai.GenerativeModel('gemini-1.5-pro')
 # Using the asynchronous method of the Gemini Python SDK
 response = await model.generate_content_async(prompt)
 elapsed = time.time() - start_time
 return "Google (Gemini 1.5 Pro)", elapsed, response.text

async def main():
 print("Sending requests to each LLM API...")

 # Execute 3 APIs in parallel
 results = await asyncio.gather(
 fetch_openai(),
 fetch_anthropic(),
 fetch_gemini()
 )

 for provider, latency, text in results:
 print(f"--- {provider} ---")
 print(f"Latency: {latency:.2f} seconds")
 print(f"Response (Excerpt): {text[:100]}...\n")

if __name__ == "__main__":
 asyncio.run(main())

By running this script, you can easily measure which model responds the fastest (minimizing $T_{total}$) in a real network environment.

8. Implementing Tool Calling (Function Calling) with Node.js

To make an LLM function not just as a chatbot but as an “AI Agent” that cooperates with external systems, Tool Calling (or Function Calling) is indispensable. Below is an example using Node.js (TypeScript) to have the OpenAI API call a weather API.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


import OpenAI from "openai";

const openai = new OpenAI({
 apiKey: process.env.OPENAI_API_KEY,
});

async function runAgent() {
 const tools = [
 {
 type: "function",
 function: {
 name: "get_weather",
 description: "Gets the current weather for the specified city.",
 parameters: {
 type: "object",
 properties: {
 location: {
 type: "string",
 description: "City name (e.g., Tokyo, New York)",
 },
 },
 required: ["location"],
 },
 },
 },
 ];

 const response = await openai.chat.completions.create({
 model: "gpt-4o",
 messages: [{ role: "user", "content": "How's the weather in Tokyo today? Will I need an umbrella?" }],
 tools: tools,
 tool_choice: "auto",
 });

 const message = response.choices[0].message;

 if (message.tool_calls) {
 const toolCall = message.tool_calls[0];
 console.log(`LLM requested a tool call: Function name = ${toolCall.function.name}`);

 const args = JSON.parse(toolCall.function.arguments);
 console.log(`Arguments: ${args.location}`);

 // Implement the logic to call the actual weather API (e.g., OpenWeatherMap) here
 // const weather = await fetchWeatherFromAPI(args.location);

 // Pass the retrieved results back to the LLM to generate the final answer
 }
}

runAgent().catch(console.error);

Claude 3.5 Sonnet and Gemini 1.5 Pro also have equivalent Tool Calling capabilities, and although there are slight differences in how schemas are defined, the basic flow is common across them.

9. RAG vs Long Context Window: Which Should You Adopt?

Currently, one of the biggest debates in enterprise AI architecture is “whether to use RAG (Retrieval-Augmented Generation) to incorporate external knowledge, or to throw it all into a massive context window (Long Context).”

Benefits and Challenges of RAG (Retrieval-Augmented Generation)

Benefits: Low cost (because only necessary chunks are put into the prompt), and it’s easy to identify the grounds (sources) for the answer.
Challenges: Because it relies on the accuracy of semantic search, it is not suitable for advanced reasoning tasks where the context is scattered across multiple documents (e.g., “Analyze the root causes of the delay in Project A chronologically from all meeting minutes from last year”).

Long Context (e.g., Gemini 1.5 Pro’s 2 Million Tokens)

Benefits: No missed information from searches. Even in “Needle In A Haystack (NIAH)” tests, Gemini 1.5 Pro and Claude 3.5 Sonnet can extract information with over 99% accuracy.
Challenges: Enormous token consumption leads to high costs, and latency ($T_{TTFT}$) increases.

Conclusion: The best practice for 2026 is a “Hybrid Approach”. The mainstream design is to use RAG with a vector database for routine Q&A, and use Long Context leveraging prompt caching for specialized tasks requiring complex analysis or full codebase reviews.

10. Comparison of Multimodal Processing Capabilities

Next-generation AI applications require the ability to directly understand not only text but also images, audio, and video.

sequenceDiagram participant User as "User" participant Client as "Frontend App" participant API as "LLM API (Multimodal)" User->>Client: Upload Video & Text Prompt Client->>API: Send Video Bytes/URI + Text Note over API: Video chunking & Audio separation Note over API: Multimodal Embedding Model API-->>Client: Return Text Summary & Timestamps Client-->>User: Display Insights

OpenAI (GPT-4o): Has extremely high image recognition accuracy and excels at reading handwritten drawings and complex graphs. Its native audio interaction with ultra-low latency (hundreds of milliseconds) using the Realtime API is also powerful.
Google (Gemini 1.5 Pro): Overwhelmingly superior in video analysis. You can input a 1-hour video file (frames + audio) as is, and it can answer pinpoint questions like, “What is the title of the document held by the person who appeared on the right edge of the screen at 12 minutes and 45 seconds?”
Anthropic (Claude 3.5 Sonnet): Its image recognition (Vision) capability is on par with GPT-4o and very excellent. It demonstrates unrivaled strength in frontend development assistance, such as passing a UI screenshot and saying, “Generate the React component code for this screen.”

11. Enterprise-Level Security and Compliance

When enterprises use LLM APIs in production environments, their biggest concerns are “Will our company’s data be used for training the AI?” and “Are compliance requirements met?”

All three companies clearly state that data sent via API (prompts and responses) will not be used for model training (Zero Data Retention / No Training on Customer Data) (*This does not apply to free consumer web chat UIs).

When an even higher level of security is required:

OpenAI: By using the Azure OpenAI Service, you can utilize Microsoft’s enterprise-grade security, SLAs, and closed-network connections via Azure Private Link.
Google: By using Google Cloud Vertex AI, strict network isolation using VPC Service Controls and data protection with CMEK (Customer-Managed Encryption Keys) are possible.
Anthropic: By using it via AWS Bedrock or Google Cloud Vertex AI, you can piggyback on the robust security infrastructure of cloud providers.

12. Conclusion: Ultimate Choice Guide by Use Case

Although we have compared them from various angles, the final conclusion to “Which one should you choose?” depends on the use case.

Complex Software Development, Code Generation, Advanced Reasoning: 👑 Winner: Claude 3.5 Sonnet (Anthropic) It currently delivers the best performance in understanding code context, refactoring, and writing natural, human-like text. The ease of use of the API and the cost efficiency due to prompt caching are also outstanding.
Parsing Ultra-Long Documents, Batch Processing of Video/Audio: 👑 Winner: Gemini 1.5 Pro (Google) Its 2-million-token context window is a unique weapon. For tasks requiring a grasp of the entire data, such as parsing hundreds of pages of PDF manuals or summarizing long meeting recordings, Gemini is second to none.
Versatility, Execution Speed, Stable Structured Output (JSON): 👑 Winner: GPT-4o / GPT-4o-mini (OpenAI) It handles any task flawlessly and has the most abundant third-party tool support. If you need reliable JSON parsing using Structured Outputs or ultra-advanced logical reasoning using the o1 model, the OpenAI ecosystem is indispensable.

Recommendation for Multi-Model Routing

The future trend is an “LLM Routing” architecture that dynamically switches models according to the difficulty and importance of the task, rather than depending on a single API (vendor lock-in). For example, you can respond to simple questions from users with the cheap and fast GPT-4o-mini or Gemini 1.5 Flash, and fallback the task to Claude 3.5 Sonnet only when complex processing is deemed necessary, thereby achieving the optimal balance of cost and performance.

The evolution of AI will not stop. Please deeply understand the strengths, weaknesses, and architectural characteristics of each API to build flexible and scalable AI applications.

A Complete Guide to llama.cpp and Customization with C++

Fri, 11 Sep 2026 11:00:00 +0900

In recent years, the evolution of Large Language Models (LLMs) has been tremendous, and their scope of application is expanding daily. However, running models with billions or tens of billions of parameters locally typically requires a high-end GPU with an enormous amount of VRAM. Breaking through this “hardware barrier” and making practical LLM inference possible on everyday PCs, Macs, and even devices like the Raspberry Pi is llama.cpp.

This article goes beyond just explaining how to use the command-line tool. It provides an extremely detailed explanation for engineers, covering the architecture of its underlying technology ggml, the mathematical background of Transformers and quantization, and how to use the C++ API to integrate and customize LLMs within your own applications.

1. Overview of llama.cpp and ggml

llama.cpp is a lightweight LLM inference engine written in C/C++, developed by Georgi Gerganov. Originally created with the goal of running Meta’s LLaMA model quickly on Apple Silicon (M1/M2 Macs), it now supports a variety of architectures and models.

Its biggest feature is that it is a pure C/C++ implementation with no external dependencies. Because it doesn’t require a massive ecosystem like Python or PyTorch and can be compiled as a single executable, deployment is incredibly easy.

The heart of llama.cpp is the tensor math library ggml. ggml was designed from the ground up to maximally optimize matrix operations in machine learning on CPUs (and some GPUs).

1.1 Why is llama.cpp so fast?

Leveraging Memory Mapping (mmap): When loading model weights into memory, it uses the OS’s mmap to avoid loading everything into RAM, enabling fast startups and saving memory.
Thorough Optimization of SIMD Instructions: It utilizes CPU-specific instruction sets like AVX2, AVX-512, ARM NEON, and Apple AMX to perform ultra-fast matrix multiplications.
Quantization: It compresses 16-bit floating-point (FP16) weights into 4-bit, 5-bit, or 8-bit integers, resolving the memory bandwidth bottleneck (more on this later).

2. Mathematical Background: Transformers and Quantization

To deeply understand llama.cpp, you need to know the mathematical formulas it calculates and how it approximates these calculations.

2.1 The Inference Process of a Transformer

Models like LLaMA adopt an auto-regressive Transformer decoder architecture. The core of text generation is the Self-Attention mechanism.

For an input hidden state matrix $X \in \mathbb{R}^{N \times d}$, the Query $Q$, Key $K$, and Value $V$ are calculated by multiplying with weight matrices:

$$ Q = X W_Q, \quad K = X W_K, \quad V = X W_V $$

Here, the output of Attention is defined as follows:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

In the inference loop of llama.cpp, the bottleneck is the multiplication of these massive matrices $W_Q, W_K, W_V$ and the feed-forward network (FFN) weight matrices with the vector $X$ ($N=1$ in the generation phase because it processes one token at a time), which is GEMV (General Matrix-Vector Multiplication).

2.2 The Mathematical Foundation of Quantization

In inference where memory access bandwidth becomes a bottleneck, quantization—representing weight parameters with a small number of bits—is essential. Here, we’ll explain the basic principles of the block-wise quantization widely used in llama.cpp (e.g., Q4_K or Q4_0).

For example, consider a block $w = [w_1, w_2, \dots, w_B]$ of length $B$ (usually 32 or 64) that is part of an FP16 weight matrix $W$. This block is approximated using 4-bit integers $q_i \in [-8, 7]$ and a single scaling factor $\Delta$ (FP16 or FP32):

$$ w_i \approx \Delta \times q_i $$

$\Delta$ is determined based on the maximum absolute value within the block:

$$ \Delta = \frac{\max_i |w_i|}{7} $$

When calculating the dot product $y = w \cdot x$ using the quantized weights, if the input vector $x$ is similarly quantized to $x_i \approx \Delta_x \times q_{x, i}$, we get:

$$ y = \sum_{i=1}^{B} w_i x_i \approx \Delta \Delta_x \sum_{i=1}^{B} q_i q_{x, i} $$

The $\sum q_i q_{x, i}$ part becomes a pure integer operation, which can be computed in parallel very quickly using SIMD instructions. This is the mathematical trick behind the astonishing speed of llama.cpp on CPUs.

3. Architecture and Inference Flow

To understand the internal workings of llama.cpp, the following Mermaid diagram shows the overall system architecture and data flow.

graph TD A["User Input (String)"] --> B["llama.cpp Tokenizer"] B --> C["Token IDs (int32 array)"] C --> D["Context Buffer (KV Cache)"] D --> E["ggml Compute Graph"] E --> F["Transformer Layers"] subgraph "ggml Engine" F --> G["Self-Attention (RoPE)"] G --> H["Feed Forward Network"] H --> F end F --> I["Logits (Vocabulary Size)"] I --> J["Sampler (Temperature, Top-K, Top-P)"] J --> K["Selected Token ID"] K --> L["llama.cpp Detokenizer"] L --> M["Output String"] K -. "Auto-regressive loop" .-> D

Text generation is an auto-regressive loop where, each time a single token is output, it is added to the KV Cache as the next input and passes through the compute graph again.

4. Environment Setup and Build Guide

Before embedding llama.cpp into a C++ project, let’s first build the source code.

4.1 Cloning the Repository

1
2


git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

4.2 Building with CMake

When embedding it into other applications as a C++ project, using CMake is the most standard approach. By enabling accelerators (backends) for your specific platform, you can speed up computations.

CPU Only (Basic Build):

1
2
3


mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

Using NVIDIA GPU (CUDA):

1
2
3


mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j 8

Using Apple Silicon (Metal):

1
2
3


mkdir build && cd build
cmake .. -DGGML_METAL=ON
cmake --build . --config Release -j 8

Upon a successful build, executable files like llama-cli and the llama library (along with the ggml library) for linking via the C++ API discussed later will be generated in the build/bin/ directory.

5. Introduction to C++ Customization: Using the llama.cpp API

From here on, we will discuss the main topic: controlling llama.cpp from C++ code. To embed an LLM into your own application (e.g., a game engine, desktop app, or embedded system) rather than just using command-line tools, you need to hit the C++ API directly.

llama.cpp primarily provides a C language interface through a header file called llama.h. We use this interface even when calling from C++.

5.1 Minimal Necessary Includes and Setup

When using llama.cpp in your project, include the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


#include "llama.h"
#include <iostream>
#include <vector>
#include <string>
#include <stdexcept>

// Macro for error handling
#define LLAMA_ASSERT(x) \
 do { \
 if (!(x)) { \
 std::cerr << "Assertion failed: " << #x << std::endl; \
 std::terminate(); \
 } \
 } while (0)

5.2 Loading the Model and Initializing the Context

First, load a .gguf format model file and allocate the context (memory space and KV cache) for inference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


int main(int argc, char ** argv) {
 if (argc < 2) {
 std::cerr << "Usage: " << argv[0] << " <model.gguf>" << std::endl;
 return 1;
 }
 std::string model_path = argv[1];

 // 1. Initialize the backend (set up environment like CPU/GPU)
 llama_backend_init();

 // 2. Get default settings for model parameters
 llama_model_params model_params = llama_model_default_params();
 model_params.n_gpu_layers = 35; // Number of layers to offload to GPU

 // 3. Load the model
 llama_model * model = llama_load_model_from_file(model_path.c_str(), model_params);
 if (model == nullptr) {
 std::cerr << "Failed to load model" << std::endl;
 return 1;
 }

 // 4. Set context parameters
 llama_context_params ctx_params = llama_context_default_params();
 ctx_params.n_ctx = 2048; // Maximum context size (number of tokens)
 ctx_params.n_threads = 8; // Number of CPU threads used for inference

 // 5. Create the context
 llama_context * ctx = llama_new_context_with_model(model, ctx_params);
 if (ctx == nullptr) {
 std::cerr << "Failed to create context" << std::endl;
 llama_free_model(model);
 return 1;
 }

 std::cout << "Model and context loaded successfully!" << std::endl;
 // ... subsequent processing

5.3 Tokenization of the Prompt

An LLM does not understand text directly; it processes strings as sequences of integer IDs (tokens). Therefore, the input string must be converted into tokens.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 std::string prompt = "Q: What is the capital of Japan?\nA:";
 std::vector<llama_token> tokens_list;
 tokens_list.resize(prompt.length() + 4); // Buffer size with some headroom

 // Whether to prepend a special token (like BOS: Begin of Sequence)
 bool add_special = true;
 // Convert string to an array of token IDs
 int n_tokens = llama_tokenize(
 model,
 prompt.c_str(),
 prompt.length(),
 tokens_list.data(),
 tokens_list.size(),
 add_special,
 false // parse_special
 );

 if (n_tokens < 0) {
 // Handling for when the buffer is insufficient, like reallocating and retrying, is necessary (omitted for brevity)
 std::cerr << "Failed to tokenize prompt" << std::endl;
 return 1;
 }
 tokens_list.resize(n_tokens);

5.4 Inference Loop and Sampling

Build a loop that feeds tokens into the model, obtains the probability distribution (Logits) of the next token, and samples from it to determine the next token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72


 // Maximum number of tokens to generate
 const int max_gen_tokens = 100;

 // Initialize structure for batch evaluation
 llama_batch batch = llama_batch_init(512, 0, 1);

 // Add prompt tokens to the batch
 for (size_t i = 0; i < tokens_list.size(); i++) {
 llama_batch_add(batch, tokens_list[i], i, { 0 }, false);
 }
 // Set to output logits (predictions) only for the very last token of the prompt
 batch.logits[batch.n_tokens - 1] = true;

 // Initial evaluation (feeding the prompt to the model)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "llama_decode() failed" << std::endl;
 return 1;
 }

 int n_cur = batch.n_tokens; // Current context length
 int n_decode = 0;

 std::cout << "\nOutput: ";

 // Initialize sampler context (settings for Temperature, Top-K, Top-P, etc.)
 llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
 llama_sampler_chain_add_top_k(smpl, 40);
 llama_sampler_chain_add_top_p(smpl, 0.9f, 1);
 llama_sampler_chain_add_temp(smpl, 0.7f);
 llama_sampler_chain_add_dist(smpl, 1234); // Seed value

 while (n_decode < max_gen_tokens) {
 // 1. Sampling: Predict the next token based on current context
 llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);

 // 2. If token is EOS (End of Sequence), break the loop
 if (llama_token_is_eog(model, new_token_id)) {
 break;
 }

 // 3. Decode token to string (text) and print
 char buf[128];
 int n_chars = llama_token_to_piece(model, new_token_id, buf, sizeof(buf), 0, false);
 if (n_chars > 0) {
 std::cout << std::string(buf, n_chars) << std::flush;
 }

 // 4. Prepare the newly generated token as the next batch
 llama_batch_clear(batch);
 llama_batch_add(batch, new_token_id, n_cur, { 0 }, true);

 // 5. Evaluate the model (update KV cache and predict next)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "Failed to evaluate" << std::endl;
 break;
 }

 n_cur += 1;
 n_decode += 1;
 }

 std::cout << std::endl;

 // Cleanup
 llama_sampler_free(smpl);
 llama_batch_free(batch);
 llama_free(ctx);
 llama_free_model(model);
 llama_backend_free();

 return 0;
}

This code implements a custom inference loop using the basic API of llama.cpp. It uses the llama_batch struct to manage token groups and executes the forward pass of the neural network using llama_decode.

6. Advanced Customization Examples: Logit Manipulation and Penalty Control in C++

When you want to go beyond simple text generation—such as forcing output in a specific format (e.g., JSON only) or suppressing specific forbidden words—you directly manipulate the Logits prior to sampling from the C++ side.

You can retrieve the array of raw scores (values before being converted to probabilities) right before the model outputs each token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Retrieve the raw logits array immediately after inference, prior to sampling
float * logits = llama_get_logits_ith(ctx, batch.n_tokens - 1);
int n_vocab = llama_n_vocab(model);

// List of forbidden token IDs (for example: 1234, 5678)
std::vector<llama_token> forbidden_tokens = { 1234, 5678 };

// Set the occurrence probability of forbidden tokens to 0 (make the Logit negative infinity)
for (llama_token bad_tok : forbidden_tokens) {
 logits[bad_tok] = -INFINITY;
}

In this way, directly interacting with the C++ API allows for “micro-millisecond interventions per inference cycle” that would be difficult or incur high overhead if done via LangChain or Python.

7. The Secrets of Performance Tuning

After finishing your C++ implementation, here are a few checkpoints for maximizing speed for actual deployment.

Optimizing Batch Processing: When handling requests from multiple users concurrently, include multiple sequences in llama_batch and call llama_decode at once (Continuous Batching). This drastically improves throughput by coalescing memory access.
Enabling Flash Attention: By setting ctx_params.flash_attn = true; in the context parameters, you can speed up Attention calculations while reducing memory usage. This setting is essential when dealing with long contexts (tens of thousands of tokens).
NUMA Support: In multi-socket server environments, properly configuring NUMA before llama_backend_init() can reduce memory access latency.

8. Conclusion

In this article, we covered everything in detail, starting from the mathematical background of llama.cpp to an explanation of its architecture, and finally, how to build a custom inference engine fully utilizing the C++ API.

While the Python ecosystem is highly convenient for prototyping, the direct control offered by C/C++ based llama.cpp demonstrates overwhelming power in production environments that demand edge device deployment, game integration, and real-time processing.

By all means, try writing C++ code yourself and experience the joy of freely manipulating LLMs in a local environment.

Reference Links

llama.cpp Official Repository

ggml - Tensor Library

Attention Is All You Need (Vaswani et al., 2017)

【2026 Edition】 The Complete Guide to Running Local LLMs in a Windows Environment

Fri, 11 Sep 2026 10:00:00 +0900

1. Introduction: Why Local LLMs on Windows Now?

As of 2026, the evolution of generative AI and Large Language Models (LLMs) shows a major paradigm shift from gigantic cloud-based API services to “local LLMs” running on personal PCs and on-premise environments. While cloud AIs like OpenAI’s GPT-5 and Anthropic’s Claude 3.5 are incredibly powerful, not all companies and individuals can send all their data to the cloud. From the perspectives of privacy, security, latency, and long-term sustainable costs, the demand for local LLMs is exploding like never before.

The evolution of the local LLM ecosystem, especially in the Windows environment, is remarkable. Until a few years ago, “Linux for AI development and execution” was common sense, but as of 2026, Windows has transformed into an extremely powerful and accessible AI platform.

In this article, based on the latest technology trends of 2026, we provide a complete guide to building, operating, and optimizing local LLMs in a Windows environment. From easy setup using Ollama for beginners to extreme optimization using llama.cpp for advanced users, and further deep dives into the mathematical approach of VRAM calculation, deep understanding of the architecture, and local fine-tuning, we will explain everything thoroughly with an overwhelming volume.

1.1 Technology Trends Surrounding Local LLMs in 2026

The major trends shaping the current local LLM ecosystem are as follows:

Complete Popularization of the GGUF Format: GGUF (GPT-Generated Unified Format), which integrates metadata and tensors into a single file, has completely become the de facto standard. With this, simply downloading a single file from Hugging Face makes it executable in any environment.
Democratization of the MoE (Mixture of Experts) Architecture: Many small but high-performance MoE models have been released. By activating only a portion of the experts during inference, they achieve performance comparable to giant models while keeping the computational load on consumer PCs low.
Advanced Abstraction and Optimization of Inference Engines: Tools like Ollama, LM Studio, and AnythingLLM have been refined so that users no longer need to be aware of complex dependencies like CUDA driver installations. Also, the native Windows support for FlashAttention 3 has dramatically improved inference speed.
Utilization of NPUs and the Rise of Windows Copilot+ PCs: Even on laptops without GPUs, the technology to run small LLMs (SLM: Small Language Models) with low power consumption using the built-in NPU (Neural Processing Unit) has entered the practical stage.

2. Hardware Requirements and OS Preparation

To run local LLMs at practical speeds (15-30 tokens per second or more), selecting the right hardware is the most important factor.

2.1 Recommended Hardware Configuration

With the evolution of AI PCs, required specs are also changing.

OS: Windows 11 Pro (24H2 or later). Essential for fully utilizing WSL2’s features, advanced memory management, and the latest DirectML APIs.
CPU: Intel Core Ultra 200 series or higher, or AMD Ryzen 9000 series or higher. When using CPU inference alongside, broad-bandwidth memory communication is indispensable.
RAM: Minimum 32GB, recommended 64GB or more. Main memory bandwidth (MB/s) becomes a crucial bottleneck during CPU inference or offloading. High-speed memory of DDR5-6000 or above is ideal.
GPU: NVIDIA RTX 4000/5000 series. The most important thing for local LLMs is not computing performance but “VRAM capacity”.
- Entry: RTX 4060 Ti (16GB version) - Best cost performance. Ideal for 8B-14B class models.
- Mid-range: RTX 4070 Ti SUPER (16GB) / RTX 4080 SUPER (16GB)
- High-end: RTX 4090 (24GB) / RTX 5090 (32GB) - Necessary to run 30B-70B class quantized models.
Storage: PCIe Gen4 or Gen5 NVMe SSD. Dramatically reduces the load times of models that are tens of gigabytes in size.

2.2 Setting up WSL2 (Windows Subsystem for Linux 2)

While many GUI tools work natively on Windows, WSL2 is extremely useful for Python development, compiling the latest tools, and the LoRA fine-tuning mentioned later. In the latest Windows 11 environment, just by installing the NVIDIA driver on the host side, the GPU (CUDA) can be transparently used from WSL2.

Open PowerShell with administrator privileges and execute the following:

1
2
3
4
5


# Install WSL2 and the latest Ubuntu
wsl --install -d Ubuntu-24.04

# Update the kernel
wsl --update

After installation, run nvidia-smi inside the WSL2 terminal, and if the GPU is recognized correctly, it is a success.

3. Local LLM Architecture and Inference Mechanism

Understanding how models generate text in a local environment and their internal structure is very useful for troubleshooting and optimization.

The following Mermaid diagram shows a typical local LLM inference pipeline.

graph TD User["User Input (Prompt)"] --> Tokenizer["Tokenizer"] Tokenizer --> Embedding["Embedding Layer"] subgraph "Transformer Block (x Layers)" Embedding --> Attn["Self-Attention"] Attn --> KVCache["KV Cache (Key/Value Storage)"] Attn --> FFN["Feed-Forward Network (FFN)"] end FFN --> Logits["Logits Calculation"] Logits --> Sampler["Sampler (Temperature, Top-K, Top-P)"] Sampler --> OutputToken["Output Token"] OutputToken --> |"Autoregressive Generation"| Tokenizer OutputToken --> Decoder["Detokenizer"] Decoder --> FinalOutput["Final Output Text"]

3.1 Two Phases: Prefill and Decode

LLM text generation is divided into two phases with different computational characteristics.

Prefill Phase (Prompt Processing): The phase that processes and understands the entire input prompt at once. Since parallel computing is possible, the computational power of the GPU (FLOPS) directly links to speed. If the prompt is long, this phase can take several seconds.
Decode Phase (Token Generation): The phase that predicts one token at a time and feeds it to the next input (autoregressive). Since parallel computing is restricted in this phase, GPU VRAM bandwidth (Memory Bandwidth) becomes the definitive bottleneck.

4. Mathematical Understanding of VRAM Consumption and Model Size

To correctly determine “which model will run on my PC?”, you need to understand the VRAM calculation formula. When VRAM shortages cause a fallback to system memory (RAM), inference speeds drop by 10x to 100x.

4.1 Base VRAM Based on Parameter Size

This is the amount of memory needed to load the model’s weights into VRAM. Calculate it using the model size $P$ (number of parameters, unit: 1 billion = 1B) and the number of bytes per parameter $B$.

$$ V_{base} = P \times B \quad \text{(GB)} $$

For example, when loading an 8B (8 billion) parameter model in FP16 (half-precision floating point, 16 bits = 2 bytes):

$$ V_{base} = 8 \times 2 = 16 \text{ GB} $$

In other words, even a GPU with 16GB of VRAM will reach its limit just by loading the model.

4.2 The Magic of Quantization

This is where “quantization” comes in. By reducing the precision of the parameters, the model size is drastically shrunk. In the case of the most common 4-bit quantization (e.g., Q4_K_M), it averages to about 0.55 bytes per parameter.

$$ V_{base\_4bit} = 8 \times 0.55 = 4.4 \text{ GB} $$

With this, if you have 16GB of VRAM, you can run an 8B model with plenty of headroom.

4.3 KV Cache Calculation (GQA Supported Version)

During inference, the “KV cache” needed to retain past context consumes VRAM. The latest models, such as Llama 3, use GQA (Grouped Query Attention) to save memory.

The KV cache consumption $V_{kv}$ (in gigabytes) is expressed by the following formula:

$$ V_{kv} = 2 \times b \times s \times l \times \left( \frac{h_{kv}}{h_q} \right) \times h_q \times d \times B_{kv} \div 10^9 $$

Simplifying this using the number of key/value heads $h_{kv}$:

$$ V_{kv} = 2 \times b \times s \times l \times h_{kv} \times d \times B_{kv} \div 10^9 $$

Where:

$b$: Batch size (usually 1 for individual local use)
$s$: Sequence length (context length, e.g., 8192)
$l$: Number of layers (e.g., 32)
$h_{kv}$: Number of KV heads (e.g., 8)
$d$: Number of dimensions per head (e.g., 128)
$B_{kv}$: Number of bytes for KV cache (2 for FP16)

Calculation example (Llama 3 8B, context 8192, FP16 cache): $V_{kv} = 2 \times 1 \times 8192 \times 32 \times 8 \times 128 \times 2 \div 10^9 \approx 1.07 \text{ GB}$

Note that the longer the context length $s$ is, the needed VRAM increases linearly.

5. Practice 1: Fastest and Shortest Setup using Ollama

Now that you understand the theory, let’s actually run an LLM in a Windows environment. As of 2026, the most user-friendly tool is “Ollama”. It provides an intuitive Docker-like CLI.

5.1 Installation and Execution

Download the Windows installer from the Ollama Official Website and run it.
Open PowerShell and enter the following command. Here, we’ll use the Japanese-compatible llama3:8b.

1

ollama run llama3:8b

The model will be downloaded on the first run. Once complete, you can interact with it directly in the terminal.

5.2 Creating a Custom AI with a Modelfile

You can easily create an AI with a specific persona. Create a Modelfile in an arbitrary location.

1
2
3
4
5
6
7
8
9


FROM llama3:8b

SYSTEM """
You are an exceptionally talented senior software engineer.
For user questions, always provide code examples and answer logically and concisely.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Build and run your custom model with the following commands:

1
2


ollama create SeniorDev -f ./Modelfile
ollama run SeniorDev

5.3 Usage from External Apps (AI Editors)

Ollama exposes an OpenAI-compatible API endpoint at http://localhost:11434. By simply setting this URL in the backend settings of VS Code extensions like Cursor or Continue.dev, and specifying the model name such as SeniorDev, a powerful local coding assistant is realized for free.

6. Practice 2: Extreme Performance Tuning with llama.cpp

If you want fine-grained memory management or want to be the first to try out the latest formats (like EXL2 or IQ quantization), you manipulate the core engine llama.cpp directly.

6.1 llama.cpp Build Steps

In a Windows environment, the best approach is building from source using CUDA Toolkit and CMake.

1
2
3
4
5
6
7
8


git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

# Configure for CUDA support and compile
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=OFF
cmake --build . --config Release -j 16

6.2 Advanced Launching in Server Mode

Host the model using the built llama-server.exe.

1
2
3
4
5
6
7


.\bin\Release\llama-server.exe `
 --model "C:\models\Llama-3-8B-Instruct.Q4_K_M.gguf" `
 --ctx-size 8192 `
 --n-gpu-layers 99 `
 --threads 8 `
 --flash-attn `
 --port 8080

--n-gpu-layers 99: Offloads all possible layers to GPU VRAM.
--flash-attn: Enables FlashAttention 3, achieving improved inference speed and reduced VRAM consumption for the KV cache.

7. GUI Frontend: LM Studio and Building Local RAG

If you’re resistant to the command line, or intuitively want to perform RAG (Retrieval-Augmented Generation), you can use a GUI.

7.1 LM Studio

LM Studio is a brilliant application that bundles model search, downloading, system requirement pre-checks, and a chat UI all into one. Just by pressing the “Local Server” button in the app, an OpenAI-compatible API starts up.

7.2 RAG Architecture using AnythingLLM

Here is the architecture diagram of a RAG environment for reading internal documents and personal notes.

graph LR Document["Document (PDF, MD)"] --> Chunking["Chunking"] Chunking --> EmbedModel["Embedding Model"] EmbedModel --> VectorDB["Vector Database"] UserQuery["User Query"] --> EmbedQuery["Query Embedding"] EmbedQuery --> VectorDB VectorDB --> |"Similarity Search"| RetrievedDocs["Extract Relevant Docs"] UserQuery --> PromptBuilder["Prompt Generation"] RetrievedDocs --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer"]

Using the AnythingLLM desktop version (Windows), just specify Ollama (LLM and Embedding) from the settings screen and set it up to use a local VectorDB (LanceDB). This architecture can be completed in minutes. A private AI is born that does not send any data externally.

8. Fine-Tuning (LoRA) on Windows WSL2

If you want to not just run locally but make the model smarter with your own data, fine-tuning using LoRA (Low-Rank Adaptation) is possible. As of 2026, by using a library called “Unsloth”, an 8B model can finish training in a few hours on a Windows WSL2 environment even with 16GB of VRAM.

Execute the following within WSL2’s Ubuntu to build the environment.

1
2
3
4


conda create --name unsloth_env python=3.11
conda activate unsloth_env
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Unsloth optimizes CUDA kernels to the extreme, providing about twice the training speed and half the VRAM consumption compared to the standard Hugging Face libraries. Just spin up a Jupyter Notebook and load your dataset (JSONL format), and training for several epochs is possible even on an RTX 4060 Ti with 12GB to 16GB of VRAM.

9. Performance Troubleshooting

Common problems faced and their solutions.

1. Inference speed is extremely slow (1-2 tokens/s)

Cause: The model doesn’t fit entirely into VRAM and is being offloaded to system memory (RAM). Solution: Check “Dedicated GPU memory” in the Task Manager. If it’s hitting the limit, decrease the context size (-c), or use a model with lower bit quantization (like Q4_K_M).

2. “CUDA out of memory” error

Cause: VRAM has been completely exhausted. This occurs especially when the context is prolonged and the KV cache becomes bloated. Solution: Intentionally restrict the values to smaller ones using num_ctx for Ollama, or -c for llama.cpp.

3. Strange Japanese Generation

Cause: Mismatch in prompt templates, or an unsupported model. Solution: Use models that include Instruct in their name, and ensure that the tool is selecting the correct template specified by the model author, such as the ChatML or Llama3 format.

10. Conclusion and Future Prospects

In 2026, building a local LLM in a Windows environment is no longer the privilege of a limited number of engineers. With the de facto standardization of the GGUF format, the emergence of refined ecosystems like Ollama and LM Studio, and hardware optimizations led by FlashAttention, anyone can easily obtain an enterprise-grade AI environment.

Please make use of the following points explained in this article:

Use mathematical VRAM calculations to logically select the optimal model size and quantization level for your PC specs.
Build your environment at maximum speed using Ollama, and dramatically improve productivity by integrating it with AI editors.
Bring out the ultimate performance of your hardware with the advanced parameter control of llama.cpp.
Build a secure local RAG system to handle confidential data with AnythingLLM.
Nurture a custom AI with your own specialized knowledge by utilizing Unsloth (WSL2).

The “democratization” of AI is no longer a buzzword, but a real system running on your Windows desktop. Free yourself from the usage costs of cloud APIs and information leak risks, and step into the world of free and powerful private AI right now.