C++ on kenji.blog

No Python! Building an AI Inference Engine Exclusively in C++

Fri, 11 Sep 2026 17:00:00 +0900

1. Introduction: Why let go of Python and build an AI inference engine in C++?

In modern AI development, Python is the de facto standard. Thanks to powerful frameworks like PyTorch and TensorFlow, complex neural networks can be built, trained, and inferred with just a few lines of code. However, behind these frameworks, low-level languages like C++ and CUDA handle the heavy computational processing. Python is merely acting as the “glue”.

So why bother eliminating Python and creating an AI inference engine exclusively in C++? There are several compelling reasons.

Extreme Performance and Low Latency: You can completely eliminate the overhead of Python’s GIL (Global Interpreter Lock) and dynamic typing. Especially in systems requiring real-time performance, delays on the millisecond scale can be fatal.
Ease of Deployment: Building a Python environment (massive library ecosystems, dependency hell) on the end user’s system is extremely difficult. With C++, you only need to distribute a single statically linked executable binary (.exe or ELF binary).
Edge Device Support: In highly resource-constrained environments like smartphones, embedded devices, and Raspberry Pi, there is no luxury to run a Python runtime that consumes gigabytes of memory.
Direct Hardware Control: Low-level control such as memory allocation timing, explicit use of SIMD instructions, and optimization of memory transfers with the GPU is possible with C++.

In this article, while drawing massive inspiration from the architecture of the “GGML” library developed by Georgi Gerganov, we will dive deep into the technical abyss and explain the process of building an inference engine from scratch using only C++ to run Large Language Models (LLMs).

2. Overview of the Inference Engine Architecture

AI inference processing is essentially a “series of massive matrix calculations”. To execute this efficiently, an inference engine needs to be composed of the following components.

graph TD A["Input Data (Tokens/Images)"] --> B["Tensor Management"] B --> C["Computation Graph (DAG)"] C --> D["Memory Arena & Allocator"] C --> E["Scheduler & Thread Pool"] E --> F["CPU Backend (AVX2/ARM NEON)"] E --> G["GPU Backend (CUDA/Metal)"] F --> H["Output Results"] G --> H

Tensor Management: Manages multidimensional array data structures and strides for each dimension.
Computation Graph: Represents the operations of each layer in the neural network as a Directed Acyclic Graph (DAG).
Memory Arena: A pre-allocated memory management mechanism to avoid the overhead of dynamic memory allocation (malloc or new).
Backend: Operations (kernels) optimized for specific hardware such as CPUs and GPUs.

We will assemble these using the powerful features of C++ (templates, pointer arithmetic, RAII, etc.).

3. The Secret of Memory Management: Memory Arena and SIMD Alignment

Memory management in an inference engine is one of the most critical factors directly linked to performance. During inference, especially as data passes through each layer of a Transformer model, a massive number of intermediate tensors are generated. If you allocate and free these with standard malloc every time, heap fragmentation and OS context switches will cause a fatal slowdown.

Therefore, we adopt an approach called the “Memory Arena”. This is a method where the maximum amount of memory required is calculated (or fixed) and allocated at once when inference begins, and memory is carved out simply by incrementing a pointer.

3.1 The Importance of Alignment

Modern CPUs support SIMD (Single Instruction, Multiple Data) instructions. Examples include AVX2/AVX-512 for Intel/AMD and NEON for ARM. These instructions process 256 bits (32 bytes) or 512 bits (64 bytes) of data at once, but the target data memory must be aligned to specific byte boundaries (usually 32 bytes or 64 bytes).

Below is an example of a C++ implementation of a memory arena that considers alignment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


#include <cstdint>
#include <cstddef>
#include <stdexcept>
#include <iostream>

struct MemoryArena {
 size_t size;
 size_t offset;
 uint8_t* data;

 MemoryArena(size_t size) : size(size), offset(0) {
 // Use posix_memalign for POSIX systems, _aligned_malloc for Windows
#ifdef _WIN32
 data = static_cast<uint8_t*>(_aligned_malloc(size, 64));
#else
 if (posix_memalign(reinterpret_cast<void**>(&data), 64, size) != 0) {
 throw std::bad_alloc();
 }
#endif
 }

 ~MemoryArena() {
#ifdef _WIN32
 _aligned_free(data);
#else
 free(data);
#endif
 }

 void* allocate(size_t bytes, size_t alignment = 64) {
 // Calculate alignment (find padding)
 size_t pad = (alignment - (offset % alignment)) % alignment;
 if (offset + pad + bytes > size) {
 throw std::runtime_error("OOM: MemoryArena out of memory");
 }
 offset += pad;
 void* ptr = data + offset;
 offset += bytes;
 return ptr;
 }

 void reset() {
 offset = 0; // Freeing memory is just resetting the pointer (O(1))
 }
};

In this way, when creating tensors, memory is always obtained through this arena. By simply calling reset() every time an inference step (such as token generation) is completed, memory can be reused instantly.

4. Tensor Data Structures and the Magic of Strides

A tensor is a generalized concept of scalars, vectors, and matrices. What is important in the implementation is that while the actual data is laid out as a one-dimensional contiguous array in memory, it has a concept called “Stride” to interpret it as multidimensional.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


enum class DataType {
 FP32,
 FP16,
 INT8, // For quantization
 INT4 // For quantization
};

struct Tensor {
 int n_dims; // Number of dimensions
 int64_t ne[4]; // Number of elements in each dimension (Number of Elements)
 size_t nb[4]; // Stride in each dimension (Number of Bytes)
 DataType type; // Data type
 void* data; // Pointer to payload

 // For computation graph
 enum OpType op;
 Tensor* src0;
 Tensor* src1;
};

The stride nb[i] represents the byte distance in memory between adjacent elements in dimension i. For example, if a matrix of size $M \times N$ (FP32, 4 bytes per element) is stored in Row-Major order, the strides would be:

nb[0] = 4 (bytes) : Movement along the column direction
nb[1] = $N \times 4$ (bytes) : Movement along the row direction

By utilizing this, operations such as “Transpose” and “View” can be achieved without memory copying, simply by swapping the stride values. It is very elegant and fast.

5. Building the Computation Graph (DAG) and Lazy Evaluation

Similar to PyTorch, our inference engine also adopts Lazy Evaluation, close to “Define-by-Run”. That is, at the time the operation function is called, the calculation is not performed; only the graph (dependencies between nodes) is built.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Tensor* tensor_add(MemoryArena& arena, Tensor* a, Tensor* b) {
 Tensor* out = create_tensor(arena, a->type, a->n_dims, a->ne);
 out->op = OpType::ADD;
 out->src0 = a;
 out->src1 = b;
 return out;
}

Tensor* tensor_mul_mat(MemoryArena& arena, Tensor* a, Tensor* b) {
 // b is often transposed
 int64_t ne[2] = { a->ne[0], b->ne[1] };
 Tensor* out = create_tensor(arena, a->type, 2, ne);
 out->op = OpType::MUL_MAT;
 out->src0 = a;
 out->src1 = b;
 return out;
}

The flow of inference processing is as follows.

graph LR A["Define Tensors"] --> B["Build Graph via Ops"] B --> C["Topological Sort"] C --> D["Allocate Memory for Outputs"] D --> E["Execute Nodes In Order"]

When evaluating the graph (forward pass), it uses topological sorting to execute nodes in order, starting from those without dependencies. Since we only do inference, there is no need to retain gradients for backpropagation, making memory management extremely simple.

6. The Core of Math and Optimization: General Matrix Multiply (GEMM)

Over 90% of the computational cost of AI inference is spent on General Matrix Multiply (GEMM). Both the attention mechanism and the feed-forward network (FFN), which are the core of the Transformer model, are ultimately massive matrix multiplications.

The product $C = A B$ (size $M \times N$) of two matrices $A$ (size $M \times K$) and $B$ (size $K \times N$) is expressed by the following formula.

$$ C_{i,j} = \sum_{k=0}^{K-1} A_{i,k} \cdot B_{k,j} $$

If this is implemented with a naive triple loop, cache misses will occur frequently, and no performance will be achieved.

6.1 Cache Blocking and SIMD Optimization on CPU

The basic strategies for speeding up GEMM on a CPU are as follows.

Loop Tiling (Cache Blocking): Divide matrices into smaller blocks that fit in the L1/L2 cache for computation.
Data Packing: Rearrange data internally so that memory access patterns become contiguous.
Utilizing SIMD: Use FMA (Fused Multiply-Add) instructions like _mm512_fmadd_ps in AVX-512 to perform many multiply-add operations in a single clock cycle.

Here is an example of a simplified vector dot product using C++ and SIMD Intrinsics.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


#include <immintrin.h> // For AVX instructions

// Fast dot product for FP32 using AVX2
float dot_product_avx2(const float* a, const float* b, int n) {
 __m256 sum256 = _mm256_setzero_ps();
 int i = 0;

 // Process 8 elements at a time (256 bits = 32 bytes = 8 * 4 bytes)
 for (; i <= n - 8; i += 8) {
 __m256 va = _mm256_loadu_ps(a + i);
 __m256 vb = _mm256_loadu_ps(b + i);
 // FMA instruction: sum256 = va * vb + sum256
 sum256 = _mm256_fmadd_ps(va, vb, sum256);
 }

 // Horizontally add the values in the SIMD register
 float result[8];
 _mm256_storeu_ps(result, sum256);
 float dot = result[0] + result[1] + result[2] + result[3] +
 result[4] + result[5] + result[6] + result[7];

 // Process the remainder
 for (; i < n; ++i) {
 dot += a[i] * b[i];
 }
 return dot;
}

Just this small tweak can yield a speed improvement of several to dozens of times compared to a naive implementation.

7. Crossing Hardware Boundaries: Integrating CUDA and Metal Backends

While pure C++ implementations alone run reasonably well on CPUs, the parallel computing power of GPUs is indispensable to run massive models like LLMs at practical speeds (e.g., generating more than 20 tokens per second). Thus, we introduce a backend abstraction layer to our engine.

7.1 Backend Abstraction

Using C++ polymorphism, we make it possible to switch the computation executor.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class Backend {
public:
 virtual ~Backend() = default;
 virtual void alloc_buffer(Tensor* t) = 0;
 virtual void free_buffer(Tensor* t) = 0;
 virtual void copy_to_device(Tensor* t) = 0;
 virtual void copy_to_host(Tensor* t) = 0;

 // Execution of various operations
 virtual void compute_add(Tensor* src0, Tensor* src1, Tensor* dst) = 0;
 virtual void compute_mul_mat(Tensor* src0, Tensor* src1, Tensor* dst) = 0;
};

7.2 Implementing the NVIDIA CUDA Backend

To leverage NVIDIA GPUs, we implement the backend using CUDA C++ extensions. While writing custom kernels is possible, for matrix multiplication, the best approach is to utilize “cuBLAS”, the premier library provided by NVIDIA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


#include <cublas_v2.h>
#include <cuda_runtime.h>

class CUDABackend : public Backend {
private:
 cublasHandle_t handle;

public:
 CUDABackend() {
 cublasCreate(&handle);
 }

 ~CUDABackend() {
 cublasDestroy(handle);
 }

 void compute_mul_mat(Tensor* src0, Tensor* src1, Tensor* dst) override {
 // Since CUDA is Column-Major by default, parameters require care
 const float alpha = 1.0f;
 const float beta = 0.0f;

 int m = src0->ne[0];
 int k = src0->ne[1];
 int n = src1->ne[1]; // Assuming src1 is transposed

 cublasSgemm(handle, CUBLAS_OP_T, CUBLAS_OP_N,
 m, n, k,
 &alpha,
 (const float*)src0->data, k,
 (const float*)src1->data, k,
 &beta,
 (float*)dst->data, m);
 cudaDeviceSynchronize();
 }
};

Data transfer (cudaMemcpy) between CUDA memory and host (CPU) memory is very heavy, so it is crucial to design the system to retain all weights (weight tensors) and intermediate tensors on VRAM as much as possible during inference.

7.3 Apple Silicon (Metal) Backend

In recent years, Mac’s M1/M2/M3 chips (Apple Silicon) have become highly excellent as AI inference machines. The reason for this lies in “Unified Memory”. Because the CPU and GPU share the same memory region, the high-cost host-to-device memory transfers via the PCIe bus, as seen in CUDA mentioned above, become completely unnecessary.

To call Metal from C++, we use Objective-C++ (.mm files) as a bridge, or use the metal-cpp library. We write kernels using Metal’s Compute Shaders (written in a C++-like manner in .metal files).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Metal shader (kernel.metal)
#include <metal_stdlib>
using namespace metal;

kernel void mul_mat_kernel(
 device const float* A [[buffer(0)]],
 device const float* B [[buffer(1)]],
 device float* C [[buffer(2)]],
 constant uint3& dims [[buffer(3)]],
 uint2 gid [[thread_position_in_grid]]
) {
 uint m = dims.x; uint k = dims.y; uint n = dims.z;
 uint row = gid.y; uint col = gid.x;

 if (row < m && col < n) {
 float sum = 0.0;
 for (uint i = 0; i < k; ++i) {
 sum += A[row * k + i] * B[i * n + col]; // Simplified
 }
 C[row * n + col] = sum;
 }
}

In the Apple Silicon environment, an optimized library for matrix multiplication called MPS (Metal Performance Shaders) is also provided, so by utilizing this in production, you can achieve astonishing inference speeds.

8. Transformer Model Specific Processing: Attention and KV Cache

State-of-the-art LLMs such as LLaMA 2/3 and GPT are based on the Transformer architecture. To implement this in C++, it is essential to construct “Scaled Dot-Product Attention,” represented by the following formula.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Also, in autoregressive token generation, the calculation results of past tokens (Key and Value) must be retained. This is called the “KV Cache (Key-Value Cache)”.

graph TD T["Current Token"] --> Q["Query"] T --> K["Key"] T --> V["Value"] K --> KCache["Append to KV Cache"] V --> VCache["Append to KV Cache"] Q --> Dot1["Q * K_Cache^T"] KCache --> Dot1 Dot1 --> Scale["Scale (1/sqrt(d))"] Scale --> Softmax["Softmax"] Softmax --> Dot2["SoftmaxOut * V_Cache"] VCache --> Dot2 Dot2 --> Out["Context Vector"]

For memory allocation of the KV Cache, a ring buffer-like operation is used, pre-allocating memory space for the maximum context length (e.g., 4096 or 8192 tokens) in the arena in advance. This prevents reallocation at every generation step.

In addition, we implement “RoPE (Rotary Position Embedding),” which has become mainstream in recent years, for positional encoding. This is a method of embedding position information as rotation vectors in complex space, and optimizing the calling of sin and cos functions in C++ (such as using lookup tables) is key to performance.

9. Extreme Optimization via Model Quantization

If you load a large-scale model (e.g., a 7 Billion parameter LLaMA model) in FP32 (32-bit floating-point), it will consume about 28GB of memory (VRAM) just for weights. If you include the KV cache and inference buffers, it easily exceeds 30GB, making it impossible to run on typical consumer GPUs.

This is where “Quantization” becomes essential. It is also the true worth of the GGML format.

Quantization is the technique of intentionally reducing the precision of weights.

FP16 (16-bit): Size halved. Almost no precision degradation.
INT8 (8-bit): Size 1/4. Slight degradation.
INT4 (4-bit): Size 1/8. Practical inference is possible by using proprietary blocking and scaling factors.

On the inference engine side, weights compressed in INT4 (or INT8) are read from memory, and immediately after being loaded into the CPU or GPU registers, they are expanded (Dequantized) to FP16 or FP32 for computation.

Surprisingly, it is faster to reduce the amount of data read from memory, even if it means increasing the amount of computation. This is because on modern hardware, the bottleneck for inference tasks is not “Compute Bound” but “Memory Bandwidth Bound”. With a C++ engine implementing INT4 quantization, it becomes possible to run local LLMs smoothly even on a MacBook Air with 8GB VRAM.

10. Performance Tuning: NUMA Architecture and Thread Pools

When performing inference using a CPU, multi-threading is essential. However, simply launching many std::thread instances is not optimal.

Modern multi-socket servers and high-end CPUs like Ryzen Threadripper employ NUMA (Non-Uniform Memory Access) architecture. Access to memory physically close to a certain CPU core (local memory) is fast, but access to memory tied to another processor becomes extremely slow.

Advanced C++ inference engines utilize the following techniques.

Thread Pinning: Fix each thread to a specific CPU core (set Affinity) to prevent cache invalidation due to context switches.
NUMA-aware Allocation: Allocate memory on the same NUMA node as the thread processing the data.
Work-stealing Thread Pool: Implement an efficient scheduler that divides each node of the computation graph into fine-grained tasks, and idle threads automatically steal and execute tasks.

By fully utilizing these, you can keep CPU utilization glued near 100% and strike throughput close to theoretical limits.

11. Conclusion: The Joy of Driving AI with C++ “Muscle”

Python is certainly convenient. In research and development or prototyping, no language can match its productivity. However, the moment you transition to the phase of “running the completed model efficiently on any device in the real world”, it is time for C++ to shine.

The sense of accomplishment you get when you see an inference engine—built by directly manipulating byte arrays in memory, pushing registers to their limits with SIMD instructions, and wrestling with GPU VRAM bandwidth—generating natural Japanese text (tokens) one after another on the console, is a “pure joy as an engineer” that you can never obtain simply by calling model.generate() in a Python framework.

AI technology tends to be a “Black Box”, but by writing everything by hand in C++, from tensor operations to memory allocation, you can deeply understand the true mechanisms of how LLMs “think”.

If you have a basic knowledge of C++ and a strong interest in current AI technology, please try taking on the challenge of developing your own inference engine. The source codes of GGML and llama.cpp should serve as the best living textbooks.

Now, throw away the heavy runtime of Python, and let’s run cutting-edge AI with the muscle of C++!

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

Fri, 11 Sep 2026 14:00:00 +0900

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

In recent years, interest in running Large Language Models (LLMs) in local environments has grown rapidly. In particular, small-scale models like TinyLLaMA (1.1B parameters) can perform inference at practical speeds even on limited-resource edge devices and typical laptops (including Windows environments). While development using Python and PyTorch is mainstream, when pursuing ultimate performance and memory efficiency, the combination of C++ and the C-based tensor library “ggml” has become the de facto standard.

This article provides an extremely detailed development guide for building an inference engine from scratch (or deeply understanding the internal structure of the existing llama.cpp) to load TinyLLaMA and generate text using C++.

1. Why C++ and ggml?

In the AI training phase, Python has an overwhelming advantage due to its flexibility and rich ecosystem. However, in the deployment or “Inference” phase, C++ becomes a powerful choice for the following reasons:

Overhead Reduction: The Python Global Interpreter Lock (GIL) and runtime overhead can be completely eliminated.
Memory Efficiency and Arena Allocation: By manually controlling memory allocation and deallocation, you can prevent unpredictable spikes caused by garbage collection.
Direct Hardware Access: By directly calling SIMD intrinsics such as AVX-512, AVX2, and ARM NEON, the CPU’s computational power can be maximized.
Zero Dependencies: ggml is a zero-dependency C/C++ library. As long as you have a compiler, it can be easily built even in an MSVC environment on Windows.

2. Overall Architecture

The flow of the entire inference pipeline is shown in the Mermaid diagram below. This is a series of processes starting from the user’s input text until the final next token is generated.

graph TD A["User Input Text"] --> B["BPE Tokenizer"] B --> C["Token IDs Array"] C --> D["Embedding Layer Lookup"] D --> E["Transformer Blocks"] E --> F["RMSNorm"] F --> G["LM Head Layer"] G --> H["Logits Array"] H --> I["Sampler Module"] I --> J["Next Token ID"] J --> K["Detokenizer"] K --> L["Output Text Chunk"] J -.-> |"Append to Context"| C

Since it is an autoregressive model, the output token is added back to the context and circulates as input for predicting the next token (the dotted line in the diagram).

3. Model Format and Memory Mapping (mmap)

The biggest hurdle in handling the weights of massive neural networks is disk I/O and memory consumption. In a C++ implementation, this is resolved with memory mapping (mmap).

3.1 How Memory Mapping Works and its Windows Implementation

Using mmap allows you to map the contents of a file directly into the virtual memory space of the process.

Zero-copy: Data is loaded directly from the disk into the kernel’s page cache, preventing extra copies into user space.
On-Demand Loading (Page Fault): The moment the CPU actually accesses that memory address, a page fault occurs, and only the required chunk (usually 4KB) is loaded into physical memory.

In a Windows environment, the Win32 APIs CreateFileMapping and MapViewOfFile are used instead of POSIX mmap.

sequenceDiagram participant OS["Windows OS"] participant RAM["Physical Memory"] participant App["C++ Application"] App->>OS: "CreateFileMapping / MapViewOfFile" OS-->>App: "Virtual Memory Address Pointer" App->>App: "Read Tensor Data at Pointer" OS->>RAM: "Page Fault / Load page from Disk" RAM-->>App: "Data ready for SIMD Compute"

3.2 Binary Structure of the GGUF Format

Converted from formats like Hugging Face’s .safetensors, the GGUF (GPT-Generated Unified Format) is the ultimate format for inference. It has the following strict binary layout:

Magic Bytes: 0x46554747 (GGUF).
Version: Format version number.
Tensor Count & Metadata Count: The number of tensors and key-value metadata pairs.
Metadata (Key-Value Pairs): Typed values and keys with string length prefixes.
Tensor Info: The name, number of dimensions, data type (FP16, Q4_K, etc.), and offset position in the file for each tensor.
Padding: Padding inserted so that tensor data is aligned to specific boundaries (usually 32 bytes or 64 bytes). This is essential for fast memory access with SIMD instructions (especially AVX).
Tensor Data: The actual aligned weight data array.

4. Mathematical Foundations of TinyLLaMA and C++ Algorithms

TinyLLaMA incorporates several advanced architectural ingenuities for efficiency. We explain the mathematical expressions to correctly implement these in C++.

4.1 RMSNorm (Root Mean Square Normalization)

It reduces computational cost by omitting mean centering from LayerNorm and only performing variance scaling.

$$ \text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \gamma $$

$d$ is the number of dimensions, and $\gamma$ is the learned scaling tensor. When implementing in C++, it is optimized by first rapidly calculating the sum of squares of the array using AVX2’s _mm256_fmadd_ps or similar, and then multiplying by the inverse square root (e.g., the _mm256_rsqrt_ps instruction).

4.2 RoPE (Rotary Position Embedding)

A technique that applies token position information as a rotation in tensor space. It can be seen as a rotation on a complex plane, applying the following rotation to adjacent dimension pairs $(x_1, x_2)$ of vector $x$:

$$ \text{RoPE}(x, m) = \begin{pmatrix} x_{1} \cos(m\theta) - x_{2} \sin(m\theta) \\ x_{1} \sin(m\theta) + x_{2} \cos(m\theta) \end{pmatrix} $$

Here, $m$ is the absolute position index of the token, and $\theta$ is a pre-calculated base frequency. In ggml, it is executed in parallel simply by adding a ggml_rope operator during inference graph construction.

4.3 Grouped-Query Attention (GQA)

In standard Multi-Head Attention (MHA), Query, Key, and Value each have the same number of heads. However, TinyLLaMA adopts Grouped-Query Attention (GQA) to drastically reduce memory bandwidth and KV cache consumption.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$

In GQA, multiple Query heads share a single Key/Value head. In the C++ implementation, before executing the matrix multiplication ggml_mul_mat, it is necessary to broadcast the KV tensors to match the number of Queries.

4.4 SwiGLU Activation Function

In the Feed-Forward Network (FFN) layer, SwiGLU is used instead of GELU.

$$ \text{SwiGLU}(x) = \text{Swish}(x W_{\text{gate}}) \otimes (x W_{\text{up}}) $$ $$ \text{Swish}(z) = z \cdot \sigma(z) = z \cdot \frac{1}{1 + e^{-z}} $$

In the computation graph, it is represented by combining the ggml_silu operator and ggml_mul.

5. Computation Graph Construction and Memory Management with ggml

ggml uses a “Define-and-Run” approach, constructing a static computation graph for inference and evaluating it later.

5.1 ggml_context and Arena Allocator

The most unique aspect of ggml is “arena allocation,” which avoids dynamic memory allocation (malloc or new) entirely within the inference loop. Upon initialization, a huge contiguous memory region (arena) is allocated, and the pointer to this region is incremented every time ggml_new_tensor or similar is called. Once one inference step is completed, simply resetting the allocation pointer to its initial position immediately finishes memory allocation for the next inference step.

5.2 Specific Example of Graph Construction

For each inference step, a computation graph like the following is assembled in memory:

graph TD A["Tokens Input ID"] --> B["Embed Lookup"] B --> C["ggml_rms_norm"] C --> D["Q / K / V Projections"] D --> E["ggml_rope Positional"] E --> F["KV Cache Store"] E --> G["KV Cache Load"] G --> H["Self Attention"] H --> I["Scale & Softmax"] I --> J["Attention Output"] J --> K["Out Projection"] K --> L["Add Residual"]

6. Quantization and Windows / SIMD Optimization

Handling TinyLLaMA (1.1B) in FP16 requires approximately 2.2GB of memory, but it can be dramatically compressed to around 600MB through 4-bit quantization (such as Q4_K).

6.1 Block Quantization Architecture

ggml does not quantize the entire tensor uniformly; instead, it does so in “block” units. In the Q4_0 format, 32 FP16 values are grouped into a single block.

Scale Factor: One FP16 value (2 bytes)
Quantized Data: 32 4-bit values (16 bytes) This minimizes the impact of local outliers.

6.2 Dot Product Acceleration with AVX2

When building for the latest x86 CPUs in a Windows environment, utilizing compiler flags like /arch:AVX2 allows SIMD processing to be performed in the following flow:

Load: Load 4-bit quantized data from memory into 256-bit AVX registers.
Expansion and Unpacking: Expand 4-bit values into Int8 or Int16 using bit masks and shift operations.
Dequantization: Multiply by the scale factor to convert to floating-point numbers.
FMA Operations: Execute multiply-add operations in parallel using activation values and _mm256_fmadd_ps (Fused Multiply-Add).

7. KV Cache Implementation Details

In autoregressive generation, the “KV cache” is an essential feature to skip the computation of Keys and Values for past tokens.

The key points for a C++ implementation are as follows:

Tensor Pre-allocation: Initialize a massive tensor for the KV cache (FP16 recommended) corresponding to the maximum context length (e.g., 2048 tokens).
Offset Copying: When calculations for token position $N$ are performed, the K and V vectors obtained in that step are stored into the $N$-th row of the KV cache tensor using ggml_cpy or similar.
Creating a View during Attention: When calculating attention, create a “view” that points only to the token portion from 0 to the $N$-th token and pass it to the matrix multiplication.

8. BPE Tokenizer and Decoding

The input string is treated as a UTF-8 byte sequence and matched against a predefined vocabulary. In C++, to speed up the vocabulary search, algorithms using a Trie (prefix tree) or a priority queue are implemented.

From the logits output by the LM Head, probabilities are scaled using the Temperature parameter, candidates are narrowed down using Top-K extraction or Top-P (Nucleus Sampling) methods, and the final next token is determined using random numbers.

9. Setting Up the C++ Project (Windows / PowerShell Environment)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cmake_minimum_required(VERSION 3.14)
project(TinyLLaMACpp)

set(CMAKE_CXX_STANDARD 17)

# Windows (MSVC) optimization and AVX2 flag settings
if(MSVC)
 add_compile_options(/O2 /arch:AVX2 /fp:fast)
 add_link_options(/STACK:8388608)
else()
 add_compile_options(-O3 -march=native -ffast-math)
endif()

add_library(ggml OBJECT ggml/ggml.c ggml/ggml-alloc.c)
target_compile_definitions(ggml PRIVATE GGML_USE_AVX2 GGML_USE_F16C GGML_USE_FMA)

add_executable(main main.cpp)
target_link_libraries(main ggml)

Example build commands in PowerShell:

1
2
3
4


mkdir build
cd build
cmake .. -G "Visual Studio 17 2022" -A x64
cmake --build . --config Release

10. Conclusion

Implementing an inference engine from scratch for small AI models like TinyLLaMA using C++ and ggml is a perfect opportunity to demystify the black box of deep learning and learn the beauty of low-level hardware control. Let’s pave the way for the future of edge AI while fully savoring the essence of systems programming, such as zero-copy loading using memory mapping, SIMD optimization, and KV cache construction.

A Complete Guide to llama.cpp and Customization with C++

Fri, 11 Sep 2026 11:00:00 +0900

In recent years, the evolution of Large Language Models (LLMs) has been tremendous, and their scope of application is expanding daily. However, running models with billions or tens of billions of parameters locally typically requires a high-end GPU with an enormous amount of VRAM. Breaking through this “hardware barrier” and making practical LLM inference possible on everyday PCs, Macs, and even devices like the Raspberry Pi is llama.cpp.

This article goes beyond just explaining how to use the command-line tool. It provides an extremely detailed explanation for engineers, covering the architecture of its underlying technology ggml, the mathematical background of Transformers and quantization, and how to use the C++ API to integrate and customize LLMs within your own applications.

1. Overview of llama.cpp and ggml

llama.cpp is a lightweight LLM inference engine written in C/C++, developed by Georgi Gerganov. Originally created with the goal of running Meta’s LLaMA model quickly on Apple Silicon (M1/M2 Macs), it now supports a variety of architectures and models.

Its biggest feature is that it is a pure C/C++ implementation with no external dependencies. Because it doesn’t require a massive ecosystem like Python or PyTorch and can be compiled as a single executable, deployment is incredibly easy.

The heart of llama.cpp is the tensor math library ggml. ggml was designed from the ground up to maximally optimize matrix operations in machine learning on CPUs (and some GPUs).

1.1 Why is llama.cpp so fast?

Leveraging Memory Mapping (mmap): When loading model weights into memory, it uses the OS’s mmap to avoid loading everything into RAM, enabling fast startups and saving memory.
Thorough Optimization of SIMD Instructions: It utilizes CPU-specific instruction sets like AVX2, AVX-512, ARM NEON, and Apple AMX to perform ultra-fast matrix multiplications.
Quantization: It compresses 16-bit floating-point (FP16) weights into 4-bit, 5-bit, or 8-bit integers, resolving the memory bandwidth bottleneck (more on this later).

2. Mathematical Background: Transformers and Quantization

To deeply understand llama.cpp, you need to know the mathematical formulas it calculates and how it approximates these calculations.

2.1 The Inference Process of a Transformer

Models like LLaMA adopt an auto-regressive Transformer decoder architecture. The core of text generation is the Self-Attention mechanism.

For an input hidden state matrix $X \in \mathbb{R}^{N \times d}$, the Query $Q$, Key $K$, and Value $V$ are calculated by multiplying with weight matrices:

$$ Q = X W_Q, \quad K = X W_K, \quad V = X W_V $$

Here, the output of Attention is defined as follows:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

In the inference loop of llama.cpp, the bottleneck is the multiplication of these massive matrices $W_Q, W_K, W_V$ and the feed-forward network (FFN) weight matrices with the vector $X$ ($N=1$ in the generation phase because it processes one token at a time), which is GEMV (General Matrix-Vector Multiplication).

2.2 The Mathematical Foundation of Quantization

In inference where memory access bandwidth becomes a bottleneck, quantization—representing weight parameters with a small number of bits—is essential. Here, we’ll explain the basic principles of the block-wise quantization widely used in llama.cpp (e.g., Q4_K or Q4_0).

For example, consider a block $w = [w_1, w_2, \dots, w_B]$ of length $B$ (usually 32 or 64) that is part of an FP16 weight matrix $W$. This block is approximated using 4-bit integers $q_i \in [-8, 7]$ and a single scaling factor $\Delta$ (FP16 or FP32):

$$ w_i \approx \Delta \times q_i $$

$\Delta$ is determined based on the maximum absolute value within the block:

$$ \Delta = \frac{\max_i |w_i|}{7} $$

When calculating the dot product $y = w \cdot x$ using the quantized weights, if the input vector $x$ is similarly quantized to $x_i \approx \Delta_x \times q_{x, i}$, we get:

$$ y = \sum_{i=1}^{B} w_i x_i \approx \Delta \Delta_x \sum_{i=1}^{B} q_i q_{x, i} $$

The $\sum q_i q_{x, i}$ part becomes a pure integer operation, which can be computed in parallel very quickly using SIMD instructions. This is the mathematical trick behind the astonishing speed of llama.cpp on CPUs.

3. Architecture and Inference Flow

To understand the internal workings of llama.cpp, the following Mermaid diagram shows the overall system architecture and data flow.

graph TD A["User Input (String)"] --> B["llama.cpp Tokenizer"] B --> C["Token IDs (int32 array)"] C --> D["Context Buffer (KV Cache)"] D --> E["ggml Compute Graph"] E --> F["Transformer Layers"] subgraph "ggml Engine" F --> G["Self-Attention (RoPE)"] G --> H["Feed Forward Network"] H --> F end F --> I["Logits (Vocabulary Size)"] I --> J["Sampler (Temperature, Top-K, Top-P)"] J --> K["Selected Token ID"] K --> L["llama.cpp Detokenizer"] L --> M["Output String"] K -. "Auto-regressive loop" .-> D

Text generation is an auto-regressive loop where, each time a single token is output, it is added to the KV Cache as the next input and passes through the compute graph again.

4. Environment Setup and Build Guide

Before embedding llama.cpp into a C++ project, let’s first build the source code.

4.1 Cloning the Repository

1
2


git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

4.2 Building with CMake

When embedding it into other applications as a C++ project, using CMake is the most standard approach. By enabling accelerators (backends) for your specific platform, you can speed up computations.

CPU Only (Basic Build):

1
2
3


mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

Using NVIDIA GPU (CUDA):

1
2
3


mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j 8

Using Apple Silicon (Metal):

1
2
3


mkdir build && cd build
cmake .. -DGGML_METAL=ON
cmake --build . --config Release -j 8

Upon a successful build, executable files like llama-cli and the llama library (along with the ggml library) for linking via the C++ API discussed later will be generated in the build/bin/ directory.

5. Introduction to C++ Customization: Using the llama.cpp API

From here on, we will discuss the main topic: controlling llama.cpp from C++ code. To embed an LLM into your own application (e.g., a game engine, desktop app, or embedded system) rather than just using command-line tools, you need to hit the C++ API directly.

llama.cpp primarily provides a C language interface through a header file called llama.h. We use this interface even when calling from C++.

5.1 Minimal Necessary Includes and Setup

When using llama.cpp in your project, include the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


#include "llama.h"
#include <iostream>
#include <vector>
#include <string>
#include <stdexcept>

// Macro for error handling
#define LLAMA_ASSERT(x) \
 do { \
 if (!(x)) { \
 std::cerr << "Assertion failed: " << #x << std::endl; \
 std::terminate(); \
 } \
 } while (0)

5.2 Loading the Model and Initializing the Context

First, load a .gguf format model file and allocate the context (memory space and KV cache) for inference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


int main(int argc, char ** argv) {
 if (argc < 2) {
 std::cerr << "Usage: " << argv[0] << " <model.gguf>" << std::endl;
 return 1;
 }
 std::string model_path = argv[1];

 // 1. Initialize the backend (set up environment like CPU/GPU)
 llama_backend_init();

 // 2. Get default settings for model parameters
 llama_model_params model_params = llama_model_default_params();
 model_params.n_gpu_layers = 35; // Number of layers to offload to GPU

 // 3. Load the model
 llama_model * model = llama_load_model_from_file(model_path.c_str(), model_params);
 if (model == nullptr) {
 std::cerr << "Failed to load model" << std::endl;
 return 1;
 }

 // 4. Set context parameters
 llama_context_params ctx_params = llama_context_default_params();
 ctx_params.n_ctx = 2048; // Maximum context size (number of tokens)
 ctx_params.n_threads = 8; // Number of CPU threads used for inference

 // 5. Create the context
 llama_context * ctx = llama_new_context_with_model(model, ctx_params);
 if (ctx == nullptr) {
 std::cerr << "Failed to create context" << std::endl;
 llama_free_model(model);
 return 1;
 }

 std::cout << "Model and context loaded successfully!" << std::endl;
 // ... subsequent processing

5.3 Tokenization of the Prompt

An LLM does not understand text directly; it processes strings as sequences of integer IDs (tokens). Therefore, the input string must be converted into tokens.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 std::string prompt = "Q: What is the capital of Japan?\nA:";
 std::vector<llama_token> tokens_list;
 tokens_list.resize(prompt.length() + 4); // Buffer size with some headroom

 // Whether to prepend a special token (like BOS: Begin of Sequence)
 bool add_special = true;
 // Convert string to an array of token IDs
 int n_tokens = llama_tokenize(
 model,
 prompt.c_str(),
 prompt.length(),
 tokens_list.data(),
 tokens_list.size(),
 add_special,
 false // parse_special
 );

 if (n_tokens < 0) {
 // Handling for when the buffer is insufficient, like reallocating and retrying, is necessary (omitted for brevity)
 std::cerr << "Failed to tokenize prompt" << std::endl;
 return 1;
 }
 tokens_list.resize(n_tokens);

5.4 Inference Loop and Sampling

Build a loop that feeds tokens into the model, obtains the probability distribution (Logits) of the next token, and samples from it to determine the next token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72


 // Maximum number of tokens to generate
 const int max_gen_tokens = 100;

 // Initialize structure for batch evaluation
 llama_batch batch = llama_batch_init(512, 0, 1);

 // Add prompt tokens to the batch
 for (size_t i = 0; i < tokens_list.size(); i++) {
 llama_batch_add(batch, tokens_list[i], i, { 0 }, false);
 }
 // Set to output logits (predictions) only for the very last token of the prompt
 batch.logits[batch.n_tokens - 1] = true;

 // Initial evaluation (feeding the prompt to the model)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "llama_decode() failed" << std::endl;
 return 1;
 }

 int n_cur = batch.n_tokens; // Current context length
 int n_decode = 0;

 std::cout << "\nOutput: ";

 // Initialize sampler context (settings for Temperature, Top-K, Top-P, etc.)
 llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
 llama_sampler_chain_add_top_k(smpl, 40);
 llama_sampler_chain_add_top_p(smpl, 0.9f, 1);
 llama_sampler_chain_add_temp(smpl, 0.7f);
 llama_sampler_chain_add_dist(smpl, 1234); // Seed value

 while (n_decode < max_gen_tokens) {
 // 1. Sampling: Predict the next token based on current context
 llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);

 // 2. If token is EOS (End of Sequence), break the loop
 if (llama_token_is_eog(model, new_token_id)) {
 break;
 }

 // 3. Decode token to string (text) and print
 char buf[128];
 int n_chars = llama_token_to_piece(model, new_token_id, buf, sizeof(buf), 0, false);
 if (n_chars > 0) {
 std::cout << std::string(buf, n_chars) << std::flush;
 }

 // 4. Prepare the newly generated token as the next batch
 llama_batch_clear(batch);
 llama_batch_add(batch, new_token_id, n_cur, { 0 }, true);

 // 5. Evaluate the model (update KV cache and predict next)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "Failed to evaluate" << std::endl;
 break;
 }

 n_cur += 1;
 n_decode += 1;
 }

 std::cout << std::endl;

 // Cleanup
 llama_sampler_free(smpl);
 llama_batch_free(batch);
 llama_free(ctx);
 llama_free_model(model);
 llama_backend_free();

 return 0;
}

This code implements a custom inference loop using the basic API of llama.cpp. It uses the llama_batch struct to manage token groups and executes the forward pass of the neural network using llama_decode.

6. Advanced Customization Examples: Logit Manipulation and Penalty Control in C++

When you want to go beyond simple text generation—such as forcing output in a specific format (e.g., JSON only) or suppressing specific forbidden words—you directly manipulate the Logits prior to sampling from the C++ side.

You can retrieve the array of raw scores (values before being converted to probabilities) right before the model outputs each token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Retrieve the raw logits array immediately after inference, prior to sampling
float * logits = llama_get_logits_ith(ctx, batch.n_tokens - 1);
int n_vocab = llama_n_vocab(model);

// List of forbidden token IDs (for example: 1234, 5678)
std::vector<llama_token> forbidden_tokens = { 1234, 5678 };

// Set the occurrence probability of forbidden tokens to 0 (make the Logit negative infinity)
for (llama_token bad_tok : forbidden_tokens) {
 logits[bad_tok] = -INFINITY;
}

In this way, directly interacting with the C++ API allows for “micro-millisecond interventions per inference cycle” that would be difficult or incur high overhead if done via LangChain or Python.

7. The Secrets of Performance Tuning

After finishing your C++ implementation, here are a few checkpoints for maximizing speed for actual deployment.

Optimizing Batch Processing: When handling requests from multiple users concurrently, include multiple sequences in llama_batch and call llama_decode at once (Continuous Batching). This drastically improves throughput by coalescing memory access.
Enabling Flash Attention: By setting ctx_params.flash_attn = true; in the context parameters, you can speed up Attention calculations while reducing memory usage. This setting is essential when dealing with long contexts (tens of thousands of tokens).
NUMA Support: In multi-socket server environments, properly configuring NUMA before llama_backend_init() can reduce memory access latency.

8. Conclusion

In this article, we covered everything in detail, starting from the mathematical background of llama.cpp to an explanation of its architecture, and finally, how to build a custom inference engine fully utilizing the C++ API.

While the Python ecosystem is highly convenient for prototyping, the direct control offered by C/C++ based llama.cpp demonstrates overwhelming power in production environments that demand edge device deployment, game integration, and real-time processing.

By all means, try writing C++ code yourself and experience the joy of freely manipulating LLMs in a local environment.

Reference Links

llama.cpp Official Repository

ggml - Tensor Library

Attention Is All You Need (Vaswani et al., 2017)