AI on kenji.blog

Latest API Usage Examples and Sample Code for Microsoft.Windows.AI

Fri, 11 Sep 2026 18:00:00 +0900

Latest API Usage Examples and Sample Code for Microsoft.Windows.AI: Exploring the Depths of Windows Copilot Runtime

1. Introduction: A New Era of Windows with Natively Integrated AI

In recent years, AI technology has evolved remarkably, resulting in a rapid paradigm shift from the utilization of Large Language Models (LLMs) on the cloud to AI inference on edge devices (local PCs). At the core of this are the “Windows Copilot Runtime” provided by Microsoft for Windows 11, and the “Microsoft.Windows.AI” APIs designed to operate it.

Application development using cloud APIs (such as OpenAI and Azure OpenAI) is easy, but it comes with challenges such as latency, privacy, and continuous costs. On the other hand, running AI models locally allows for the realization of ultra-low latency applications that work even offline without letting sensitive data leave the device.

In this article, we will thoroughly explain in extreme detail the implementation methods for local AI features, which will be essential for future Windows application development, along with practical sample code in C# and C++. We will cover everything from architecture to performance tuning. We won’t just hit the APIs, but will delve deep into the advanced technical details such as utilizing the underlying hardware (NPU and GPU) and integrating with DirectML.

2. Overall Architecture of Windows Copilot Runtime

Windows Copilot Runtime is a set of AI stacks designed so that developers can easily integrate AI models on Windows and extract maximum performance. This runtime abstracts hardware acceleration at the OS level and provides developers with a unified interface.

graph TD App["Windows Application (C# / C++)"] --> API["Microsoft.Windows.AI APIs"] App --> ORT["ONNX Runtime"] API --> WCR["Windows Copilot Runtime (OS Layer)"] WCR --> SLM["Local Models (Phi-Silica, etc.)"] ORT --> DML["DirectML Execution Provider"] SLM --> DML DML --> DXCore["DXCore / DirectX 12"] DXCore --> NPU["NPU (Neural Processing Unit)"] DXCore --> GPU["GPU (Graphics Processing Unit)"] DXCore --> CPU["CPU"]

As the architecture diagram above shows, applications can directly access small language models built into the OS (SLMs: such as Phi-Silica) by utilizing the high-level Microsoft.Windows.AI APIs. Additionally, when using custom models, it is also possible to explicitly use hardware acceleration via ONNX Runtime and DirectML. Since the OS layer optimizes the distribution of workloads to the CPU, GPU, and NPU, developers can build high-performance AI apps without being deeply conscious of hardware differences.

3. Hardware Acceleration and Mathematical Evaluation of NPUs

The latest Copilot+ PCs are equipped with NPUs (Neural Processing Units), processors specialized for AI processing. The performance of an NPU is generally evaluated in TOPS (Tera Operations Per Second).

In AI model inference, the computational capability of GEMM (General Matrix Multiply) in particular determines the throughput. The theoretical peak performance $P_{\text{peak}}$ of hardware is estimated by the following formula:

$$ P_{\text{peak}} = f \times N_{\text{cores}} \times N_{\text{MACs/core}} \times 2 $$

Here,

$f$ is the NPU clock frequency (Hz)
$N_{\text{cores}}$ is the number of cores in the NPU
$N_{\text{MACs/core}}$ is the number of MAC (Multiply-Accumulate) units per core
The final $2$ is because one MAC operation counts as two operations (FLOPs/OPs): multiplication and addition.

For example, for an NPU with a frequency of 1.5GHz, 4 cores, and 4096 MACs per core:

$$ P_{\text{peak}} = 1.5 \times 10^9 \times 4 \times 4096 \times 2 \approx 49.15 \text{ TOPS} $$

This mathematically demonstrates a performance that clears the 40 TOPS requirement for Windows 11 Copilot+ PCs.

Furthermore, AI model inference (the decode phase), especially for LLMs, tends to be Memory-Bound. The theoretical bandwidth $BW$ of system memory is calculated as follows:

$$ BW = f_{\text{mem}} \times W_{\text{bus}} \times \frac{2}{8} $$

In the case of LPDDR5x-8533 memory ($f_{\text{mem}} = 8533 \text{ MT/s}$) and a 128-bit bus ($W_{\text{bus}} = 128$), the bandwidth is approximately $136 \text{ GB/s}$. In AI application optimization, how to conserve this bandwidth is crucial, and the model Quantization discussed later becomes essential.

4. Development Environment Setup

To use the latest Windows AI APIs, you need to prepare the following environment and toolchain:

OS: Windows 11 version 24H2 or later (strongly recommended to have an NPU-equipped device meeting the Copilot+ PC requirements)
SDK: Windows App SDK (v1.5 or later, AI extension supported version)
Development Environment: Visual Studio 2022 (v17.10 or later), with the “Desktop development with C++” and “.NET desktop development” workloads
Packages: Install Microsoft.Windows.AI and Microsoft.ML.OnnxRuntime.DirectML via NuGet

1
2
3
4
5
6


<!-- Example .csproj configuration -->
<ItemGroup>
 <PackageReference Include="Microsoft.Windows.AI" Version="1.0.0-preview1" />
 <PackageReference Include="Microsoft.WindowsAppSDK" Version="1.5.240311000" />
 <PackageReference Include="Microsoft.ML.OnnxRuntime.DirectML" Version="1.17.1" />
</ItemGroup>

5. [Deep Dive 1] Utilizing Local Language Models (Phi-Silica) with C#

Windows Copilot Runtime natively incorporates the highly efficient small language model “Phi-Silica” developed by Microsoft as a standard OS component. This enables advanced natural language processing (text summarization, code generation, chatbots) in offline environments without downloading gigabyte-sized models from the network.

Below is an advanced C# sample code to build an AI chatbot using the Microsoft.Windows.AI.Generative namespace. It supports streaming responses and generates text in real-time without blocking the UI thread.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84


using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Windows.AI.Generative;

namespace WindowsAI.Sample
{
 public class LocalLanguageModelService
 {
 private LanguageModel _languageModel;
 private bool _isInitialized = false;

 /// <summary>
 /// Initializes the language model. Checks NPU availability and loads the model onto the optimal device.
 /// </summary>
 public async Task InitializeAsync()
 {
 if (_isInitialized) return;

 Console.WriteLine("Checking system requirements and availability for the local AI model (Phi-Silica)...");

 // Check if the model is available on the system (if unsupported, a download might be prompted)
 var availability = await LanguageModel.CheckAvailabilityAsync();
 if (availability != LanguageModelAvailability.Available)
 {
 throw new InvalidOperationException($"Local AI model is currently unavailable. Status: {availability}");
 }

 // Create a model instance (mapping to memory space and NPU initialization happen at this time)
 _languageModel = await LanguageModel.CreateAsync();
 _isInitialized = true;

 Console.WriteLine("Language model initialization completed. Hardware acceleration via DirectML is active.");
 }

 /// <summary>
 /// Receives user prompt and generates response stream.
 /// </summary>
 public async Task GenerateResponseStreamAsync(string prompt, CancellationToken cancellationToken)
 {
 if (!_isInitialized) await InitializeAsync();

 Console.WriteLine($"\n[User Input]: {prompt}\n[AI Assistant]: ");

 // Set generation hyperparameters
 var options = new LanguageModelOptions
 {
 Temperature = 0.7f,
 TopP = 0.9f,
 MaxTokens = 2048,
 RepetitionPenalty = 1.1f
 };

 // Construct conversation context
 var context = new LanguageModelContext();
 context.AddSystemMessage("You are an advanced AI assistant running directly on a Windows local NPU. Think logically step-by-step and answer concisely.");
 context.AddUserMessage(prompt);

 try
 {
 // Call the streaming inference API
 var responseStream = _languageModel.GenerateResponseStreamAsync(context, options);

 // Asynchronously iterate over the chunks returned as IAsyncEnumerable
 await foreach (var chunk in responseStream.WithCancellation(cancellationToken))
 {
 // Output generated tokens (chunks) to the console in real-time
 // For UI apps, reflect this to a TextBox, etc. using DispatcherQueue here
 Console.Write(chunk.Text);
 }
 Console.WriteLine();
 }
 catch (OperationCanceledException)
 {
 Console.WriteLine("\n[Generation was cancelled by the user or system]");
 }
 catch (Exception ex)
 {
 Console.WriteLine($"\n[A fatal error occurred: {ex.Message}]");
 }
 }
 }
}

5.1 Explanation of the C# Architecture

The core of this code lies in the pre-execution validation using LanguageModel.CheckAvailabilityAsync() and the asynchronous streaming via GenerateResponseStreamAsync. When the Copilot Runtime running in the OS background receives this API call, it internally launches the ONNX Runtime and selects the optimal Execution Provider (DirectML + NPU on many modern PCs) according to the system configuration.

Developers can integrate cutting-edge AI inference pipelines into their applications with just a few lines of C# code, without needing to be aware of the model’s tensor shapes, tokenizer implementations, or memory management of KV caches at all.

6. [Deep Dive 2] High-Speed Custom Model Inference using C++ and DirectML

When dealing with specific domains (such as unique image segmentation, speech recognition, custom object detection models, etc.) that cannot be covered by the OS standard language models alone, developers need to directly operate ONNX Runtime and DirectML, which are located at a lower layer of Microsoft.Windows.AI.

By using C++, memory allocation can be optimized to the limit, bringing out the peak performance of the NPU/GPU. Below is a core implementation of an advanced initialization and inference pipeline for executing ONNX format custom models (e.g., YOLOv8) via DirectML in C++.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99


#include <iostream>
#include <vector>
#include <string>
#include <stdexcept>
#include <onnxruntime_cxx_api.h>
#include <dml_provider_factory.h>

class CustomVisionAIProcessor {
private:
 Ort::Env env;
 Ort::Session session{nullptr};
 Ort::AllocatorWithDefaultOptions allocator;

public:
 CustomVisionAIProcessor(const std::wstring& modelPath)
 : env(ORT_LOGGING_LEVEL_WARNING, "VisionAIProcessor") {

 Ort::SessionOptions sessionOptions;
 // Optimize the number of threads
 sessionOptions.SetIntraOpNumThreads(1);
 // Set graph optimization level to maximum
 sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

 // 1. Add DirectML Execution Provider (DML EP)
 // device_id = 0 is the system's default recommended adapter (NPU or high-performance GPU)
 const OrtApi& ortApi = Ort::GetApi();
 OrtDmlApi* dmlApi = nullptr;
 OrtStatus* status = ortApi.GetExecutionProviderApi("DML", ORT_API_VERSION, reinterpret_cast<void**>(&dmlApi));

 if (status == nullptr && dmlApi != nullptr) {
 Ort::ThrowOnError(dmlApi->SessionOptionsAppendExecutionProvider_DML(sessionOptions, 0));
 std::cout << "[Info] Successfully attached DirectML Execution Provider." << std::endl;
 } else {
 std::cerr << "[Warning] Failed to retrieve DirectML API. Running in CPU fallback mode." << std::endl;
 if (status != nullptr) ortApi.ReleaseStatus(status);
 }

 // 2. Load model and create session
 try {
 session = Ort::Session(env, modelPath.c_str(), sessionOptions);
 std::cout << "[Info] ONNX model successfully loaded and computation graph compiled." << std::endl;
 } catch (const Ort::Exception& e) {
 std::cerr << "[Error] Failed to load model: " << e.what() << std::endl;
 throw;
 }
 }

 void RunInference(const std::vector<float>& imageTensor, const std::vector<int64_t>& inputShape) {
 // 3. Create input tensor buffer
 Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
 Ort::Value inputTensor = Ort::Value::CreateTensor<float>(
 memoryInfo,
 const_cast<float*>(imageTensor.data()),
 imageTensor.size(),
 inputShape.data(),
 inputShape.size()
 );

 // 4. Dynamically retrieve input/output node names
 auto inputNamePtr = session.GetInputNameAllocated(0, allocator);
 auto outputNamePtr = session.GetOutputNameAllocated(0, allocator);
 const char* inputNames[] = { inputNamePtr.get() };
 const char* outputNames[] = { outputNamePtr.get() };

 // 5. Execute inference (offloaded to NPU/GPU via DirectML)
 std::cout << "[Info] Starting inference engine execution..." << std::endl;
 auto startTime = std::chrono::high_resolution_clock::now();

 auto outputTensors = session.Run(Ort::RunOptions{nullptr}, inputNames, &inputTensor, 1, outputNames, 1);

 auto endTime = std::chrono::high_resolution_clock::now();
 auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime);

 // 6. Retrieve and analyze result tensor
 float* outputData = outputTensors.front().GetTensorMutableData<float>();
 size_t outputSize = outputTensors.front().GetTensorTypeAndShapeInfo().GetElementCount();

 std::cout << "[Result] Inference completed: " << duration.count() << " ms" << std::endl;
 std::cout << "[Result] Output tensor element count: " << outputSize << std::endl;
 // *After this, implement NMS (Non-Maximum Suppression) or bounding box drawing processing for the output tensor
 }
};

int main() {
 try {
 // ONNX model path to execute
 CustomVisionAIProcessor processor(L"models/yolov8n_quantized.onnx");

 // Dummy image data for inference (Batch size 1 x 3 channels x 640 x 640)
 std::vector<int64_t> inputShape = {1, 3, 640, 640};
 std::vector<float> dummyImage(1 * 3 * 640 * 640, 0.5f);

 processor.RunInference(dummyImage, inputShape);
 } catch (const std::exception& e) {
 std::cerr << "Program terminated abnormally: " << e.what() << std::endl;
 return -1;
 }
 return 0;
}

6.1 The Importance of Memory Management and Zero-Copy Inference in C++

The greatest advantage of using DirectML in C++ is that tight integration with DirectX 12 (DX12) is possible. The above code includes standard data copying from CPU memory for educational purposes, but in actual game engines and video processing applications, there are many cases where images (textures) are already held in the GPU or NPU memory space using DX12.

In this case, by utilizing the advanced binding features of OrtDmlApi, you can achieve “Zero-Copy Inference”, which directly maps DX12 resources as ONNX Runtime tensors. By doing this, the data transfer overhead across the PCIe bus (consumption of the aforementioned bandwidth $BW$) completely vanishes, dramatically improving the frame rate in real-time video processing.

7. Performance Optimization and Best Practices

Below, we summarize essential optimization strategies for developing top-tier AI applications leveraging Windows AI APIs and DirectML.

7.1 Model Quantization and the Olive Toolkit

To unleash the true power of an NPU, it is an absolute requirement to Quantize the weights and activations of AI models from FP32 (single-precision floating-point) to INT8 or INT4. The NPU architecture is specialized for integer arithmetic, theoretically achieving 4x the throughput and massive power savings with INT8 compared to FP32.

By using the Olive (ONNX Live) toolchain provided by Microsoft, you can automatically optimize models like PyTorch for the Windows environment. Olive strongly supports specific attention optimizations for Transformer models and hardware-specific graph compilation.

7.2 The Trade-off: Batch Processing vs. Interactive Streaming

In API calls, by batching multiple inference requests together, you can increase NPU utilization (Compute Utilization). However, for interactive UIs like chatbots, the time until the first token is displayed (TTFT: Time To First Token) dictates the user experience (UX) more than throughput. Therefore, for interactive UIs, the best practice is to set the batch size to 1 and design to prioritize streaming generation.

7.3 Integration with Background Tasks and the OS

Local AI inference consumes significant power and system resources. Integrating with the Windows App Lifecycle API is required to suspend or throttle the resource consumption of low-priority inference tasks when the application moves to the background.

sequenceDiagram participant User as "User" participant App as "Windows App (C#)" participant API as "Microsoft.Windows.AI" participant OS as "Windows Copilot Runtime" participant ORT as "ONNX Runtime (DML)" participant NPU as "NPU Hardware" User->>App: "Input Prompt" App->>API: "GenerateResponseStreamAsync()" API->>OS: "Dispatch Inference Job" OS->>ORT: "Graph Execution Request" ORT->>NPU: "Execute Command List via DirectML" NPU-->>ORT: "Computation Completed (1 Token Generated)" ORT-->>OS: "Tensor Result" OS-->>API: "Decoded Text" API-->>App: "IAsyncEnumerable Chunks" App-->>User: "Real-time Text Rendering to UI" Note over ORT,NPU: "Rapidly repeat this loop until completion"

This sequence diagram illustrates the beauty of asynchronous processing where data streams continuously from the lowest layer NPU hardware to the presentation layer of the application, without ever blocking the UI thread.

8. Future Outlook and Evolution of Windows AI

The Microsoft.Windows.AI APIs and Copilot Runtime are undergoing rapid evolution. Future updates for developers are expected to bring paradigm shifts such as the following:

OS Native Integration of Multimodal APIs: Seamlessly process text, audio, images, and even live video feeds simultaneously, providing cross-modal AI inference natively at the OS level.
System-level Support for RAG (Retrieval-Augmented Generation): Connecting a personal collection of documents within a local PC or Windows Search indices with AI models within a secure OS sandbox, allowing the construction of ultra-advanced personal AI assistants while completely protecting user privacy.
Dynamic Resource Scaling of NPUs: A mechanism where the Windows kernel scheduler dynamically switches the NPU execution context to guarantee QoS (Quality of Service) when multiple AI applications (e.g., background noise cancellation and foreground code generation) are running concurrently.

9. Conclusion: The Future of Applications Transformed by Local AI

The Windows 11 Copilot Runtime and Microsoft.Windows.AI APIs have brought an extremely powerful weapon called “Local AI” to all Windows developers. You no longer need to rely entirely on cloud APIs. It is possible to eliminate latency, firmly protect privacy, and deliver next-generation AI experiences to users that fully operate even offline.

By utilizing the knowledge of system standard language model integration using C#, mathematical performance evaluation, and extreme hardware optimization using C++ and DirectML explained in this article, please create next-generation “AI-native” Windows applications with your own hands. The boundless possibilities brought by AI stretch right out ahead of the code you write.

※Note: This article is written based on the preview version APIs and the latest specifications as of September 2026. Because API specifications and hardware requirements may change due to Windows updates, always refer to the official Microsoft Learn documentation when implementing.

No Python! Building an AI Inference Engine Exclusively in C++

Fri, 11 Sep 2026 17:00:00 +0900

1. Introduction: Why let go of Python and build an AI inference engine in C++?

In modern AI development, Python is the de facto standard. Thanks to powerful frameworks like PyTorch and TensorFlow, complex neural networks can be built, trained, and inferred with just a few lines of code. However, behind these frameworks, low-level languages like C++ and CUDA handle the heavy computational processing. Python is merely acting as the “glue”.

So why bother eliminating Python and creating an AI inference engine exclusively in C++? There are several compelling reasons.

Extreme Performance and Low Latency: You can completely eliminate the overhead of Python’s GIL (Global Interpreter Lock) and dynamic typing. Especially in systems requiring real-time performance, delays on the millisecond scale can be fatal.
Ease of Deployment: Building a Python environment (massive library ecosystems, dependency hell) on the end user’s system is extremely difficult. With C++, you only need to distribute a single statically linked executable binary (.exe or ELF binary).
Edge Device Support: In highly resource-constrained environments like smartphones, embedded devices, and Raspberry Pi, there is no luxury to run a Python runtime that consumes gigabytes of memory.
Direct Hardware Control: Low-level control such as memory allocation timing, explicit use of SIMD instructions, and optimization of memory transfers with the GPU is possible with C++.

In this article, while drawing massive inspiration from the architecture of the “GGML” library developed by Georgi Gerganov, we will dive deep into the technical abyss and explain the process of building an inference engine from scratch using only C++ to run Large Language Models (LLMs).

2. Overview of the Inference Engine Architecture

AI inference processing is essentially a “series of massive matrix calculations”. To execute this efficiently, an inference engine needs to be composed of the following components.

graph TD A["Input Data (Tokens/Images)"] --> B["Tensor Management"] B --> C["Computation Graph (DAG)"] C --> D["Memory Arena & Allocator"] C --> E["Scheduler & Thread Pool"] E --> F["CPU Backend (AVX2/ARM NEON)"] E --> G["GPU Backend (CUDA/Metal)"] F --> H["Output Results"] G --> H

Tensor Management: Manages multidimensional array data structures and strides for each dimension.
Computation Graph: Represents the operations of each layer in the neural network as a Directed Acyclic Graph (DAG).
Memory Arena: A pre-allocated memory management mechanism to avoid the overhead of dynamic memory allocation (malloc or new).
Backend: Operations (kernels) optimized for specific hardware such as CPUs and GPUs.

We will assemble these using the powerful features of C++ (templates, pointer arithmetic, RAII, etc.).

3. The Secret of Memory Management: Memory Arena and SIMD Alignment

Memory management in an inference engine is one of the most critical factors directly linked to performance. During inference, especially as data passes through each layer of a Transformer model, a massive number of intermediate tensors are generated. If you allocate and free these with standard malloc every time, heap fragmentation and OS context switches will cause a fatal slowdown.

Therefore, we adopt an approach called the “Memory Arena”. This is a method where the maximum amount of memory required is calculated (or fixed) and allocated at once when inference begins, and memory is carved out simply by incrementing a pointer.

3.1 The Importance of Alignment

Modern CPUs support SIMD (Single Instruction, Multiple Data) instructions. Examples include AVX2/AVX-512 for Intel/AMD and NEON for ARM. These instructions process 256 bits (32 bytes) or 512 bits (64 bytes) of data at once, but the target data memory must be aligned to specific byte boundaries (usually 32 bytes or 64 bytes).

Below is an example of a C++ implementation of a memory arena that considers alignment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


#include <cstdint>
#include <cstddef>
#include <stdexcept>
#include <iostream>

struct MemoryArena {
 size_t size;
 size_t offset;
 uint8_t* data;

 MemoryArena(size_t size) : size(size), offset(0) {
 // Use posix_memalign for POSIX systems, _aligned_malloc for Windows
#ifdef _WIN32
 data = static_cast<uint8_t*>(_aligned_malloc(size, 64));
#else
 if (posix_memalign(reinterpret_cast<void**>(&data), 64, size) != 0) {
 throw std::bad_alloc();
 }
#endif
 }

 ~MemoryArena() {
#ifdef _WIN32
 _aligned_free(data);
#else
 free(data);
#endif
 }

 void* allocate(size_t bytes, size_t alignment = 64) {
 // Calculate alignment (find padding)
 size_t pad = (alignment - (offset % alignment)) % alignment;
 if (offset + pad + bytes > size) {
 throw std::runtime_error("OOM: MemoryArena out of memory");
 }
 offset += pad;
 void* ptr = data + offset;
 offset += bytes;
 return ptr;
 }

 void reset() {
 offset = 0; // Freeing memory is just resetting the pointer (O(1))
 }
};

In this way, when creating tensors, memory is always obtained through this arena. By simply calling reset() every time an inference step (such as token generation) is completed, memory can be reused instantly.

4. Tensor Data Structures and the Magic of Strides

A tensor is a generalized concept of scalars, vectors, and matrices. What is important in the implementation is that while the actual data is laid out as a one-dimensional contiguous array in memory, it has a concept called “Stride” to interpret it as multidimensional.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


enum class DataType {
 FP32,
 FP16,
 INT8, // For quantization
 INT4 // For quantization
};

struct Tensor {
 int n_dims; // Number of dimensions
 int64_t ne[4]; // Number of elements in each dimension (Number of Elements)
 size_t nb[4]; // Stride in each dimension (Number of Bytes)
 DataType type; // Data type
 void* data; // Pointer to payload

 // For computation graph
 enum OpType op;
 Tensor* src0;
 Tensor* src1;
};

The stride nb[i] represents the byte distance in memory between adjacent elements in dimension i. For example, if a matrix of size $M \times N$ (FP32, 4 bytes per element) is stored in Row-Major order, the strides would be:

nb[0] = 4 (bytes) : Movement along the column direction
nb[1] = $N \times 4$ (bytes) : Movement along the row direction

By utilizing this, operations such as “Transpose” and “View” can be achieved without memory copying, simply by swapping the stride values. It is very elegant and fast.

5. Building the Computation Graph (DAG) and Lazy Evaluation

Similar to PyTorch, our inference engine also adopts Lazy Evaluation, close to “Define-by-Run”. That is, at the time the operation function is called, the calculation is not performed; only the graph (dependencies between nodes) is built.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Tensor* tensor_add(MemoryArena& arena, Tensor* a, Tensor* b) {
 Tensor* out = create_tensor(arena, a->type, a->n_dims, a->ne);
 out->op = OpType::ADD;
 out->src0 = a;
 out->src1 = b;
 return out;
}

Tensor* tensor_mul_mat(MemoryArena& arena, Tensor* a, Tensor* b) {
 // b is often transposed
 int64_t ne[2] = { a->ne[0], b->ne[1] };
 Tensor* out = create_tensor(arena, a->type, 2, ne);
 out->op = OpType::MUL_MAT;
 out->src0 = a;
 out->src1 = b;
 return out;
}

The flow of inference processing is as follows.

graph LR A["Define Tensors"] --> B["Build Graph via Ops"] B --> C["Topological Sort"] C --> D["Allocate Memory for Outputs"] D --> E["Execute Nodes In Order"]

When evaluating the graph (forward pass), it uses topological sorting to execute nodes in order, starting from those without dependencies. Since we only do inference, there is no need to retain gradients for backpropagation, making memory management extremely simple.

6. The Core of Math and Optimization: General Matrix Multiply (GEMM)

Over 90% of the computational cost of AI inference is spent on General Matrix Multiply (GEMM). Both the attention mechanism and the feed-forward network (FFN), which are the core of the Transformer model, are ultimately massive matrix multiplications.

The product $C = A B$ (size $M \times N$) of two matrices $A$ (size $M \times K$) and $B$ (size $K \times N$) is expressed by the following formula.

$$ C_{i,j} = \sum_{k=0}^{K-1} A_{i,k} \cdot B_{k,j} $$

If this is implemented with a naive triple loop, cache misses will occur frequently, and no performance will be achieved.

6.1 Cache Blocking and SIMD Optimization on CPU

The basic strategies for speeding up GEMM on a CPU are as follows.

Loop Tiling (Cache Blocking): Divide matrices into smaller blocks that fit in the L1/L2 cache for computation.
Data Packing: Rearrange data internally so that memory access patterns become contiguous.
Utilizing SIMD: Use FMA (Fused Multiply-Add) instructions like _mm512_fmadd_ps in AVX-512 to perform many multiply-add operations in a single clock cycle.

Here is an example of a simplified vector dot product using C++ and SIMD Intrinsics.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


#include <immintrin.h> // For AVX instructions

// Fast dot product for FP32 using AVX2
float dot_product_avx2(const float* a, const float* b, int n) {
 __m256 sum256 = _mm256_setzero_ps();
 int i = 0;

 // Process 8 elements at a time (256 bits = 32 bytes = 8 * 4 bytes)
 for (; i <= n - 8; i += 8) {
 __m256 va = _mm256_loadu_ps(a + i);
 __m256 vb = _mm256_loadu_ps(b + i);
 // FMA instruction: sum256 = va * vb + sum256
 sum256 = _mm256_fmadd_ps(va, vb, sum256);
 }

 // Horizontally add the values in the SIMD register
 float result[8];
 _mm256_storeu_ps(result, sum256);
 float dot = result[0] + result[1] + result[2] + result[3] +
 result[4] + result[5] + result[6] + result[7];

 // Process the remainder
 for (; i < n; ++i) {
 dot += a[i] * b[i];
 }
 return dot;
}

Just this small tweak can yield a speed improvement of several to dozens of times compared to a naive implementation.

7. Crossing Hardware Boundaries: Integrating CUDA and Metal Backends

While pure C++ implementations alone run reasonably well on CPUs, the parallel computing power of GPUs is indispensable to run massive models like LLMs at practical speeds (e.g., generating more than 20 tokens per second). Thus, we introduce a backend abstraction layer to our engine.

7.1 Backend Abstraction

Using C++ polymorphism, we make it possible to switch the computation executor.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


class Backend {
public:
 virtual ~Backend() = default;
 virtual void alloc_buffer(Tensor* t) = 0;
 virtual void free_buffer(Tensor* t) = 0;
 virtual void copy_to_device(Tensor* t) = 0;
 virtual void copy_to_host(Tensor* t) = 0;

 // Execution of various operations
 virtual void compute_add(Tensor* src0, Tensor* src1, Tensor* dst) = 0;
 virtual void compute_mul_mat(Tensor* src0, Tensor* src1, Tensor* dst) = 0;
};

7.2 Implementing the NVIDIA CUDA Backend

To leverage NVIDIA GPUs, we implement the backend using CUDA C++ extensions. While writing custom kernels is possible, for matrix multiplication, the best approach is to utilize “cuBLAS”, the premier library provided by NVIDIA.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


#include <cublas_v2.h>
#include <cuda_runtime.h>

class CUDABackend : public Backend {
private:
 cublasHandle_t handle;

public:
 CUDABackend() {
 cublasCreate(&handle);
 }

 ~CUDABackend() {
 cublasDestroy(handle);
 }

 void compute_mul_mat(Tensor* src0, Tensor* src1, Tensor* dst) override {
 // Since CUDA is Column-Major by default, parameters require care
 const float alpha = 1.0f;
 const float beta = 0.0f;

 int m = src0->ne[0];
 int k = src0->ne[1];
 int n = src1->ne[1]; // Assuming src1 is transposed

 cublasSgemm(handle, CUBLAS_OP_T, CUBLAS_OP_N,
 m, n, k,
 &alpha,
 (const float*)src0->data, k,
 (const float*)src1->data, k,
 &beta,
 (float*)dst->data, m);
 cudaDeviceSynchronize();
 }
};

Data transfer (cudaMemcpy) between CUDA memory and host (CPU) memory is very heavy, so it is crucial to design the system to retain all weights (weight tensors) and intermediate tensors on VRAM as much as possible during inference.

7.3 Apple Silicon (Metal) Backend

In recent years, Mac’s M1/M2/M3 chips (Apple Silicon) have become highly excellent as AI inference machines. The reason for this lies in “Unified Memory”. Because the CPU and GPU share the same memory region, the high-cost host-to-device memory transfers via the PCIe bus, as seen in CUDA mentioned above, become completely unnecessary.

To call Metal from C++, we use Objective-C++ (.mm files) as a bridge, or use the metal-cpp library. We write kernels using Metal’s Compute Shaders (written in a C++-like manner in .metal files).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


// Metal shader (kernel.metal)
#include <metal_stdlib>
using namespace metal;

kernel void mul_mat_kernel(
 device const float* A [[buffer(0)]],
 device const float* B [[buffer(1)]],
 device float* C [[buffer(2)]],
 constant uint3& dims [[buffer(3)]],
 uint2 gid [[thread_position_in_grid]]
) {
 uint m = dims.x; uint k = dims.y; uint n = dims.z;
 uint row = gid.y; uint col = gid.x;

 if (row < m && col < n) {
 float sum = 0.0;
 for (uint i = 0; i < k; ++i) {
 sum += A[row * k + i] * B[i * n + col]; // Simplified
 }
 C[row * n + col] = sum;
 }
}

In the Apple Silicon environment, an optimized library for matrix multiplication called MPS (Metal Performance Shaders) is also provided, so by utilizing this in production, you can achieve astonishing inference speeds.

8. Transformer Model Specific Processing: Attention and KV Cache

State-of-the-art LLMs such as LLaMA 2/3 and GPT are based on the Transformer architecture. To implement this in C++, it is essential to construct “Scaled Dot-Product Attention,” represented by the following formula.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Also, in autoregressive token generation, the calculation results of past tokens (Key and Value) must be retained. This is called the “KV Cache (Key-Value Cache)”.

graph TD T["Current Token"] --> Q["Query"] T --> K["Key"] T --> V["Value"] K --> KCache["Append to KV Cache"] V --> VCache["Append to KV Cache"] Q --> Dot1["Q * K_Cache^T"] KCache --> Dot1 Dot1 --> Scale["Scale (1/sqrt(d))"] Scale --> Softmax["Softmax"] Softmax --> Dot2["SoftmaxOut * V_Cache"] VCache --> Dot2 Dot2 --> Out["Context Vector"]

For memory allocation of the KV Cache, a ring buffer-like operation is used, pre-allocating memory space for the maximum context length (e.g., 4096 or 8192 tokens) in the arena in advance. This prevents reallocation at every generation step.

In addition, we implement “RoPE (Rotary Position Embedding),” which has become mainstream in recent years, for positional encoding. This is a method of embedding position information as rotation vectors in complex space, and optimizing the calling of sin and cos functions in C++ (such as using lookup tables) is key to performance.

9. Extreme Optimization via Model Quantization

If you load a large-scale model (e.g., a 7 Billion parameter LLaMA model) in FP32 (32-bit floating-point), it will consume about 28GB of memory (VRAM) just for weights. If you include the KV cache and inference buffers, it easily exceeds 30GB, making it impossible to run on typical consumer GPUs.

This is where “Quantization” becomes essential. It is also the true worth of the GGML format.

Quantization is the technique of intentionally reducing the precision of weights.

FP16 (16-bit): Size halved. Almost no precision degradation.
INT8 (8-bit): Size 1/4. Slight degradation.
INT4 (4-bit): Size 1/8. Practical inference is possible by using proprietary blocking and scaling factors.

On the inference engine side, weights compressed in INT4 (or INT8) are read from memory, and immediately after being loaded into the CPU or GPU registers, they are expanded (Dequantized) to FP16 or FP32 for computation.

Surprisingly, it is faster to reduce the amount of data read from memory, even if it means increasing the amount of computation. This is because on modern hardware, the bottleneck for inference tasks is not “Compute Bound” but “Memory Bandwidth Bound”. With a C++ engine implementing INT4 quantization, it becomes possible to run local LLMs smoothly even on a MacBook Air with 8GB VRAM.

10. Performance Tuning: NUMA Architecture and Thread Pools

When performing inference using a CPU, multi-threading is essential. However, simply launching many std::thread instances is not optimal.

Modern multi-socket servers and high-end CPUs like Ryzen Threadripper employ NUMA (Non-Uniform Memory Access) architecture. Access to memory physically close to a certain CPU core (local memory) is fast, but access to memory tied to another processor becomes extremely slow.

Advanced C++ inference engines utilize the following techniques.

Thread Pinning: Fix each thread to a specific CPU core (set Affinity) to prevent cache invalidation due to context switches.
NUMA-aware Allocation: Allocate memory on the same NUMA node as the thread processing the data.
Work-stealing Thread Pool: Implement an efficient scheduler that divides each node of the computation graph into fine-grained tasks, and idle threads automatically steal and execute tasks.

By fully utilizing these, you can keep CPU utilization glued near 100% and strike throughput close to theoretical limits.

11. Conclusion: The Joy of Driving AI with C++ “Muscle”

Python is certainly convenient. In research and development or prototyping, no language can match its productivity. However, the moment you transition to the phase of “running the completed model efficiently on any device in the real world”, it is time for C++ to shine.

The sense of accomplishment you get when you see an inference engine—built by directly manipulating byte arrays in memory, pushing registers to their limits with SIMD instructions, and wrestling with GPU VRAM bandwidth—generating natural Japanese text (tokens) one after another on the console, is a “pure joy as an engineer” that you can never obtain simply by calling model.generate() in a Python framework.

AI technology tends to be a “Black Box”, but by writing everything by hand in C++, from tensor operations to memory allocation, you can deeply understand the true mechanisms of how LLMs “think”.

If you have a basic knowledge of C++ and a strong interest in current AI technology, please try taking on the challenge of developing your own inference engine. The source codes of GGML and llama.cpp should serve as the best living textbooks.

Now, throw away the heavy runtime of Python, and let’s run cutting-edge AI with the muscle of C++!

How to Tune TinyLLaMA the Fastest in an On-Premises Environment

Fri, 11 Sep 2026 16:00:00 +0900

1. Introduction: Why TinyLLaMA and On-Premises Now?

The evolution of Large Language Models (LLMs) is proceeding at an incredible speed, but along with it, the number of model parameters continues to inflate to the scale of hundreds of billions. While ultra-giant models like GPT-4 and Claude 3 boast unparalleled performance, the computational costs required for inference and training, as well as security and data privacy concerns when using external APIs, present significant hurdles for companies. In business operations that handle highly sensitive internal data or personal information in particular, sending data to public LLM APIs on the cloud is often unacceptable from a compliance perspective (such as GDPR or APPI).

This is where Small Language Models (SLMs) and local operation in an on-premises environment are stepping into the spotlight. Among them, “TinyLLaMA” is a compact model with only 1.1B (1.1 billion) parameters, yet it has been pre-trained on a massive dataset of approximately 3 trillion tokens, demonstrating astonishing performance compared to models in the same class.

In this article, we provide a complete guide to fine-tuning TinyLLaMA “fastest and highly efficiently” for your company’s specific tasks in an on-premises environment (local servers or workstations). We will comprehensively explain everything from the mathematical background to the latest optimization technologies and specific PyTorch implementation code.

2. TinyLLaMA Architecture and Features

TinyLLaMA follows the LLaMA (Large Language Model Meta AI) architecture developed by Meta. While keeping the number of parameters down to 1.1B, it uses the same technology stack as LLaMA 2, which features highly compatible ecosystem integration.

Major Architectural Components

RMSNorm (Root Mean Square Normalization): A normalization technique that improves computational efficiency by omitting mean subtraction from conventional LayerNorm calculations. It improves throughput while maintaining training stability.
SwiGLU Activation Function: In the Feed Forward Network (FFN), SwiGLU is adopted instead of the conventional ReLU or GELU. Mathematically, it is expressed as follows: $$ \text{SwiGLU}(x, W, V) = \text{Swish}(xW) \otimes (xV) $$ Here, $\otimes$ represents the element-wise product (Hadamard product), and the Swish function is $\text{Swish}(z) = z \cdot \sigma(\beta z)$. This significantly improves expressive power.
RoPE (Rotary Position Embedding): A method that combines the advantages of absolute position encoding and relative position encoding. It has high generalization performance even when the sequence length is extended.
Grouped Query Attention (GQA): An intermediate approach between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), grouping key and value heads to save memory bandwidth and dramatically improve inference speed.

The following Mermaid diagram illustrates the overall data flow and Transformer block structure of TinyLLaMA.

graph TD A["Input Text"] --> B["Tokenizer (BPE)"] B --> C["Embedding Layer"] C --> D["Transformer Blocks (x22 Layers for TinyLLaMA)"] D --> E["RMSNorm (Final)"] E --> F["Linear Projection (Vocab Size)"] F --> G["Output Probabilities (Softmax)"] subgraph "Transformer Block Anatomy" D1["Input Hidden State"] --> D2["RMSNorm"] D2 --> D3["Grouped Query Attention (GQA)"] D3 --> D4["Residual Add"] D4 --> D5["RMSNorm"] D5 --> D6["SwiGLU FFN"] D6 --> D7["Residual Add"] D7 --> D8["Output to Next Layer"] D1 -.-> D4 D4 -.-> D7 end

3. Breakthrough in Fine-Tuning: LoRA and QLoRA

To perform full-parameter fine-tuning in an on-premises environment, even a 1.1B model consumes tens of GB of VRAM (Video RAM) to maintain optimizer states and gradients. To learn efficiently with limited resources, PEFT (Parameter-Efficient Fine-Tuning) techniques such as “LoRA” and its quantized extension “QLoRA” are essential.

3.1 Mathematical Background of LoRA (Low-Rank Adaptation)

LoRA is a method that fixes (freezes) pre-trained weight matrices and approximates the weight updates ($\Delta W$) as the product of two small, low-rank matrices.

Let the pre-trained weights be $W_0 \in \mathbb{R}^{d \times k}$. In full fine-tuning, $W_0$ itself is updated to $W_0 + \Delta W$, but in LoRA, the update matrix $\Delta W$ is decomposed as follows:

$$ \Delta W = B \times A $$

Here, $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, where $r$ is a hyperparameter called Rank, and it is a very small value (usually 8, 16, 32, etc.) that satisfies $r \ll \min(d, k)$.

The forward pass calculation is as follows:

$$ h = W_0 x + \Delta W x = W_0 x + B A x $$

In the initial state, matrix $A$ is randomly initialized with a normal distribution (Gaussian distribution), and matrix $B$ is initialized as a zero matrix. As a result, $\Delta W$ at the start of training is zero, allowing training to start with the base model’s output perfectly preserved.

graph LR X["Input Vector x"] --> W0["Frozen Pre-trained Weight (W_0)"] X --> A["Trainable LoRA Matrix A (r x k)"] A --> B["Trainable LoRA Matrix B (d x r)"] W0 --> Add["Vector Addition"] B --> Add Add --> Y["Output Vector h"]

3.2 The Innovativeness of QLoRA (Quantized LoRA)

QLoRA pushes the LoRA approach further by quantizing the base model $W_0$ into 4-bit precision (NormalFloat 4, NF4) before loading it into memory. This drastically reduces VRAM consumption.

QLoRA incorporates three important technologies:

4-bit NormalFloat (NF4) Quantization: A theoretically optimal data type optimized for normally distributed weights.
Double Quantization: By quantizing the quantization constants (scale factors) themselves, it saves even more memory.
Paged Optimizers: Uses NVIDIA’s unified memory feature to temporarily evict optimizer states to CPU RAM when VRAM is insufficient.

With this, fine-tuning that typically requires 16GB to 24GB of VRAM can comfortably run even on consumer-grade GPUs (like RTX 3060 12GB or RTX 4070).

4. Hardware Requirements and Setup in an On-Premises Environment

When tuning TinyLLaMA (1.1B) with QLoRA, hardware requirements are kept extremely low.

Recommended Hardware Specifications

GPU: NVIDIA RTX 3060 (12GB), RTX 3090/4090 (24GB), or NVIDIA A10G/A100, etc. It can run with a minimum of 8GB of VRAM, but 12GB or more is recommended to increase the batch size.
CPU: A modern CPU with 8 or more cores (Intel Core i7/i9, AMD Ryzen 7/9)
RAM: 32GB or more (crucial as a swap destination from VRAM when using Paged Optimizers)
Storage: NVMe SSD (to speed up dataset loading and model saving)

Software Environment Setup

Here is the setup procedure assuming an Ubuntu 22.04 LTS environment. We will use Python 3.10 or later.

1
2
3
4
5
6
7
8
9


# Create and activate a virtual environment
python3 -m venv tinyllama_env
source tinyllama_env/bin/activate

# Install PyTorch (for CUDA 12.1)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install transformer-related libraries
pip install transformers datasets peft trl accelerate bitsandbytes

5. Optimization Techniques for the Fastest Tuning

To complete tuning not just by running scripts but doing so at the “fastest” speed, you need to combine the following optimization techniques.

5.1 Flash Attention 2

The standard Attention mechanism has time and space complexity of $O(N^2)$ for sequence length $N$. Flash Attention 2 optimizes memory access between the GPU’s SRAM and HBM (High Bandwidth Memory), eliminating the IO bottleneck without reducing the computational complexity. This boosts the training speed by several times and drastically reduces memory consumption.

5.2 Gradient Checkpointing

Instead of storing all intermediate activations computed during the forward pass in VRAM, this technique saves only a portion of them and recomputes them when needed during the backward pass. The computation time increases by about 20%, but memory consumption is drastically reduced, allowing for a larger batch size and resulting in improved overall throughput.

5.3 Mixed Precision Training and Bfloat16

To maximize the use of GPU Tensor Cores, calculations during training are performed in bfloat16 (Brain Floating Point). Since the bit length of the exponent is the same as float32 compared to float16, the risk of overflow and underflow is extremely low, stabilizing the training process.

6. Practice: QLoRA Fine-Tuning Code for TinyLLaMA

Let’s break down the PyTorch script for the fastest tuning incorporating all the optimizations mentioned above. Here, we will use the SFTTrainer from Hugging Face’s trl (Transformer Reinforcement Learning) library.

6.1 Dataset Preparation and Model Loading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


import torch
from datasets import load_dataset
from transformers import (
 AutoModelForCausalLM,
 AutoTokenizer,
 BitsAndBytesConfig,
 TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# 1. Specify model and tokenizer
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

# 2. 4-bit quantization settings for QLoRA
bnb_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_use_double_quant=True,
 bnb_4bit_quant_type="nf4",
 bnb_4bit_compute_dtype=torch.bfloat16 # Perform calculations in bfloat16
)

# 3. Load model (Enable Flash Attention 2)
print("Loading model...")
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 quantization_config=bnb_config,
 device_map="auto",
 use_flash_attention_2=True # Key to maximum speed
)

# 4. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right" # Set to right to avoid bugs during fp16/bf16 training

6.2 Applying the LoRA Adapter and Formatting the Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# 5. Prepare for k-bit training and enable gradient checkpointing
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

# 6. LoRA configuration
peft_config = LoraConfig(
 r=16, # Rank
 lora_alpha=32, # Scaling factor
 lora_dropout=0.05,
 bias="none",
 task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"] # Targeting all Linear layers improves performance
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# Example output: trainable params: 14,286,848 || all params: 1,114,335,232 || trainable%: 1.282%

# 7. Load dataset (Using Japanese Instruction dataset as an example here)
# In practice, you would load an on-premises private JSONL file, etc.
dataset = load_dataset("kunishou/databricks-dolly-15k-ja", split="train")

def format_instruction(sample):
 """
 Formats the string according to ChatML format or prompt template
 """
 prompt = f"<|im_start|>user\n{sample['instruction']}"
 if sample.get("input", "") != "":
 prompt += f"\n{sample['input']}"
 prompt += f"<|im_end|>\n<|im_start|>assistant\n{sample['output']}<|im_end|>"
 return {"text": prompt}

dataset = dataset.map(format_instruction)

6.3 Executing the Training

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


# 8. Set training arguments
training_args = TrainingArguments(
 output_dir="./tinyllama-lora-output",
 per_device_train_batch_size=8, # Increase if you have spare VRAM
 gradient_accumulation_steps=2, # Effective batch size = 8 * 2 = 16
 optim="paged_adamw_32bit", # VRAM savings with Paged Optimizer
 save_steps=100,
 logging_steps=10,
 learning_rate=2e-4,
 fp16=False,
 bf16=True, # Mixed precision training (bfloat16)
 max_grad_norm=0.3,
 max_steps=500, # 500 steps for testing. Specify by epochs in production
 warmup_ratio=0.03,
 group_by_length=True,
 lr_scheduler_type="cosine",
)

# 9. Start training with SFTTrainer
trainer = SFTTrainer(
 model=model,
 train_dataset=dataset,
 peft_config=peft_config,
 dataset_text_field="text",
 max_seq_length=1024, # Adjust according to the expected input length
 tokenizer=tokenizer,
 args=training_args,
)

print("Starting training...")
trainer.train()

# 10. Save the LoRA adapter
trainer.model.save_pretrained("./tinyllama-lora-final")
tokenizer.save_pretrained("./tinyllama-lora-final")
print("Training complete and model saved.")

7. Performance Evaluation and Troubleshooting

Here are common issues encountered when running training in an on-premises environment and their solutions.

OOM (Out Of Memory) occurs:
- Lower per_device_train_batch_size to 1.
- Maintain the effective batch size by increasing gradient_accumulation_steps.
- Shorten max_seq_length from 2048 to 1024 or 512.
Loss doesn’t drop or diverges:
- The learning rate (learning_rate) might be too high. Try lowering it from 2e-4 to around 5e-5.
- If using Float16 instead of Bfloat16, gradient underflow might be occurring. Ensure bf16=True is set.
Mysterious strings are generated during inference:
- Ensure padding_side="right" is correctly set. Also, verify that the dataset format (<|im_start|> and other special tokens) is consistent with the base model’s pre-training.

8. Model Deployment After Tuning

Once tuning is complete, what gets saved is not the “entire base model,” but only the “LoRA adapter (delta weights),” which is a few MB to tens of MB in size. To perform fast inference, you need to merge (integrate) these LoRA weights back into the original base model and export it as a single model.

Model Merging Script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

output_dir = "./tinyllama-lora-final"

# Load model and adapter in FP16/BF16
model = AutoPeftModelForCausalLM.from_pretrained(
 output_dir,
 device_map="auto",
 torch_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(output_dir)

# Merge weights and save
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./tinyllama-merged", safe_serialization=True)
tokenizer.save_pretrained("./tinyllama-merged")
print("Model merged and saved successfully!")

Launching an Ultra-Fast Inference Server with vLLM

For deployment in an on-premises environment, we strongly recommend using vLLM or TGI (Text Generation Inference) instead of Hugging Face’s standard pipeline to maximize inference speed (Tokens per second). vLLM uses PagedAttention technology to prevent GPU memory fragmentation and dramatically improves the processing capability for concurrent requests.

The following Mermaid diagram shows the pipeline from training to deploying an inference server.

graph TD A["Raw Private Data"] --> B["Preprocessing & Formatting (JSONL)"] B --> C["QLoRA Fine-Tuning (SFTTrainer)"] C --> D["LoRA Adapter Weights (.safetensors)"] D --> E["Merge with Base TinyLLaMA 1.1B"] E --> F["Merged Model"] F --> G["Deploy via vLLM Server"] G --> H["API Endpoint / UI (e.g. Chatbot)"]

Starting an API server using vLLM can be completed with the following single command:

1
2
3
4
5
6


python -m vllm.entrypoints.openai.api_server \
 --model ./tinyllama-merged \
 --host 0.0.0.0 \
 --port 8000 \
 --max-model-len 2048 \
 --dtype bfloat16

Now, an OpenAI API-compatible endpoint is built in your on-premises environment, allowing you to utilize local AI securely and quickly.

9. Conclusion

In this article, we explained the methods for performing the fastest and most memory-efficient fine-tuning in an on-premises environment for “TinyLLaMA”, a model that is highly performant despite being lightweight with 1.1B parameters.

LoRA / QLoRA enables full-fledged LLM tuning even on consumer GPUs.
Leveraging Flash Attention 2 and Gradient Checkpointing optimizes training time and VRAM consumption to the limit.
Deployment using vLLM achieves high throughput even in production environments.

Operating a local LLM on-premises is not only for protecting data confidentiality but also serves as the ultimate weapon for building specialized AI at low cost for specific domains (legal, medical, internal regulations, etc.). We encourage you to refer to this guide and nurture your own TinyLLaMA.

How Should Programmers Survive in the AI Era?

Fri, 11 Sep 2026 15:00:00 +0900

How Should Programmers Survive in the AI Era? The End of Coding and the Dawn of a New Engineering

As of 2026, the field of software development is undergoing a period of unprecedented and dramatic change. Until just a few years ago, the concept of “AI writing code” was limited to the role of an “auxiliary tool” for programmers, such as generating boilerplate code or auto-completing functions at best. However, due to the astonishing evolution of Large Language Models (LLMs), the situation has been fundamentally overturned. Modern AI is not just a “smart typewriter,” but has transformed into an “autonomous junior engineer” capable of autonomously assembling an entire system in an instant, from front-end to back-end logic, database schema design, and even building CI/CD pipelines, given a requirements definition document.

In such an era, how should we “programmers” or “software engineers” survive? As the economic value of the act of “writing code” itself rapidly deflates, a “coder” who merely knows the syntax of a specific programming language and is familiar with the APIs of a specific framework is rapidly being weeded out from the market.

In this article, we will examine the survival strategy for programmers in the AI era in extreme detail from technical, mathematical, and philosophical perspectives. This is not merely a career theory, but a redefinition of the academic discipline of software engineering itself.

1. The History of Abstraction and the Redefinition of “Programming”

Looking back at the history of software engineering, it is clear that it has always been a history of “Abstraction.” We have constantly built layers to describe more complex systems in languages closer to human understanding.

Early computer scientists used punch cards to directly manipulate the physical switches of hardware, giving instructions to computers in machine language (a sequence of 0s and 1s). Later, assembly languages emerged, allowing humans to operate hardware using mnemonics that were easier to understand. As time progressed, high-level languages like C and Fortran appeared, successfully encapsulating complex hardware details such as memory management and CPU registers. With the subsequent emergence of modern languages like Java, Python, Ruby, and TypeScript, programmers could focus more on “What” they wanted the computer to do, rather than “How” to make the computer operate.

The advent of AI (LLMs) is the latest and greatest paradigm shift in this history of abstraction. If the evolution of programming languages was the “hiding of hardware,” the evolution of LLMs is the “hiding of syntax.”

graph TD A["Machine Code / Punch Cards (1940s)"] --> B["Assembly Language (1950s)"] B --> C["Compiled High-Level Languages (1970s)"] C --> D["Managed / Scripting Languages (1990s)"] D --> E["Natural Language via LLMs (2020s)"] E --> F["Autonomous AI Agents (2026-)"] style E fill:#f9f,stroke:#333,stroke-width:2px style F fill:#fbb,stroke:#333,stroke-width:2px

The era where developers operated pointers while worrying about memory leaks, or wrote hundreds of lines of boilerplate code to parse JSON, is over. Using natural language (such as English or Japanese), the language with the highest level of abstraction for humanity, to define systems has become the standard of “programming” in 2026.

2. A Mathematical Model of Productivity: Riding the Wave of Exponential Growth

Let’s quantitatively evaluate the productivity improvement brought about by AI using a mathematical model. Individual productivity in traditional software development, $P_{traditional}$, could be modeled as a linear combination of an individual’s skill level $S$, domain experience $E$, and tool efficiency $T$.

$$ P_{traditional} = c_1 \cdot S + c_2 \cdot E + c_3 \cdot T $$

However, in modern development utilizing AI, the capability of AI, $A(t)$, acts as a “powerful multiplier (Leverage)” that amplifies human capabilities. Because AI capabilities grow exponentially over time $t$ (an AI version of Moore’s Law), productivity in the AI era, $P_{AI}(t)$, can be expressed by the following equation:

$$ P_{AI}(t) = \alpha \cdot S_{core} \cdot e^{\beta \cdot A(t)} $$

Here, each variable means the following:

$\alpha$: The baseline human productivity coefficient
$S_{core}$: “Core human skills” that cannot be replaced by AI (architecture design, understanding business requirements, ethical judgment, etc.)
$A(t)$: The absolute capability of the AI model at time $t$ (number of parameters, context window, reasoning ability)
$\beta$: A coefficient indicating how effectively AI tools can be utilized (the quality of prompt engineering and the refinement of collaborative workflows with AI)

The important insight derived from this formula is that in a world where $A(t)$ increases exponentially, traditional skills such as mere typing speed or memorization of a specific language have extremely little impact on overall productivity. Instead, the coefficient $\beta$ to multiply the exponential growth of AI, and $S_{core}$, which is the domain that AI cannot cover, become the dominant factors determining an engineer’s market value.

3. Probability of Automation

So, what kind of tasks will be automated, and what tasks will remain in human hands? The probability that a certain task $T$ will be completely automated by AI, $P_{auto}(T)$, can be formulated as follows:

$$ P_{auto}(T) = 1 - \exp\left(-\lambda \cdot \frac{\text{Predictability}(T)}{\text{Complexity}(T) \times \text{Context Dependency}(T)}\right) $$

$\text{Predictability}(T)$: The predictability of the task (how much of a pattern exists in past data)
$\text{Complexity}(T)$: The complexity of the task
$\text{Context Dependency}(T)$: The strength of “implicit context (domain-specific knowledge or human relationships)” that the task depends on
$\lambda$: The rate of technological advancement of AI

Tasks with high predictability and low context dependency, such as writing API routing logic or creating a simple CRUD screen, will have $P_{auto} \approx 1$ and be almost completely automated. On the other hand, tasks with extremely high context dependency, such as “how to safely integrate a new microservice with an existing legacy system” or “how to design an authentication flow that satisfies the legal department’s requirements without compromising the user experience,” are difficult to automate.

4. Return from Syntax to Architecture (Structure)

Clearly separating what AI is good at and what humans are good at is an absolute condition for survival.

graph LR Sub1["AI's Domains of Excellence"] Sub2["Human's Domains of Excellence"] A["Code Generation from Spec"] --> Sub1 B["Syntax Error & Bug Fixing"] --> Sub1 C["Boilerplate / Test Generation"] --> Sub1 D["Log Analysis & Pattern Matching"] --> Sub1 E["System Architecture Design"] --> Sub2 F["Resolving Ambiguous Requirements"] --> Sub2 G["Cross-team Negotiation"] --> Sub2 H["Ethical Judgement / Responsibility"] --> Sub2

AI surpasses humans in “local optimization.” There is no way humans can compete with the speed and accuracy of writing a single function, a single class, or a single module. However, AI is extremely vulnerable to “global optimization” and “Missing Context.”

Future programmers must change their roles from “workers who write code” to “architects who orchestrate the countless components generated by AI.” Having a bird’s-eye view of the entire system, where to draw microservice boundaries, how to resolve the trade-off between availability and consistency in the CAP theorem according to the business context, and how to control technical debt. These are highly intellectual tasks that can only be done by humans who understand the big picture and business goals.

5. Requirements Engineering is the “True Prompt Engineering”

The term “prompt engineering,” which we hear a lot lately, is often misunderstood as a “hack to trick AI into getting the desired output.” However, the essence of prompt engineering in software development is undeniably “advanced Requirements Engineering.”

In order to give instructions to AI in natural language and have it output the intended software, the following elements must be strictly verbalized:

Purpose (Why): Why is this feature needed? What is the business value?
Constraints: Performance requirements (latency, throughput), security requirements, cost constraints.
Edge Cases: Fallback processing when the user provides unexpected input.
Interfaces: Integration specifications with existing systems.

Vague instructions (prompts) can only produce vague and fragile systems. The ability to deeply interview “what the customer really wanted,” sort out conflicting requirements, and create a logically sound specification document (prompt). This is the ultimate “coding skill” in the AI era. Programmers will spend more time facing Notion or Markdown files instead of code editors, elaborately describing what a system should look like in text.

6. The Overwhelming Advantage of Domain Knowledge

Because AI has trained on open-source code and public documents from around the world, it is well-versed in general web technologies and algorithms. However, there is data that AI cannot access. That is “your company’s unique business rules” and “domain knowledge deeply rooted in a specific industry (medical, financial, manufacturing, etc.).”

For example, suppose you are developing an electronic medical record system at a medical startup. AI knows “how to create a tabular UI in React” and the “general data structure of HL7 FHIR.” However, it has not learned the tacit knowledge of “in what order doctors at a specific department of Hospital A view patient data, and what kind of UI can minimize the risk of medical errors.”

In a world where technology itself becomes commoditized (general-purpose), the true value of an engineer is born at the intersection of “technology” and “business domain.” Rather than competing on technical skills alone, the market will be driven by those who possess deep expertise in a specific domain—such as medical, financial, logistics, or entertainment—and can solve the challenges of that domain using the powerful tool of AI.

7. The “Trolley Problem” of Software Development: Who Takes Responsibility?

As reliance on AI increases, we face serious philosophical and ethical issues. This is the problem of “where responsibility lies” in software engineering.

If the code autonomously generated by AI causes a serious bug in a production environment, resulting in hundreds of millions of yen in losses for a company, or causes a malfunction in a life-critical medical system, who will take the responsibility? The company that developed the AI model? Or the engineer who input the prompt? You cannot “fire” or “arrest” AI.

The role of “human” as the entity that assumes “Accountability (legal and ethical responsibility)” for the impact a system has on society will never disappear, no matter how much technology advances. Rather, as the code generation process becomes more black-boxed, humans will bear a heavier responsibility as the “Approver” and “Supervisor” of the system.

Auditing whether the architecture or code proposed by AI meets security standards, whether there are ethical issues (e.g., whether it contains bias), and whether it complies with regulations, and then giving the final GO sign. This act of “taking responsibility” itself will become an important part of an engineer’s job.

8. AI Pair Programming and Cognitive Load Management

When working alongside AI, the nature of human “Cognitive Load” is also changing. The cognitive load of writing code from scratch is completely different from the cognitive load of “reading and reviewing” hundreds of lines of unknown code generated by AI.

According to cognitive load theory in psychology, human working memory is quickly depleted when processing complex information that does not match existing schemas (knowledge structures in the brain). While the code generated by AI sometimes contains advanced optimizations that humans might not think of, it can also contain “hallucinations” that ignore the context.

To prevent this, it is necessary to systematize the review process for AI.

sequenceDiagram participant H as "Human Engineer (Architect)" participant A as "AI Agent" participant S as "CI/CD & Testing" H->>A: "Define strict requirements & constraints" A->>H: "Propose Architecture & Initial Code" Note over H,A: Review Phase: High Cognitive Load H->>A: "Critique design choices, ask for refactoring" A->>S: "Generate Final Code & Push" S-->>H: "Automated Test Results & Static Analysis" H->>H: "Final Approval & Responsibility Assumption"

More than the skill of “writing,” humans need to push their skills of “Code Reading & Auditing (reading quickly and instantly spotting logical flaws)” to the limit. The importance of Test-Driven Development (TDD) has increased even further in the AI era. Before letting AI write code, the mainstream approach will be for a human or another AI to write strict test code, and then have AI modify the code until it passes those tests.

9. Concrete Survival Strategy: What You Should Learn Starting Tomorrow

Based on the analysis so far, here is a concrete action plan for programmers to survive in the AI era.

Thoroughly Relearn the “Fundamentals” of Technology: You can leave the usage of frameworks to AI. However, a deep understanding of how OS works, network protocols (TCP/IP, HTTP/3), internal database structures (B-Tree, transaction isolation levels), and data structures and algorithms is absolutely necessary. To judge whether AI’s output is correct, a solid foundation in computer science is indispensable.
Master Cloud Architecture and Distributed Systems: Focus on how to combine cloud resources like AWS, GCP, and Azure to build scalable systems, rather than individual pieces of code. Understand the concept of IaC (Infrastructure as Code) such as Terraform, and cultivate the ability to design an entire system as code.
Become an Expert in a Business Domain: Deeply study the business models, legal regulations, and behavioral psychology of the users in the industry you belong to. Go beyond the boundaries of an engineer and acquire a perspective closer to that of a Product Manager (PM).
Polish Communication and Facilitation Skills: The process of resolving the “ambiguity” between humans and building consensus cannot be replaced by AI. Soft skills to communicate with stakeholders and discover real issues will become the most valuable skills.
Use AI to the Fullest as a “Colleague”: Do not fear the evolution of AI tools, but utilize them as your most powerful weapons. Use the latest LLMs and AI coding agents on a daily basis to accumulate the “tacit knowledge” of where AI fails and how to tweak prompts to draw out the best performance.

Conclusion: Do Not Fear, Ride the Wave

The automation of programming by AI does not mean the “death” of the programmer profession. Rather, it is a “Renaissance” that liberates us from the “non-essential labor” in software development, such as fixing typos, troubleshooting environment setups, and writing boring boilerplate.

Historically, when automated looms appeared, and when spreadsheet software (Excel) appeared, pessimism about jobs disappearing was rampant. However, the reality is that the dramatic increase in productivity created new demands, and more advanced jobs were born. The same thing will happen in the software world. By making it possible to “build systems cheaply,” software will penetrate into every area that had not been IT-enabled before due to cost constraints, and the problems (What) that engineers should solve will expand infinitely.

We programmers are now given the chance to evolve from craftsmen who write code to “orchestra conductors” who direct the mighty intelligence of AI. Instead of staying on the shore out of fear of the technological wave, let’s ride that wave early and set sail on a journey to create larger, more valuable systems. The AI era is the era when “engineering” in the truest sense begins.

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

Fri, 11 Sep 2026 14:00:00 +0900

Development Guide for Small AI Models (TinyLLaMA, etc.) Using C++

In recent years, interest in running Large Language Models (LLMs) in local environments has grown rapidly. In particular, small-scale models like TinyLLaMA (1.1B parameters) can perform inference at practical speeds even on limited-resource edge devices and typical laptops (including Windows environments). While development using Python and PyTorch is mainstream, when pursuing ultimate performance and memory efficiency, the combination of C++ and the C-based tensor library “ggml” has become the de facto standard.

This article provides an extremely detailed development guide for building an inference engine from scratch (or deeply understanding the internal structure of the existing llama.cpp) to load TinyLLaMA and generate text using C++.

1. Why C++ and ggml?

In the AI training phase, Python has an overwhelming advantage due to its flexibility and rich ecosystem. However, in the deployment or “Inference” phase, C++ becomes a powerful choice for the following reasons:

Overhead Reduction: The Python Global Interpreter Lock (GIL) and runtime overhead can be completely eliminated.
Memory Efficiency and Arena Allocation: By manually controlling memory allocation and deallocation, you can prevent unpredictable spikes caused by garbage collection.
Direct Hardware Access: By directly calling SIMD intrinsics such as AVX-512, AVX2, and ARM NEON, the CPU’s computational power can be maximized.
Zero Dependencies: ggml is a zero-dependency C/C++ library. As long as you have a compiler, it can be easily built even in an MSVC environment on Windows.

2. Overall Architecture

The flow of the entire inference pipeline is shown in the Mermaid diagram below. This is a series of processes starting from the user’s input text until the final next token is generated.

graph TD A["User Input Text"] --> B["BPE Tokenizer"] B --> C["Token IDs Array"] C --> D["Embedding Layer Lookup"] D --> E["Transformer Blocks"] E --> F["RMSNorm"] F --> G["LM Head Layer"] G --> H["Logits Array"] H --> I["Sampler Module"] I --> J["Next Token ID"] J --> K["Detokenizer"] K --> L["Output Text Chunk"] J -.-> |"Append to Context"| C

Since it is an autoregressive model, the output token is added back to the context and circulates as input for predicting the next token (the dotted line in the diagram).

3. Model Format and Memory Mapping (mmap)

The biggest hurdle in handling the weights of massive neural networks is disk I/O and memory consumption. In a C++ implementation, this is resolved with memory mapping (mmap).

3.1 How Memory Mapping Works and its Windows Implementation

Using mmap allows you to map the contents of a file directly into the virtual memory space of the process.

Zero-copy: Data is loaded directly from the disk into the kernel’s page cache, preventing extra copies into user space.
On-Demand Loading (Page Fault): The moment the CPU actually accesses that memory address, a page fault occurs, and only the required chunk (usually 4KB) is loaded into physical memory.

In a Windows environment, the Win32 APIs CreateFileMapping and MapViewOfFile are used instead of POSIX mmap.

sequenceDiagram participant OS["Windows OS"] participant RAM["Physical Memory"] participant App["C++ Application"] App->>OS: "CreateFileMapping / MapViewOfFile" OS-->>App: "Virtual Memory Address Pointer" App->>App: "Read Tensor Data at Pointer" OS->>RAM: "Page Fault / Load page from Disk" RAM-->>App: "Data ready for SIMD Compute"

3.2 Binary Structure of the GGUF Format

Converted from formats like Hugging Face’s .safetensors, the GGUF (GPT-Generated Unified Format) is the ultimate format for inference. It has the following strict binary layout:

Magic Bytes: 0x46554747 (GGUF).
Version: Format version number.
Tensor Count & Metadata Count: The number of tensors and key-value metadata pairs.
Metadata (Key-Value Pairs): Typed values and keys with string length prefixes.
Tensor Info: The name, number of dimensions, data type (FP16, Q4_K, etc.), and offset position in the file for each tensor.
Padding: Padding inserted so that tensor data is aligned to specific boundaries (usually 32 bytes or 64 bytes). This is essential for fast memory access with SIMD instructions (especially AVX).
Tensor Data: The actual aligned weight data array.

4. Mathematical Foundations of TinyLLaMA and C++ Algorithms

TinyLLaMA incorporates several advanced architectural ingenuities for efficiency. We explain the mathematical expressions to correctly implement these in C++.

4.1 RMSNorm (Root Mean Square Normalization)

It reduces computational cost by omitting mean centering from LayerNorm and only performing variance scaling.

$$ \text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \gamma $$

$d$ is the number of dimensions, and $\gamma$ is the learned scaling tensor. When implementing in C++, it is optimized by first rapidly calculating the sum of squares of the array using AVX2’s _mm256_fmadd_ps or similar, and then multiplying by the inverse square root (e.g., the _mm256_rsqrt_ps instruction).

4.2 RoPE (Rotary Position Embedding)

A technique that applies token position information as a rotation in tensor space. It can be seen as a rotation on a complex plane, applying the following rotation to adjacent dimension pairs $(x_1, x_2)$ of vector $x$:

$$ \text{RoPE}(x, m) = \begin{pmatrix} x_{1} \cos(m\theta) - x_{2} \sin(m\theta) \\ x_{1} \sin(m\theta) + x_{2} \cos(m\theta) \end{pmatrix} $$

Here, $m$ is the absolute position index of the token, and $\theta$ is a pre-calculated base frequency. In ggml, it is executed in parallel simply by adding a ggml_rope operator during inference graph construction.

4.3 Grouped-Query Attention (GQA)

In standard Multi-Head Attention (MHA), Query, Key, and Value each have the same number of heads. However, TinyLLaMA adopts Grouped-Query Attention (GQA) to drastically reduce memory bandwidth and KV cache consumption.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$

In GQA, multiple Query heads share a single Key/Value head. In the C++ implementation, before executing the matrix multiplication ggml_mul_mat, it is necessary to broadcast the KV tensors to match the number of Queries.

4.4 SwiGLU Activation Function

In the Feed-Forward Network (FFN) layer, SwiGLU is used instead of GELU.

$$ \text{SwiGLU}(x) = \text{Swish}(x W_{\text{gate}}) \otimes (x W_{\text{up}}) $$ $$ \text{Swish}(z) = z \cdot \sigma(z) = z \cdot \frac{1}{1 + e^{-z}} $$

In the computation graph, it is represented by combining the ggml_silu operator and ggml_mul.

5. Computation Graph Construction and Memory Management with ggml

ggml uses a “Define-and-Run” approach, constructing a static computation graph for inference and evaluating it later.

5.1 ggml_context and Arena Allocator

The most unique aspect of ggml is “arena allocation,” which avoids dynamic memory allocation (malloc or new) entirely within the inference loop. Upon initialization, a huge contiguous memory region (arena) is allocated, and the pointer to this region is incremented every time ggml_new_tensor or similar is called. Once one inference step is completed, simply resetting the allocation pointer to its initial position immediately finishes memory allocation for the next inference step.

5.2 Specific Example of Graph Construction

For each inference step, a computation graph like the following is assembled in memory:

graph TD A["Tokens Input ID"] --> B["Embed Lookup"] B --> C["ggml_rms_norm"] C --> D["Q / K / V Projections"] D --> E["ggml_rope Positional"] E --> F["KV Cache Store"] E --> G["KV Cache Load"] G --> H["Self Attention"] H --> I["Scale & Softmax"] I --> J["Attention Output"] J --> K["Out Projection"] K --> L["Add Residual"]

6. Quantization and Windows / SIMD Optimization

Handling TinyLLaMA (1.1B) in FP16 requires approximately 2.2GB of memory, but it can be dramatically compressed to around 600MB through 4-bit quantization (such as Q4_K).

6.1 Block Quantization Architecture

ggml does not quantize the entire tensor uniformly; instead, it does so in “block” units. In the Q4_0 format, 32 FP16 values are grouped into a single block.

Scale Factor: One FP16 value (2 bytes)
Quantized Data: 32 4-bit values (16 bytes) This minimizes the impact of local outliers.

6.2 Dot Product Acceleration with AVX2

When building for the latest x86 CPUs in a Windows environment, utilizing compiler flags like /arch:AVX2 allows SIMD processing to be performed in the following flow:

Load: Load 4-bit quantized data from memory into 256-bit AVX registers.
Expansion and Unpacking: Expand 4-bit values into Int8 or Int16 using bit masks and shift operations.
Dequantization: Multiply by the scale factor to convert to floating-point numbers.
FMA Operations: Execute multiply-add operations in parallel using activation values and _mm256_fmadd_ps (Fused Multiply-Add).

7. KV Cache Implementation Details

In autoregressive generation, the “KV cache” is an essential feature to skip the computation of Keys and Values for past tokens.

The key points for a C++ implementation are as follows:

Tensor Pre-allocation: Initialize a massive tensor for the KV cache (FP16 recommended) corresponding to the maximum context length (e.g., 2048 tokens).
Offset Copying: When calculations for token position $N$ are performed, the K and V vectors obtained in that step are stored into the $N$-th row of the KV cache tensor using ggml_cpy or similar.
Creating a View during Attention: When calculating attention, create a “view” that points only to the token portion from 0 to the $N$-th token and pass it to the matrix multiplication.

8. BPE Tokenizer and Decoding

The input string is treated as a UTF-8 byte sequence and matched against a predefined vocabulary. In C++, to speed up the vocabulary search, algorithms using a Trie (prefix tree) or a priority queue are implemented.

From the logits output by the LM Head, probabilities are scaled using the Temperature parameter, candidates are narrowed down using Top-K extraction or Top-P (Nucleus Sampling) methods, and the final next token is determined using random numbers.

9. Setting Up the C++ Project (Windows / PowerShell Environment)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cmake_minimum_required(VERSION 3.14)
project(TinyLLaMACpp)

set(CMAKE_CXX_STANDARD 17)

# Windows (MSVC) optimization and AVX2 flag settings
if(MSVC)
 add_compile_options(/O2 /arch:AVX2 /fp:fast)
 add_link_options(/STACK:8388608)
else()
 add_compile_options(-O3 -march=native -ffast-math)
endif()

add_library(ggml OBJECT ggml/ggml.c ggml/ggml-alloc.c)
target_compile_definitions(ggml PRIVATE GGML_USE_AVX2 GGML_USE_F16C GGML_USE_FMA)

add_executable(main main.cpp)
target_link_libraries(main ggml)

Example build commands in PowerShell:

1
2
3
4


mkdir build
cd build
cmake .. -G "Visual Studio 17 2022" -A x64
cmake --build . --config Release

10. Conclusion

Implementing an inference engine from scratch for small AI models like TinyLLaMA using C++ and ggml is a perfect opportunity to demystify the black box of deep learning and learn the beauty of low-level hardware control. Let’s pave the way for the future of edge AI while fully savoring the essence of systems programming, such as zero-copy loading using memory mapping, SIMD optimization, and KV cache construction.

[RAG Implementation Guide] How to Load Your Own Documents into Local AI

Fri, 11 Sep 2026 13:00:00 +0900

Introduction

In recent years, the evolution of Large Language Models (LLMs) has been remarkable, and many AIs, led by ChatGPT and Claude, have permeated our daily lives and business operations. However, general LLMs have a distinct weakness. That is, they only know “public information at the time of their training”. Naturally, they cannot answer questions about “private documents” such as internal company regulations, personal notes, and unpublished project materials. Forcing them to answer increases the risk of generating plausible lies (hallucinations) that differ from the facts.

Therefore, the technological architecture known as RAG (Retrieval-Augmented Generation) is currently spreading explosively worldwide. By using RAG, it becomes possible to dynamically provide unique knowledge to the LLM from an external database and have it generate accurate and well-founded answers based on it.

Furthermore, when handling enterprise domains or personal confidential information, sending data to cloud-based APIs like OpenAI is often unacceptable under security policies. What is required there is the construction of “Local RAG” combined with Local AI (an LLM that operates entirely on your own PC or on-premise server).

In this article, we will thoroughly explain everything from the fundamental theory of RAG, specific implementation methods of Local RAG using Python, mathematical background (how vector search works), to advanced techniques for running the system in production.

1. Overall Architecture of RAG

RAG is not a single AI model, but a system architecture where multiple components work together. It broadly consists of two phases: the “Ingestion (Data Loading) Phase” and the “Retrieval & Generation Phase”.

The Mermaid diagram below shows the overall picture of a RAG system.

graph TD subgraph "Ingestion Phase (Preparation)" Doc["Custom Documents (PDF, TXT, etc.)"] --> Loader["Document Loader"] Loader --> Splitter["Text Splitting (Chunking)"] Splitter --> EmbedModel1["Embedding Model"] EmbedModel1 --> VectorDB["Vector Database"] end subgraph "Inference Phase (User Query)" User["User Question (Query)"] --> EmbedModel2["Embedding Model"] EmbedModel2 --> QueryVector["Query Vector"] QueryVector --> Search["Similarity Search (Vector Search)"] VectorDB --> Search Search --> Context["Relevant Chunk Extraction (Context)"] User --> PromptBuilder["Prompt Builder"] Context --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer Generation"] end

Ingestion Phase (Preparation)

Document Loading: Loads unstructured data such as PDFs, Word documents, and text files.
Chunking (Text Splitting): Splits long texts into meaningful chunks to fit within the LLM’s input limit (context window) and to improve search accuracy.
Embedding (Vectorization): Inputs the split chunks into an Embedding Model and converts them into an array of numerical values (vectors) with hundreds to thousands of dimensions.
Saving to Database: Saves the converted vectors and their associated original text data into a Vector Database (Vector DB).

Inference Phase (Runtime)

Query Vectorization: Vectorizes the user’s question text using the same embedding model used in the preparation phase.
Similarity Search: Performs a similarity calculation between the query vector and the document vectors in the database, and retrieves the top few semantic (highly relevant) text chunks.
Prompt Building: Combines the retrieved relevant text as “context (background knowledge)” with the user’s question text to create an input prompt for the LLM.
Answer Generation: The LLM, receiving the augmented prompt, generates an answer based on the provided context information.

2. Deep Understanding of Vector Search and Embeddings

At the core of RAG is “Vector Search (Semantic Search)”. While traditional keyword search (like BM25) is based on exact word matching and frequency, vector search is based on “semantic similarity”. For example, even if the words are different, like “dog” and “puppy”, or “PC” and “computer”, they will be hit in the search if their meanings are close.

What is an Embedding Model?

An embedding model is a neural network that takes natural language text as input and outputs a fixed-length dense vector. Common models (e.g., text-embedding-3-small or the open-source multilingual-e5-large) map text into a vector of real numbers with 384 or 1024 dimensions.

In this multi-dimensional space (latent space), the model is trained so that sentences with similar meanings are closer in distance in the coordinate space.

Mathematical Background of Similarity Calculation: Cosine Similarity

When a vector database searches for relevant documents, the most commonly used distance metric is Cosine Similarity. Unlike Euclidean distance (absolute spatial distance), cosine similarity focuses on the “angle between two vectors”. Since it is less affected by the length of the sentence (the norm of the vector), it is highly suitable for calculating text similarity.

Expressed mathematically, the cosine similarity between vectors $\mathbf{A}$ and $\mathbf{B}$ is as follows:

$$ \text{Cosine Similarity}(\mathbf{A}, \mathbf{B}) = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}} $$

$\mathbf{A} \cdot \mathbf{B}$ represents the Dot Product.
$\|\mathbf{A}\|$ represents the L2 norm (length) of vector $\mathbf{A}$.
$n$ is the number of dimensions of the vector.

Cosine similarity takes a value from -1 to 1.

Close to 1: The directions of the two vectors are almost the same (meanings are very similar).
Close to 0: The two vectors are orthogonal (unrelated).
Close to -1: The two vectors are in opposite directions (meanings are opposite).

Recent Vector DBs (Chroma, FAISS, Qdrant, etc.) employ an Approximate Nearest Neighbor (ANN) algorithm called HNSW (Hierarchical Navigable Small World), optimizing them to search for documents with high cosine similarity in milliseconds even from millions of vector data.

3. Technology Stack for Building Local RAG

To build a fully local RAG that does not rely on the cloud, we leverage the open-source ecosystem. The recommended technology stack is introduced below.

Large Language Model (LLM)
- Tools: Ollama or Llama.cpp
- Models: Lightweight, high-performance open models like Llama-3-8B-Instruct, Gemma-2-9B-It, Qwen2-7B-Instruct. For Japanese tasks, Japanese-tuned models like Llama-3-ELYZA-JP-8B are suitable.
Embedding Model (Embedding)
- Models: intfloat/multilingual-e5-large or BAAI/bge-m3. When running locally, it is common to download them from Hugging Face and run them with Sentence-Transformers.
Vector Database (Vector DB)
- ChromaDB: Python-based and extremely easy to set up. Ideal for local development.
- FAISS: A fast vector search library developed by Meta.
- Qdrant / Milvus: For larger scale and production environments.
Orchestration Framework
- LangChain: The de facto standard for chaining components together.
- LlamaIndex: A data connection framework specifically specialized for RAG.

This time, we will implement it using the easiest combination to introduce: LangChain + ChromaDB + Ollama + HuggingFaceEmbeddings.

4. Implementation Tutorial: Building a Full Local RAG with Python

From here, we will build a local RAG while actually writing Python code. Please install Ollama on your PC in advance and have it running in the background. Also, pull a model on Ollama (e.g., ollama run llama3).

Step 1: Installing Required Libraries

1
2


pip install langchain langchain-community langchain-huggingface
pip install chromadb sentence-transformers pypdf

Step 2: Complete Implementation Code

Below is the complete Python script to load a PDF file, vectorize it, and have the local LLM answer questions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93


import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain_core.prompts import PromptTemplate
from langchain.chains import RetrievalQA

def main():
 # 1. Document Loading
 print("Loading document...")
 # Specify the path of the PDF you want to load
 file_path = "sample_company_policy.pdf"
 loader = PyPDFLoader(file_path)
 documents = loader.load()

 # 2. Chunking (Text Splitting)
 # Split into appropriate sizes without breaking the meaning of the sentences
 text_splitter = RecursiveCharacterTextSplitter(
 chunk_size=500, # Maximum number of characters per chunk
 chunk_overlap=50, # Overlap characters between previous and next chunks (prevents context disconnection)
 separators=["\n\n", "\n", "。", "、", " ", ""]
 )
 chunks = text_splitter.split_documents(documents)
 print(f"Split into {len(chunks)} chunks.")

 # 3. Initialization of Embedding Model (Local HuggingFace Model)
 # Using a multilingual model with strong Japanese support
 print("Loading embedding model...")
 embeddings = HuggingFaceEmbeddings(
 model_name="intfloat/multilingual-e5-large",
 model_kwargs={'device': 'cpu'} # 'cuda' or 'mps' if you have a GPU
 )

 # 4. Building Vector Database (Chroma)
 print("Building vector database...")
 persist_directory = "./chroma_db"
 vectorstore = Chroma.from_documents(
 documents=chunks,
 embedding=embeddings,
 persist_directory=persist_directory
 )
 # Creating a Retriever. Configured to get the top 3 relevant documents
 retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

 # 5. Initialization of Local LLM (Ollama)
 print("Connecting to Local LLM...")
 # Make sure to get the model in advance with 'ollama pull llama3' etc.
 llm = Ollama(model="llama3")

 # 6. Definition of Prompt Template
 prompt_template = """You are an excellent assistant familiar with company regulations and internal information.
Please answer the user's question in detail based ONLY on the following context (background information).
If you cannot find the answer from the context, please do not guess and honestly answer "I don't know from the provided information."

[Context]
{context}

[Question]
{question}

[Answer]:
"""
 PROMPT = PromptTemplate(
 template=prompt_template,
 input_variables=["context", "question"]
 )

 # 7. Building RAG Chain
 qa_chain = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff",
 retriever=retriever,
 return_source_documents=True, # Set whether to return source documents
 chain_type_kwargs={"prompt": PROMPT}
 )

 # 8. Executing a Question
 query = "Please tell me about the conditions for transportation expenses payment regarding remote work."
 print(f"\nQuestion: {query}\n")

 result = qa_chain.invoke({"query": query})

 print("[Answer]")
 print(result['result'])
 print("\n---")
 print("[Reference Sources]")
 for doc in result['source_documents']:
 print(f"- Page {doc.metadata.get('page', 'Unknown')}: {doc.page_content[:50]}...")

if __name__ == "__main__":
 main()

Explanation of Key Code Points

RecursiveCharacterTextSplitter: This is the most recommended splitter for dividing natural language. It attempts to split in the order of paragraphs (\n\n), lines (\n), and periods (。), keeping semantic blocks together as much as possible while fitting within the specified chunk_size. By setting chunk_overlap, you prevent context boundaries from being cut off and information from being lost.
HuggingFaceEmbeddings: intfloat/multilingual-e5-large is a very powerful open-source embedding model that supports multiple languages. You can vectorize text locally on your memory offline without using a cloud API (like OpenAI’s text-embedding-ada-002).
ChromaDB: Since it runs in-memory or on local storage (SQLite-based), there is no need to spin up a complex database server. By specifying persist_directory, you can skip the vectorization process on subsequent runs and load the DB from disk.

5. Advanced RAG Techniques

The basic RAG system (Naive RAG) built in the above tutorial works, but if high answer accuracy is required in a production environment, the introduction of advanced techniques like the following becomes necessary.

5.1 Hybrid Search

While vector search is good at capturing “meaning”, it can be poor at strict keyword searches like “specific proper nouns”, “product model numbers”, and “employee IDs”. Therefore, by running semantic search via vector search and keyword search using algorithms like BM25 in parallel, and integrating both results by scoring them (using techniques like Reciprocal Rank Fusion; RRF), you can drastically reduce search misses.

5.2 Re-ranking

Vector search is fast, but it does not necessarily evaluate the exact contextual relevance of the context. A general pipeline for improving search accuracy is as follows:

First-stage Retrieval: Retrieve a broad and shallow range of relevant chunks (about 20-30) from the Vector DB.
Re-ranking: Use another heavier machine learning model called a Cross-Encoder (e.g., bge-reranker) to input pairs of the user’s query and the retrieved chunks, and recalculate their semantic relevance scores.
Selection: Pass only the top 3-5 with the highest scores as the final context to the LLM prompt.

This technique prevents irrelevant noise information from being passed to the LLM, significantly increasing the precision of the answers.

graph LR Query["Query"] --> VSearch["Vector Search (Top 20)"] VSearch --> Reranker["Re-ranker Model (Cross-Encoder)"] Query --> Reranker Reranker --> TopK["High Precision Top 3"] TopK --> LLM["LLM Generation"]

5.3 Semantic Chunking and Parent Document Retrieval

Instead of mechanically splitting text by a fixed number of characters, there is a technique called “Semantic Chunking” that uses AI to detect shifts in meaning and splits the text accordingly. Also, in the “Parent Document Retriever” technique, you vectorize in very small units (like sentences) for high-precision search, but when passing it to the LLM, you provide the “original large paragraph (parent document)” containing that sentence, thus providing sufficient context to the LLM.

6. Challenges and Countermeasures when Operating Local RAG

There are unique hurdles when building and operating RAG in a local environment.

VRAM (Video Memory) Exhaustion: To run a Local LLM at a practical speed (dozens of tokens per second), you need to load the model into the GPU’s VRAM. Running an 8B class model in fp16 (16-bit floating point) requires about 16GB of VRAM, but by using Quantization technologies (compressing to 4-bit or 8-bit, such as GGUF or AWQ formats), it is possible to run it fast enough even with 8GB of VRAM (like a standard gaming PC). Llama.cpp and Ollama support these quantization formats by default.
Context Window Limits: If the amount of retrieved context is too large, it may exceed the LLM’s input limit (token limit), or the model might forget the middle part of the information (Lost in the middle phenomenon). Adjusting the number of chunks to extract and carefully selecting them through the aforementioned re-ranking techniques are essential.
Data Freshness Management: When a source document is updated, the corresponding document’s vector in the vector database also needs to be updated or deleted (CRUD operations). Since ChromaDB supports updates based on document IDs, it is practical to manage file hash values and set up a batch process to sync only the differences.

Conclusion

RAG (Retrieval-Augmented Generation) is a powerful paradigm that evolves AI from a general-purpose assistant into your “exclusive expert” or an “expert specialized in internal business”.

We found that even with highly confidential requirements where cloud services cannot be used, a complete “Local RAG” environment can be relatively easily built by combining the open-source ecosystem such as Ollama, LangChain, and ChromaDB.

Based on the advanced approaches such as mathematical understanding of vector space, text splitting, and re-ranking explained in this article, please try developing an original AI system using your own data. The speed of evolution in local AI is astounding, and the system you build today can instantly update its performance tomorrow simply by swapping in a smarter lightweight model that appears.

This blog will continue to publish deep-dive articles on AI technology and RAG. If you have any questions or feedback, please share them in the comments section.

ChatGPT, Gemini, and Claude APIs Thoroughly Compared! Which Should You Choose?

Fri, 11 Sep 2026 12:00:00 +0900

ChatGPT, Gemini, and Claude APIs Thoroughly Compared! Which Should You Choose?

AI technology is evolving remarkably, especially in the field of Large Language Models (LLM), where OpenAI’s ChatGPT (GPT series), Google’s Gemini, and Anthropic’s Claude are engaged in a fierce three-way battle for supremacy. As of 2026, each company is releasing new models and API features on a scale of months, or even weeks. For developers and corporate IT architects, the question of “which API to integrate into products” has become a highly critical decision that dictates the success of a project.

In this article, we will thoroughly compare and explain these top 3 AI providers’ APIs from a developer’s perspective. We go beyond just listing specifications, and delve into architecture design, detailed pricing structures, mathematical analysis of latency, concrete implementation examples using Python and Node.js, and the latest cost-optimization techniques such as prompt caching.

Our goal is for this to serve as a complete guide for readers to select the most optimal LLM API for their own use cases and build scalable, cost-effective AI applications.

1. Philosophy and Design Concepts of Each LLM API

When choosing technology, it is very important to first understand the philosophies each company uses to build their models and APIs.

1.1 OpenAI (ChatGPT)

OpenAI has set its mission to “achieve Artificial General Intelligence (AGI)” and constantly drives the de facto standard of the industry. They offer a diverse range of models tailored to use cases, such as GPT-4o, GPT-4o-mini, and the inference-specialized o1 model. Their ecosystem is the most mature, and they have the most abundant libraries and documentation, both official and unofficial.

1.2 Google (Gemini)

Google advocates “AI First” and uses the scalability of its own infrastructure (TPU networks) to its maximum advantage. Gemini 1.5 Pro/Flash’s greatest feature is its overwhelming context window of up to 2 million tokens, allowing it to process lengthy documents or hours of video/audio all at once. Its strong integration with Google Cloud (Vertex AI) is also attractive for enterprises.

1.3 Anthropic (Claude)

Anthropic is a company founded by former OpenAI members, adopting a unique safety approach called “Constitutional AI”. Claude 3.5 Sonnet and Opus have garnered enthusiastic support from many developers for their high reasoning capabilities, code generation skills, and above all, their “human-like natural dialogue” and “low hallucinations”.

2. Thorough Comparison of Model Family Specifications

We compare the specifications of the flagship models as of 2026.

Provider	Flagship Model	Max Context Length	Key Strengths	Recommended Use Cases
OpenAI	GPT-4o	128K	Speed, vision, audio support	Interactive apps, general tasks
OpenAI	o1-preview	128K	Advanced logical reasoning, math, coding	Complex algorithm generation, research
Google	Gemini 1.5 Pro	2,000K	Ultra-long text processing, multimodal (video/audio)	Analyzing massive codebases, video summarization
Google	Gemini 1.5 Flash	2,000K	Low latency, high throughput, overwhelmingly low cost	Real-time processing, batch processing of large data
Anthropic	Claude 3.5 Sonnet	200K	Coding capabilities, natural text generation	Software development assistance, advanced customer support
Anthropic	Claude 3.5 Haiku	200K	Ultra-fast response, cost performance	Edge AI, real-time chatbots

3. Deep Dive into Architecture: Behind the API Requests

When an LLM API is called, what kind of processing takes place on the backend? To optimize performance, we need to understand this architecture.

The following Mermaid diagram shows the overall picture from when an API request is sent by the client to when tokens are returned in a stream.

graph TD A["Client Application"] -->|HTTP/REST or gRPC| B["API Gateway"] B --> C["Load Balancer"] C --> D["Inference Cluster"] D --> E["Tokenizer (BPE / SentencePiece)"] E --> F["KV Cache & Attention Mechanism"] F --> G["Transformer Blocks (Forward Pass)"] G --> H["Output Layer (Logits)"] H --> I["Sampler (Temperature, Top-p, Top-k)"] I --> J["Detokenizer"] J -->|Streaming Response (Chunk)| A

3.1 Tokenization Algorithms

The text input into the API is internally divided into units called “tokens”.

OpenAI (tiktoken): Uses Byte-Pair Encoding (BPE). It compresses English extremely efficiently, but for non-alphabet languages like Japanese, the number of tokens tends to inflate.
Google (Gemini): Uses SentencePiece (Unigram Language Model). It handles multiple languages well, and tends to express even Japanese text with relatively few tokens.
Anthropic (Claude): Uses a customized version of BPE. Multilingual support has been strengthened, and since Claude 3, token efficiency for Japanese has also significantly improved.

4. Mathematical Analysis of Latency and Performance

In real-time applications, latency is directly tied to the User Experience (UX). The latency of an LLM API, $T_{total}$, can be mathematically modeled as follows:

$$ T_{total} = T_{network} + T_{TTFT} + (N \times T_{TPOT}) $$

Here, each variable has the following meaning:

$T_{network}$: Network Round Trip Time (RTT).
$T_{TTFT}$ (Time To First Token): The time until the first character is generated. It depends heavily on the cost of attention calculation, which is proportional to the square of the prompt length (number of input tokens).
$N$: The total number of tokens output.
$T_{TPOT}$ (Time Per Output Token): Generation time per token. Because it is an autoregressive model, it is calculated serially depending on the previous output.

4.1 Computational Complexity of the Self-Attention Mechanism

The computational complexity of Self-Attention in the Transformer architecture increases quadratically with respect to the input sequence length $L$.

$$ \text{Complexity} = O(L^2 \cdot d) $$

Here, $d$ is the dimensionality of the embedding vector. Because of this constraint, $T_{TTFT}$ usually worsens dramatically as the prompt gets longer. However, Google’s Gemini 1.5 employs innovative optimization architectures like “Ring Attention” and “Block-wise Compute,” successfully generating the first token in a realistic amount of time (a few seconds to tens of seconds) even when a long text of 2 million tokens is input.

5. Pricing Systems and Cost Optimization Strategies

API costs are fundamentally calculated based on the number of input and output tokens.

$$ Cost = (Tokens_{in} \times Rate_{in}) + (Tokens_{out} \times Rate_{out}) $$

However, the latest APIs have introduced new mechanisms to drastically reduce costs.

5.1 Prompt Caching

Sending a massive system prompt or a large amount of documents retrieved by RAG every time incurs an enormous cost. To address this, each company provides caching features.

In Anthropic (Claude) and Google (Gemini), caching specific text blocks can significantly reduce input costs (up to 90%).

The cost model when using a cache is as follows:

$$ Cost_{cached} = (Tokens_{cache\_write} \times Rate_{cache\_write}) + (Tokens_{cache\_read} \times Rate_{cache\_read}) + (Tokens_{out} \times Rate_{out}) $$

Here, $Rate_{cache\_read}$ is set to about 10% to 25% of the regular $Rate_{in}$. This makes it possible to run a chatbot cheaply while constantly keeping tens of thousands of lines of a codebase as background knowledge.

5.2 Batch API

For tasks that do not require real-time processing (log analysis, bulk data classification, etc.), OpenAI and Anthropic offer a Batch API. This is a powerful mechanism where you can send requests in bulk and receive results within 24 hours at half the regular API price (50% off).

6. Developer Experience (DX) and SDK Comparison

We compare the SDKs (Software Development Kits) provided by each company from the perspective of development efficiency.

6.1 OpenAI API

The most widely used, with the fastest support for third-party libraries (LangChain, LlamaIndex, etc.). Furthermore, the Structured Outputs feature guarantees a response that complies with a JSON schema with 100% accuracy, making system integration extremely easy.

6.2 Anthropic API (Claude)

The SDK interface is refined, and its TypeScript type definitions are highly regarded for being very easy to handle. The structure of the Message API is particularly intuitive, allowing simple coding for multimodal requests containing multiple images.

6.3 Google Gemini API

There are two access methods: via Google Cloud Vertex AI and via AI Studio (Google Gen AI SDK), which might slightly confuse beginners. However, the Vertex AI SDK for enterprises is fully integrated with GCP’s IAM (Identity and Access Management) system, enabling the construction of secure development environments.

7. Practical! Integrated Testing Implementation of Multiple APIs with Python

Here, we will use Python to implement a script that sends asynchronous requests simultaneously to three APIs—OpenAI, Anthropic, and Gemini—and compares their latency.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


import asyncio
import time
import os
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
import google.generativeai as genai

# Initialize clients
openai_client = AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
genai.configure(api_key=os.environ.get("GEMINI_API_KEY"))

prompt = "Please explain the basics of quantum computing and its impact on current cryptographic technologies in an easy-to-understand way for beginners."

async def fetch_openai():
 start_time = time.time()
 response = await openai_client.chat.completions.create(
 model="gpt-4o",
 messages=[{"role": "user", "content": prompt}],
 max_tokens=1024
 )
 elapsed = time.time() - start_time
 return "OpenAI (GPT-4o)", elapsed, response.choices[0].message.content

async def fetch_anthropic():
 start_time = time.time()
 response = await anthropic_client.messages.create(
 model="claude-3-5-sonnet-20240620",
 messages=[{"role": "user", "content": prompt}],
 max_tokens=1024
 )
 elapsed = time.time() - start_time
 return "Anthropic (Claude 3.5 Sonnet)", elapsed, response.content[0].text

async def fetch_gemini():
 start_time = time.time()
 model = genai.GenerativeModel('gemini-1.5-pro')
 # Using the asynchronous method of the Gemini Python SDK
 response = await model.generate_content_async(prompt)
 elapsed = time.time() - start_time
 return "Google (Gemini 1.5 Pro)", elapsed, response.text

async def main():
 print("Sending requests to each LLM API...")

 # Execute 3 APIs in parallel
 results = await asyncio.gather(
 fetch_openai(),
 fetch_anthropic(),
 fetch_gemini()
 )

 for provider, latency, text in results:
 print(f"--- {provider} ---")
 print(f"Latency: {latency:.2f} seconds")
 print(f"Response (Excerpt): {text[:100]}...\n")

if __name__ == "__main__":
 asyncio.run(main())

By running this script, you can easily measure which model responds the fastest (minimizing $T_{total}$) in a real network environment.

8. Implementing Tool Calling (Function Calling) with Node.js

To make an LLM function not just as a chatbot but as an “AI Agent” that cooperates with external systems, Tool Calling (or Function Calling) is indispensable. Below is an example using Node.js (TypeScript) to have the OpenAI API call a weather API.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


import OpenAI from "openai";

const openai = new OpenAI({
 apiKey: process.env.OPENAI_API_KEY,
});

async function runAgent() {
 const tools = [
 {
 type: "function",
 function: {
 name: "get_weather",
 description: "Gets the current weather for the specified city.",
 parameters: {
 type: "object",
 properties: {
 location: {
 type: "string",
 description: "City name (e.g., Tokyo, New York)",
 },
 },
 required: ["location"],
 },
 },
 },
 ];

 const response = await openai.chat.completions.create({
 model: "gpt-4o",
 messages: [{ role: "user", "content": "How's the weather in Tokyo today? Will I need an umbrella?" }],
 tools: tools,
 tool_choice: "auto",
 });

 const message = response.choices[0].message;

 if (message.tool_calls) {
 const toolCall = message.tool_calls[0];
 console.log(`LLM requested a tool call: Function name = ${toolCall.function.name}`);

 const args = JSON.parse(toolCall.function.arguments);
 console.log(`Arguments: ${args.location}`);

 // Implement the logic to call the actual weather API (e.g., OpenWeatherMap) here
 // const weather = await fetchWeatherFromAPI(args.location);

 // Pass the retrieved results back to the LLM to generate the final answer
 }
}

runAgent().catch(console.error);

Claude 3.5 Sonnet and Gemini 1.5 Pro also have equivalent Tool Calling capabilities, and although there are slight differences in how schemas are defined, the basic flow is common across them.

9. RAG vs Long Context Window: Which Should You Adopt?

Currently, one of the biggest debates in enterprise AI architecture is “whether to use RAG (Retrieval-Augmented Generation) to incorporate external knowledge, or to throw it all into a massive context window (Long Context).”

Benefits and Challenges of RAG (Retrieval-Augmented Generation)

Benefits: Low cost (because only necessary chunks are put into the prompt), and it’s easy to identify the grounds (sources) for the answer.
Challenges: Because it relies on the accuracy of semantic search, it is not suitable for advanced reasoning tasks where the context is scattered across multiple documents (e.g., “Analyze the root causes of the delay in Project A chronologically from all meeting minutes from last year”).

Long Context (e.g., Gemini 1.5 Pro’s 2 Million Tokens)

Benefits: No missed information from searches. Even in “Needle In A Haystack (NIAH)” tests, Gemini 1.5 Pro and Claude 3.5 Sonnet can extract information with over 99% accuracy.
Challenges: Enormous token consumption leads to high costs, and latency ($T_{TTFT}$) increases.

Conclusion: The best practice for 2026 is a “Hybrid Approach”. The mainstream design is to use RAG with a vector database for routine Q&A, and use Long Context leveraging prompt caching for specialized tasks requiring complex analysis or full codebase reviews.

10. Comparison of Multimodal Processing Capabilities

Next-generation AI applications require the ability to directly understand not only text but also images, audio, and video.

sequenceDiagram participant User as "User" participant Client as "Frontend App" participant API as "LLM API (Multimodal)" User->>Client: Upload Video & Text Prompt Client->>API: Send Video Bytes/URI + Text Note over API: Video chunking & Audio separation Note over API: Multimodal Embedding Model API-->>Client: Return Text Summary & Timestamps Client-->>User: Display Insights

OpenAI (GPT-4o): Has extremely high image recognition accuracy and excels at reading handwritten drawings and complex graphs. Its native audio interaction with ultra-low latency (hundreds of milliseconds) using the Realtime API is also powerful.
Google (Gemini 1.5 Pro): Overwhelmingly superior in video analysis. You can input a 1-hour video file (frames + audio) as is, and it can answer pinpoint questions like, “What is the title of the document held by the person who appeared on the right edge of the screen at 12 minutes and 45 seconds?”
Anthropic (Claude 3.5 Sonnet): Its image recognition (Vision) capability is on par with GPT-4o and very excellent. It demonstrates unrivaled strength in frontend development assistance, such as passing a UI screenshot and saying, “Generate the React component code for this screen.”

11. Enterprise-Level Security and Compliance

When enterprises use LLM APIs in production environments, their biggest concerns are “Will our company’s data be used for training the AI?” and “Are compliance requirements met?”

All three companies clearly state that data sent via API (prompts and responses) will not be used for model training (Zero Data Retention / No Training on Customer Data) (*This does not apply to free consumer web chat UIs).

When an even higher level of security is required:

OpenAI: By using the Azure OpenAI Service, you can utilize Microsoft’s enterprise-grade security, SLAs, and closed-network connections via Azure Private Link.
Google: By using Google Cloud Vertex AI, strict network isolation using VPC Service Controls and data protection with CMEK (Customer-Managed Encryption Keys) are possible.
Anthropic: By using it via AWS Bedrock or Google Cloud Vertex AI, you can piggyback on the robust security infrastructure of cloud providers.

12. Conclusion: Ultimate Choice Guide by Use Case

Although we have compared them from various angles, the final conclusion to “Which one should you choose?” depends on the use case.

Complex Software Development, Code Generation, Advanced Reasoning: 👑 Winner: Claude 3.5 Sonnet (Anthropic) It currently delivers the best performance in understanding code context, refactoring, and writing natural, human-like text. The ease of use of the API and the cost efficiency due to prompt caching are also outstanding.
Parsing Ultra-Long Documents, Batch Processing of Video/Audio: 👑 Winner: Gemini 1.5 Pro (Google) Its 2-million-token context window is a unique weapon. For tasks requiring a grasp of the entire data, such as parsing hundreds of pages of PDF manuals or summarizing long meeting recordings, Gemini is second to none.
Versatility, Execution Speed, Stable Structured Output (JSON): 👑 Winner: GPT-4o / GPT-4o-mini (OpenAI) It handles any task flawlessly and has the most abundant third-party tool support. If you need reliable JSON parsing using Structured Outputs or ultra-advanced logical reasoning using the o1 model, the OpenAI ecosystem is indispensable.

Recommendation for Multi-Model Routing

The future trend is an “LLM Routing” architecture that dynamically switches models according to the difficulty and importance of the task, rather than depending on a single API (vendor lock-in). For example, you can respond to simple questions from users with the cheap and fast GPT-4o-mini or Gemini 1.5 Flash, and fallback the task to Claude 3.5 Sonnet only when complex processing is deemed necessary, thereby achieving the optimal balance of cost and performance.

The evolution of AI will not stop. Please deeply understand the strengths, weaknesses, and architectural characteristics of each API to build flexible and scalable AI applications.

A Complete Guide to llama.cpp and Customization with C++

Fri, 11 Sep 2026 11:00:00 +0900

In recent years, the evolution of Large Language Models (LLMs) has been tremendous, and their scope of application is expanding daily. However, running models with billions or tens of billions of parameters locally typically requires a high-end GPU with an enormous amount of VRAM. Breaking through this “hardware barrier” and making practical LLM inference possible on everyday PCs, Macs, and even devices like the Raspberry Pi is llama.cpp.

This article goes beyond just explaining how to use the command-line tool. It provides an extremely detailed explanation for engineers, covering the architecture of its underlying technology ggml, the mathematical background of Transformers and quantization, and how to use the C++ API to integrate and customize LLMs within your own applications.

1. Overview of llama.cpp and ggml

llama.cpp is a lightweight LLM inference engine written in C/C++, developed by Georgi Gerganov. Originally created with the goal of running Meta’s LLaMA model quickly on Apple Silicon (M1/M2 Macs), it now supports a variety of architectures and models.

Its biggest feature is that it is a pure C/C++ implementation with no external dependencies. Because it doesn’t require a massive ecosystem like Python or PyTorch and can be compiled as a single executable, deployment is incredibly easy.

The heart of llama.cpp is the tensor math library ggml. ggml was designed from the ground up to maximally optimize matrix operations in machine learning on CPUs (and some GPUs).

1.1 Why is llama.cpp so fast?

Leveraging Memory Mapping (mmap): When loading model weights into memory, it uses the OS’s mmap to avoid loading everything into RAM, enabling fast startups and saving memory.
Thorough Optimization of SIMD Instructions: It utilizes CPU-specific instruction sets like AVX2, AVX-512, ARM NEON, and Apple AMX to perform ultra-fast matrix multiplications.
Quantization: It compresses 16-bit floating-point (FP16) weights into 4-bit, 5-bit, or 8-bit integers, resolving the memory bandwidth bottleneck (more on this later).

2. Mathematical Background: Transformers and Quantization

To deeply understand llama.cpp, you need to know the mathematical formulas it calculates and how it approximates these calculations.

2.1 The Inference Process of a Transformer

Models like LLaMA adopt an auto-regressive Transformer decoder architecture. The core of text generation is the Self-Attention mechanism.

For an input hidden state matrix $X \in \mathbb{R}^{N \times d}$, the Query $Q$, Key $K$, and Value $V$ are calculated by multiplying with weight matrices:

$$ Q = X W_Q, \quad K = X W_K, \quad V = X W_V $$

Here, the output of Attention is defined as follows:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

In the inference loop of llama.cpp, the bottleneck is the multiplication of these massive matrices $W_Q, W_K, W_V$ and the feed-forward network (FFN) weight matrices with the vector $X$ ($N=1$ in the generation phase because it processes one token at a time), which is GEMV (General Matrix-Vector Multiplication).

2.2 The Mathematical Foundation of Quantization

In inference where memory access bandwidth becomes a bottleneck, quantization—representing weight parameters with a small number of bits—is essential. Here, we’ll explain the basic principles of the block-wise quantization widely used in llama.cpp (e.g., Q4_K or Q4_0).

For example, consider a block $w = [w_1, w_2, \dots, w_B]$ of length $B$ (usually 32 or 64) that is part of an FP16 weight matrix $W$. This block is approximated using 4-bit integers $q_i \in [-8, 7]$ and a single scaling factor $\Delta$ (FP16 or FP32):

$$ w_i \approx \Delta \times q_i $$

$\Delta$ is determined based on the maximum absolute value within the block:

$$ \Delta = \frac{\max_i |w_i|}{7} $$

When calculating the dot product $y = w \cdot x$ using the quantized weights, if the input vector $x$ is similarly quantized to $x_i \approx \Delta_x \times q_{x, i}$, we get:

$$ y = \sum_{i=1}^{B} w_i x_i \approx \Delta \Delta_x \sum_{i=1}^{B} q_i q_{x, i} $$

The $\sum q_i q_{x, i}$ part becomes a pure integer operation, which can be computed in parallel very quickly using SIMD instructions. This is the mathematical trick behind the astonishing speed of llama.cpp on CPUs.

3. Architecture and Inference Flow

To understand the internal workings of llama.cpp, the following Mermaid diagram shows the overall system architecture and data flow.

graph TD A["User Input (String)"] --> B["llama.cpp Tokenizer"] B --> C["Token IDs (int32 array)"] C --> D["Context Buffer (KV Cache)"] D --> E["ggml Compute Graph"] E --> F["Transformer Layers"] subgraph "ggml Engine" F --> G["Self-Attention (RoPE)"] G --> H["Feed Forward Network"] H --> F end F --> I["Logits (Vocabulary Size)"] I --> J["Sampler (Temperature, Top-K, Top-P)"] J --> K["Selected Token ID"] K --> L["llama.cpp Detokenizer"] L --> M["Output String"] K -. "Auto-regressive loop" .-> D

Text generation is an auto-regressive loop where, each time a single token is output, it is added to the KV Cache as the next input and passes through the compute graph again.

4. Environment Setup and Build Guide

Before embedding llama.cpp into a C++ project, let’s first build the source code.

4.1 Cloning the Repository

1
2


git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

4.2 Building with CMake

When embedding it into other applications as a C++ project, using CMake is the most standard approach. By enabling accelerators (backends) for your specific platform, you can speed up computations.

CPU Only (Basic Build):

1
2
3


mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

Using NVIDIA GPU (CUDA):

1
2
3


mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j 8

Using Apple Silicon (Metal):

1
2
3


mkdir build && cd build
cmake .. -DGGML_METAL=ON
cmake --build . --config Release -j 8

Upon a successful build, executable files like llama-cli and the llama library (along with the ggml library) for linking via the C++ API discussed later will be generated in the build/bin/ directory.

5. Introduction to C++ Customization: Using the llama.cpp API

From here on, we will discuss the main topic: controlling llama.cpp from C++ code. To embed an LLM into your own application (e.g., a game engine, desktop app, or embedded system) rather than just using command-line tools, you need to hit the C++ API directly.

llama.cpp primarily provides a C language interface through a header file called llama.h. We use this interface even when calling from C++.

5.1 Minimal Necessary Includes and Setup

When using llama.cpp in your project, include the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


#include "llama.h"
#include <iostream>
#include <vector>
#include <string>
#include <stdexcept>

// Macro for error handling
#define LLAMA_ASSERT(x) \
 do { \
 if (!(x)) { \
 std::cerr << "Assertion failed: " << #x << std::endl; \
 std::terminate(); \
 } \
 } while (0)

5.2 Loading the Model and Initializing the Context

First, load a .gguf format model file and allocate the context (memory space and KV cache) for inference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


int main(int argc, char ** argv) {
 if (argc < 2) {
 std::cerr << "Usage: " << argv[0] << " <model.gguf>" << std::endl;
 return 1;
 }
 std::string model_path = argv[1];

 // 1. Initialize the backend (set up environment like CPU/GPU)
 llama_backend_init();

 // 2. Get default settings for model parameters
 llama_model_params model_params = llama_model_default_params();
 model_params.n_gpu_layers = 35; // Number of layers to offload to GPU

 // 3. Load the model
 llama_model * model = llama_load_model_from_file(model_path.c_str(), model_params);
 if (model == nullptr) {
 std::cerr << "Failed to load model" << std::endl;
 return 1;
 }

 // 4. Set context parameters
 llama_context_params ctx_params = llama_context_default_params();
 ctx_params.n_ctx = 2048; // Maximum context size (number of tokens)
 ctx_params.n_threads = 8; // Number of CPU threads used for inference

 // 5. Create the context
 llama_context * ctx = llama_new_context_with_model(model, ctx_params);
 if (ctx == nullptr) {
 std::cerr << "Failed to create context" << std::endl;
 llama_free_model(model);
 return 1;
 }

 std::cout << "Model and context loaded successfully!" << std::endl;
 // ... subsequent processing

5.3 Tokenization of the Prompt

An LLM does not understand text directly; it processes strings as sequences of integer IDs (tokens). Therefore, the input string must be converted into tokens.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 std::string prompt = "Q: What is the capital of Japan?\nA:";
 std::vector<llama_token> tokens_list;
 tokens_list.resize(prompt.length() + 4); // Buffer size with some headroom

 // Whether to prepend a special token (like BOS: Begin of Sequence)
 bool add_special = true;
 // Convert string to an array of token IDs
 int n_tokens = llama_tokenize(
 model,
 prompt.c_str(),
 prompt.length(),
 tokens_list.data(),
 tokens_list.size(),
 add_special,
 false // parse_special
 );

 if (n_tokens < 0) {
 // Handling for when the buffer is insufficient, like reallocating and retrying, is necessary (omitted for brevity)
 std::cerr << "Failed to tokenize prompt" << std::endl;
 return 1;
 }
 tokens_list.resize(n_tokens);

5.4 Inference Loop and Sampling

Build a loop that feeds tokens into the model, obtains the probability distribution (Logits) of the next token, and samples from it to determine the next token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72


 // Maximum number of tokens to generate
 const int max_gen_tokens = 100;

 // Initialize structure for batch evaluation
 llama_batch batch = llama_batch_init(512, 0, 1);

 // Add prompt tokens to the batch
 for (size_t i = 0; i < tokens_list.size(); i++) {
 llama_batch_add(batch, tokens_list[i], i, { 0 }, false);
 }
 // Set to output logits (predictions) only for the very last token of the prompt
 batch.logits[batch.n_tokens - 1] = true;

 // Initial evaluation (feeding the prompt to the model)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "llama_decode() failed" << std::endl;
 return 1;
 }

 int n_cur = batch.n_tokens; // Current context length
 int n_decode = 0;

 std::cout << "\nOutput: ";

 // Initialize sampler context (settings for Temperature, Top-K, Top-P, etc.)
 llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
 llama_sampler_chain_add_top_k(smpl, 40);
 llama_sampler_chain_add_top_p(smpl, 0.9f, 1);
 llama_sampler_chain_add_temp(smpl, 0.7f);
 llama_sampler_chain_add_dist(smpl, 1234); // Seed value

 while (n_decode < max_gen_tokens) {
 // 1. Sampling: Predict the next token based on current context
 llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);

 // 2. If token is EOS (End of Sequence), break the loop
 if (llama_token_is_eog(model, new_token_id)) {
 break;
 }

 // 3. Decode token to string (text) and print
 char buf[128];
 int n_chars = llama_token_to_piece(model, new_token_id, buf, sizeof(buf), 0, false);
 if (n_chars > 0) {
 std::cout << std::string(buf, n_chars) << std::flush;
 }

 // 4. Prepare the newly generated token as the next batch
 llama_batch_clear(batch);
 llama_batch_add(batch, new_token_id, n_cur, { 0 }, true);

 // 5. Evaluate the model (update KV cache and predict next)
 if (llama_decode(ctx, batch) != 0) {
 std::cerr << "Failed to evaluate" << std::endl;
 break;
 }

 n_cur += 1;
 n_decode += 1;
 }

 std::cout << std::endl;

 // Cleanup
 llama_sampler_free(smpl);
 llama_batch_free(batch);
 llama_free(ctx);
 llama_free_model(model);
 llama_backend_free();

 return 0;
}

This code implements a custom inference loop using the basic API of llama.cpp. It uses the llama_batch struct to manage token groups and executes the forward pass of the neural network using llama_decode.

6. Advanced Customization Examples: Logit Manipulation and Penalty Control in C++

When you want to go beyond simple text generation—such as forcing output in a specific format (e.g., JSON only) or suppressing specific forbidden words—you directly manipulate the Logits prior to sampling from the C++ side.

You can retrieve the array of raw scores (values before being converted to probabilities) right before the model outputs each token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


// Retrieve the raw logits array immediately after inference, prior to sampling
float * logits = llama_get_logits_ith(ctx, batch.n_tokens - 1);
int n_vocab = llama_n_vocab(model);

// List of forbidden token IDs (for example: 1234, 5678)
std::vector<llama_token> forbidden_tokens = { 1234, 5678 };

// Set the occurrence probability of forbidden tokens to 0 (make the Logit negative infinity)
for (llama_token bad_tok : forbidden_tokens) {
 logits[bad_tok] = -INFINITY;
}

In this way, directly interacting with the C++ API allows for “micro-millisecond interventions per inference cycle” that would be difficult or incur high overhead if done via LangChain or Python.

7. The Secrets of Performance Tuning

After finishing your C++ implementation, here are a few checkpoints for maximizing speed for actual deployment.

Optimizing Batch Processing: When handling requests from multiple users concurrently, include multiple sequences in llama_batch and call llama_decode at once (Continuous Batching). This drastically improves throughput by coalescing memory access.
Enabling Flash Attention: By setting ctx_params.flash_attn = true; in the context parameters, you can speed up Attention calculations while reducing memory usage. This setting is essential when dealing with long contexts (tens of thousands of tokens).
NUMA Support: In multi-socket server environments, properly configuring NUMA before llama_backend_init() can reduce memory access latency.

8. Conclusion

In this article, we covered everything in detail, starting from the mathematical background of llama.cpp to an explanation of its architecture, and finally, how to build a custom inference engine fully utilizing the C++ API.

While the Python ecosystem is highly convenient for prototyping, the direct control offered by C/C++ based llama.cpp demonstrates overwhelming power in production environments that demand edge device deployment, game integration, and real-time processing.

By all means, try writing C++ code yourself and experience the joy of freely manipulating LLMs in a local environment.

Reference Links

llama.cpp Official Repository

ggml - Tensor Library

Attention Is All You Need (Vaswani et al., 2017)

【2026 Edition】 The Complete Guide to Running Local LLMs in a Windows Environment

Fri, 11 Sep 2026 10:00:00 +0900

1. Introduction: Why Local LLMs on Windows Now?

As of 2026, the evolution of generative AI and Large Language Models (LLMs) shows a major paradigm shift from gigantic cloud-based API services to “local LLMs” running on personal PCs and on-premise environments. While cloud AIs like OpenAI’s GPT-5 and Anthropic’s Claude 3.5 are incredibly powerful, not all companies and individuals can send all their data to the cloud. From the perspectives of privacy, security, latency, and long-term sustainable costs, the demand for local LLMs is exploding like never before.

The evolution of the local LLM ecosystem, especially in the Windows environment, is remarkable. Until a few years ago, “Linux for AI development and execution” was common sense, but as of 2026, Windows has transformed into an extremely powerful and accessible AI platform.

In this article, based on the latest technology trends of 2026, we provide a complete guide to building, operating, and optimizing local LLMs in a Windows environment. From easy setup using Ollama for beginners to extreme optimization using llama.cpp for advanced users, and further deep dives into the mathematical approach of VRAM calculation, deep understanding of the architecture, and local fine-tuning, we will explain everything thoroughly with an overwhelming volume.

1.1 Technology Trends Surrounding Local LLMs in 2026

The major trends shaping the current local LLM ecosystem are as follows:

Complete Popularization of the GGUF Format: GGUF (GPT-Generated Unified Format), which integrates metadata and tensors into a single file, has completely become the de facto standard. With this, simply downloading a single file from Hugging Face makes it executable in any environment.
Democratization of the MoE (Mixture of Experts) Architecture: Many small but high-performance MoE models have been released. By activating only a portion of the experts during inference, they achieve performance comparable to giant models while keeping the computational load on consumer PCs low.
Advanced Abstraction and Optimization of Inference Engines: Tools like Ollama, LM Studio, and AnythingLLM have been refined so that users no longer need to be aware of complex dependencies like CUDA driver installations. Also, the native Windows support for FlashAttention 3 has dramatically improved inference speed.
Utilization of NPUs and the Rise of Windows Copilot+ PCs: Even on laptops without GPUs, the technology to run small LLMs (SLM: Small Language Models) with low power consumption using the built-in NPU (Neural Processing Unit) has entered the practical stage.

2. Hardware Requirements and OS Preparation

To run local LLMs at practical speeds (15-30 tokens per second or more), selecting the right hardware is the most important factor.

2.1 Recommended Hardware Configuration

With the evolution of AI PCs, required specs are also changing.

OS: Windows 11 Pro (24H2 or later). Essential for fully utilizing WSL2’s features, advanced memory management, and the latest DirectML APIs.
CPU: Intel Core Ultra 200 series or higher, or AMD Ryzen 9000 series or higher. When using CPU inference alongside, broad-bandwidth memory communication is indispensable.
RAM: Minimum 32GB, recommended 64GB or more. Main memory bandwidth (MB/s) becomes a crucial bottleneck during CPU inference or offloading. High-speed memory of DDR5-6000 or above is ideal.
GPU: NVIDIA RTX 4000/5000 series. The most important thing for local LLMs is not computing performance but “VRAM capacity”.
- Entry: RTX 4060 Ti (16GB version) - Best cost performance. Ideal for 8B-14B class models.
- Mid-range: RTX 4070 Ti SUPER (16GB) / RTX 4080 SUPER (16GB)
- High-end: RTX 4090 (24GB) / RTX 5090 (32GB) - Necessary to run 30B-70B class quantized models.
Storage: PCIe Gen4 or Gen5 NVMe SSD. Dramatically reduces the load times of models that are tens of gigabytes in size.

2.2 Setting up WSL2 (Windows Subsystem for Linux 2)

While many GUI tools work natively on Windows, WSL2 is extremely useful for Python development, compiling the latest tools, and the LoRA fine-tuning mentioned later. In the latest Windows 11 environment, just by installing the NVIDIA driver on the host side, the GPU (CUDA) can be transparently used from WSL2.

Open PowerShell with administrator privileges and execute the following:

1
2
3
4
5


# Install WSL2 and the latest Ubuntu
wsl --install -d Ubuntu-24.04

# Update the kernel
wsl --update

After installation, run nvidia-smi inside the WSL2 terminal, and if the GPU is recognized correctly, it is a success.

3. Local LLM Architecture and Inference Mechanism

Understanding how models generate text in a local environment and their internal structure is very useful for troubleshooting and optimization.

The following Mermaid diagram shows a typical local LLM inference pipeline.

graph TD User["User Input (Prompt)"] --> Tokenizer["Tokenizer"] Tokenizer --> Embedding["Embedding Layer"] subgraph "Transformer Block (x Layers)" Embedding --> Attn["Self-Attention"] Attn --> KVCache["KV Cache (Key/Value Storage)"] Attn --> FFN["Feed-Forward Network (FFN)"] end FFN --> Logits["Logits Calculation"] Logits --> Sampler["Sampler (Temperature, Top-K, Top-P)"] Sampler --> OutputToken["Output Token"] OutputToken --> |"Autoregressive Generation"| Tokenizer OutputToken --> Decoder["Detokenizer"] Decoder --> FinalOutput["Final Output Text"]

3.1 Two Phases: Prefill and Decode

LLM text generation is divided into two phases with different computational characteristics.

Prefill Phase (Prompt Processing): The phase that processes and understands the entire input prompt at once. Since parallel computing is possible, the computational power of the GPU (FLOPS) directly links to speed. If the prompt is long, this phase can take several seconds.
Decode Phase (Token Generation): The phase that predicts one token at a time and feeds it to the next input (autoregressive). Since parallel computing is restricted in this phase, GPU VRAM bandwidth (Memory Bandwidth) becomes the definitive bottleneck.

4. Mathematical Understanding of VRAM Consumption and Model Size

To correctly determine “which model will run on my PC?”, you need to understand the VRAM calculation formula. When VRAM shortages cause a fallback to system memory (RAM), inference speeds drop by 10x to 100x.

4.1 Base VRAM Based on Parameter Size

This is the amount of memory needed to load the model’s weights into VRAM. Calculate it using the model size $P$ (number of parameters, unit: 1 billion = 1B) and the number of bytes per parameter $B$.

$$ V_{base} = P \times B \quad \text{(GB)} $$

For example, when loading an 8B (8 billion) parameter model in FP16 (half-precision floating point, 16 bits = 2 bytes):

$$ V_{base} = 8 \times 2 = 16 \text{ GB} $$

In other words, even a GPU with 16GB of VRAM will reach its limit just by loading the model.

4.2 The Magic of Quantization

This is where “quantization” comes in. By reducing the precision of the parameters, the model size is drastically shrunk. In the case of the most common 4-bit quantization (e.g., Q4_K_M), it averages to about 0.55 bytes per parameter.

$$ V_{base\_4bit} = 8 \times 0.55 = 4.4 \text{ GB} $$

With this, if you have 16GB of VRAM, you can run an 8B model with plenty of headroom.

4.3 KV Cache Calculation (GQA Supported Version)

During inference, the “KV cache” needed to retain past context consumes VRAM. The latest models, such as Llama 3, use GQA (Grouped Query Attention) to save memory.

The KV cache consumption $V_{kv}$ (in gigabytes) is expressed by the following formula:

$$ V_{kv} = 2 \times b \times s \times l \times \left( \frac{h_{kv}}{h_q} \right) \times h_q \times d \times B_{kv} \div 10^9 $$

Simplifying this using the number of key/value heads $h_{kv}$:

$$ V_{kv} = 2 \times b \times s \times l \times h_{kv} \times d \times B_{kv} \div 10^9 $$

Where:

$b$: Batch size (usually 1 for individual local use)
$s$: Sequence length (context length, e.g., 8192)
$l$: Number of layers (e.g., 32)
$h_{kv}$: Number of KV heads (e.g., 8)
$d$: Number of dimensions per head (e.g., 128)
$B_{kv}$: Number of bytes for KV cache (2 for FP16)

Calculation example (Llama 3 8B, context 8192, FP16 cache): $V_{kv} = 2 \times 1 \times 8192 \times 32 \times 8 \times 128 \times 2 \div 10^9 \approx 1.07 \text{ GB}$

Note that the longer the context length $s$ is, the needed VRAM increases linearly.

5. Practice 1: Fastest and Shortest Setup using Ollama

Now that you understand the theory, let’s actually run an LLM in a Windows environment. As of 2026, the most user-friendly tool is “Ollama”. It provides an intuitive Docker-like CLI.

5.1 Installation and Execution

Download the Windows installer from the Ollama Official Website and run it.
Open PowerShell and enter the following command. Here, we’ll use the Japanese-compatible llama3:8b.

1

ollama run llama3:8b

The model will be downloaded on the first run. Once complete, you can interact with it directly in the terminal.

5.2 Creating a Custom AI with a Modelfile

You can easily create an AI with a specific persona. Create a Modelfile in an arbitrary location.

1
2
3
4
5
6
7
8
9


FROM llama3:8b

SYSTEM """
You are an exceptionally talented senior software engineer.
For user questions, always provide code examples and answer logically and concisely.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Build and run your custom model with the following commands:

1
2


ollama create SeniorDev -f ./Modelfile
ollama run SeniorDev

5.3 Usage from External Apps (AI Editors)

Ollama exposes an OpenAI-compatible API endpoint at http://localhost:11434. By simply setting this URL in the backend settings of VS Code extensions like Cursor or Continue.dev, and specifying the model name such as SeniorDev, a powerful local coding assistant is realized for free.

6. Practice 2: Extreme Performance Tuning with llama.cpp

If you want fine-grained memory management or want to be the first to try out the latest formats (like EXL2 or IQ quantization), you manipulate the core engine llama.cpp directly.

6.1 llama.cpp Build Steps

In a Windows environment, the best approach is building from source using CUDA Toolkit and CMake.

1
2
3
4
5
6
7
8


git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

# Configure for CUDA support and compile
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=OFF
cmake --build . --config Release -j 16

6.2 Advanced Launching in Server Mode

Host the model using the built llama-server.exe.

1
2
3
4
5
6
7


.\bin\Release\llama-server.exe `
 --model "C:\models\Llama-3-8B-Instruct.Q4_K_M.gguf" `
 --ctx-size 8192 `
 --n-gpu-layers 99 `
 --threads 8 `
 --flash-attn `
 --port 8080

--n-gpu-layers 99: Offloads all possible layers to GPU VRAM.
--flash-attn: Enables FlashAttention 3, achieving improved inference speed and reduced VRAM consumption for the KV cache.

7. GUI Frontend: LM Studio and Building Local RAG

If you’re resistant to the command line, or intuitively want to perform RAG (Retrieval-Augmented Generation), you can use a GUI.

7.1 LM Studio

LM Studio is a brilliant application that bundles model search, downloading, system requirement pre-checks, and a chat UI all into one. Just by pressing the “Local Server” button in the app, an OpenAI-compatible API starts up.

7.2 RAG Architecture using AnythingLLM

Here is the architecture diagram of a RAG environment for reading internal documents and personal notes.

graph LR Document["Document (PDF, MD)"] --> Chunking["Chunking"] Chunking --> EmbedModel["Embedding Model"] EmbedModel --> VectorDB["Vector Database"] UserQuery["User Query"] --> EmbedQuery["Query Embedding"] EmbedQuery --> VectorDB VectorDB --> |"Similarity Search"| RetrievedDocs["Extract Relevant Docs"] UserQuery --> PromptBuilder["Prompt Generation"] RetrievedDocs --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer"]

Using the AnythingLLM desktop version (Windows), just specify Ollama (LLM and Embedding) from the settings screen and set it up to use a local VectorDB (LanceDB). This architecture can be completed in minutes. A private AI is born that does not send any data externally.

8. Fine-Tuning (LoRA) on Windows WSL2

If you want to not just run locally but make the model smarter with your own data, fine-tuning using LoRA (Low-Rank Adaptation) is possible. As of 2026, by using a library called “Unsloth”, an 8B model can finish training in a few hours on a Windows WSL2 environment even with 16GB of VRAM.

Execute the following within WSL2’s Ubuntu to build the environment.

1
2
3
4


conda create --name unsloth_env python=3.11
conda activate unsloth_env
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Unsloth optimizes CUDA kernels to the extreme, providing about twice the training speed and half the VRAM consumption compared to the standard Hugging Face libraries. Just spin up a Jupyter Notebook and load your dataset (JSONL format), and training for several epochs is possible even on an RTX 4060 Ti with 12GB to 16GB of VRAM.

9. Performance Troubleshooting

Common problems faced and their solutions.

1. Inference speed is extremely slow (1-2 tokens/s)

Cause: The model doesn’t fit entirely into VRAM and is being offloaded to system memory (RAM). Solution: Check “Dedicated GPU memory” in the Task Manager. If it’s hitting the limit, decrease the context size (-c), or use a model with lower bit quantization (like Q4_K_M).

2. “CUDA out of memory” error

Cause: VRAM has been completely exhausted. This occurs especially when the context is prolonged and the KV cache becomes bloated. Solution: Intentionally restrict the values to smaller ones using num_ctx for Ollama, or -c for llama.cpp.

3. Strange Japanese Generation

Cause: Mismatch in prompt templates, or an unsupported model. Solution: Use models that include Instruct in their name, and ensure that the tool is selecting the correct template specified by the model author, such as the ChatML or Llama3 format.

10. Conclusion and Future Prospects

In 2026, building a local LLM in a Windows environment is no longer the privilege of a limited number of engineers. With the de facto standardization of the GGUF format, the emergence of refined ecosystems like Ollama and LM Studio, and hardware optimizations led by FlashAttention, anyone can easily obtain an enterprise-grade AI environment.

Please make use of the following points explained in this article:

Use mathematical VRAM calculations to logically select the optimal model size and quantization level for your PC specs.
Build your environment at maximum speed using Ollama, and dramatically improve productivity by integrating it with AI editors.
Bring out the ultimate performance of your hardware with the advanced parameter control of llama.cpp.
Build a secure local RAG system to handle confidential data with AnythingLLM.
Nurture a custom AI with your own specialized knowledge by utilizing Unsloth (WSL2).

The “democratization” of AI is no longer a buzzword, but a real system running on your Windows desktop. Free yourself from the usage costs of cloud APIs and information leak risks, and step into the world of free and powerful private AI right now.