Windows on kenji.blog

Latest API Usage Examples and Sample Code for Microsoft.Windows.AI

Fri, 11 Sep 2026 18:00:00 +0900

Latest API Usage Examples and Sample Code for Microsoft.Windows.AI: Exploring the Depths of Windows Copilot Runtime

1. Introduction: A New Era of Windows with Natively Integrated AI

In recent years, AI technology has evolved remarkably, resulting in a rapid paradigm shift from the utilization of Large Language Models (LLMs) on the cloud to AI inference on edge devices (local PCs). At the core of this are the “Windows Copilot Runtime” provided by Microsoft for Windows 11, and the “Microsoft.Windows.AI” APIs designed to operate it.

Application development using cloud APIs (such as OpenAI and Azure OpenAI) is easy, but it comes with challenges such as latency, privacy, and continuous costs. On the other hand, running AI models locally allows for the realization of ultra-low latency applications that work even offline without letting sensitive data leave the device.

In this article, we will thoroughly explain in extreme detail the implementation methods for local AI features, which will be essential for future Windows application development, along with practical sample code in C# and C++. We will cover everything from architecture to performance tuning. We won’t just hit the APIs, but will delve deep into the advanced technical details such as utilizing the underlying hardware (NPU and GPU) and integrating with DirectML.

2. Overall Architecture of Windows Copilot Runtime

Windows Copilot Runtime is a set of AI stacks designed so that developers can easily integrate AI models on Windows and extract maximum performance. This runtime abstracts hardware acceleration at the OS level and provides developers with a unified interface.

graph TD App["Windows Application (C# / C++)"] --> API["Microsoft.Windows.AI APIs"] App --> ORT["ONNX Runtime"] API --> WCR["Windows Copilot Runtime (OS Layer)"] WCR --> SLM["Local Models (Phi-Silica, etc.)"] ORT --> DML["DirectML Execution Provider"] SLM --> DML DML --> DXCore["DXCore / DirectX 12"] DXCore --> NPU["NPU (Neural Processing Unit)"] DXCore --> GPU["GPU (Graphics Processing Unit)"] DXCore --> CPU["CPU"]

As the architecture diagram above shows, applications can directly access small language models built into the OS (SLMs: such as Phi-Silica) by utilizing the high-level Microsoft.Windows.AI APIs. Additionally, when using custom models, it is also possible to explicitly use hardware acceleration via ONNX Runtime and DirectML. Since the OS layer optimizes the distribution of workloads to the CPU, GPU, and NPU, developers can build high-performance AI apps without being deeply conscious of hardware differences.

3. Hardware Acceleration and Mathematical Evaluation of NPUs

The latest Copilot+ PCs are equipped with NPUs (Neural Processing Units), processors specialized for AI processing. The performance of an NPU is generally evaluated in TOPS (Tera Operations Per Second).

In AI model inference, the computational capability of GEMM (General Matrix Multiply) in particular determines the throughput. The theoretical peak performance $P_{\text{peak}}$ of hardware is estimated by the following formula:

$$ P_{\text{peak}} = f \times N_{\text{cores}} \times N_{\text{MACs/core}} \times 2 $$

Here,

$f$ is the NPU clock frequency (Hz)
$N_{\text{cores}}$ is the number of cores in the NPU
$N_{\text{MACs/core}}$ is the number of MAC (Multiply-Accumulate) units per core
The final $2$ is because one MAC operation counts as two operations (FLOPs/OPs): multiplication and addition.

For example, for an NPU with a frequency of 1.5GHz, 4 cores, and 4096 MACs per core:

$$ P_{\text{peak}} = 1.5 \times 10^9 \times 4 \times 4096 \times 2 \approx 49.15 \text{ TOPS} $$

This mathematically demonstrates a performance that clears the 40 TOPS requirement for Windows 11 Copilot+ PCs.

Furthermore, AI model inference (the decode phase), especially for LLMs, tends to be Memory-Bound. The theoretical bandwidth $BW$ of system memory is calculated as follows:

$$ BW = f_{\text{mem}} \times W_{\text{bus}} \times \frac{2}{8} $$

In the case of LPDDR5x-8533 memory ($f_{\text{mem}} = 8533 \text{ MT/s}$) and a 128-bit bus ($W_{\text{bus}} = 128$), the bandwidth is approximately $136 \text{ GB/s}$. In AI application optimization, how to conserve this bandwidth is crucial, and the model Quantization discussed later becomes essential.

4. Development Environment Setup

To use the latest Windows AI APIs, you need to prepare the following environment and toolchain:

OS: Windows 11 version 24H2 or later (strongly recommended to have an NPU-equipped device meeting the Copilot+ PC requirements)
SDK: Windows App SDK (v1.5 or later, AI extension supported version)
Development Environment: Visual Studio 2022 (v17.10 or later), with the “Desktop development with C++” and “.NET desktop development” workloads
Packages: Install Microsoft.Windows.AI and Microsoft.ML.OnnxRuntime.DirectML via NuGet

1
2
3
4
5
6


<!-- Example .csproj configuration -->
<ItemGroup>
 <PackageReference Include="Microsoft.Windows.AI" Version="1.0.0-preview1" />
 <PackageReference Include="Microsoft.WindowsAppSDK" Version="1.5.240311000" />
 <PackageReference Include="Microsoft.ML.OnnxRuntime.DirectML" Version="1.17.1" />
</ItemGroup>

5. [Deep Dive 1] Utilizing Local Language Models (Phi-Silica) with C#

Windows Copilot Runtime natively incorporates the highly efficient small language model “Phi-Silica” developed by Microsoft as a standard OS component. This enables advanced natural language processing (text summarization, code generation, chatbots) in offline environments without downloading gigabyte-sized models from the network.

Below is an advanced C# sample code to build an AI chatbot using the Microsoft.Windows.AI.Generative namespace. It supports streaming responses and generates text in real-time without blocking the UI thread.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84


using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Windows.AI.Generative;

namespace WindowsAI.Sample
{
 public class LocalLanguageModelService
 {
 private LanguageModel _languageModel;
 private bool _isInitialized = false;

 /// <summary>
 /// Initializes the language model. Checks NPU availability and loads the model onto the optimal device.
 /// </summary>
 public async Task InitializeAsync()
 {
 if (_isInitialized) return;

 Console.WriteLine("Checking system requirements and availability for the local AI model (Phi-Silica)...");

 // Check if the model is available on the system (if unsupported, a download might be prompted)
 var availability = await LanguageModel.CheckAvailabilityAsync();
 if (availability != LanguageModelAvailability.Available)
 {
 throw new InvalidOperationException($"Local AI model is currently unavailable. Status: {availability}");
 }

 // Create a model instance (mapping to memory space and NPU initialization happen at this time)
 _languageModel = await LanguageModel.CreateAsync();
 _isInitialized = true;

 Console.WriteLine("Language model initialization completed. Hardware acceleration via DirectML is active.");
 }

 /// <summary>
 /// Receives user prompt and generates response stream.
 /// </summary>
 public async Task GenerateResponseStreamAsync(string prompt, CancellationToken cancellationToken)
 {
 if (!_isInitialized) await InitializeAsync();

 Console.WriteLine($"\n[User Input]: {prompt}\n[AI Assistant]: ");

 // Set generation hyperparameters
 var options = new LanguageModelOptions
 {
 Temperature = 0.7f,
 TopP = 0.9f,
 MaxTokens = 2048,
 RepetitionPenalty = 1.1f
 };

 // Construct conversation context
 var context = new LanguageModelContext();
 context.AddSystemMessage("You are an advanced AI assistant running directly on a Windows local NPU. Think logically step-by-step and answer concisely.");
 context.AddUserMessage(prompt);

 try
 {
 // Call the streaming inference API
 var responseStream = _languageModel.GenerateResponseStreamAsync(context, options);

 // Asynchronously iterate over the chunks returned as IAsyncEnumerable
 await foreach (var chunk in responseStream.WithCancellation(cancellationToken))
 {
 // Output generated tokens (chunks) to the console in real-time
 // For UI apps, reflect this to a TextBox, etc. using DispatcherQueue here
 Console.Write(chunk.Text);
 }
 Console.WriteLine();
 }
 catch (OperationCanceledException)
 {
 Console.WriteLine("\n[Generation was cancelled by the user or system]");
 }
 catch (Exception ex)
 {
 Console.WriteLine($"\n[A fatal error occurred: {ex.Message}]");
 }
 }
 }
}

5.1 Explanation of the C# Architecture

The core of this code lies in the pre-execution validation using LanguageModel.CheckAvailabilityAsync() and the asynchronous streaming via GenerateResponseStreamAsync. When the Copilot Runtime running in the OS background receives this API call, it internally launches the ONNX Runtime and selects the optimal Execution Provider (DirectML + NPU on many modern PCs) according to the system configuration.

Developers can integrate cutting-edge AI inference pipelines into their applications with just a few lines of C# code, without needing to be aware of the model’s tensor shapes, tokenizer implementations, or memory management of KV caches at all.

6. [Deep Dive 2] High-Speed Custom Model Inference using C++ and DirectML

When dealing with specific domains (such as unique image segmentation, speech recognition, custom object detection models, etc.) that cannot be covered by the OS standard language models alone, developers need to directly operate ONNX Runtime and DirectML, which are located at a lower layer of Microsoft.Windows.AI.

By using C++, memory allocation can be optimized to the limit, bringing out the peak performance of the NPU/GPU. Below is a core implementation of an advanced initialization and inference pipeline for executing ONNX format custom models (e.g., YOLOv8) via DirectML in C++.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99


#include <iostream>
#include <vector>
#include <string>
#include <stdexcept>
#include <onnxruntime_cxx_api.h>
#include <dml_provider_factory.h>

class CustomVisionAIProcessor {
private:
 Ort::Env env;
 Ort::Session session{nullptr};
 Ort::AllocatorWithDefaultOptions allocator;

public:
 CustomVisionAIProcessor(const std::wstring& modelPath)
 : env(ORT_LOGGING_LEVEL_WARNING, "VisionAIProcessor") {

 Ort::SessionOptions sessionOptions;
 // Optimize the number of threads
 sessionOptions.SetIntraOpNumThreads(1);
 // Set graph optimization level to maximum
 sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);

 // 1. Add DirectML Execution Provider (DML EP)
 // device_id = 0 is the system's default recommended adapter (NPU or high-performance GPU)
 const OrtApi& ortApi = Ort::GetApi();
 OrtDmlApi* dmlApi = nullptr;
 OrtStatus* status = ortApi.GetExecutionProviderApi("DML", ORT_API_VERSION, reinterpret_cast<void**>(&dmlApi));

 if (status == nullptr && dmlApi != nullptr) {
 Ort::ThrowOnError(dmlApi->SessionOptionsAppendExecutionProvider_DML(sessionOptions, 0));
 std::cout << "[Info] Successfully attached DirectML Execution Provider." << std::endl;
 } else {
 std::cerr << "[Warning] Failed to retrieve DirectML API. Running in CPU fallback mode." << std::endl;
 if (status != nullptr) ortApi.ReleaseStatus(status);
 }

 // 2. Load model and create session
 try {
 session = Ort::Session(env, modelPath.c_str(), sessionOptions);
 std::cout << "[Info] ONNX model successfully loaded and computation graph compiled." << std::endl;
 } catch (const Ort::Exception& e) {
 std::cerr << "[Error] Failed to load model: " << e.what() << std::endl;
 throw;
 }
 }

 void RunInference(const std::vector<float>& imageTensor, const std::vector<int64_t>& inputShape) {
 // 3. Create input tensor buffer
 Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
 Ort::Value inputTensor = Ort::Value::CreateTensor<float>(
 memoryInfo,
 const_cast<float*>(imageTensor.data()),
 imageTensor.size(),
 inputShape.data(),
 inputShape.size()
 );

 // 4. Dynamically retrieve input/output node names
 auto inputNamePtr = session.GetInputNameAllocated(0, allocator);
 auto outputNamePtr = session.GetOutputNameAllocated(0, allocator);
 const char* inputNames[] = { inputNamePtr.get() };
 const char* outputNames[] = { outputNamePtr.get() };

 // 5. Execute inference (offloaded to NPU/GPU via DirectML)
 std::cout << "[Info] Starting inference engine execution..." << std::endl;
 auto startTime = std::chrono::high_resolution_clock::now();

 auto outputTensors = session.Run(Ort::RunOptions{nullptr}, inputNames, &inputTensor, 1, outputNames, 1);

 auto endTime = std::chrono::high_resolution_clock::now();
 auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime);

 // 6. Retrieve and analyze result tensor
 float* outputData = outputTensors.front().GetTensorMutableData<float>();
 size_t outputSize = outputTensors.front().GetTensorTypeAndShapeInfo().GetElementCount();

 std::cout << "[Result] Inference completed: " << duration.count() << " ms" << std::endl;
 std::cout << "[Result] Output tensor element count: " << outputSize << std::endl;
 // *After this, implement NMS (Non-Maximum Suppression) or bounding box drawing processing for the output tensor
 }
};

int main() {
 try {
 // ONNX model path to execute
 CustomVisionAIProcessor processor(L"models/yolov8n_quantized.onnx");

 // Dummy image data for inference (Batch size 1 x 3 channels x 640 x 640)
 std::vector<int64_t> inputShape = {1, 3, 640, 640};
 std::vector<float> dummyImage(1 * 3 * 640 * 640, 0.5f);

 processor.RunInference(dummyImage, inputShape);
 } catch (const std::exception& e) {
 std::cerr << "Program terminated abnormally: " << e.what() << std::endl;
 return -1;
 }
 return 0;
}

6.1 The Importance of Memory Management and Zero-Copy Inference in C++

The greatest advantage of using DirectML in C++ is that tight integration with DirectX 12 (DX12) is possible. The above code includes standard data copying from CPU memory for educational purposes, but in actual game engines and video processing applications, there are many cases where images (textures) are already held in the GPU or NPU memory space using DX12.

In this case, by utilizing the advanced binding features of OrtDmlApi, you can achieve “Zero-Copy Inference”, which directly maps DX12 resources as ONNX Runtime tensors. By doing this, the data transfer overhead across the PCIe bus (consumption of the aforementioned bandwidth $BW$) completely vanishes, dramatically improving the frame rate in real-time video processing.

7. Performance Optimization and Best Practices

Below, we summarize essential optimization strategies for developing top-tier AI applications leveraging Windows AI APIs and DirectML.

7.1 Model Quantization and the Olive Toolkit

To unleash the true power of an NPU, it is an absolute requirement to Quantize the weights and activations of AI models from FP32 (single-precision floating-point) to INT8 or INT4. The NPU architecture is specialized for integer arithmetic, theoretically achieving 4x the throughput and massive power savings with INT8 compared to FP32.

By using the Olive (ONNX Live) toolchain provided by Microsoft, you can automatically optimize models like PyTorch for the Windows environment. Olive strongly supports specific attention optimizations for Transformer models and hardware-specific graph compilation.

7.2 The Trade-off: Batch Processing vs. Interactive Streaming

In API calls, by batching multiple inference requests together, you can increase NPU utilization (Compute Utilization). However, for interactive UIs like chatbots, the time until the first token is displayed (TTFT: Time To First Token) dictates the user experience (UX) more than throughput. Therefore, for interactive UIs, the best practice is to set the batch size to 1 and design to prioritize streaming generation.

7.3 Integration with Background Tasks and the OS

Local AI inference consumes significant power and system resources. Integrating with the Windows App Lifecycle API is required to suspend or throttle the resource consumption of low-priority inference tasks when the application moves to the background.

sequenceDiagram participant User as "User" participant App as "Windows App (C#)" participant API as "Microsoft.Windows.AI" participant OS as "Windows Copilot Runtime" participant ORT as "ONNX Runtime (DML)" participant NPU as "NPU Hardware" User->>App: "Input Prompt" App->>API: "GenerateResponseStreamAsync()" API->>OS: "Dispatch Inference Job" OS->>ORT: "Graph Execution Request" ORT->>NPU: "Execute Command List via DirectML" NPU-->>ORT: "Computation Completed (1 Token Generated)" ORT-->>OS: "Tensor Result" OS-->>API: "Decoded Text" API-->>App: "IAsyncEnumerable Chunks" App-->>User: "Real-time Text Rendering to UI" Note over ORT,NPU: "Rapidly repeat this loop until completion"

This sequence diagram illustrates the beauty of asynchronous processing where data streams continuously from the lowest layer NPU hardware to the presentation layer of the application, without ever blocking the UI thread.

8. Future Outlook and Evolution of Windows AI

The Microsoft.Windows.AI APIs and Copilot Runtime are undergoing rapid evolution. Future updates for developers are expected to bring paradigm shifts such as the following:

OS Native Integration of Multimodal APIs: Seamlessly process text, audio, images, and even live video feeds simultaneously, providing cross-modal AI inference natively at the OS level.
System-level Support for RAG (Retrieval-Augmented Generation): Connecting a personal collection of documents within a local PC or Windows Search indices with AI models within a secure OS sandbox, allowing the construction of ultra-advanced personal AI assistants while completely protecting user privacy.
Dynamic Resource Scaling of NPUs: A mechanism where the Windows kernel scheduler dynamically switches the NPU execution context to guarantee QoS (Quality of Service) when multiple AI applications (e.g., background noise cancellation and foreground code generation) are running concurrently.

9. Conclusion: The Future of Applications Transformed by Local AI

The Windows 11 Copilot Runtime and Microsoft.Windows.AI APIs have brought an extremely powerful weapon called “Local AI” to all Windows developers. You no longer need to rely entirely on cloud APIs. It is possible to eliminate latency, firmly protect privacy, and deliver next-generation AI experiences to users that fully operate even offline.

By utilizing the knowledge of system standard language model integration using C#, mathematical performance evaluation, and extreme hardware optimization using C++ and DirectML explained in this article, please create next-generation “AI-native” Windows applications with your own hands. The boundless possibilities brought by AI stretch right out ahead of the code you write.

※Note: This article is written based on the preview version APIs and the latest specifications as of September 2026. Because API specifications and hardware requirements may change due to Windows updates, always refer to the official Microsoft Learn documentation when implementing.

【2026 Edition】 The Complete Guide to Running Local LLMs in a Windows Environment

Fri, 11 Sep 2026 10:00:00 +0900

1. Introduction: Why Local LLMs on Windows Now?

As of 2026, the evolution of generative AI and Large Language Models (LLMs) shows a major paradigm shift from gigantic cloud-based API services to “local LLMs” running on personal PCs and on-premise environments. While cloud AIs like OpenAI’s GPT-5 and Anthropic’s Claude 3.5 are incredibly powerful, not all companies and individuals can send all their data to the cloud. From the perspectives of privacy, security, latency, and long-term sustainable costs, the demand for local LLMs is exploding like never before.

The evolution of the local LLM ecosystem, especially in the Windows environment, is remarkable. Until a few years ago, “Linux for AI development and execution” was common sense, but as of 2026, Windows has transformed into an extremely powerful and accessible AI platform.

In this article, based on the latest technology trends of 2026, we provide a complete guide to building, operating, and optimizing local LLMs in a Windows environment. From easy setup using Ollama for beginners to extreme optimization using llama.cpp for advanced users, and further deep dives into the mathematical approach of VRAM calculation, deep understanding of the architecture, and local fine-tuning, we will explain everything thoroughly with an overwhelming volume.

1.1 Technology Trends Surrounding Local LLMs in 2026

The major trends shaping the current local LLM ecosystem are as follows:

Complete Popularization of the GGUF Format: GGUF (GPT-Generated Unified Format), which integrates metadata and tensors into a single file, has completely become the de facto standard. With this, simply downloading a single file from Hugging Face makes it executable in any environment.
Democratization of the MoE (Mixture of Experts) Architecture: Many small but high-performance MoE models have been released. By activating only a portion of the experts during inference, they achieve performance comparable to giant models while keeping the computational load on consumer PCs low.
Advanced Abstraction and Optimization of Inference Engines: Tools like Ollama, LM Studio, and AnythingLLM have been refined so that users no longer need to be aware of complex dependencies like CUDA driver installations. Also, the native Windows support for FlashAttention 3 has dramatically improved inference speed.
Utilization of NPUs and the Rise of Windows Copilot+ PCs: Even on laptops without GPUs, the technology to run small LLMs (SLM: Small Language Models) with low power consumption using the built-in NPU (Neural Processing Unit) has entered the practical stage.

2. Hardware Requirements and OS Preparation

To run local LLMs at practical speeds (15-30 tokens per second or more), selecting the right hardware is the most important factor.

2.1 Recommended Hardware Configuration

With the evolution of AI PCs, required specs are also changing.

OS: Windows 11 Pro (24H2 or later). Essential for fully utilizing WSL2’s features, advanced memory management, and the latest DirectML APIs.
CPU: Intel Core Ultra 200 series or higher, or AMD Ryzen 9000 series or higher. When using CPU inference alongside, broad-bandwidth memory communication is indispensable.
RAM: Minimum 32GB, recommended 64GB or more. Main memory bandwidth (MB/s) becomes a crucial bottleneck during CPU inference or offloading. High-speed memory of DDR5-6000 or above is ideal.
GPU: NVIDIA RTX 4000/5000 series. The most important thing for local LLMs is not computing performance but “VRAM capacity”.
- Entry: RTX 4060 Ti (16GB version) - Best cost performance. Ideal for 8B-14B class models.
- Mid-range: RTX 4070 Ti SUPER (16GB) / RTX 4080 SUPER (16GB)
- High-end: RTX 4090 (24GB) / RTX 5090 (32GB) - Necessary to run 30B-70B class quantized models.
Storage: PCIe Gen4 or Gen5 NVMe SSD. Dramatically reduces the load times of models that are tens of gigabytes in size.

2.2 Setting up WSL2 (Windows Subsystem for Linux 2)

While many GUI tools work natively on Windows, WSL2 is extremely useful for Python development, compiling the latest tools, and the LoRA fine-tuning mentioned later. In the latest Windows 11 environment, just by installing the NVIDIA driver on the host side, the GPU (CUDA) can be transparently used from WSL2.

Open PowerShell with administrator privileges and execute the following:

1
2
3
4
5


# Install WSL2 and the latest Ubuntu
wsl --install -d Ubuntu-24.04

# Update the kernel
wsl --update

After installation, run nvidia-smi inside the WSL2 terminal, and if the GPU is recognized correctly, it is a success.

3. Local LLM Architecture and Inference Mechanism

Understanding how models generate text in a local environment and their internal structure is very useful for troubleshooting and optimization.

The following Mermaid diagram shows a typical local LLM inference pipeline.

graph TD User["User Input (Prompt)"] --> Tokenizer["Tokenizer"] Tokenizer --> Embedding["Embedding Layer"] subgraph "Transformer Block (x Layers)" Embedding --> Attn["Self-Attention"] Attn --> KVCache["KV Cache (Key/Value Storage)"] Attn --> FFN["Feed-Forward Network (FFN)"] end FFN --> Logits["Logits Calculation"] Logits --> Sampler["Sampler (Temperature, Top-K, Top-P)"] Sampler --> OutputToken["Output Token"] OutputToken --> |"Autoregressive Generation"| Tokenizer OutputToken --> Decoder["Detokenizer"] Decoder --> FinalOutput["Final Output Text"]

3.1 Two Phases: Prefill and Decode

LLM text generation is divided into two phases with different computational characteristics.

Prefill Phase (Prompt Processing): The phase that processes and understands the entire input prompt at once. Since parallel computing is possible, the computational power of the GPU (FLOPS) directly links to speed. If the prompt is long, this phase can take several seconds.
Decode Phase (Token Generation): The phase that predicts one token at a time and feeds it to the next input (autoregressive). Since parallel computing is restricted in this phase, GPU VRAM bandwidth (Memory Bandwidth) becomes the definitive bottleneck.

4. Mathematical Understanding of VRAM Consumption and Model Size

To correctly determine “which model will run on my PC?”, you need to understand the VRAM calculation formula. When VRAM shortages cause a fallback to system memory (RAM), inference speeds drop by 10x to 100x.

4.1 Base VRAM Based on Parameter Size

This is the amount of memory needed to load the model’s weights into VRAM. Calculate it using the model size $P$ (number of parameters, unit: 1 billion = 1B) and the number of bytes per parameter $B$.

$$ V_{base} = P \times B \quad \text{(GB)} $$

For example, when loading an 8B (8 billion) parameter model in FP16 (half-precision floating point, 16 bits = 2 bytes):

$$ V_{base} = 8 \times 2 = 16 \text{ GB} $$

In other words, even a GPU with 16GB of VRAM will reach its limit just by loading the model.

4.2 The Magic of Quantization

This is where “quantization” comes in. By reducing the precision of the parameters, the model size is drastically shrunk. In the case of the most common 4-bit quantization (e.g., Q4_K_M), it averages to about 0.55 bytes per parameter.

$$ V_{base\_4bit} = 8 \times 0.55 = 4.4 \text{ GB} $$

With this, if you have 16GB of VRAM, you can run an 8B model with plenty of headroom.

4.3 KV Cache Calculation (GQA Supported Version)

During inference, the “KV cache” needed to retain past context consumes VRAM. The latest models, such as Llama 3, use GQA (Grouped Query Attention) to save memory.

The KV cache consumption $V_{kv}$ (in gigabytes) is expressed by the following formula:

$$ V_{kv} = 2 \times b \times s \times l \times \left( \frac{h_{kv}}{h_q} \right) \times h_q \times d \times B_{kv} \div 10^9 $$

Simplifying this using the number of key/value heads $h_{kv}$:

$$ V_{kv} = 2 \times b \times s \times l \times h_{kv} \times d \times B_{kv} \div 10^9 $$

Where:

$b$: Batch size (usually 1 for individual local use)
$s$: Sequence length (context length, e.g., 8192)
$l$: Number of layers (e.g., 32)
$h_{kv}$: Number of KV heads (e.g., 8)
$d$: Number of dimensions per head (e.g., 128)
$B_{kv}$: Number of bytes for KV cache (2 for FP16)

Calculation example (Llama 3 8B, context 8192, FP16 cache): $V_{kv} = 2 \times 1 \times 8192 \times 32 \times 8 \times 128 \times 2 \div 10^9 \approx 1.07 \text{ GB}$

Note that the longer the context length $s$ is, the needed VRAM increases linearly.

5. Practice 1: Fastest and Shortest Setup using Ollama

Now that you understand the theory, let’s actually run an LLM in a Windows environment. As of 2026, the most user-friendly tool is “Ollama”. It provides an intuitive Docker-like CLI.

5.1 Installation and Execution

Download the Windows installer from the Ollama Official Website and run it.
Open PowerShell and enter the following command. Here, we’ll use the Japanese-compatible llama3:8b.

1

ollama run llama3:8b

The model will be downloaded on the first run. Once complete, you can interact with it directly in the terminal.

5.2 Creating a Custom AI with a Modelfile

You can easily create an AI with a specific persona. Create a Modelfile in an arbitrary location.

1
2
3
4
5
6
7
8
9


FROM llama3:8b

SYSTEM """
You are an exceptionally talented senior software engineer.
For user questions, always provide code examples and answer logically and concisely.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Build and run your custom model with the following commands:

1
2


ollama create SeniorDev -f ./Modelfile
ollama run SeniorDev

5.3 Usage from External Apps (AI Editors)

Ollama exposes an OpenAI-compatible API endpoint at http://localhost:11434. By simply setting this URL in the backend settings of VS Code extensions like Cursor or Continue.dev, and specifying the model name such as SeniorDev, a powerful local coding assistant is realized for free.

6. Practice 2: Extreme Performance Tuning with llama.cpp

If you want fine-grained memory management or want to be the first to try out the latest formats (like EXL2 or IQ quantization), you manipulate the core engine llama.cpp directly.

6.1 llama.cpp Build Steps

In a Windows environment, the best approach is building from source using CUDA Toolkit and CMake.

1
2
3
4
5
6
7
8


git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

# Configure for CUDA support and compile
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=OFF
cmake --build . --config Release -j 16

6.2 Advanced Launching in Server Mode

Host the model using the built llama-server.exe.

1
2
3
4
5
6
7


.\bin\Release\llama-server.exe `
 --model "C:\models\Llama-3-8B-Instruct.Q4_K_M.gguf" `
 --ctx-size 8192 `
 --n-gpu-layers 99 `
 --threads 8 `
 --flash-attn `
 --port 8080

--n-gpu-layers 99: Offloads all possible layers to GPU VRAM.
--flash-attn: Enables FlashAttention 3, achieving improved inference speed and reduced VRAM consumption for the KV cache.

7. GUI Frontend: LM Studio and Building Local RAG

If you’re resistant to the command line, or intuitively want to perform RAG (Retrieval-Augmented Generation), you can use a GUI.

7.1 LM Studio

LM Studio is a brilliant application that bundles model search, downloading, system requirement pre-checks, and a chat UI all into one. Just by pressing the “Local Server” button in the app, an OpenAI-compatible API starts up.

7.2 RAG Architecture using AnythingLLM

Here is the architecture diagram of a RAG environment for reading internal documents and personal notes.

graph LR Document["Document (PDF, MD)"] --> Chunking["Chunking"] Chunking --> EmbedModel["Embedding Model"] EmbedModel --> VectorDB["Vector Database"] UserQuery["User Query"] --> EmbedQuery["Query Embedding"] EmbedQuery --> VectorDB VectorDB --> |"Similarity Search"| RetrievedDocs["Extract Relevant Docs"] UserQuery --> PromptBuilder["Prompt Generation"] RetrievedDocs --> PromptBuilder PromptBuilder --> LocalLLM["Local LLM"] LocalLLM --> Answer["Final Answer"]

Using the AnythingLLM desktop version (Windows), just specify Ollama (LLM and Embedding) from the settings screen and set it up to use a local VectorDB (LanceDB). This architecture can be completed in minutes. A private AI is born that does not send any data externally.

8. Fine-Tuning (LoRA) on Windows WSL2

If you want to not just run locally but make the model smarter with your own data, fine-tuning using LoRA (Low-Rank Adaptation) is possible. As of 2026, by using a library called “Unsloth”, an 8B model can finish training in a few hours on a Windows WSL2 environment even with 16GB of VRAM.

Execute the following within WSL2’s Ubuntu to build the environment.

1
2
3
4


conda create --name unsloth_env python=3.11
conda activate unsloth_env
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Unsloth optimizes CUDA kernels to the extreme, providing about twice the training speed and half the VRAM consumption compared to the standard Hugging Face libraries. Just spin up a Jupyter Notebook and load your dataset (JSONL format), and training for several epochs is possible even on an RTX 4060 Ti with 12GB to 16GB of VRAM.

9. Performance Troubleshooting

Common problems faced and their solutions.

1. Inference speed is extremely slow (1-2 tokens/s)

Cause: The model doesn’t fit entirely into VRAM and is being offloaded to system memory (RAM). Solution: Check “Dedicated GPU memory” in the Task Manager. If it’s hitting the limit, decrease the context size (-c), or use a model with lower bit quantization (like Q4_K_M).

2. “CUDA out of memory” error

Cause: VRAM has been completely exhausted. This occurs especially when the context is prolonged and the KV cache becomes bloated. Solution: Intentionally restrict the values to smaller ones using num_ctx for Ollama, or -c for llama.cpp.

3. Strange Japanese Generation

Cause: Mismatch in prompt templates, or an unsupported model. Solution: Use models that include Instruct in their name, and ensure that the tool is selecting the correct template specified by the model author, such as the ChatML or Llama3 format.

10. Conclusion and Future Prospects

In 2026, building a local LLM in a Windows environment is no longer the privilege of a limited number of engineers. With the de facto standardization of the GGUF format, the emergence of refined ecosystems like Ollama and LM Studio, and hardware optimizations led by FlashAttention, anyone can easily obtain an enterprise-grade AI environment.

Please make use of the following points explained in this article:

Use mathematical VRAM calculations to logically select the optimal model size and quantization level for your PC specs.
Build your environment at maximum speed using Ollama, and dramatically improve productivity by integrating it with AI editors.
Bring out the ultimate performance of your hardware with the advanced parameter control of llama.cpp.
Build a secure local RAG system to handle confidential data with AnythingLLM.
Nurture a custom AI with your own specialized knowledge by utilizing Unsloth (WSL2).

The “democratization” of AI is no longer a buzzword, but a real system running on your Windows desktop. Free yourself from the usage costs of cloud APIs and information leak risks, and step into the world of free and powerful private AI right now.