Society on kenji.blog

Can Technology Bridge the Social Divide? (A Proposal from an Engineer)

Sat, 12 Sep 2026 12:00:00 +0900

Introduction: On the Occasion of My Memorable 100th Post

It has been several years since I started this blog, during which I have repeatedly shared technical explanations, daily development memos, and occasionally reflections on the relationship between technology and society. And now, this article marks the memorable “100th” post. I would like to express my deepest gratitude to all the readers who have continued to read up to this point.

On this milestone of the 100th post, there is a theme I desperately wanted to write down. That is the extremely significant and fundamental question in modern society: “Can technology bridge the social divide?”

The early internet (Web 1.0) was spoken of as a utopia of “democratization of knowledge,” where anyone could freely broadcast and access information. The subsequent era of social media (Web 2.0) was supposed to connect people around the world and realize a “flat world.” However, as of 2026, what is the reality we face? Political polarization, the spread of conspiracy theories, the proliferation of fake news, and the formation of “echo chambers” and “filter bubbles” that reject mutual understanding. Rather than connecting people, technology seems to have become a powerful engine accelerating Social Divide.

We engineers are not just entities who write code and build systems. Behind the architectures we design, the algorithms we select, and the objective functions we optimize, lie the “rules” that govern how society should be. In this article, from the perspective of an engineer, I would like to deeply explore and discuss how the current social divide is technically generated using mathematics and network theory, and simultaneously, the concrete technical approaches (bridging algorithms, decentralized SNS protocols) to overcome it.

Chapter 1: The Mathematical Structure of “Echo Chambers” through the Lens of Network Theory

When discussing the social divide, the structural analysis of communities using “Network Theory (Graph Theory)” is an unavoidable starting point. Human relationships on social media can be modeled as a giant graph where users are “nodes (vertices)” and follows or interactions between users are “edges.”

One of the most important metrics characterizing division is the “Clustering Coefficient.” The clustering coefficient $C_i$ of a certain user $i$ represents the probability that the friends of user $i$ are also friends with each other, and is defined by the following formula:

$$ C_i = \frac{2e_i}{k_i(k_i - 1)} $$

Here, $k_i$ is the degree (number of friends) of user $i$, and $e_i$ is the actual number of edges existing among those $k_i$ friends. On social media, the phenomenon where local networks (dense subgraphs) with abnormally high clustering coefficients are formed becomes the foundation of so-called “echo chambers.”

Behind the formation of echo chambers works the principle of “Homophily” in sociology. As the saying “birds of a feather flock together” goes, humans have a tendency to easily connect with others who have similar attributes or ideologies. Expressing this as a probabilistic model, the probability $P(u, v)$ that an edge is formed between user $u$ and user $v$ can be assumed to be inversely proportional to their ideological distance $d(u,v)$.

$$ P(u, v) \propto e^{-\beta \cdot d(u,v)} $$

The parameter $\beta > 0$ is a constant indicating the strength of homophily. When a platform’s recommendation algorithm continuously presents “content and users that the user likes (= similar to themselves),” the value of this $\beta$ is artificially pushed up. As a result, edges (Weak Ties) between groups with different ideologies drastically decrease, and the entire network splits into multiple mutually isolated clusters.

The following Mermaid diagram visualizes the concept of a divided network and the bridging that connects it.

graph TD
subgraph "Cluster A (Conservative Echo Chamber)"
A1["User A1"] --- A2["User A2"]
A2["User A2"] --- A3["User A3"]
A3["User A3"] --- A4["User A4"]
A4["User A4"] --- A1["User A1"]
A1["User A1"] --- A3["User A3"]
end
subgraph "Cluster B (Liberal Echo Chamber)"
B1["User B1"] --- B2["User B2"]
B2["User B2"] --- B3["User B3"]
B3["User B3"] --- B4["User B4"]
B4["User B4"] --- B1["User B1"]
B2["User B2"] --- B4["User B4"]
end
A2["User A2 (Bridge Node)"] -. "Cross-cutting Edge (Bridging)" .- B2["User B2 (Bridge Node)"]
classDef cluster fill:#f9f9f9,stroke:#333,stroke-width:2px;
classDef node fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
classDef bridge fill:#ffecb3,stroke:#ff6f00,stroke-width:2px,stroke-dasharray: 5 5;
class A1,A3,A4,B1,B3,B4 node;
class A2,B2 bridge;

In this way, as long as the algorithm continues to adopt the objective function $J(\theta) = \sum \log P(\text{engage} | \text{user}, \text{content})$ that only optimizes for engagement (click-through rate, dwell time), the system will fall into a local optimum (strengthening of echo chambers) and move further away from the global optimum (formation of a healthy public square).

Chapter 2: Acceleration of Polarization by Algorithms and the Information Diffusion Model

To think about how information spreads within an echo chamber, let’s apply the “SIR model,” a mathematical model for infectious diseases, to information diffusion.

$S$ (Susceptible) : Users who have not yet been exposed to the information
$I$ (Infected) : Users who believe the information and are spreading it
$R$ (Recovered/Removed) : Users who have lost interest in the information or realized it’s fake and stopped spreading it

The differential equations for information propagation are expressed as follows:

$$ \frac{dS}{dt} = -\alpha S I $$$$ \frac{dI}{dt} = \alpha S I - \gamma I $$$$ \frac{dR}{dt} = \gamma I $$

Here, $\alpha$ is the “infection rate (ease of information spread),” and $\gamma$ is the “recovery rate (information saturation/forgetting).” What’s interesting is that empirical studies show polarizing content that incites anger or fear has a significantly higher $\alpha$ compared to general information. Furthermore, since there are fewer opportunities to encounter contradictory information within an echo chamber, $\gamma$ becomes extremely low. In other words, when an algorithm tries to maximize engagement, it inevitably learns to preferentially distribute content with high $\alpha$ and low $\gamma$—namely, “extreme views and fake news.” This is the mechanism by which AI unintentionally accelerates social division.

Chapter 3: Technical Solution (1) Bridging Algorithms and Community Notes

So, how should we confront this structural flaw? The first approach is the introduction of a “Bridging Algorithm.”

If engagement-based recommendation algorithms reward “homogeneity,” bridging algorithms reward “bridging heterogeneity.” A prime successful example of this is the “Community Notes” algorithm introduced on X (formerly Twitter).

Community Notes is not a simple majority vote. In a majority vote, the opinion of the echo chamber with the most people would always win. The revolutionary aspect of Community Notes is that it highly evaluates “notes that people who usually disagree (belonging to different clusters) happen to uniformly rate as ‘helpful’.”

To realize this, a machine learning technique called Matrix Factorization is used. The predicted score $\hat{r}_{u,n}$ of the rating (helpful or not) that user $u$ gives to note $n$ is modeled as follows:

$$ \hat{r}_{u,n} = \mu + i_u + i_n + \mathbf{f}_u \cdot \mathbf{f}_n $$

$\mu$ : Overall baseline (average rating tendency)
$i_u$ : Rating bias of user $u$ (e.g., a person who always gives high ratings)
$i_n$ : General quality of note $n$ (whether it’s easy for anyone to understand)
$\mathbf{f}_u$ : Latent feature vector of user $u$ (e.g., ideological stance)
$\mathbf{f}_n$ : Latent feature vector of note $n$

The algorithm learns each parameter to minimize the error between the actual rating data and the predicted scores. What is crucial here is that the final decision to display a note is not based on a simple average rating, but on “the parameter $i_n$ indicating the general quality of the note.”

If a certain note receives a massive number of high ratings from a specific biased group (e.g., only right-wing or only left-wing), those high ratings are absorbed by the latent vector term $\mathbf{f}_u \cdot \mathbf{f}_n$, and $i_n$ does not become high. However, if it receives high ratings from both the right ($\mathbf{f}_u > 0$) and the left ($\mathbf{f}_u < 0$), it can no longer be explained solely by the dot product of the latent vectors, and as a result, the algorithm learns that “this note itself is universally excellent ($i_n$ is high).”

Through this mathematical approach, it becomes possible to algorithmically discover and evaluate “consensus formation across echo chambers.” This is a highly powerful technical breakthrough for bridging social divisions.

While bridging algorithms are powerful, the structural issue remains that a single giant corporation (centralized platform) monopolizes the algorithm. The algorithm can be changed at any time by a single management policy of the platform.

The second approach to this is a paradigm shift at the architecture level via “Decentralized Social Protocols.” Currently, ActivityPub (adopted by Mastodon, etc.) and AT Protocol (adopted by Bluesky) are attracting significant attention.

AT Protocol (Authenticated Transfer Protocol), in particular, has an extremely beautiful design philosophy of “separating data and algorithms.”

graph TD
subgraph "User Control Layer"
Client["Client App (Bluesky etc.)"]
end
subgraph "Data Layer (Federated)"
PDS1["PDS (Personal Data Server) A"]
PDS2["PDS (Personal Data Server) B"]
end
subgraph "Indexing & App Layer"
Relay["Relay (Big Graph Server)"]
AppView["AppView"]
end
subgraph "Algorithmic Layer (Composable)"
FeedGen1["Feed Generator (Chronological)"]
FeedGen2["Feed Generator (Bridging Algorithm)"]
Labeler["Moderation Labeler (Fact Checkers)"]
end
Client -->|"Reads/Writes"| PDS1
Client -->|"Views"| AppView
PDS1 -->|"Syncs via WebSocket"| Relay
PDS2 -->|"Syncs via WebSocket"| Relay
Relay -->|"Indexes"| AppView
AppView -.->|"Requests Feed"| FeedGen1
AppView -.->|"Requests Feed"| FeedGen2
AppView -.->|"Gets Labels"| Labeler

The greatest achievement of the AT Protocol is that it has decoupled “feed generation (algorithms)” and “moderation (labeling)” from the main platform, making them freely selectable and composable by the users themselves (Custom Feeds / Stackable Moderation).

Until now, we could choose “which SNS to use,” but we could not choose “which algorithm will bathe us in information.” In the world of AT Protocol, one person can choose a “chronological” feed, another can install an “academic feed that provides counterarguments to their opinions,” and yet another can subscribe to “moderation labels from a third-party organization that hides inappropriate words.”

Backed by cryptographic technology (DID: Decentralized Identifiers) and data structures (Merkle Search Trees: MST), this protocol restores “informational self-determination” to users. As algorithms are no longer black boxes but compete and are selected in an open market, it holds the potential to transform the incentive structure from engagement-supremacy algorithms to ones that prioritize the mental health of users and the soundness of society.

Thus far, I have described the analysis using network theory and the concrete technologies to overcome it (Matrix Factorization in Community Notes, decentralized architecture of AT Protocol). However, ultimately, what bridges social divisions is not mere code or mathematical formulas. It is the “human will and philosophy” that creates them.

In the world of software engineering, there is a great culture called “Open Source.” Starting with Linux, most of the foundational technologies that build the internet have been created by strangers around the world cooperating, debating, and merging code across ideologies and borders. The open-source community possesses a mechanism not to eliminate conflicts, but to elevate them into constructive consensus building through “pull requests” and “code reviews.”

I believe that this open-source philosophy itself will serve as a hint to repair our divided modern society. Making systems transparent, entrusting the choice of algorithms to users, and designing a decentralized public space (Public Square) where diverse values can coexist. That is a critically important social responsibility imposed on modern engineers.

Code is law, and architecture is politics. A single line of code we write, a single API endpoint we define, or a database schema we design shapes the cognition of millions or hundreds of millions of users; it can accelerate social division, or it can build bridges that encourage dialogue.

Conclusion: Finishing the 100th Article

“Can technology bridge the social divide?”

My answer to this question is: “Technology alone cannot bridge it, but properly designed technology can become a ‘scaffolding’ for humans to overcome the divide.”

It is impossible to completely erase fundamental human biases (homophily and confirmation bias). However, it is possible to stop the rampage of algorithms that solely pursue engagement, introduce mathematical models like Community Notes that evaluate “bridging,” and return choices to users through autonomous decentralized architectures like the AT Protocol.

This blog marks its 100th post with this entry. In previous articles, I have focused on the so-called “How,” such as language specifications and how to use frameworks. However, in the coming era where AI automatically generates code and all technologies become commoditized, what is most important for us engineers are the ethical and philosophical questions of “What (what to make)” and “Why (why make it).”

Technology is not magic. It is a mirror of humanity. If society is divided, it is because the systems we built are reflecting and amplifying that division. That is precisely why I believe that by rewriting the systems, we can slightly, but surely, change the state of society for the better.

From the 101st post onward, as an engineer, I would like to continue standing at the intersection of code and society, deepening my thoughts. Thank you very much for reading this long piece to the end. I hope that the networks of the future will not be walls that divide us, but bridges for us to understand each other.

(End)

Deepfake and Information Literacy: How to Technically Detect Fake News

Sat, 12 Sep 2026 12:00:00 +0900

Introduction: The Era Where the Boundary Between Reality and Fiction Melts

In the 2020s, the evolution of Generative AI has been progressing at an unprecedented speed. It is now possible to generate content—text, audio, images, and even videos—that is indistinguishable from what humans create, in just a few seconds. While this technological leap brings tremendous benefits to creative fields, it has also created a serious social threat: the flood of sophisticated forged content known as “Deepfakes.”

Deepfakes threaten society in various forms, such as fake speeches by politicians, scams impersonating corporate CEOs (an evolution of BEC scams), and pornography that defames celebrities. Especially during election periods, the spread of fake news via deepfakes has escalated to a point where it shakes the very foundation of democracy.

In such an era, what is required of us is an update to our “information literacy.” The common sense of “believing what you see with your own eyes” is no longer valid. In this article, starting from the technical background of how deepfakes are generated, we will explain at a very deep level—incorporating mathematical formulas and code—the cutting-edge digital forensic techniques to “technically” detect them, and the frameworks (such as C2PA) for society as a whole to counter fake information.

1. The Mechanisms of Generative AI Supporting Deepfakes

To understand deepfakes, you must first know the mechanisms of the generative AI that forms their foundation. Currently, the two representative architectures used for generating high-definition images and videos are “GAN (Generative Adversarial Networks)” and “Diffusion Models.”

1.1 Generative Adversarial Networks (GAN)

Proposed by Ian Goodfellow and others in 2014, GANs ignited the deepfake technology trend. In a GAN, two neural networks take on roles like a “forger” and a “police officer,” and by competing with each other (adversarial training), they generate extremely realistic data.

Generator ($G$): Takes random noise (latent variable $z$) as input and generates data (such as images) that looks exactly like the real thing.
Discriminator ($D$): Determines whether the inputted data is “Real” coming from an actual dataset or “Fake” created by the generator.

These two networks proceed with training to optimize a loss function formulated as the following Minimax game:

$$ \min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_{z}(z)}[\log(1 - D(G(z)))] $$

Here, $x$ is real data, and $z$ is a latent variable (noise). The discriminator $D$ tries to maximize this formula (accurately distinguishing between real and fake), and the generator $G$ tries to minimize it (fooling the discriminator). When this training reaches an equilibrium state (Nash equilibrium), the generator becomes able to generate data indistinguishable from the real thing.

flowchart LR
Z["Latent Variable (Latent Vector Z)"] --> G["Generator"]
G --> F["Generated Image (Fake Image)"]
R["Actual Image (Real Image)"] --> D["Discriminator"]
F --> D
D --> O["Truth/False Judgment (Real/Fake)"]
O -.->|"Loss Feedback"| G
O -.->|"Loss Feedback"| D

1.2 Diffusion Models

In recent years, “Diffusion Models” have emerged as the foundational technology for Midjourney and Stable Diffusion, boasting image quality and stability that surpass GANs. A diffusion model consists of a “forward diffusion process,” which gradually adds noise to data, and a “reverse diffusion process,” which restores the original data from noise.

In the Forward Process, Gaussian noise is added step by step over time $t$ to a clean image $x_0$. This process is expressed as a Markov chain with the following formula:

$$ q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I}) $$

Here, $\beta_t$ is a schedule parameter that controls the variance of the noise. After a sufficient number of steps $T$, $x_T$ becomes complete random noise.

In the Reverse Process, a neural network (usually a U-Net architecture) learns to predict the noise from the noisy image $x_t$ and restore the previous step $x_{t-1}$. By combining this process with conditioning (such as text prompts), it becomes possible to generate any image from zero (noise).

2. Digital Forensics: Techniques to Search for Traces of Generated Artifacts

No matter how advanced generative models become, “mathematical and statistical traces (artifacts)” invisible to humans always remain in AI-generated data. Detection technologies (deepfake detectors) capture these subtle traces through various approaches.

2.1 Frequency Domain Analysis and DCT (Discrete Cosine Transform)

Human eyes are sensitive to spatial changes (spatial domain) in an image’s color and brightness, but insensitive to frequency changes (frequency domain). Images generated by GANs or diffusion models, even if they look perfect at first glance, produce peculiar frequency patterns (such as checkerboard artifacts) during the upsampling process (enlargement from low to high resolution).

To detect this, the Discrete Cosine Transform (DCT) is often used. DCT represents an image as a sum of cosine waves of different frequencies. The formula for the 2D DCT is as follows:

$$ X_{k_1, k_2} = \sum_{n_1=0}^{N_1-1} \sum_{n_2=0}^{N_2-1} x_{n_1, n_2} \cos\left[\frac{\pi}{N_1}\left(n_1 + \frac{1}{2}\right)k_1\right] \cos\left[\frac{\pi}{N_2}\left(n_2 + \frac{1}{2}\right)k_2\right] $$

Generated images tend to have an abnormal energy distribution in the high-frequency components (fine noise and abrupt edge changes) compared to natural images. The following Python code is a simple example of extracting the energy of high-frequency components from an image using DCT.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


import cv2
import numpy as np
import scipy.fftpack

def extract_high_frequency_features(image_path):
 # Load image and convert to grayscale
 img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
 if img is None:
 raise ValueError("Image not found")

 # Apply 2D Discrete Cosine Transform (DCT)
 # First apply 1D DCT to rows, then 1D DCT to columns
 dct_result = scipy.fftpack.dct(scipy.fftpack.dct(img.T, norm='ortho').T, norm='ortho')

 # Extract high-frequency components (mask the top-left low-frequency components to zero)
 rows, cols = dct_result.shape
 mask = np.ones((rows, cols))

 # Mask the low-frequency region (10% of the whole)
 mask[:int(rows*0.1), :int(cols*0.1)] = 0

 high_freq_features = dct_result * mask

 # Calculate the amount of energy in the high-frequency region
 energy = np.sum(np.abs(high_freq_features))

 return energy

# Comparing natural images with generated images often reveals a statistically significant difference in the energy value

This unnaturalness in the frequency domain arises because while AI can learn “local consistency at the pixel level,” it struggles to perfectly mimic the “global frequency characteristics of the entire image.”

3. Detection of Biological Signals: Confirming the “Beat of Life” via rPPG

In addition to detection technologies for images (still images), a groundbreaking approach to deepfake detection in videos is the extraction of biological signals.

As long as a human is alive, blood circulates through the body in sync with the heartbeat. Because hemoglobin in the blood absorbs specific wavelengths (especially green light, around 530nm) well, the color of the facial skin changes minutely (at a level invisible to the human eye) in time with the heartbeat. The technology that uses this principle to estimate the heart rate contactlessly from standard RGB camera video is called rPPG (remote Photoplethysmography).

The basic rPPG model based on light absorption and reflection is expressed by the Beer-Lambert law as follows:

$$ I(t) = I_0(t) e^{-\left( \mu_{dc} + \mu_{ac}(t) \right) d} $$

Here, $I(t)$ is the light intensity observed by the camera, $I_0(t)$ is the light source intensity, $\mu_{dc}$ is the static light absorption coefficient by tissue, $\mu_{ac}(t)$ is the dynamic light absorption coefficient due to blood flow fluctuation (heartbeat), and $d$ is the path length of the light.

Deepfake videos (such as FaceSwap, which swaps faces, or Lip-sync, which matches lip movements to audio) pursue visual realism on a frame-by-frame basis, but they cannot reproduce the minute blood flow changes (heartbeat signals) along the time axis. Therefore, when attempting to extract an rPPG signal from a deepfake video, one obtains a noisy, unnatural signal that differs from the regular rhythm of a natural human heart rate (typically in the range of 60-100 bpm).

flowchart LR
V["Input Video (Video Stream)"] --> F["Face Detection & Tracking (Face Tracking)"]
F --> R["Region of Interest Extraction (ROI Extraction)"]
R --> S["Spatial Pooling"]
S --> B["Bandpass Filter"]
B --> H["Heartbeat Signal Extraction (Heartbeat Signal)"]
H --> A["Truth/False Classification & Anomaly Detection (Fake/Real Classification)"]

Below is a conceptual implementation example of a pipeline for extracting rPPG signals from video using Python.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


import cv2
import numpy as np
from scipy import signal

def extract_rppg_signal(video_path):
 cap = cv2.VideoCapture(video_path)
 green_signals = []

 while cap.isOpened():
 ret, frame = cap.read()
 if not ret:
 break

 # 1. Face detection and extraction of ROI (Region of Interest: e.g., forehead or cheeks)
 # roi = detect_face_and_extract_roi(frame)
 # Here, for simplicity, the central part of the entire frame is used as the ROI
 h, w = frame.shape[:2]
 roi = frame[int(h*0.3):int(h*0.6), int(w*0.4):int(w*0.6)]

 # 2. Extract Green channel from RGB space
 # Because hemoglobin in the blood absorbs green light the most
 g_channel = roi[:, :, 1]

 # 3. Spatial pooling (calculating the mean value)
 mean_g = np.mean(g_channel)
 green_signals.append(mean_g)

 cap.release()

 if len(green_signals) == 0:
 return None

 # 4. Noise removal with a bandpass filter
 # Extract the human heart rate frequency band (e.g., 0.7Hz - 2.5Hz = 42 - 150 bpm)
 fps = 30.0 # Assumed frame rate
 nyquist = 0.5 * fps
 low = 0.7 / nyquist
 high = 2.5 / nyquist
 b, a = signal.butter(3, [low, high], btype='bandpass')
 filtered_signal = signal.filtfilt(b, a, green_signals)

 return filtered_signal

# By analyzing the frequency spectrum of the extracted filtered_signal,
# if no clear peak (heartbeat) exists, it is determined that the probability of it being a deepfake is high.

4. The Never-ending “Cat-and-Mouse Game”: Adversarial Training and Evasion Techniques

As introduced so far, advanced forensic techniques such as frequency analysis and biological signals (rPPG) do exist. However, in the world of AI, there is no “absolute barrier.” As soon as a detection technology is published in a paper, attackers (deepfake creators) immediately improve their generative models to evade that detector.

For example, suppose a detector identifies deepfakes by detecting “anomalies in the frequency domain.” Attackers will incorporate this detector itself as the “Discriminator” of a new GAN and retrain the Generator. Then, the Generator evolves to output “images that are indistinguishable from natural images even in the frequency domain.”

Furthermore, there are already reports of research (Anti-Forensics) attempting to fool rPPG-based detection systems by intentionally adding artificial “minute color fluctuations (fake heartbeat signals)” to videos in post-processing.

Detection and generation are truly engaged in a never-ending cat-and-mouse game of “shield and spear.” For this reason, it is pointed out that the approach of judging authenticity by retrospectively analyzing only the output data (images and videos) (passive detection) will eventually reach its limit.

5. Fundamental Countermeasures: Provenance Proof and the C2PA Framework

As retrospective detection approaches its limits, an active defense approach that cryptographically guarantees the “Provenance” of data is rapidly being promoted worldwide. Constructing the global standard framework for this is the C2PA (Coalition for Content Provenance and Authenticity).

C2PA is a consortium established with the participation of major companies such as Adobe, Microsoft, Intel, BBC, and Sony. It defines technical specifications to embed the provenance of digital content (who shot it, when, with which camera, and what edits were made) directly into the content itself in a tamper-evident manner.

5.1 How C2PA Works

The core technologies of C2PA are digital signatures using Public Key Infrastructure (PKI) and content hash binding.

Manifest Generation (Manifest): The moment a photo is taken with a camera, or when it is edited with software, a metadata called a “Manifest” is generated, which includes the operation history, device information, and creator information.
Cryptographic Signature (Digital Signature): A digital signature is applied to the Manifest and the hash value of the image itself (a summary of the pixel data) using a hardware or software private key.
Embedding in the Asset: The signed Manifest (C2PA credential) is embedded in the header information of file formats like JPEG or MP4.

If an attacker attempts to tamper with a part of the image or attach fake metadata to an AI-generated image, the hash value of the image itself will change, causing the digital signature verification to fail and instantly revealing the tampering.

flowchart TD
C["Creator / Camera"] --> M["Manifest Generation"]
M --> S["Signature and Binding (Cryptographic Signature)"]
S --> A["Asset with C2PA Manifest"]
A --> P["Platform (Social Media Platform)"]
P --> V["Validation Process"]
V --> U["Display on User Screen (Content Credentials UI)"]

5.2 Visualization with the “Content Credentials” Icon

In systems compliant with the C2PA standard, when users view images on social media or news sites, an icon reading “CR (Content Credentials)” is displayed in the corner of the image. By clicking this, anyone can transparently check the history of the image, such as whether it was “generated by AI,” “shot with an actual camera,” or “color-corrected in Photoshop.”

Currently, major AI vendors like OpenAI (DALL-E 3) and Google have started attaching C2PA metadata to generated images, and camera manufacturers like Leica and Sony are proceeding to implement C2PA signature functions at the hardware level. The paradigm of society is shifting from “detecting fakes” to “proving authenticity (Zero-Trust approach).”

6. Next-Generation Information Literacy: What We Can Do

Technical countermeasures (such as deepfake detectors or provenance proofs like C2PA) are merely infrastructure to protect society. Ultimately, it is our human brains that decide whether to consume and spread information.

“Information literacy” in the AI era means adopting the following attitudes:

Avoid Reflexive Spreading (Stop and Think) Especially when exposed to shocking footage or content that incites anger (information appealing to emotions), stop for a moment and halt your hands from reposting or sharing. The primary goal of deepfake creators is to hack human emotions and make the information spread.
Verify the Source of the Information (Verify the Source) Is the information transmitted by a reliable news organization? Is it accompanied by a provenance proof (Content Credentials) like C2PA? It is crucial to develop the habit of cross-checking information sources.
Healthy Skepticism that “Everything Might be Fake” (Healthy Skepticism) There is no need to become pessimistic, but the old common sense of “video = fact” must be discarded. We must consume information with the premise that audio, video, and text can all be easily forged in this era.

Conclusion

The evolution of AI technology has opened Pandora’s box. It is no longer possible to erase the technology itself that creates deepfakes.

However, as explained in this article, engineers are confronting the threat of fake news with a variety of approaches, such as frequency analysis, biological signal detection, and provenance proof (C2PA) using cryptography. By combining these technical shields (defenses) with the social shield of “information literacy” that each of us possesses, we should be able to navigate the wave of fiction brought by AI and protect the value of truth.

Precisely because we are in an era where the boundary between reality and fiction is melting, the human “will” to try and discern the truth has become more important than ever.

How SNS Algorithms Influence Our Thinking and Tech Selection

Sat, 12 Sep 2026 12:00:00 +0900

1. Introduction: Democratization of Tech Information and the Rise of Algorithms

In modern software engineering, much of the technical information we consume daily passes through Social Networking Services (SNS) like X (formerly Twitter), Hacker News, Reddit, and LinkedIn, or news aggregators. There was once an era where we gathered information autonomously and chronologically through mailing lists, expert-run blogs, or RSS readers. However, with the explosive increase in frameworks and tools created every day, it has become common practice to entrust information curation to “Recommendation Algorithms” provided by platforms in order to optimize our limited cognitive resources (disposable time and attention).

This paradigm shift has brought the immense benefit of efficiently discovering valuable technical articles and groundbreaking open-source projects. On the other hand, it has also caused a highly critical side effect. That is the fact that “the technical trends and best practices we see are distorted not by pure technical superiority or objective evaluation, but by the algorithm’s ’engagement optimization function’.”

In this article, we mathematically and structurally unravel how the advanced machine learning algorithms running behind the scenes of SNS shape our cognition and influence decision-making in tech selection. Furthermore, we deeply consider the dangers of “Hype Driven Development (HDD),” where one is swept away by the hype created by algorithms, and concrete approaches to break free from it and make objective, robust tech selections.

2. Evolution and Mechanism of Recommendation Algorithms

When we open an SNS, the content displayed on our timeline (feed) is not random. There are machine learning models highly tuned to maximize user retention time and improve ad revenue. First, let’s look at the foundational technologies behind these.

2.1 Collaborative Filtering and Matrix Factorization

“Collaborative Filtering” has served as a powerful baseline from the dawn of recommendation systems to the present. In particular, “Matrix Factorization,” which represents the interaction between users and items (posts or articles) as a matrix and maps them into a latent feature space, is widely used.

Given an evaluation matrix $R \in \mathbb{R}^{M \times N}$ with $M$ users and $N$ items, matrix factorization approximates this large, sparse matrix as the product of a low-dimensional latent feature matrix $U \in \mathbb{R}^{M \times K}$ (user features) and $V \in \mathbb{R}^{N \times K}$ (item features) ($K \ll M, N$).

$$ R \approx U \times V^T $$

The predicted score (probability of engagement) $\hat{r}_{ij}$ of item $j$ for a specific user $i$ is calculated as the inner product of their respective latent feature vectors.

$$ \hat{r}_{ij} = \mathbf{u}_i \cdot \mathbf{v}_j $$

This model is trained to minimize the following loss function ($\lambda$ is a regularization term to prevent overfitting).

$$ \mathcal{L} = \sum_{(i,j) \in \Omega} (r_{ij} - \mathbf{u}_i \cdot \mathbf{v}_j)^2 + \lambda (\|\mathbf{u}_i\|^2 + \|\mathbf{v}_j\|^2) $$

Impact on Tech Selection: This algorithm brings “Person A who is interested in Rust” and “Person B who is interested in Rust” closer together in the latent space. If Person A “likes” a post about an emerging Web framework, posts about that framework will appear on Person B’s timeline with a high probability. Because of this, a phenomenon occurs where specific technologies become locally popular within engineer groups that prefer certain tech stacks.

2.2 Deep Learning Recommendation Model (DLRM)

In recent years, deep learning-based architectures, represented by the Deep Learning Recommendation Model (DLRM), have become widespread, centered around companies like Meta (formerly Facebook). DLRM receives a wide variety of features as input, such as the user’s past behavior history and item metadata, and predicts the Click-Through Rate (CTR) and the like.

The characteristic of DLRM lies in converting sparse categorical features (e.g., User ID, followed hashtags) into dense vectors through an “Embedding Table,” and combining them with continuous dense features (e.g., days since account creation, past average retention time).

$$ \mathbf{e}_{\text{sparse}} = \text{EmbeddingLookup}(\mathbf{x}_{\text{sparse}}) $$$$ \mathbf{h}_{\text{dense}} = \text{BottomMLP}(\mathbf{x}_{\text{dense}}) $$

After combining these through concatenation or interactions like inner products (Feature Interaction), they are input into a Top Multilayer Perceptron (Top MLP) to output the final probability of CTR, etc., using a sigmoid function $\sigma$.

$$ \hat{y} = \sigma(\text{TopMLP}(\text{Interact}(\mathbf{e}_{\text{sparse}}, \mathbf{h}_{\text{dense}}))) $$

Impact on Tech Selection: Giant models like DLRM capture even extremely subtle signals (for example, a slight increase in retention time for “posts with videos” or “posts containing specific buzzwords”) and reflect them in the prediction score. As a result, technical information that includes “radical titles (e.g., ‘React is dead’, ‘The End of Microservices’)” or “visually flashy demos” tends to be algorithmically favored.

2.3 Reinforcement Learning and Multi-Armed Bandits

Recommendation systems must constantly explore the user’s latest preferences. Here is where the “Multi-Armed Bandit problem” comes in. It optimizes the trade-off between “Exploitation” (presenting reliable content based on existing preferences) and “Exploration” (discovering new trends).

In UCB (Upper Confidence Bound), a representative algorithm, the score for selecting arm (content group) $a$ at time $t$ is calculated as follows.

$$ a_t = \arg\max_{a} \left( \hat{\mu}_a + c \sqrt{\frac{\ln t}{N_a(t)}} \right) $$

Here, $\hat{\mu}_a$ is the average reward (engagement rate) of arm $a$ so far, $N_a(t)$ is the number of times it has been selected, and $c$ is a parameter that adjusts the degree of exploration.

Impact on Tech Selection: The algorithm temporarily gives an exploration bonus to posts about newly introduced frameworks or libraries (those with a low number of trials $N_a(t)$) and exposes them to a random group of users. In this initial “exploration phase,” if the reaction from influencers and others is good, $\hat{\mu}_a$ sharply rises, rapidly developing into a buzz (viral). This is the mechanism of “suddenly everyone starts talking about that technology.”

3. The Mathematics of Echo Chambers and Filter Bubbles

As algorithms become more optimized, users find themselves surrounded only by “information they find comfortable or information that reinforces their existing beliefs.” This is the Echo Chamber phenomenon and the Filter Bubble.

In network theory, the tendency for similar individuals to connect is called “Homophily.” In a graph $G=(V, E)$, edges (follow relationships and information propagation) between nodes (users) are more likely to form the higher the similarity of attributes.

SNS recommendation algorithms artificially accelerate this homophily. For example, suppose there is a community of engineers promoting “Serverless Architecture” and another supporting “On-Premises Bare Metal.” The algorithm learns to lower the weight of cross-cutting ties between different communities and strengthen edges within the same community (because opposing opinions often cause user churn and carry the risk of lowering engagement. Or conversely, it might trigger engagement through extreme anger, but the former tends to be more common in tech circles).

As a result, a completely divided technological reality is created, where on your timeline it looks like “companies all over the world are migrating to serverless,” while on someone else’s timeline it looks like “Cloud Repatriation is a global trend.”

4. Hype Driven Development (HDD) Created by Algorithms

The combination of echo chambers and powerful recommendation models triggers one of the biggest anti-patterns in the engineering industry: Hype Driven Development (HDD). HDD is the phenomenon of adopting new technologies simply because “they are trending on SNS” or “they are the latest trend,” without deeply considering the actual merits, trade-offs, or compatibility with one’s own business requirements.

The Mermaid diagram below shows how SNS algorithms spin the feedback loop of HDD.

graph TD
A["Engineers post the 'overwhelming benefits' of a new tech"] --> B["Algorithm measures initial CTR and retention time (Exploration)"]
B --> C["Judged as high engagement, exposure expands to similar users' TLs"]
C --> D["Users stimulated by FOMO (Fear Of Missing Out) further spread it"]
D --> E["Occurrence of Frequency Illusion: 'It's becoming the industry standard'"]
E --> F["Introduced into real projects without sufficient verification (HDD)"]
F --> A

What is terrifying about this loop is that the “Baader-Meinhof phenomenon (Frequency Illusion)” is intentionally triggered by the algorithm. Once you see the name of a new state management library, the algorithm captures it as a signal and fills your feed with topics about that library from the next day. The human brain misidentifies this as a “global pandemic.”

The chart below illustrates the difference in lifecycle between technologies overly hyped on SNS and “Boring Technology” that is plain and dull but robust.

xychart-beta
title Technology Lifecycle and Evaluation Trends
x-axis ["Month 0", "Month 6", "Month 12", "Month 18", "Month 24", "Month 30", "Month 36"]
y-axis "Number of Mentions / Hype Level on SNS" 0 --> 100
line [10, 85, 95, 45, 20, 10, 5]
line [15, 20, 25, 35, 50, 65, 80]

(Note: In the graph above, the line that sharply rises and falls indicates the “Hyped Technology,” while the line that slowly and steadily rises indicates “Boring Technology”)

Hyped technologies face realistic problems such as “lack of documentation,” “critical bugs in edge cases,” and “maintainer burnout” 6 to 12 months after introduction, and rapidly disappear from SNS. However, once technical debt is embedded into a system, removing it costs an enormous amount.

5. “Escaping the Algorithm” Strategies in Tech Selection

So, how should we make objective and calm tech selections under the dominance of these algorithms? Here are some concrete strategies not to hack the algorithm, but to “step off” from it.

5.1 Returning to Primary Information: Source Code and RFCs

The most reliable defense is to shift your information sources from SNS aggregations to Primary Sources.

Read the Source Code: Instead of believing SNS posts saying “This library is blazingly fast,” actually open GitHub and check the core logic’s time complexity and memory allocation mechanisms.
Follow RFCs (Request for Comments): Many mature open-source projects (React, Rust, Python, etc.) adopt the RFC process when introducing new features. RFCs objectively and logically describe “Why this feature is necessary,” “What the design trade-offs are,” and “What the alternatives are,” without worrying about algorithm engagement. This is where true technical value lies.

5.2 Close Reading of Academic Papers and Whitepapers

When it comes to foundational tech selections like distributed systems, databases, or machine learning model architectures, you should directly read papers published in ACM, IEEE, or arXiv, or detailed whitepapers published by companies (e.g., Google’s Spanner paper, Amazon’s Dynamo paper), rather than a few lines of summary on SNS.

SNS posts are optimized to “steal readers’ attention,” whereas peer-reviewed papers are optimized for “factual accuracy and reproducibility.” The evaluation functions are entirely different.

5.3 Building an In-House Decision-Making Framework

To prevent HDD at the team or organizational level, a process is needed to eliminate personal intuition or reasons like “because I saw it on Twitter.” A prime example of this is the introduction of ADR (Architecture Decision Records).

When introducing a new technology, you must always document the following items and undergo a review:

Context: Why is the new technology necessary? What are the current issues?
Decision: What will be adopted?
Consequences: What are the trade-offs? (What is gained at the expense of what?)

By enforcing this process, “Hype” can be transformed into “Engineering.”

5.4 The Philosophy of the Boring Technology Club

There is a famous mantra in the tech community: “Choose Boring Technology.” This is a teaching that Innovation Tokens (the limited resources an organization can spend on new, unknown technologies) should not be wasted on selecting infrastructure or frameworks that are not directly tied to the core value of the business.

SNS algorithms prefer “novelty.” However, to build a robust system that can withstand real-world operations, what is needed is “boring” technology (like PostgreSQL, Redis, or standard REST APIs) that has over 10 years of operational track record and yields millions of Google search hits for disaster recovery procedures.

6. Conclusion: How We Should Face Technology

SNS recommendation algorithms are powerful tools that broaden our technical horizons and provide encounters with wonderful communities. However, as long as their internal structures (Matrix Factorization, DLRM, Multi-Armed Bandits) have “engagement maximization” as their supreme imperative, the information output is inevitably biased.

We need to acquire the literacy to treat the information flowing into our timelines not as “facts” or “absolute trends,” but merely as a single “signal.”

Stepping out of the echo chamber, reading source code with our own hands, following RFC discussions, deciphering the math in papers, and facing the true challenges of our business domains. That alone is the only path to practicing true software engineering without being swallowed by the waves of algorithms.

The Current State and Challenges of IT Education in Japan: After the Mandatory Programming Education

Sat, 12 Sep 2026 12:00:00 +0900

1. Introduction: The Light and Shadow Brought by Mandatory Programming Education

With the mandatory implementation of programming education in elementary schools in 2020, its expansion in junior high school technology and home economics classes in 2021, and the mandatory introduction of the new subject “Information I” in high schools in 2022, IT and information education in Japan have experienced an unprecedented paradigm shift in recent years. At the root of this series of policies lies a very pressing national demand: the cultivation of logical thinking skills (programming thinking) necessary to survive in the Society 5.0 (Super Smart Society) era, and the resolution of the chronic shortage of highly skilled IT professionals in the industry.

However, looking at the front lines of education, it has become apparent that there is a massive disconnect between the ideal envisioned by the government and reality. The most serious issue is the complete conflation of “learning programming as a tool” and “mastering the academic discipline of computer science.” Furthermore, structural challenges are piling up, including the technical limitations due to the hardware specs of the nationwide IT infrastructure, and the lack of professional skill sets among the teachers who are supposed to instruct the students.

This article summarizes the “aftermath” of the mandatory programming education in Japan and explores in detail and technically the fundamental and structural problems facing IT education today. It examines these issues from the perspectives of computer science theory, hardware architecture constraints, and global industrial competitiveness. Spanning approximately 10,000 characters, this is not just an educational discourse, but an essay that considers the future of Japan from the viewpoint of software engineering.

2. The Trap of Visual Programming: The Deep and Steep Chasm from Scratch to Text Coding

The de facto standard in elementary school programming education is visual programming languages (block programming), represented by “Scratch” developed by the MIT Media Lab. The fact that it allows beginners to visually and intuitively learn the three basic algorithmic control structures—sequence, selection, and iteration—by combining puzzle-like blocks using an intuitive graphical interface makes it a great invention that deserves high praise as introductory education.

However, there is a major pitfall here, which can be called the “trap of abstraction.” That is the cruel reality that “transitioning from visual programming to a full-fledged text-based programming language (Python, JavaScript, C++, Rust, etc.) is extremely difficult, and many learners drop out at this stage.”

The Wall of Abstraction and the Black-Boxing of Computer Science

Visual programming environments like Scratch highly abstract and intentionally hide (encapsulate) the fundamental elements of computer science, such as the complex syntax of programming, strict type systems, and memory lifecycle management. While this is excellent for lowering the cognitive load on beginners, it becomes a massive barrier when advancing to genuine engineering. In actual software development, an understanding of variable scope (local and global variables), complex data structures (arrays, linked lists, hash tables, binary search trees, graphs), pointer manipulation, and memory heap and stack regions is absolutely essential.

The following Mermaid diagram visualizes the learning hurdles and dropout points that beginners face during the transition from visual programming to authentic computer science.

flowchart TD
A["Elementary School: Scratch (Visual/Block-based)"] --> B{"Junior High: The Wall of Transition to Text Languages"}
B -->|"Frustration from strict syntax errors"| C["Dropout (Syntax Allergy)"]
B -->|"Lack of conceptual understanding of variables and static typing"| D["Dropout (The Wall of Types)"]
B -->|"Successful Transition"| E["High School: Information I (Basics of Python/JavaScript, etc.)"]
E --> F{"The Wall of Algorithm Design and Data Structures"}
F -->|"Lack of understanding of time and space complexity"| G["Inefficient Code (Performance degradation from mass-producing O(N^2))"]
F -->|"Black-boxing of memory management and references"| H["Becoming a coder who only does superficial API calls"]
F -->|"Conceptual Breakthrough"| I["Full-scale CS Learning (C/C++, Java, Low-level Architecture)"]
I --> J["Highly skilled IT professionals coveted by the industry"]
classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
classDef error fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
classDef success fill:#ccffcc,stroke:#00cc00,stroke-width:2px;
class C,D,G,H error;
class J success;

As is evident from this flowchart, merely accumulating the experience of “writing code to move characters on a screen” will not cultivate true software engineers who can design scalable distributed system architectures and optimize performance down to the millisecond. Between the task of assembling colorful Scratch blocks with a mouse and reading the C source code of the Linux kernel to trace the behavior of the TCP/IP stack, there lies an absolute conceptual disconnect that cannot be dismissed simply as “a difference in the language used.”

3. The Limits of Coding Without “Mathematics” and “Discrete Logic”: An Approach from Computational Complexity Theory

The greatest weakness and a potentially fatal flaw in Japan’s programming education curriculum is the overwhelming lack of integration between “coding skills” and “mathematics / discrete mathematics.” In top-tier computer science education, such as in the United States and India, the emphasis is placed on algorithm efficiency, mathematical logic, and mathematical proofs rather than the syntax of the programming language itself. This is because code is merely a translation of mathematical formulas.

The Absolute Dominance of Time Complexity and Space Complexity (Big O Notation)

When evaluating and designing software performance, the concepts of Time Complexity and Space Complexity are unavoidable. Landau’s asymptotic notation (Big O Notation) indicates how execution time and memory consumption increase when the input data size to an algorithm is $N$.

Mathematically, $f(x) = O(g(x))$ is strictly defined as follows:

$$ \exists C > 0, \exists x_0 > 0, \forall x > x_0, |f(x)| \le C \cdot |g(x)| $$

In Japan’s information education, when learning about data sorting, for example, there are cases where it simply ends with calling a built-in method like array.sort() in Python. However, what is truly required in information engineering is the mathematical understanding and proof of why the simple Bubble Sort is never used in practical domains, and why Quick Sort, Merge Sort, or Timsort are adopted as standard libraries.

Below are the average time complexities of representative sorting algorithms.

Bubble Sort: $O(N^2)$
Selection Sort: $O(N^2)$
Insertion Sort: $O(N^2)$
Merge Sort: $O(N \log N)$
Quick Sort: $O(N \log N)$
Heap Sort: $O(N \log N)$

For instance, the time complexity $T(N)$ of Merge Sort is expressed by the following recurrence relation based on the Divide and Conquer paradigm.

$$ T(N) = 2T\left(\frac{N}{2}\right) + O(N) $$

By expanding and solving this recursive relation using the Master Theorem, the ideal computational complexity $T(N) = O(N \log N)$ is derived.

$$ T(N) = \Theta(N \log_2 N) $$

In modern big data analytics and web-scale traffic processing, the order of $N$ is massive, reaching hundreds of millions or billions. If an ignorant programmer implements an inefficient $O(N^2)$ algorithm, it would require a staggering $10^{12}$ (1 trillion) useless comparison operations for data size $N = 10^6$, effectively causing the system to freeze and crash. On the other hand, $O(N \log N)$ would complete in about $2 \times 10^7$ (20 million) operations. To claim “I can program” without this cruelly rigorous mathematical backing is like building a skyscraper without knowing structural mechanics, which is extremely dangerous.

4. The Black-Boxing of Memory Management and System Architecture

At an even deeper layer is the complete omission of the understanding of Memory Management and CPU architecture. Learners who have only been taught high-level languages with Garbage Collection (GC), like Python and JavaScript, in schools today will never in their lives be conscious of where variables and objects are physically allocated in RAM (heap vs. stack), how they are assigned, and when and how they are freed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


// Example of explicit and direct memory allocation and pointer manipulation in C
#include <stdio.h>
#include <stdlib.h>

int main() {
 int n = 1000000;
 // Dynamically allocate memory consecutively in the heap (System call to the OS)
 int *array = (int*)malloc(n * sizeof(int));

 if (array == NULL) {
 fprintf(stderr, "Memory allocation failed! Out of memory.\n");
 return 1;
 }

 // Initializing the array using pointer arithmetic
 for(int i = 0; i < n; i++) {
 *(array + i) = i * 2; // Equivalent to array[i] = i * 2
 }

 // Explicit resource deallocation to prevent Memory Leaks
 free(array);
 array = NULL; // Prevent dangling pointers

 return 0;
}

Knowledge of pointers (direct references to memory addresses), data placement to maximize CPU cache hierarchy (L1/L2/L3 cache) hit rates (Data Locality), and Race Conditions and mutual exclusion (Mutex/Semaphores) in multi-threaded environments are absolutely essential for developing high-performance backend systems, 3D game engines, or embedded systems for IoT. The current curriculum by the Ministry of Education, Culture, Sports, Science and Technology is focused solely on “running superficial applications” and has severely deviated from the original academic goal of “understanding the depths of computer science.”

5. The Wall of Databases and Persistence: The Absence of Relational Algebra

In modern applications, data saving and retrieval (persistence) is an unavoidable theme. However, much of school education remains stuck in “data processing in memory,” which disappears once the program finishes executing. The mathematical theory behind Relational Databases (RDBMS) and SQL, namely the “Relational Algebra” proposed by Dr. Edgar F. Codd, is rarely taught.

Database operations are defined by the following basic operations based on set theory:

Selection ($\sigma$): Extracting tuples (rows) that satisfy a condition
Projection ($\pi$): Extracting specific attributes (columns)
Join ($\bowtie$): Conditional intersection of multiple relations

Furthermore, learning the structure of the “B-Tree index” to instantly search for the desired data from a vast number of records is the best practical application of data structures. The B-Tree minimizes disk I/O operations while guaranteeing a search speed of $O(\log N)$. Without knowing the ACID properties (Atomicity, Consistency, Isolation, Durability) of a transaction, it is impossible to build robust systems.

While superficial security education like “Let’s make passwords complex” and “Don’t click on suspicious links” is conducted in information literacy education, the mathematics of “Cryptography” that fundamentally supports internet society is almost never taught.

The HTTPS communications and digital signatures we use every day are protected by public-key cryptography, such as RSA cryptography. The security of RSA relies on the mathematical difficulty (considered an NP-intermediate problem) that “the prime factorization of massive integers cannot be solved within a realistic time frame by current classical computers.”

The mathematical formulas underlying RSA cryptography are beautiful applications of Euler’s totient function and Fermat’s Little Theorem.

Choose two massive prime numbers $p$ and $q$
Calculate $n = p \times q$ (This becomes part of the public key)
Calculate $\phi(n) = (p-1)(q-1)$
Choose $e$ and $d$ such that $e \times d \equiv 1 \pmod{\phi(n)}$
Encryption: $C \equiv M^e \pmod{n}$
Decryption: $M \equiv C^d \pmod{n}$

In this way, programming education only truly unleashes its power when closely linked with mathematics education. The process of translating mathematical formulas into code and implementing them in society is the true essence of science.

7. The GIGA School Concept and the Hopeless Limits of Infrastructure: Chromebooks and Cloud IDEs

When discussing IT education in Japan, one cannot ignore the “GIGA School Concept,” a national project promoted by the Ministry of Education with a massive budget. This initiative to provide “one device per student” and high-speed network environments to elementary and junior high school students nationwide was expected to be a catalyst to catch up on the delay in digitalization. However, the hardware specs and architectures of the devices actually distributed have become a severe hindrance to full-scale programming education.

Low-Spec Devices and the Loss of Local Development Environments

Many of the devices introduced as standard under the GIGA School Concept are extremely cheap Chromebooks, iPads, or budget Windows devices. Their standard specs are as follows:

CPU: Intel Celeron or budget ARM processors
Memory (RAM): 4GB (barely enough to run a modern OS)
Storage (eMMC): 32GB to 64GB (extremely slow I/O speeds)

Due to these weak hardware constraints, it is virtually impossible to set up the “local development environments” that professional engineers use daily. Attempting to launch Linux containers using Docker, running heavy IDEs like Visual Studio Code with full features, or starting local Node.js or Python servers to install heavy libraries will immediately lead to memory exhaustion and system freezes.

As a result, educational frontlines are forced to rely entirely on cloud IDEs that run in the browser (such as Google Colaboratory, Replit, or lightweight web tools proprietary to textbook publishers).

flowchart LR
subgraph "GIGA Devices (Chromebook / iPad / Budget Windows)"
A["Web Browser (UI rendering only)"]
end
subgraph "Remote Cloud Infrastructure (AWS / GCP, etc.)"
B["Cloud IDE Web Server"]
C["Backend Compile/Execution Environment"]
D["Persistent File Storage"]
end
A -->|"HTTP/WebSocket Communication: Severe latency due to narrow school network lines"| B
B <--> C
B <--> D

Complete reliance on cloud IDEs causes the following critical educational deficiencies:

Lack of understanding of file systems and OS architecture: Without a local environment, students never acquire UNIX literacy—essential knowledge that IT engineers should use as naturally as breathing, such as directory structures, absolute and relative paths, environment variables, file permissions, and OS operations via the CLI (Command Line Interface).
Network latency and infrastructure vulnerabilities: Because constant connectivity is assumed, there are frequent nationwide incidents where school network bandwidth becomes congested the moment all students access it simultaneously, causing browsers to freeze and learning to completely stop.
Deprivation of version control (Git) experience: Students are robbed of the opportunity to have the concepts of Git and GitHub hammered into them through a black terminal screen, preventing them from learning how to manage source code history and collaborate globally.

When professional software engineers develop, operating within a terminal (shell) is the absolute foundation. Without the gritty experience of running commands like ls, cd, grep, chmod, and git rebase to interact directly with the local OS kernel, cultivating true IT talent is impossible. Playing solely within the sandbox of a Chromebook will never produce full-stack engineers who can oversee the entire system.

8. The Despairing Gap with the World: The Disconnect Between Industry Demands and School Education

The final and arguably national crisis-level challenge facing Japan’s IT education is the overwhelming decline in competitiveness in a global context.

Fierce Computer Science Education in Other Countries

In the UK, the subject “Computing” has been mandatory from age 5 (Key Stage 1) since as early as 2014. Their curriculum goes far beyond mere “programming experiences,” dealing with highly academic and systematic computer science, from logical algorithm design and understanding logic circuits via Boolean algebra, to network topologies and hardware architecture.

In the US, there are rigorous standard K-12 curriculums established by the CSTA (Computer Science Teachers Association). In the AP (Advanced Placement) Computer Science A course taken by high school students, they are tested on authentic object-oriented programming using Java, polymorphism, recursion, implementation of data structures, and algorithmic complexity evaluation at a level comparable to a first-year university course. The fierce STEM education in countries like India and China, and the depth of the elite talent they produce, hardly need further mention.

The Despairing Disconnect Between Required Skills and Taught Skills

The requirements that modern industries—especially globally expanding mega-ventures and tech giants (like GAFAM)—demand of new graduate software engineers are advancing at a terrifying speed every year. Extensive and deep expertise is required, including the construction of cloud-native infrastructure (AWS, GCP, Kubernetes), the design of distributed systems using microservice architectures, the implementation of machine learning pipelines, and advanced security knowledge.

The graph below conceptually illustrates the despairing gap between the skill attainment levels provided by Japan’s current school education and the skill levels demanded by the frontline industry.

xychart-beta
title Skills Provided by Japanese School Education vs. Skill Levels Demanded by Industry
x-axis ["Visual Languages", "Basic Syntax/Variables", "Algorithms/Complexity", "OS/Networks", "DB/System Design", "Cloud/Distributed Arch"]
y-axis "Attainment / Demand (%)" 0 --> 100
line "Current Attainment Level in School Education" [95, 60, 15, 5, 2, 0]
line "Level Demanded by Industry / Tech Companies" [0, 20, 85, 90, 95, 100]

To bridge this massive gap (Death Valley), a radical paradigm shift in school education and enormous investment are required. With an overwhelming nationwide shortage of specialized “Information” teachers, math, science, or technology and home economics teachers are currently teaching programming on the side with insufficient training. Under this system, Japan will never be able to produce top-tier engineers who can compete globally.

9. The Plummeting Value of “Coding” in the AI Era (LLMs)

Further complicating the situation is the explosive spread of Large Language Models (LLMs) like ChatGPT and AI coding assistants like GitHub Copilot. In an era where AI can instantly generate perfect code from natural language instructions and even write test codes, the market value of so-called “Coders” who merely “know Python syntax” or “know how to call an API” is rapidly plummeting.

What is required of human engineers in the AI era is not the memorization of programming language syntax. It is the following abilities:

Requirements Definition and Domain Modeling: The ability to extract complex real-world problems to be solved and model them as a system.
Architecture Design: The ability to draw a system-wide blueprint that guarantees scalability, availability, and maintainability.
Mathematical and Logical Verification: The ability to theoretically verify and prove whether the AI-generated code has security holes or computational bottlenecks.

Ironically, all of these lie not in “superficial programming,” but in the deep and abstract realms of “computer science and mathematics.” If Japanese education is only teaching “downstream skills that are easily replaced by AI,” it must be called a national loss.

10. Towards the Integration of Mathematical Sciences and Programming: A Proposal for Next-Generation Education

The urgent task for future IT education in Japan is to break away from “making programming the goal or a mere tool” and to return to the “exploration of computer science as a mathematical science.” A programming language is merely a tool to express thought, and the mathematical and logical structures underlying it possess universal value that will not fade even as times change.

For example, at the core of Artificial Intelligence (AI) and machine learning, linear algebra (matrix operations and tensors), multivariable calculus (gradient descent), and probability and statistics (Bayesian inference and information theory) are intricately intertwined. The optimization of weights in deep learning neural networks is formulated by the Chain Rule using partial derivatives and backpropagation.

$$ \frac{\partial L}{\partial w_{ij}^{(l)}} = \frac{\partial L}{\partial z_i^{(l+1)}} \cdot \frac{\partial z_i^{(l+1)}}{\partial w_{ij}^{(l)}} = \delta_i^{(l+1)} \cdot a_j^{(l)} $$

Professionals who can translate such advanced mathematical formulas into code, and optimally implement Parallel Computing while remaining conscious of GPU (CUDA) and TPU hardware architectures, are the ones who will lead the next-generation IT industry. That is why we must immediately steer away from superficial education that just makes students memorize syntax, and shift towards profound education that questions the First Principles of computation.

11. Conclusion: The Steep Path to a True IT Nation and Our Resolve

There is no doubt that making programming education mandatory in the 2020s was a solid step forward in terms of making Japanese society as a whole widely recognize the “importance of IT and information.” However, it is merely “warm-up exercises” in a long journey.

We must step beyond the fun of moving a cat character in Scratch, move students with the mathematical beauty of an $O(N \log N)$ algorithm, and teach them the excitement of conversing with servers around the world via TCP packets from a black terminal screen. We must rebuild new educational infrastructures to overcome the hardware constraints of the GIGA School Concept, train and deploy instructors with advanced CS expertise, and sometimes boldly involve external professional engineers in school education.

The challenges facing Japan’s IT education are extremely deep, persistent, and complex. However, if we do not avert our eyes from these issues, and if industry, academia, and government work together in earnest to build an ecosystem that continuously produces not just “laborers who can write code according to specifications,” but “genuine engineers who can design and create systems from scratch,” Japan will once again be able to lead the world as a true IT nation.

How we fight through the “aftermath” of mandatory programming—the most difficult and important phase—is testing the absolute seriousness and resolve of us adults right now.

This article outlined the computational complexity theory and the infrastructural limits of the GIGA School Concept. We plan to cover more specialized computer science topics (such as details of distributed system algorithms and low-level memory management techniques) sequentially in future series.

The Deepening of the 'New Digital Divide' Brought by the Evolution of Generative AI

Sat, 12 Sep 2026 12:00:00 +0900

1. Introduction: Historical Transition of the Digital Divide and the New Paradigm

Since the popularization of the internet, we have often heard the term “digital divide” (information gap). The early digital divide was primarily about “physical access rights.” In other words, it was a simple scenario where whether or not one had a computer or high-speed internet connection determined access to information and economic opportunities. Later, as smartphones and broadband connections became commoditized, the focus of the divide shifted to “IT literacy” (information utilization capability). This involved the software and cognitive aspects, such as whether one could appropriately search for information using search engines or master software.

However, the sudden emergence of Generative AI and the evolution of Large Language Models (LLMs) in the 2020s are fundamentally overturning this concept of the digital divide. What we are facing now is not merely a “divide in access to information” or a “divide in software operation skills.” It is a “divide in the ability to orchestrate (direct and integrate) AI,” a profound and irreversible “3rd Digital Divide” that determines whether an individual’s productivity is amplified exponentially or if they are left behind by the evolution of AI and lose relative value.

In this article, we will unravel in great detail the true nature of this new digital divide brought about by Generative AI from three layers: the mathematical model of productivity, hardware architecture and cost, and the cognitive aspects of human beings.

2. From “Access” to “Orchestration”: The Arrival of the 3rd Digital Divide

Past software tools were essentially “passive instruments.” The limitation of traditional software was that it returned deterministic results in response to the user’s explicit input (e.g., entering a formula in spreadsheet software to get a calculated result). However, current Generative AI, especially LLMs based on the Transformer architecture (GPT-4, Claude 3.5, Llama 3, etc.), act as “fragments of active intelligence.”

Due to this paradigm shift, the required skill set for humans has dramatically changed from the “ability to operate tools” to the “ability to combine multiple AI agents and tools, and to design and direct autonomous workflows (AI Orchestration).” This can be called “AI Orchestration Literacy.”

Below is the transition of the digital divide from the past to the present.

flowchart TD
A["1st Divide: Access to Hardware and Infrastructure (1990s-2000s)"] --> B["2nd Divide: IT Literacy and Information Retrieval Skills (2010s)"]
B --> C["3rd Divide: Prompting and Orchestration of Generative AI (2020s-)"]
C --> D["Designing Autonomous Task Execution by AI"]
C --> E["Integration of Multiple AI Agents (Agentic Workflows)"]
C --> F["Advanced Information Verification and Hallucination Detection"]

Moving beyond the boundaries of prompt engineering, we have now entered a stage where systems are made to autonomously solve problems using multi-agent frameworks like LangChain, AutoGen, and CrewAI. Between the “class that draws the blueprints and lets AI execute them” and the “class that still performs routine work manually,” a divergence in productivity is occurring at a speed that humanity has never experienced before.

3. The Matthew Effect of Productivity: Visualizing the Divide through a Mathematical Approach

The “Matthew Effect,” derived from the New Testament saying “For to everyone who has, more will be given, and he will have abundance; but from him who does not have, even what he has will be taken away,” refers in sociology and economics to a phenomenon where early advantages lead to cumulative benefits. With the introduction of Generative AI, this Matthew Effect is strongly manifesting in the labor market and knowledge production.

The productivity of an individual who effectively uses AI grows exponentially, not linearly, with time. This is because the time saved by AI can be further invested in building more advanced AI systems, optimizing prompts, and self-learning. Let us express this with a mathematical model.

The productivity of a non-AI user $P_{human}(t)$ and the productivity of an AI orchestrator $P_{AI}(t)$ at a given time $t$ can be represented by the following models, respectively.

$$ P_{human}(t) = P_0 (1 + r_{human})^t $$

Here, $P_0$ is the initial productivity, and $r_{human}$ is the natural human learning rate (growth rate based on the experience curve). Generally, $r_{human}$ is very small, and growth tends to be arithmetic.

On the other hand, the productivity of a user who fully utilizes AI combines the capability improvement rate of the AI model being used, $r_{model}$, and the compound interest effect of workflow automation by AI, $\alpha$.

$$ P_{AI}(t) = P_0 \cdot \exp\left( \int_0^t (r_{human} + \alpha \cdot r_{model}(\tau)) d\tau \right) $$

Because the AI model itself is evolving exponentially (an increase in parameter count and computational complexity based on scaling laws), $r_{model}(t)$ itself increases over time. As a result, the difference in productivity between the two, $\Delta P(t)$, rapidly widens.

$$ \Delta P(t) = P_{AI}(t) - P_{human}(t) $$

The graph below visually shows this divergence.

xychart-beta
title Productivity Divergence Over Time (The Matthew Effect)
x-axis ["Year 1", "Year 2", "Year 3", "Year 4", "Year 5", "Year 6"]
y-axis "Output Volume" 0 --> 200
line [10, 15, 30, 60, 110, 180]
line [10, 12, 14, 16, 18, 20]

(Note: The blue line represents the productivity of the AI orchestrator, and the lower line represents the productivity of the non-AI user)

In the first year, the difference seems negligible, but each time the AI model evolves from GPT-3 to GPT-4, and further to its next generation, AI users enjoy a dramatic leap in productivity simply by plugging the new model into their existing automation pipelines. It becomes mathematically closer to impossible over time for non-AI users to close this gap.

4. The Hardware Divide: The Wall of Local Inference and the Trap of Cloud APIs

The 3rd Digital Divide creates a new hardware gap not just in software skills, but in “access to compute (computational resources)” needed to run cutting-edge AI models.

To utilize Large Language Models, there are mainly two approaches: “using Cloud APIs” or “running the model locally for inference.” Both have their pros and cons, which are forming a new economic and physical wall.

Limitations and Running Costs of Cloud APIs

State-of-the-art frontier models provided by OpenAI, Anthropic, and Google (GPT-4o, Claude 3.5 Sonnet, etc.) are generally accessed via API. However, if you build a highly autonomous agent (Agentic Workflow) that generates tens of thousands of API calls per day, the costs explode.

The total cost of the API, $C_{cloud}$, depends on the volume of input and output tokens.

$$ C_{cloud} = \sum_{i=1}^{N} \left( c_{in} \cdot T_{in}^{(i)} + c_{out} \cdot T_{out}^{(i)} \right) $$

(Where $N$ is the number of requests, $T$ is the number of tokens, and $c$ is the unit price per token)

When continuously performing large-scale data processing or vectorization for RAG (Retrieval-Augmented Generation), this variable cost can become a fatal burden for individual developers and small to medium-sized enterprises.

The Wall of Local LLMs and VRAM

From the perspective of avoiding cloud costs and maintaining data privacy, the demand for running open-weight models like Meta’s Llama 3 and Mistral locally is increasing. However, the physical divide known as the “Wall of VRAM (Video RAM)” stands in the way here.

The inference speed of an LLM depends more strongly on Memory Bandwidth rather than the calculation performance (FLOPS) of the GPU (it has a Memory-bound nature). Assuming the number of parameters of the model is $P$ and the precision is 16-bit (2 bytes), just loading the model into memory requires at least $2P$ bytes of VRAM. For example, a 70 billion (70B) parameter model demands over 140GB of VRAM.

$$ VRAM_{required} \approx \left( \frac{P \times bits\_per\_weight}{8} \right) + Context\_Memory $$

Even with high-end GPUs available to general consumers (like the NVIDIA RTX 4090), VRAM is limited to 24GB, making it impossible to run a 70B class model as is. Here, “Quantization” technologies like AWQ and GGUF have emerged, and a technical struggle is taking place to find a compromise by compressing weights to 4-bit or 8-bit, but performance degradation (worsening of Perplexity) due to quantization is inevitable.

Furthermore, in recent years, “AI PCs” equipped with NPUs (Neural Processing Units) have appeared, but the TOPS (Tera Operations Per Second) of current NPUs can only handle lightweight, small-scale models (SLMs: Small Language Models) at best. To truly perform highly advanced inference locally, you need the capital to build a multi-GPU environment costing millions of yen. This is the true nature of the “capital-intensive digital divide” in AI.

5. The Cognitive Divide: The Loop of Hallucination and Verification

What is more terrifying than the disparity in hardware and skills is the “Cognitive Divide.” AI generates highly fluent and persuasive text, but at the same time, it causes “hallucinations,” outputting completely baseless information as if it were plausible.

The divide that arises here is the separation between “the class that critically examines and verifies (fact-checks) AI output” and “the class that blindly believes AI output as an authoritative truth.” The former utilizes AI as a powerful brainstorming and drafting tool, performing quality assurance (QA) on the final output using their own domain knowledge. The latter sends incorrect information out into the world as is, which not only ruins their own credibility but also contributes to polluting the internet’s information space with spam-like content.

The process of the Cognitive Verification Loop to prevent this is shown below.

flowchart TD
A["Human Intent"] --> B["Prompt Input to AI (Prompting)"]
B --> C["Generation by AI Model (Generation)"]
C --> D{"Cognitive Verification"}
D -- Doubts / Logical Failures Exist --> E["Fact-checking using RAG or external tools"]
E --> F["Readjusting / Refining Prompts"]
F --> B
D -- Facts and Logic are Valid --> G["Final adjustments based on human domain knowledge"]
G --> H["Output of Final Deliverable"]

To iterate through this loop, one needs not only to understand how to use AI but also to possess deep “domain knowledge” and “critical thinking” concerning the output domain. Ironically, the more AI evolves, the more the requirements for humans shift away from basic operational skills to highly advanced cognitive abilities, such as philosophical and logical reasoning and the cultural sophistication to distinguish truth from falsehood.

6. The New Class Society: AI Orchestrators and Manual Workers

In a future where these disparities have reached their limits (or a reality currently unfolding), the labor market will polarize in unprecedented ways.

1. AI Orchestrators (Top 1-5%) In their fields of expertise, they construct workflows that autonomously run multiple AI agents. They delegate the majority of processes, such as research, coding, data analysis, and report generation, to AI, specializing themselves in “process design,” “exception handling,” and “final decision-making.” Their productivity reaches tens to hundreds of times that of traditional workers, creating immense economic value.

2. Traditional Knowledge Workers / Manual Workers These are people who write code with their own hands, operate Excel with their own hands, and write text with their own hands. Their jobs will gradually be replaced by AI, or they will be relegated to “end-point monitoring and maintenance” of systems created by AI orchestrators, or forced into “labor in physical space.” Intellectual labor that does not utilize AI faces the risk of entirely losing its market competitiveness.

Amidst this overwhelming divide, how should individuals, corporations, and society adapt?

Individual Strategies: Adapting to the Paradigm Shift

The most important thing is to discard the underestimation that “AI is just a chatbot.” It is necessary to develop the habit of treating AI as an “advanced intern” or a “team of experts” and constantly thinking about how you can break down your own work processes and delegate them to AI (Task Decomposition). Also, even if you cannot program, learning about the concept of APIs and data structuring (like JSON) enables powerful automation by combining No-Code/Low-Code tools (Zapier, Make, etc.) with AI.

Corporate Strategies: AI-Native Organizational Design

For companies, simply “distributing ChatGPT accounts” is not enough. It requires infrastructure investment, such as redesigning the entire workflow around AI (BPR: Business Process Re-engineering), building a secure RAG environment, and fine-tuning local models with internal proprietary knowledge. Furthermore, introducing new KPIs to evaluate employees’ AI orchestration capabilities is also required.

At the national and societal levels, safety nets and education are needed so that the 3rd Digital Divide does not lead to severe economic disparities and social unrest. For instance, public support for the R&D of open-source AI models and making “Critical AI Literacy” compulsory in educational institutions can be considered. In addition, updating appropriate legal regulations and antitrust laws to prevent the “monopolization of AI models and computing resources” by Big Tech companies should be brought to the table for discussion.

8. Conclusion: Ride the Wave of Evolution, or Be Swallowed by It

The “new digital divide” caused by Generative AI is restructuring our society more rapidly and broadly than any technological innovation in the past. This divide appears as a difference in hardware computational resources, the ability to invest in Cloud APIs, and above all, the “cognitive and logical skills to orchestrate AI.”

As the Matthew Effect of productivity indicates, this gap will expand unbridgeably over time. What we must do now is neither to fear the evolution of AI nor to blindly believe in it. It is to deeply understand the characteristics of AI, the greatest Intelligence Amplifier in human history, and decisively execute an “intellectual self-transformation” to update our own thinking and workflows.

Will we stand on this side of the new digital divide, or remain on the other side? That choice is entrusted to our daily learning and actions, right at this very moment.

If you have any opinions on this article or specific case studies on introducing AI orchestration, please send them to the comment section or the author’s social media.

The Trade-off Between Privacy and Convenience: The Fate of Personal Information in the Era of Big Data

Sat, 12 Sep 2026 12:00:00 +0900

The Trade-off Between Privacy and Convenience: The Fate of Personal Information in the Era of Big Data

In today’s digital society, we generate enormous amounts of data in our daily lives. A wide variety of “big data”—such as smartphone location information, social media posts, online shopping purchase histories, and health data recorded by wearable devices—is constantly being collected. This data is essential for the evolution of AI (Artificial Intelligence) and the provision of personalized services, making our lives more convenient and richer.

However, on the other hand, the risk of privacy infringement associated with the collection and use of personal information has emerged as a serious social issue. The risks lurking behind convenience have reached an undeniable scale, including data breaches, the provision of data to third parties without user consent, and concerns about a surveillance society by the state. This article provides an extremely detailed technical explanation of how technology and legal regulations are approaching the modern dilemma of this “trade-off between privacy and convenience,” along with the latest trends.

1. The Paradigm of a Data-Driven Society and the Evolution of Data Architecture

To collect and utilize data efficiently, companies are adopting various data architectures. There is an ongoing paradigm shift from the once-mainstream “Data Warehouse” to a “Data Lake” that centrally manages all data, including unstructured data, and now to a “Data Mesh,” which is a decentralized architecture.

Centralized Data Lakes and Anonymization Pipelines

A data lake is a storage repository that stores large amounts of raw data in its native format. However, using raw data containing PII (Personally Identifiable Information) directly for analysis causes serious compliance violations. Therefore, a strict “Anonymization Pipeline” is implemented between the data lake and the analysis environment.

The figure below shows the flow of an anonymization pipeline in a typical centralized data lake.

flowchart TD
A["Data Sources (Web, IoT, Mobile)"] -->|"Ingestion"| B["Raw Data Zone (Untouched)"]
B -->|"ETL Process"| C["Anonymization & Cleansing Pipeline"]
C -->|"Pseudonymization / Tokenization"| D["Trusted Zone (k-anonymized)"]
D -->|"Feature Engineering"| E["Refined Zone (Ready for ML)"]
E -->|"Model Training"| F["BI Tools & ML Models"]
C -->|"Audit Logs"| G["Security & Compliance Hub"]

In such pipelines, processes like hashing, masking, and encryption are automatically applied when data flows in. However, as discussed later, simple masking or pseudonymization cannot completely eliminate the risk of “Re-identification” through matching with other data sources.

2. A Deep Understanding of Privacy-Enhancing Technologies (PETs)

The key to balancing privacy and data utilization is “Privacy-Enhancing Technologies (PETs)”. Here, we provide detailed mathematical definitions and technical implementations of the major PETs that play extremely important roles in modern big data analysis and machine learning.

2.1 K-Anonymity and Its Extensions

Proposed by Latanya Sweeney and Pierangela Samarati in 1998, “k-anonymity” is a foundational concept for privacy protection in data publication. It means ensuring that every record in a dataset is indistinguishable from at least $k-1$ other records.

Attributes in a database are broadly classified into the following three categories:

Explicit Identifiers: Information that can directly identify an individual, such as names and social security numbers (these are usually deleted or encrypted).
Quasi-Identifiers (QIs): Information that cannot identify an individual on its own, such as age, gender, and zip code, but can be used to identify them when combined.
Sensitive Attributes: Information that should be protected, such as medical conditions or annual income.

K-anonymity guarantees that there are always at least $k$ combinations of quasi-identifiers (Equivalence Classes). However, k-anonymity is vulnerable to “Homogeneity Attacks” and “Background Knowledge Attacks”. For example, if all $k$ people belonging to an equivalence class have the same medical condition (sensitive attribute), the condition will be identified even if k-anonymity is maintained.

To overcome this, the following extended models have been proposed:

l-diversity: Guarantees that sensitive attributes have at least $l$ different values in each equivalence class.
t-closeness: Ensures that the distance (such as Earth Mover’s Distance) between the distribution of sensitive attributes in each equivalence class and the distribution of sensitive attributes in the entire dataset is less than or equal to a threshold $t$.

2.2 Differential Privacy (DP)

Overcoming the limitations of the k-anonymity model, “Differential Privacy” proposed by Cynthia Dwork et al. in 2006 is now widely adopted as the strongest and most mathematically rigorous privacy standard. Tech giants such as Apple, Google, and Microsoft apply this $\epsilon$-differential privacy when collecting telemetry and statistical data from users.

Mathematical Definition of Differential Privacy

A randomized algorithm $\mathcal{M}$ satisfies $\epsilon$-differential privacy if, for any two adjacent datasets $D$ and $D'$ differing by only one record (i.e., $\|D - D'\|_1 = 1$), and for any subset of outputs $S \subseteq \text{Range}(\mathcal{M})$, the following inequality holds:

$$ \Pr[\mathcal{M}(D) \in S] \le e^\epsilon \Pr[\mathcal{M}(D') \in S] $$

Here, $\epsilon$ (privacy budget) is a non-negative parameter that controls the level of privacy protection. The smaller the $\epsilon$, the stronger the privacy protection, but the utility of the data decreases.

Furthermore, $(\epsilon, \delta)$-differential privacy, a relaxed model that allows the privacy guarantee to be broken with a very small probability $\delta$, is also widely used.

$$ \Pr[\mathcal{M}(D) \in S] \le e^\epsilon \Pr[\mathcal{M}(D') \in S] + \delta $$

Laplace Mechanism

A typical method for achieving differential privacy is the “Laplace Mechanism,” which intentionally adds noise (random numbers) following a specific distribution to the true output result of a query. How much noise should be added depends on the “Global Sensitivity” $\Delta f$ of the function $f$.

The global sensitivity $\Delta f$ is defined as the maximum change in the output of the function $f$ for any adjacent datasets $D, D'$.

$$ \Delta f = \max_{D, D'} \| f(D) - f(D') \|_1 $$

The Laplace mechanism adds noise $Y$ sampled from the Laplace distribution $\text{Lap}(b)$ with scale parameter $b = \frac{\Delta f}{\epsilon}$ to the result of the function $f(D)$.

$$ \mathcal{M}(D) = f(D) + Y, \quad Y \sim \text{Lap}\left(\frac{\Delta f}{\epsilon}\right) $$

The probability density function of the Laplace distribution is as follows:

$$ p(x \mid b) = \frac{1}{2b} \exp\left( - \frac{|x|}{b} \right) $$

By this noise injection, it becomes impossible to infer whether a specific individual is included in the dataset from the output result. Companies leverage DP as a technology that masks individual data itself while maintaining the utility of statistical trends (mean, variance, counts, etc.) of the entire data.

2.3 Federated Learning (FL)

Traditional machine learning took a centralized approach, where massive amounts of data were aggregated on a central server to train models, like the aforementioned data lakes. However, sending sensitive data such as medical images or smartphone input histories to a central server involves significant privacy risks.

Therefore, “Federated Learning” was proposed by Google in 2016. In federated learning, rather than moving the data itself, the “computational processing of the model” is moved to the edge devices (smartphones, hospital servers, etc.) where the data resides.

flowchart TD
Server["Central Aggregation Server"]
Device1["Edge Device 1 (Smartphone)"]
Device2["Edge Device 2 (Smartphone)"]
Device3["Edge Device 3 (Smartphone)"]
Server -->|"1. Broadcast Global Model Weights"| Device1
Server -->|"1. Broadcast Global Model Weights"| Device2
Server -->|"1. Broadcast Global Model Weights"| Device3
Device1 -->|"2. Local Training on Private Data"| Device1
Device2 -->|"2. Local Training on Private Data"| Device2
Device3 -->|"2. Local Training on Private Data"| Device3
Device1 -->|"3. Transmit Model Gradients/Updates"| Server
Device2 -->|"3. Transmit Model Gradients/Updates"| Server
Device3 -->|"3. Transmit Model Gradients/Updates"| Server
Server -->|"4. Aggregation (FedAvg)"| Server
Server -->|"5. Update Global Model"| Server

Federated Averaging (FedAvg) Algorithm

A representative aggregation algorithm in federated learning is FedAvg. Each client $k$ performs several epochs of local training using Stochastic Gradient Descent (SGD) on their own dataset $D_k$ (of size $n_k$) to calculate the updated weights $w_{t+1}^k$.

The central server receives the weights from the $K$ participating clients and updates the global model weights $w_{t+1}$ by taking a weighted average of these weights according to the data size. Letting the total number of data samples be $n = \sum_{k=1}^K n_k$, the update equation is as follows:

$$ w_{t+1} = \sum_{k=1}^K \frac{n_k}{n} w_{t+1}^k $$

This makes it possible to build intelligent AI models without an individual’s raw data (message history, photos, etc.) ever leaving the device. Typical applications include improving the next-word prediction feature of Google Keyboard (Gboard) and Apple’s FaceID and Hey Siri voice recognition models.

2.4 Homomorphic Encryption (HE)

“Homomorphic Encryption” is a “magical” cryptographic technology that allows computations (such as addition and multiplication) to be performed on data while it remains in an encrypted state. With normal encryption methods, when performing computations on data, it must first be decrypted (returned to plaintext), but decrypting on a cloud server creates a security vulnerability.

By using homomorphic encryption, the following properties are achieved. Letting the encryption function be $E(\cdot)$, the addition and multiplication of plaintexts $m_1$ and $m_2$ become possible with operations ($\oplus$ and $\otimes$) directly on the ciphertext.

$$ E(m_1 + m_2) = E(m_1) \oplus E(m_2) $$$$ E(m_1 \times m_2) = E(m_1) \otimes E(m_2) $$

Homomorphic encryption is divided into “Partially Homomorphic Encryption (PHE),” which allows either addition or multiplication, and “Fully Homomorphic Encryption (FHE),” which allows both addition and multiplication an unlimited number of times. Since Craig Gentry constructed the first FHE scheme using lattice-based cryptography in 2009, it has been a major breakthrough in cryptography.

Currently, challenges such as computational cost and increased ciphertext size (overhead) remain, but it is expected to be applied to the secure analysis of medical data on the cloud and secure multi-party computation among financial institutions.

In parallel with technological evolution, the development of legal frameworks is advancing rapidly worldwide. When companies utilize big data, complying with these legal regulations is a strict requirement. Let’s compare the two most influential regulatory frameworks.

Enforced in May 2018, the EU’s GDPR (General Data Protection Regulation) is recognized as the global “gold standard” for personal data protection. GDPR applies to all organizations that handle the data of individuals within the EU, and violations result in massive fines of either up to 4% of global annual turnover or 20 million euros, whichever is higher.

Key Features of GDPR:

Opt-in Principle: Explicit and freely given prior consent from users is required for data collection and processing.
Right to be Forgotten / Right to Erasure: Users have the right to request companies to completely erase their personal data. Data must also be deleted from data lake backups, which is an extremely difficult technical requirement.
Data Controllers and Data Processors: It strictly defines the responsibilities of those who determine the purposes of using the data (controllers) and those who process the data according to those instructions (processors).

California Consumer Privacy Act (CCPA/CPRA)

While there is no comprehensive privacy law at the federal level in the US, the CCPA (California Consumer Privacy Act) enacted in California in 2020 serves as the de facto national standard. It was later further strengthened by the CPRA (California Privacy Rights Act).

Key Features of CCPA:

Opt-out Principle: Unlike GDPR’s “prior consent,” data collection is possible without prior consent, but it is mandatory to provide users with a clear opt-out link stating “Do Not Sell My Personal Information.”
Right to Access Data: Consumers can request the disclosure of specific pieces of information collected by the company, their categories, sources, and whether they have been sold to third parties.

These regulations strongly demand that companies adopt “Privacy by Design”—embedding privacy protection from the design stage of systems and processes.

4. Implementation Challenges in the Data Ecosystem

Let’s look at the implementation perspective when applying privacy-enhancing technologies and legal regulations to actual big data environments. For example, assume a case of implementing k-anonymization or differential privacy in a data lake using Python and Pandas, or PySpark.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


# Conceptual implementation of data aggregation applying differential privacy (Python)
import numpy as np
import pandas as pd

def laplace_mechanism(true_value, sensitivity, epsilon):
 """
 Function to add Laplace noise to the true value
 """
 scale = sensitivity / epsilon
 noise = np.random.laplace(loc=0, scale=scale)
 return true_value + noise

def get_dp_average_salary(dataframe, epsilon=1.0):
 """
 Calculate the average salary with guaranteed differential privacy
 """
 # Actual calculation
 true_sum = dataframe['salary'].sum()
 true_count = len(dataframe)

 # Application of differential privacy (based on sensitivity assumption)
 # Assume the fluctuation of the maximum salary as sensitivity (more strictly, clipping is necessary)
 max_salary_diff = 100000

 # Adding noise (DP can be applied to both the sum and the count)
 noisy_sum = laplace_mechanism(true_sum, max_salary_diff, epsilon / 2)
 noisy_count = laplace_mechanism(true_count, 1, epsilon / 2)

 return noisy_sum / noisy_count

# Execution within the data pipeline
# dp_avg_salary = get_dp_average_salary(raw_df, epsilon=0.5)

As seen in this code snippet, the implementation of differential privacy itself is as simple as adding noise; however, in actual operations, managing the “privacy budget ($\epsilon$)” becomes extremely difficult. Issuing multiple queries to the same dataset consumes the privacy budget (based on the composition theorem), and ultimately it is necessary to build a mechanism (Privacy Budget Management) that locks the entire dataset or rejects queries.

5. Future Outlook and Ethical Issues

The trade-off between big data and privacy is not a zero-sum game. With the evolution of PETs such as differential privacy, federated learning, and homomorphic encryption, a new data utilization paradigm of “sharing insights without sharing data” is becoming a reality.

Furthermore, in recent years, connected with the concepts of “Data Mesh” and “Web3” (decentralized web), the movement to take back “Data Sovereignty” from giant platformers to individuals is accelerating. A future is being discussed where individual data is stored in Personal Data Stores (PDS) or data wallets, and users themselves control the licensing and monetization of their data.

However, technological solutions are not perfect. In federated learning, there is a threat of “Poisoning Attacks,” where malicious clients send fraudulent model updates to corrupt the global model. In differential privacy, an ethical issue has also been pointed out where data of minorities is drowned out by noise, creating bias in AI models.

Conclusion

The fate of personal information in the era of big data poses a fundamental question that goes beyond a mere technical issue: what kind of society do we desire? How can we protect individual dignity and privacy while enjoying convenience? We can only arrive at a sustainable solution through a trinity: the establishment of legal regulations, the continuous innovation of privacy-enhancing technologies, and the high literacy of each of us who provide the data. Privacy and convenience are no longer a trade-off, but will evolve into “essential requirements” that can be achieved simultaneously through the latest technologies.

Society on kenji.blog

Can Technology Bridge the Social Divide? (A Proposal from an Engineer)

Introduction: On the Occasion of My Memorable 100th Post

Chapter 1: The Mathematical Structure of “Echo Chambers” through the Lens of Network Theory

Chapter 2: Acceleration of Polarization by Algorithms and the Information Diffusion Model

Chapter 3: Technical Solution (1) Bridging Algorithms and Community Notes

Chapter 4: Technical Solution (2) Decentralized SNS Protocols (AT Protocol / ActivityPub)

Chapter 5: The Philosophy of Open Source and the Social Responsibility of Engineers

Conclusion: Finishing the 100th Article

Deepfake and Information Literacy: How to Technically Detect Fake News

Introduction: The Era Where the Boundary Between Reality and Fiction Melts

1. The Mechanisms of Generative AI Supporting Deepfakes

1.1 Generative Adversarial Networks (GAN)

1.2 Diffusion Models

2. Digital Forensics: Techniques to Search for Traces of Generated Artifacts

2.1 Frequency Domain Analysis and DCT (Discrete Cosine Transform)

3. Detection of Biological Signals: Confirming the “Beat of Life” via rPPG

4. The Never-ending “Cat-and-Mouse Game”: Adversarial Training and Evasion Techniques

5. Fundamental Countermeasures: Provenance Proof and the C2PA Framework

5.1 How C2PA Works

5.2 Visualization with the “Content Credentials” Icon

6. Next-Generation Information Literacy: What We Can Do

Conclusion

How SNS Algorithms Influence Our Thinking and Tech Selection

1. Introduction: Democratization of Tech Information and the Rise of Algorithms

2. Evolution and Mechanism of Recommendation Algorithms

2.1 Collaborative Filtering and Matrix Factorization

2.2 Deep Learning Recommendation Model (DLRM)

2.3 Reinforcement Learning and Multi-Armed Bandits

3. The Mathematics of Echo Chambers and Filter Bubbles

4. Hype Driven Development (HDD) Created by Algorithms

5. “Escaping the Algorithm” Strategies in Tech Selection

5.1 Returning to Primary Information: Source Code and RFCs

5.2 Close Reading of Academic Papers and Whitepapers

5.3 Building an In-House Decision-Making Framework

5.4 The Philosophy of the Boring Technology Club

6. Conclusion: How We Should Face Technology

The Current State and Challenges of IT Education in Japan: After the Mandatory Programming Education

1. Introduction: The Light and Shadow Brought by Mandatory Programming Education

2. The Trap of Visual Programming: The Deep and Steep Chasm from Scratch to Text Coding

The Wall of Abstraction and the Black-Boxing of Computer Science

3. The Limits of Coding Without “Mathematics” and “Discrete Logic”: An Approach from Computational Complexity Theory

The Absolute Dominance of Time Complexity and Space Complexity (Big O Notation)

4. The Black-Boxing of Memory Management and System Architecture

5. The Wall of Databases and Persistence: The Absence of Relational Algebra

6. Security and Cryptography: The Social Infrastructure Supported by the Difficulty of Prime Factorization

7. The GIGA School Concept and the Hopeless Limits of Infrastructure: Chromebooks and Cloud IDEs

Low-Spec Devices and the Loss of Local Development Environments

8. The Despairing Gap with the World: The Disconnect Between Industry Demands and School Education

Fierce Computer Science Education in Other Countries

The Despairing Disconnect Between Required Skills and Taught Skills

9. The Plummeting Value of “Coding” in the AI Era (LLMs)

10. Towards the Integration of Mathematical Sciences and Programming: A Proposal for Next-Generation Education

11. Conclusion: The Steep Path to a True IT Nation and Our Resolve

The Deepening of the 'New Digital Divide' Brought by the Evolution of Generative AI

1. Introduction: Historical Transition of the Digital Divide and the New Paradigm

2. From “Access” to “Orchestration”: The Arrival of the 3rd Digital Divide

3. The Matthew Effect of Productivity: Visualizing the Divide through a Mathematical Approach

4. The Hardware Divide: The Wall of Local Inference and the Trap of Cloud APIs

Limitations and Running Costs of Cloud APIs

The Wall of Local LLMs and VRAM

5. The Cognitive Divide: The Loop of Hallucination and Verification

6. The New Class Society: AI Orchestrators and Manual Workers

7. Strategies and Social Prescriptions for Surviving the Stratified Society

Individual Strategies: Adapting to the Paradigm Shift

Corporate Strategies: AI-Native Organizational Design

Social Prescriptions: AI Infrastructure as a Public Good

8. Conclusion: Ride the Wave of Evolution, or Be Swallowed by It

The Trade-off Between Privacy and Convenience: The Fate of Personal Information in the Era of Big Data

The Trade-off Between Privacy and Convenience: The Fate of Personal Information in the Era of Big Data

1. The Paradigm of a Data-Driven Society and the Evolution of Data Architecture

Centralized Data Lakes and Anonymization Pipelines

2. A Deep Understanding of Privacy-Enhancing Technologies (PETs)

2.1 K-Anonymity and Its Extensions

2.2 Differential Privacy (DP)

Mathematical Definition of Differential Privacy

Laplace Mechanism

2.3 Federated Learning (FL)

Federated Averaging (FedAvg) Algorithm

2.4 Homomorphic Encryption (HE)