<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Statistics on kenji.blog</title><link>http://kenji.blog/en/tags/statistics/</link><description>Recent content in Statistics on kenji.blog</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>kenjinote</copyright><lastBuildDate>Thu, 10 Sep 2026 21:00:00 +0900</lastBuildDate><atom:link href="http://kenji.blog/en/tags/statistics/index.xml" rel="self" type="application/rss+xml"/><item><title>Your Friends Have More Friends Than You Do: The Friendship Paradox</title><link>http://kenji.blog/en/p/friendship-paradox/</link><pubDate>Thu, 10 Sep 2026 21:00:00 +0900</pubDate><guid>http://kenji.blog/en/p/friendship-paradox/</guid><description>&lt;img src="http://kenji.blog/p/friendship-paradox/img/friendship_paradox.jpg" alt="Featured image of post Your Friends Have More Friends Than You Do: The Friendship Paradox" />&lt;p>&amp;ldquo;People around me seem to have more friends and have more fun than I do&amp;hellip;&amp;rdquo;
Have you ever felt this way while scrolling through social media?&lt;/p>
&lt;p>Actually, you feeling this way is not because of your personality or lack of popularity. It is a mathematical fact proven by network theory and statistics, known as the &lt;strong>&amp;ldquo;Friendship Paradox&amp;rdquo;&lt;/strong>.&lt;/p>
&lt;p>Discovered in 1991 by sociologist Scott Feld, this paradox explains the counterintuitive phenomenon that &amp;ldquo;most people have fewer friends than their friends do.&amp;rdquo;&lt;/p>
&lt;h2 id="why-do-friends-have-more-friends">Why do &amp;ldquo;Friends Have More Friends&amp;rdquo;?
&lt;/h2>&lt;p>To put it simply, this is due to a simple sampling bias: &lt;strong>&amp;ldquo;People with many friends (popular people) appear on many people&amp;rsquo;s friend lists.&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>Let&amp;rsquo;s consider a simple network (graph).&lt;/p>
&lt;div class="mermaid">graph TD
A["Alice (1 friend)"] --- C["Charlie (3 friends)"]
B["Bob (1 friend)"] --- C
C --- D["David (1 friend)"]
style A fill:#4FC3F7,stroke:#333,stroke-width:2px
style B fill:#4FC3F7,stroke:#333,stroke-width:2px
style C fill:#FF9800,stroke:#333,stroke-width:4px
style D fill:#4FC3F7,stroke:#333,stroke-width:2px&lt;/div>
&lt;p>In this small world, there are four people: Alice, Bob, Charlie, and David.
Charlie is the &amp;ldquo;popular one&amp;rdquo; and is friends with all three of the others. The other three are only friends with Charlie.&lt;/p>
&lt;p>Let&amp;rsquo;s look at the number of friends each person has.&lt;/p>
&lt;ul>
&lt;li>Alice&amp;rsquo;s number of friends: 1&lt;/li>
&lt;li>Bob&amp;rsquo;s number of friends: 1&lt;/li>
&lt;li>David&amp;rsquo;s number of friends: 1&lt;/li>
&lt;li>Charlie&amp;rsquo;s number of friends: 3
&lt;strong>The average number of friends for everyone&lt;/strong> is $(1 + 1 + 1 + 3) / 4 = 1.5$ friends.&lt;/li>
&lt;/ul>
&lt;p>Next, let&amp;rsquo;s calculate the average of the &amp;ldquo;number of friends their friends have&amp;rdquo; for each person.&lt;/p>
&lt;ul>
&lt;li>Number of friends of Alice&amp;rsquo;s friend (Charlie): 3&lt;/li>
&lt;li>Number of friends of Bob&amp;rsquo;s friend (Charlie): 3&lt;/li>
&lt;li>Number of friends of David&amp;rsquo;s friend (Charlie): 3&lt;/li>
&lt;li>Average number of friends of Charlie&amp;rsquo;s friends (Alice, Bob, David): $(1 + 1 + 1) / 3 = 1$&lt;/li>
&lt;/ul>
&lt;p>Now, let&amp;rsquo;s compare each &amp;ldquo;person&amp;rdquo; with the &amp;ldquo;average of their friends&amp;rdquo;.&lt;/p>
&lt;ul>
&lt;li>Alice: Herself (1) &amp;lt; Friends&amp;rsquo; average (3)&lt;/li>
&lt;li>Bob: Himself (1) &amp;lt; Friends&amp;rsquo; average (3)&lt;/li>
&lt;li>David: Himself (1) &amp;lt; Friends&amp;rsquo; average (3)&lt;/li>
&lt;li>Charlie: Himself (3) &amp;gt; Friends&amp;rsquo; average (1)&lt;/li>
&lt;/ul>
&lt;p>3 out of 4 people (75% of the people) are in a situation where &amp;ldquo;their friends have more friends than they do.&amp;rdquo; The presence of the popular Charlie strongly pulls up the &amp;ldquo;friends&amp;rsquo; average&amp;rdquo; for everyone around him.&lt;/p>
&lt;h2 id="mathematical-proof-variance-is-key">Mathematical Proof: Variance is Key
&lt;/h2>&lt;p>Let&amp;rsquo;s express this with a mathematical formula.
In network theory, let the number of friends (degree) of a person $v$ be $k(v)$. Let the overall average number of friends in the network be $\mu$, and the variance of the number of friends be $\sigma^2$.&lt;/p>
&lt;p>According to Feld&amp;rsquo;s proof, the expected value of the &amp;ldquo;number of friends of a randomly chosen friend&amp;rdquo; is as follows:&lt;/p>
$$ \text{Average number of friends of friends} = \mu + \frac{\sigma^2}{\mu} $$
&lt;p>The variance $\sigma^2$ is always a value of 0 or greater. In other words, except for the impossible situation where everyone has exactly the same number of friends ($\sigma^2 = 0$), the following inequality always holds.&lt;/p>
$$ \mu + \frac{\sigma^2}{\mu} > \mu $$
&lt;p>&lt;strong>The &amp;ldquo;average number of friends of friends&amp;rdquo; will always be greater than the &amp;ldquo;overall average number of friends.&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>In the real world and on social media (like X or Instagram), a tiny fraction of people have millions of followers (friends), while the vast majority only have dozens to hundreds. Because the variance $\sigma^2$ is extremely large, the effect of this paradox becomes even more intense.&lt;/p>
&lt;h2 id="application-pandemics-and-vaccination">Application: Pandemics and Vaccination
&lt;/h2>&lt;p>The Friendship Paradox goes beyond the psychology of social media. It has a highly effective application in real-world social issues, particularly &lt;strong>infectious disease control&lt;/strong>.&lt;/p>
&lt;p>Suppose we have a limited number of vaccines and are unsure who to vaccinate. There is a more effective method than random vaccination.&lt;/p>
&lt;ol>
&lt;li>Choose people at random.&lt;/li>
&lt;li>Vaccinate not the person themselves, but &lt;strong>the person they named as a &amp;ldquo;friend&amp;rdquo;&lt;/strong>.&lt;/li>
&lt;/ol>
&lt;p>Why is that? Because of the Friendship Paradox, the &amp;ldquo;friends&amp;rdquo; of randomly chosen people have a higher probability of having more connections (being a hub) on average. By prioritizing vaccines for people with many connections, we can dramatically slow the spread of infection throughout the entire network.&lt;/p>
&lt;h2 id="conclusion">Conclusion
&lt;/h2>&lt;p>When you look at social media and feel &amp;ldquo;everyone has more friends and a better social life than me,&amp;rdquo; it is not your illusion, but a mathematical inevitability created by the structure of networks.&lt;/p>
&lt;p>Because popular people show up in many people&amp;rsquo;s networks, we are inevitably forced to observe a sample consisting mostly of &amp;ldquo;above-average popular people.&amp;rdquo; The next time you are about to feel down on social media, please remember this formula.&lt;/p>
$$ \mu + \frac{\sigma^2}{\mu} > \mu $$</description></item><item><title>Simpson's Paradox: The Mysterious Phenomenon of Winning in the Parts but Losing in the Whole</title><link>http://kenji.blog/en/p/simpsons-paradox/</link><pubDate>Thu, 10 Sep 2026 07:00:00 +0900</pubDate><guid>http://kenji.blog/en/p/simpsons-paradox/</guid><description>&lt;img src="http://kenji.blog/p/simpsons-paradox/img/simpsons_paradox.jpg" alt="Featured image of post Simpson's Paradox: The Mysterious Phenomenon of Winning in the Parts but Losing in the Whole" />&lt;h2 id="1-which-hospital-should-you-get-surgery-at">1. Which hospital should you get surgery at?
&lt;/h2>&lt;p>You have fallen seriously ill and must undergo surgery.
Before you, there are two choices: Hospital A and Hospital B. You requested the &amp;ldquo;success rate&amp;rdquo; data for surgeries at each hospital.&lt;/p>
&lt;p>&lt;strong>[Overall Success Rate]&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hospital A&lt;/strong>: 900 out of 1000 people succeeded (Success rate &lt;strong>90%&lt;/strong>)&lt;/li>
&lt;li>&lt;strong>Hospital B&lt;/strong>: 800 out of 1000 people succeeded (Success rate &lt;strong>80%&lt;/strong>)&lt;/li>
&lt;/ul>
&lt;p>Looking at this, anyone would think &amp;ldquo;Hospital A is better!&amp;rdquo;
However, since you have a cautious personality, you decided to investigate in more detail how the data changes depending on the state of the disease (mild or severe).&lt;/p>
&lt;p>&lt;strong>[Success Rate for Mild Patients]&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hospital A&lt;/strong>: 99 out of 100 people succeeded (Success rate &lt;strong>99%&lt;/strong>)&lt;/li>
&lt;li>&lt;strong>Hospital B&lt;/strong>: 870 out of 900 people succeeded (Success rate &lt;strong>96%&lt;/strong>)
$\rightarrow$ For mild cases, &lt;strong>Hospital A wins (99% &amp;gt; 96%)&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>[Success Rate for Severe Patients]&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hospital A&lt;/strong>: 801 out of 900 people succeeded (Success rate &lt;strong>89%&lt;/strong>)&lt;/li>
&lt;li>&lt;strong>Hospital B&lt;/strong>: 70 out of 100 people succeeded (Success rate &lt;strong>70%&lt;/strong>)
$\rightarrow$ Even for severe cases, &lt;strong>Hospital A wins (89% &amp;gt; 70%)&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>Wait? Don&amp;rsquo;t you think something is strange?&lt;/p>
&lt;p>For &amp;ldquo;mild&amp;rdquo; patients, Hospital A has a higher success rate.
For &amp;ldquo;severe&amp;rdquo; patients, Hospital A has a higher success rate.
And yet, when you calculate the &amp;ldquo;overall&amp;rdquo; success rate for all patients combined&amp;hellip;?&lt;/p>
&lt;ul>
&lt;li>Hospital A Overall: $(99 + 801) / 1000 =$ &lt;strong>90%&lt;/strong>&lt;/li>
&lt;li>Hospital B Overall: $(870 + 70) / 1000 =$ &lt;strong>94%&lt;/strong>&amp;hellip; Wait, according to the previous calculation, wasn&amp;rsquo;t it &lt;strong>80%?&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>Wait a minute, let&amp;rsquo;s look at the first data again.
The first data was like this.&lt;/p>
&lt;ul>
&lt;li>Hospital A Overall Success Rate: &lt;strong>90%&lt;/strong>&lt;/li>
&lt;li>Hospital B Overall Success Rate: &lt;strong>80%&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>However, if we recalculate using the broken-down data,
Hospital B&amp;rsquo;s overall success rate should be $(870 + 70) / 1000 = 940 / 1000 = $ &lt;strong>94%&lt;/strong>.&lt;/p>
&lt;p>&lt;strong>&amp;hellip;Well, you&amp;rsquo;ve been fooled!&lt;/strong>
Actually, this trick of numbers is exactly the terrifying trap of statistics that we will explain this time.
Let me show you the correct data once more.&lt;/p>
&lt;hr>
&lt;h2 id="2-to-you-who-were-fooled-the-real-data">2. To you who were fooled: The real data
&lt;/h2>&lt;p>&lt;strong>[Success Rate for Mild Patients]&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hospital A&lt;/strong>: 870 out of 900 people succeeded (Success rate &lt;strong>96%&lt;/strong>)&lt;/li>
&lt;li>&lt;strong>Hospital B&lt;/strong>: 99 out of 100 people succeeded (Success rate &lt;strong>99%&lt;/strong>)
$\rightarrow$ For mild cases, &lt;strong>Hospital B wins (99% &amp;gt; 96%)&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>[Success Rate for Severe Patients]&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hospital A&lt;/strong>: 30 out of 100 people succeeded (Success rate &lt;strong>30%&lt;/strong>)&lt;/li>
&lt;li>&lt;strong>Hospital B&lt;/strong>: 315 out of 900 people succeeded (Success rate &lt;strong>35%&lt;/strong>)
$\rightarrow$ Even for severe cases, &lt;strong>Hospital B wins (35% &amp;gt; 30%)&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>In other words, whether mild or severe, &lt;strong>Hospital B is overwhelmingly superior&lt;/strong>.&lt;/p>
&lt;p>Now, let&amp;rsquo;s combine this into an &amp;ldquo;overall&amp;rdquo; figure.&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hospital A Overall&lt;/strong>: $(870 + 30) / (900 + 100) = 900 / 1000 =$ &lt;strong>Success rate 90%&lt;/strong>&lt;/li>
&lt;li>&lt;strong>Hospital B Overall&lt;/strong>: $(99 + 315) / (100 + 900) = 414 / 1000 =$ &lt;strong>Success rate 41%&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>Surprisingly, even though Hospital B wins in everything when looking at the &amp;ldquo;parts&amp;rdquo;, Hospital A overwhelmingly wins when combining into the &amp;ldquo;whole&amp;rdquo;!
This is the phenomenon known as &lt;strong>&amp;ldquo;Simpson&amp;rsquo;s Paradox&amp;rdquo;&lt;/strong>.&lt;/p>
&lt;div class="mermaid">graph TD
subgraph "Partial data (B wins)"
Light["Mild: Hospital B wins (99% > 96%)"]
Heavy["Severe: Hospital B wins (35% > 30%)"]
end
subgraph "Overall data (A wins)"
Total["Overall sum: Hospital A overwhelmingly wins (90% > 41%)"]
end
Light -->|Summing them up somehow reverses the result| Total
Heavy -->|Summing them up somehow reverses the result| Total
style Total fill:#ff9999,stroke:#333,stroke-width:2px&lt;/div>
&lt;hr>
&lt;h2 id="3-why-does-this-bizarre-reversal-occur">3. Why does this bizarre reversal occur?
&lt;/h2>&lt;p>The true nature of this paradox lies in &lt;strong>&amp;ldquo;bias in the denominator (population size)&amp;rdquo;&lt;/strong> and &lt;strong>&amp;ldquo;hidden variables (confounding factors)&amp;rdquo;&lt;/strong>.&lt;/p>
&lt;p>Look closely at the data.&lt;/p>
&lt;ul>
&lt;li>Hospital A accepts &lt;strong>a massive number of &amp;ldquo;mild patients who are easy to cure&amp;rdquo; (900 people)&lt;/strong>.&lt;/li>
&lt;li>Hospital B accepts &lt;strong>a massive number of &amp;ldquo;severe patients who are difficult to cure&amp;rdquo; (900 people)&lt;/strong>.&lt;/li>
&lt;/ul>
&lt;p>Because Hospital B has highly skilled doctors, it was like a &amp;ldquo;last resort&amp;rdquo; hospital taking in many difficult severe patients turned away by others. Naturally, the success rate for severe patients is lower (35%). Hospital B&amp;rsquo;s &amp;ldquo;overall success rate&amp;rdquo; was dragged down by the low success rate of this massive number of severe patients, making it appear lower overall (41%).&lt;/p>
&lt;p>Conversely, since Hospital A handles mostly simple mild patients, its overall success rate appeared high (90%), but when compared under the same conditions (severe vs. severe, mild vs. mild), its skills were inferior to Hospital B.&lt;/p>
&lt;p>Expressed in mathematical formulas, the cause is the property of adding fractions.
In general, even if $\frac{a}{b} &lt; \frac{A}{B}$ and $\frac{c}{d} &lt; \frac{C}{D}$,
&lt;/p>
$$ \frac{a+c}{b+d} &lt; \frac{A+C}{B+D} $$
&lt;p>
does not always hold true. When the sizes of the denominators are extremely different, the direction of the inequality sign can reverse.&lt;/p>
&lt;hr>
&lt;h2 id="4-simpsons-paradox-occurring-in-the-real-world">4. &amp;ldquo;Simpson&amp;rsquo;s Paradox&amp;rdquo; occurring in the real world
&lt;/h2>&lt;p>This paradox is not merely an arithmetic puzzle; it frequently occurs in real society and has sparked major controversies.&lt;/p>
&lt;h3 id="the-1973-uc-berkeley-gender-bias-suspicions">The 1973 UC Berkeley Gender Bias Suspicions
&lt;/h3>&lt;p>When investigating the admission rates for graduate school at UC Berkeley, the &amp;ldquo;male admission rate (44%)&amp;rdquo; was significantly higher than the &amp;ldquo;female admission rate (35%)&amp;rdquo;, which became an issue as apparent discrimination against women.
However, when the data was broken down and analyzed &amp;ldquo;by department&amp;rdquo;, an astonishing fact came to light.
In almost all departments, &lt;strong>the admission rate for women was higher than for men&lt;/strong>.&lt;/p>
&lt;p>Why did the overall numbers reverse?
Actually, women applied in large numbers to &amp;ldquo;departments with low admission rates (highly competitive)&amp;rdquo;, while men applied in large numbers to &amp;ldquo;departments with high admission rates (easier to get into)&amp;rdquo;.&lt;/p>
&lt;h3 id="efficacy-data-of-covid-19-vaccines">Efficacy Data of COVID-19 Vaccines
&lt;/h3>&lt;p>There was a time when data circulated stating &amp;ldquo;people who received the vaccine have a higher mortality rate than those unvaccinated&amp;rdquo;, which caused an uproar.
This was also the result of ignoring data by age group (a hidden variable).
Because the vaccine was prioritized for &amp;ldquo;the elderly (who inherently have a higher mortality rate)&amp;rdquo;, simply summing up the overall mortality rate caused an extreme skew of elderly people in the vaccinated group, making the mortality rate appear artificially higher.&lt;/p>
&lt;p>When comparing by age group, it was confirmed that &amp;ldquo;people who received the vaccine had a lower mortality rate&amp;rdquo; in all age groups.&lt;/p>
&lt;hr>
&lt;h2 id="5-conclusion-data-doesnt-lie-but-people-can-lie-with-data">5. Conclusion: Data doesn&amp;rsquo;t lie, but people can lie with data
&lt;/h2>&lt;p>Simpson&amp;rsquo;s Paradox warns of &lt;strong>&amp;ldquo;the danger of making judgments based solely on overall data like averages or totals&amp;rdquo;&lt;/strong>.&lt;/p>
&lt;p>The world is overflowing with companies, politicians, and media that cherry-pick only &amp;ldquo;overall numbers&amp;rdquo; to appeal in ways convenient to them.
Even if told &amp;ldquo;Our Product A has higher overall satisfaction than Competitor&amp;rsquo;s Product B!&amp;rdquo;, if you break it down into &amp;ldquo;young demographics&amp;rdquo; and &amp;ldquo;elderly demographics&amp;rdquo;, Competitor&amp;rsquo;s Product B might be winning in both groups.&lt;/p>
&lt;p>When looking at data, having a skeptical eye that doesn&amp;rsquo;t get fooled by superficial &amp;ldquo;overall&amp;rdquo; numbers and asks, &amp;ldquo;Is there an extreme bias in the proportion of groups due to variables hidden behind (age, gender, severity, etc.)?&amp;rdquo; becomes the strongest weapon for surviving the modern information society.&lt;/p></description></item><item><title>Monty Hall Problem: The Trap of Probability Theory that Betrays Intuition and its Complete Resolution using Bayesian Inference</title><link>http://kenji.blog/en/p/monty-hall-problem/</link><pubDate>Thu, 10 Sep 2026 00:00:00 +0900</pubDate><guid>http://kenji.blog/en/p/monty-hall-problem/</guid><description>&lt;img src="http://kenji.blog/p/monty-hall-problem/img/monty_hall.jpg" alt="Featured image of post Monty Hall Problem: The Trap of Probability Theory that Betrays Intuition and its Complete Resolution using Bayesian Inference" />&lt;h2 id="1-the-stage-is-a-tv-quiz-show-what-would-you-do">1. The Stage is a TV Quiz Show: What Would You Do?
&lt;/h2>&lt;p>In 1990, the following question was submitted by a reader to the column &amp;ldquo;Ask Marilyn&amp;rdquo; in the American news magazine &lt;em>Parade&lt;/em>.&lt;/p>
&lt;blockquote>
&lt;p>You are a contestant on a TV game show. In front of you are &lt;strong>3 doors (A, B, C)&lt;/strong>.
Behind one door is a &lt;strong>new car (the prize)&lt;/strong>, and behind the remaining two doors are &lt;strong>goats (the blanks)&lt;/strong>.&lt;/p>
&lt;ol>
&lt;li>First, you choose &lt;strong>Door A&lt;/strong>.&lt;/li>
&lt;li>Then, the host, Monty Hall, who knows what is behind each door, opens &lt;strong>Door B&lt;/strong>, which has a goat.&lt;/li>
&lt;li>Monty says to you, &lt;strong>&amp;ldquo;You may now change your choice to Door C if you like. What will you do?&amp;rdquo;&lt;/strong>&lt;/li>
&lt;/ol>
&lt;p>So, &lt;strong>should you change your door?&lt;/strong>&lt;/p>
&lt;/blockquote>
&lt;p>Intuitively, it seems: &amp;ldquo;The remaining doors are A and C. Since the new car is completely random between the two, the probability of winning is $\frac{1}{2}$ (50%) for each. So it doesn&amp;rsquo;t matter whether you change or not.&amp;rdquo;&lt;/p>
&lt;p>However, columnist Marilyn vos Savant (recognized by the Guinness Book of Records as having the highest IQ) replied, &lt;strong>&amp;ldquo;You should change. If you change, your probability of winning doubles.&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>This answer caused a sensation across the United States, bringing in a storm of harsh criticism with about 10,000 letters of protest (about 1,000 of which were from scholars with PhDs in mathematics), saying things like &amp;ldquo;You do not understand the basics of probability&amp;rdquo; and &amp;ldquo;That&amp;rsquo;s female logic.&amp;rdquo;
However, to get straight to the conclusion, &lt;strong>Marilyn&amp;rsquo;s answer was mathematically entirely correct&lt;/strong>.&lt;/p>
&lt;hr>
&lt;h2 id="2-the-gap-between-intuition-and-mathematics-branching-probabilities-in-mermaid">2. The Gap Between Intuition and Mathematics: Branching Probabilities in Mermaid
&lt;/h2>&lt;p>Why does our intuition create the illusion that it is &amp;ldquo;$\frac{1}{2}$&amp;rdquo;?
First, let&amp;rsquo;s visualize all the patterns of the game.&lt;/p>
&lt;div class="mermaid">graph TD
Start["Game Start"] --> CarA["Car is behind Door A (Prob. 1/3)"]
Start --> CarB["Car is behind Door B (Prob. 1/3)"]
Start --> CarC["Car is behind Door C (Prob. 1/3)"]
CarA --> PickA1["You pick Door A"]
CarB --> PickA2["You pick Door A"]
CarC --> PickA3["You pick Door A"]
PickA1 --> HostB_or_C["Host opens B or C"]
PickA2 --> HostC["Host must open C"]
PickA3 --> HostB["Host must open B"]
HostB_or_C --> Stay1["Stay: Lose..."]
HostB_or_C --> Switch1["Switch: Win!"]
HostC --> Stay2["Stay: Lose..."]
HostC --> Switch2["Switch: Win!"]
HostB --> Stay3["Stay: Lose..."]
HostB --> Switch3["Switch: Win!"]
style Switch2 fill:#bbf,stroke:#333,stroke-width:2px
style Switch3 fill:#bbf,stroke:#333,stroke-width:2px
style Stay1 fill:#f99,stroke:#333,stroke-width:2px&lt;/div>
&lt;p>Assuming you chose &amp;ldquo;Door A&amp;rdquo;, the following three scenarios occur with equal probability ($\frac{1}{3}$).&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Scenario 1 (Car is A):&lt;/strong> The host opens either B or C, both of which have goats. If you change your door, you &lt;strong>lose&lt;/strong>.&lt;/li>
&lt;li>&lt;strong>Scenario 2 (Car is B):&lt;/strong> The host can only open C, which has a goat. If you change your door, you &lt;strong>win&lt;/strong>.&lt;/li>
&lt;li>&lt;strong>Scenario 3 (Car is C):&lt;/strong> The host can only open B, which has a goat. If you change your door, you &lt;strong>win&lt;/strong>.&lt;/li>
&lt;/ol>
&lt;p>In other words, in 2 out of 3 times (Scenarios 2 and 3), you are in a state where &lt;strong>&amp;ldquo;you will definitely win if you change doors&amp;rdquo;&lt;/strong>.
Therefore, the win rate when you change doors is $\frac{2}{3}$, which is &lt;strong>double&lt;/strong> the win rate of $\frac{1}{3}$ when you do not change.&lt;/p>
&lt;hr>
&lt;h2 id="3-strict-proof-using-bayes-theorem">3. Strict Proof Using Bayes&amp;rsquo; Theorem
&lt;/h2>&lt;p>To strictly solve this problem mathematically, we use &amp;ldquo;Bayes&amp;rsquo; Theorem&amp;rdquo; to calculate conditional probabilities.&lt;/p>
$$ P(H|E) = \frac{P(E|H) P(H)}{P(E)} $$
&lt;p>Here, we define the events as follows:&lt;/p>
&lt;ul>
&lt;li>$C_A, C_B, C_C$ : The events that the new car is behind doors A, B, and C, respectively. The prior probabilities are $P(C_A) = P(C_B) = P(C_C) = \frac{1}{3}$&lt;/li>
&lt;li>Suppose you initially selected &lt;strong>Door A&lt;/strong>.&lt;/li>
&lt;li>$M_B$ : The event that the host opens &lt;strong>Door B&lt;/strong>, which has a goat.&lt;/li>
&lt;/ul>
&lt;p>What we want to find is &amp;ldquo;the probability that the new car is behind Door C given that the host opened Door B&amp;rdquo;, i.e., the posterior probability $P(C_C|M_B)$.&lt;/p>
&lt;p>First, let&amp;rsquo;s consider the probability $P(M_B|C_X)$ that the host opens Door B depending on where the new car is.&lt;/p>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>When the new car is behind Door A ($C_A$)&lt;/strong>
The host can open either B or C randomly.
&lt;/p>
$$ P(M_B|C_A) = \frac{1}{2} $$
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When the new car is behind Door B ($C_B$)&lt;/strong>
The host cannot open the door with the new car, so the probability of opening B is zero.
&lt;/p>
$$ P(M_B|C_B) = 0 $$
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When the new car is behind Door C ($C_C$)&lt;/strong>
The host cannot open A (which you picked) or C (where the new car is), so they must inevitably open B.
&lt;/p>
$$ P(M_B|C_C) = 1 $$
&lt;/li>
&lt;/ol>
&lt;p>Next, we find the total probability $P(M_B)$ that the host opens Door B using the &amp;ldquo;Law of Total Probability&amp;rdquo;.&lt;/p>
$$ P(M_B) = P(M_B|C_A)P(C_A) + P(M_B|C_B)P(C_B) + P(M_B|C_C)P(C_C) $$
$$ P(M_B) = \left(\frac{1}{2} \times \frac{1}{3}\right) + \left(0 \times \frac{1}{3}\right) + \left(1 \times \frac{1}{3}\right) = \frac{1}{6} + 0 + \frac{1}{3} = \frac{1}{2} $$
&lt;p>Finally, we apply Bayes&amp;rsquo; Theorem to calculate the posterior probabilities for Door A and Door C.&lt;/p>
&lt;p>&lt;strong>Probability that the new car is behind Door A (if you stay):&lt;/strong>
&lt;/p>
$$ P(C_A|M_B) = \frac{P(M_B|C_A) P(C_A)}{P(M_B)} = \frac{\frac{1}{2} \times \frac{1}{3}}{\frac{1}{2}} = \frac{1}{3} $$
&lt;p>&lt;strong>Probability that the new car is behind Door C (if you switch):&lt;/strong>
&lt;/p>
$$ P(C_C|M_B) = \frac{P(M_B|C_C) P(C_C)}{P(M_B)} = \frac{1 \times \frac{1}{3}}{\frac{1}{2}} = \frac{2}{3} $$
&lt;p>The mathematical proof also clearly demonstrates that &lt;strong>&amp;ldquo;changing doors doubles your probability of winning (2/3)&amp;rdquo;&lt;/strong>.&lt;/p>
&lt;hr>
&lt;h2 id="4-cognitive-bias-the-value-of-information-as-conditioning">4. Cognitive Bias: The Value of Information as &amp;ldquo;Conditioning&amp;rdquo;
&lt;/h2>&lt;p>Why did even many genius mathematicians intuitively get this problem wrong?
The answer lies in the &amp;ldquo;equiprobability bias&amp;rdquo; and the &amp;ldquo;failure to update information&amp;rdquo; built into the human brain.&lt;/p>
&lt;h3 id="41-equiprobability-bias">4.1. Equiprobability Bias
&lt;/h3>&lt;p>When presented with unknown options, humans have a tendency to unconsciously assign that &amp;ldquo;the probabilities of the remaining options are always equal.&amp;rdquo;
The moment we see two doors remaining, our brain automatically labels them as &amp;ldquo;$50\%$ : $50\%$&amp;rdquo;.&lt;/p>
&lt;h3 id="42-the-information-of-the-hosts-intent">4.2. The Information of the Host&amp;rsquo;s &amp;ldquo;Intent&amp;rdquo;
&lt;/h3>&lt;p>The biggest reason intuition goes wrong is overlooking the fact that &lt;strong>the host&amp;rsquo;s actions are not random&lt;/strong>.
If the rule was &amp;ldquo;the host opens a door randomly without knowing where the car is, and it just happened to be a goat&amp;rdquo; (known as the Monty Fall problem), then the probabilities for Door A and Door C would both be $\frac{1}{2}$.&lt;/p>
&lt;p>However, in the actual Monty Hall problem, the host operates under the following strict constraints:&lt;/p>
&lt;ol>
&lt;li>They cannot open the door chosen by the contestant.&lt;/li>
&lt;li>They cannot open the door with the new car.&lt;/li>
&lt;/ol>
&lt;p>Because of these constraints, the very act of the host &amp;ldquo;opening Door B&amp;rdquo; gives us &lt;strong>massive information about Door C&lt;/strong>. It contains the unspoken message, &amp;ldquo;I couldn&amp;rsquo;t open Door C (because the new car is there).&amp;rdquo;&lt;/p>
&lt;hr>
&lt;h2 id="5-correcting-intuition-with-an-extreme-example">5. Correcting Intuition with an Extreme Example
&lt;/h2>&lt;p>If you&amp;rsquo;re still not convinced, try increasing the number of doors to &lt;strong>1,000,000&lt;/strong>.&lt;/p>
&lt;ol>
&lt;li>You pick &lt;strong>Door 1&lt;/strong> out of 1,000,000 doors. (Probability of winning is $\frac{1}{1,000,000}$)&lt;/li>
&lt;li>The host, who knows everything, opens &lt;strong>all 999,998 doors&lt;/strong> with goats behind them out of the remaining 999,999 doors.&lt;/li>
&lt;li>The only doors closed are &amp;ldquo;Door 1&amp;rdquo; which you picked, and &amp;ldquo;Door 777,777&amp;rdquo; which the host deliberately left closed.&lt;/li>
&lt;/ol>
&lt;p>Now, do you change?
In this case, unless you believe you pulled off a &amp;ldquo;one in a million&amp;rdquo; miracle right at the start, you should change. Realistically, it should be intuitively clear that the probability of the new car being behind &lt;strong>&amp;ldquo;the single door the host absolutely could not open&amp;rdquo;&lt;/strong> is $\frac{999,999}{1,000,000}$.&lt;/p>
&lt;p>The Monty Hall problem (with 3 doors) is simply a scaled-down phenomenon of this &amp;ldquo;1,000,000 doors&amp;rdquo; scenario.&lt;/p>
&lt;div class="mermaid">pie title "Effect of Switching Doors (100 Simulations)"
"Win by switching (approx. 66.7%)" : 67
"Win by staying (approx. 33.3%)" : 33&lt;/div>
&lt;h2 id="6-conclusion-life-and-business-lessons-from-probability-theory">6. Conclusion: Life and Business Lessons from Probability Theory
&lt;/h2>&lt;p>The Monty Hall problem goes beyond a mere quiz and teaches us important lessons.&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Intuition is often wrong&lt;/strong>: The human brain has not evolved to intuitively process complex conditional probabilities. In important decision-making, relying solely on intuition is dangerous.&lt;/li>
&lt;li>&lt;strong>Update probabilities with new information (Bayesian updating)&lt;/strong>: When situations change and new information (such as which door the host opened) is provided, the key to success is whether you can flexibly update your probabilities and strategies without clinging to existing beliefs.&lt;/li>
&lt;/ol>
&lt;p>The small decision to &amp;ldquo;change your door&amp;rdquo; just might double the probability of getting a &amp;ldquo;new car&amp;rdquo; in your life.&lt;/p></description></item><item><title>The Birthday Paradox: Over 50% with just 23 people? The magic of "combinations" that deceives intuition</title><link>http://kenji.blog/en/p/birthday-paradox/</link><pubDate>Thu, 10 Sep 2026 00:00:00 +0900</pubDate><guid>http://kenji.blog/en/p/birthday-paradox/</guid><description>&lt;img src="http://kenji.blog/p/birthday-paradox/img/birthday_paradox.jpg" alt="Featured image of post The Birthday Paradox: Over 50% with just 23 people? The magic of "combinations" that deceives intuition" />&lt;h2 id="1-a-test-of-intuition-how-many-people-does-it-take-for-the-probability-to-exceed-50">1. A Test of Intuition: How many people does it take for the probability to exceed 50%?
&lt;/h2>&lt;p>People are gathering at a party venue.
Here, &lt;strong>how many people do you think are needed at a minimum for the probability of having at least one pair with the exact same birthday (month and day) to exceed 50%?&lt;/strong> (*Excluding leap years, assuming a year has 365 days, and each birthday is equally probable).&lt;/p>
&lt;p>Human intuition tends to calculate like this:
&amp;ldquo;There are 365 days in a year. If we pop people into 365 slots and expect an overlap, we&amp;rsquo;d probably need at least about 180 people. Even with a conservative estimate, shouldn&amp;rsquo;t there be 50 to 60 people for the probability to be half?&amp;rdquo;&lt;/p>
&lt;p>However, the correct answer derived by mathematics is a mere &lt;strong>&amp;ldquo;23 people&amp;rdquo;&lt;/strong>.
In a typical school class (about 30 to 40 people), the probability of a pair sharing a birthday jumps to about 70% to 89%. With 50 people, the probability reaches 97%, making it &amp;ldquo;more unusual not to have people with the same birthday.&amp;rdquo;&lt;/p>
&lt;p>Why does our intuition deviate so much from actual probability?&lt;/p>
&lt;hr>
&lt;h2 id="2-why-intuition-fails-the-difference-between-me-and-someone-and-someone-and-someone">2. Why Intuition Fails: The Difference Between &amp;ldquo;Me and Someone&amp;rdquo; and &amp;ldquo;Someone and Someone&amp;rdquo;
&lt;/h2>&lt;p>The biggest reason intuition fails in this problem is that we unconsciously think about the &lt;strong>&amp;ldquo;probability that someone has the same birthday as a specific person (e.g., yourself).&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>If you enter the venue and look for &amp;ldquo;someone with the same birthday as me,&amp;rdquo; the probability that someone among the 23 people shares your birthday is only &lt;strong>about 6.1%&lt;/strong>. (For this probability to exceed 50%, you would actually need 253 people).&lt;/p>
&lt;p>However, what the Birthday Paradox asks is not a pair of &amp;ldquo;me and someone.&amp;rdquo; It only requires one match among &lt;strong>&amp;ldquo;all possible combinations between everyone present at the venue (Person A and Person B, Person B and Person C, Person C and Person A&amp;hellip;).&amp;rdquo;&lt;/strong>&lt;/p>
&lt;div class="mermaid">graph TD
subgraph "Illusion of Intuition: 'Self'-centered comparison"
You["You"] --- P1["Person A"]
You --- P2["Person B"]
You --- P3["Person C"]
You --- P4["Person D"]
style You fill:#ff9999,stroke:#333,stroke-width:4px
end
subgraph "Reality: Round-robin comparison of 'Everyone with Everyone'"
A["Person A"] --- B["Person B"]
A --- C["Person C"]
A --- D["Person D"]
B --- C
B --- D
C --- D
end&lt;/div>
&lt;p>Even in a group of just 4 people, a comparison centered around &amp;ldquo;yourself&amp;rdquo; yields 3 pairs, but a comparison among everyone yields 6 pairs (${}_4 C_2 = 6$).
When the number of people increases to 23, the combinations of pairs explosively increase to a whopping &lt;strong>253 pairs&lt;/strong> (${}_{23} C_2$).
With as many as 253 pairs, doesn&amp;rsquo;t it start to feel unsurprising that at least one of those pairs might hit the &amp;ldquo;1 in 365&amp;rdquo; chance?&lt;/p>
&lt;hr>
&lt;h2 id="3-mathematical-proof-a-brilliant-solution-using-the-complementary-event">3. Mathematical Proof: A Brilliant Solution Using the Complementary Event
&lt;/h2>&lt;p>Calculating the &amp;ldquo;probability of at least one pair sharing a birthday&amp;rdquo; head-on is difficult (because there are too many patterns, such as exactly one pair matching, two pairs matching, three people having the same birthday&amp;hellip;).
Therefore, we use a fundamental technique in probability theory: the &lt;strong>&amp;ldquo;complementary event.&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>A complementary event refers to the &amp;ldquo;probability of something not happening.&amp;rdquo;
In other words, we calculate the &lt;strong>&amp;ldquo;probability that everyone&amp;rsquo;s birthday is different (not a single pair overlaps),&amp;rdquo;&lt;/strong> and subtract it from 100% (1) to get the probability we want.&lt;/p>
$$ P(\text{At least 2 people share a birthday}) = 1 - P(\text{Everyone has a different birthday}) $$
&lt;p>Now, let&amp;rsquo;s imagine people entering the venue one by one and calculate.&lt;/p>
&lt;ol>
&lt;li>&lt;strong>1st person&lt;/strong>: There&amp;rsquo;s no worry of overlapping with anyone. The probability is $\frac{365}{365}$.&lt;/li>
&lt;li>&lt;strong>2nd person&lt;/strong>: Must have a different birthday from the 1st person. The remaining 364 days are safe. The probability is $\frac{364}{365}$.&lt;/li>
&lt;li>&lt;strong>3rd person&lt;/strong>: Must have a different birthday from the previous 2 people. The remaining 363 days are safe. The probability is $\frac{363}{365}$.&lt;/li>
&lt;/ol>
&lt;p>Multiplying this up to the $n$-th person gives the general formula for the probability $P(n)'$ that everyone has a different birthday.&lt;/p>
$$ P(n)' = \frac{365}{365} \times \frac{364}{365} \times \frac{363}{365} \times \dots \times \frac{365 - (n - 1)}{365} $$
$$ P(n)' = \prod_{k=1}^{n-1} \left(1 - \frac{k}{365}\right) $$
&lt;p>Therefore, the sought &amp;ldquo;probability $P(n)$ that at least 2 people share a birthday&amp;rdquo; is as follows:&lt;/p>
$$ P(n) = 1 - \prod_{k=1}^{n-1} \left(1 - \frac{k}{365}\right) $$
&lt;p>If we substitute the number of people $n$ into this formula, we can see the probability rises at an astonishing speed.&lt;/p>
&lt;ul>
&lt;li>When $n = 10$, the probability is about &lt;strong>11.7%&lt;/strong>&lt;/li>
&lt;li>When $n = 23$, the probability is about &lt;strong>50.7%&lt;/strong> (it crosses 50% here!)&lt;/li>
&lt;li>When $n = 40$, the probability is about &lt;strong>89.1%&lt;/strong>&lt;/li>
&lt;li>When $n = 70$, the probability is about &lt;strong>99.9%&lt;/strong>&lt;/li>
&lt;/ul>
&lt;div class="mermaid">pie title "Probability when 23 people gather"
"There is a pair with the same birthday (50.7%)" : 50.7
"Everyone is different (49.3%)" : 49.3&lt;/div>
&lt;hr>
&lt;h2 id="4-approximate-calculation-via-taylor-expansion">4. Approximate Calculation via Taylor Expansion
&lt;/h2>&lt;p>Calculating 23 multiplications by hand is tedious, so let&amp;rsquo;s use a mathematical approximation formula to understand it a bit more intuitively.&lt;/p>
&lt;p>Consider the Taylor expansion of the exponential function $e^{-x}$. When $x$ is sufficiently small, the following approximation holds:
&lt;/p>
$$ e^{-x} \approx 1 - x $$
&lt;p>Applying this to each term $\left(1 - \frac{k}{365}\right)$ from earlier:
&lt;/p>
$$ 1 - \frac{k}{365} \approx e^{-\frac{k}{365}} $$
&lt;p>Multiplying all of these together (which becomes addition by the laws of exponents):
&lt;/p>
$$ P(n)' \approx e^{-\frac{1}{365}} \times e^{-\frac{2}{365}} \times \dots \times e^{-\frac{n-1}{365}} $$
$$ P(n)' \approx \exp\left(-\sum_{k=1}^{n-1} \frac{k}{365}\right) $$
&lt;p>The sum from 1 to $n-1$ is $\frac{n(n-1)}{2}$ (that is, the number of combinations ${}_n C_2$), so:
&lt;/p>
$$ P(n)' \approx \exp\left(-\frac{n(n-1)}{2 \times 365}\right) $$
&lt;p>Using this formula, we find $n$ when the probability is 50% ($0.5$).
&lt;/p>
$$ 0.5 = e^{-\frac{n(n-1)}{730}} $$
&lt;p>
Taking the natural logarithm of both sides ($\ln 0.5 \approx -0.693$):
&lt;/p>
$$ -0.693 = -\frac{n(n-1)}{730} $$
$$ n(n-1) = 0.693 \times 730 \approx 505.89 $$
&lt;p>Approximating as $n^2 \approx 506$, we get $n = \sqrt{506} \approx 22.49$
The answer &lt;strong>$n \approx 23$&lt;/strong> is beautifully derived!&lt;/p>
&lt;hr>
&lt;h2 id="5-application-to-daily-life-and-hash-collisions">5. Application to Daily Life and &amp;ldquo;Hash Collisions&amp;rdquo;
&lt;/h2>&lt;p>This paradox is not just a party trick. It plays a critically important role in the &lt;strong>cryptography and information security&lt;/strong> that supports modern IT society.&lt;/p>
&lt;p>Computer systems use a mechanism called a &amp;ldquo;hash function&amp;rdquo; to quickly verify the identity of passwords or files. A hash function returns a random string of a fixed length (a hash value) no matter what data is put in.
However, the phenomenon where these hash values coincidentally turn out to be the same is called a &lt;strong>&amp;ldquo;Hash Collision.&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>Hash collisions occur due to the exact same principle as the Birthday Paradox.
Contrary to human intuition, which assumes &amp;ldquo;since the number of possible hash values is astronomical, collisions would rarely happen,&amp;rdquo; it is surprisingly easy for an attacker to randomly generate a massive amount of data and find a &amp;ldquo;pair that matches (has the same birthday).&amp;rdquo;&lt;/p>
&lt;p>This is called a &lt;strong>&amp;ldquo;Birthday Attack.&amp;rdquo;&lt;/strong>
Engineers designing security systems assume this mathematical fact that &amp;ldquo;collisions happen far faster than intuition suggests,&amp;rdquo; and ensure safety by setting the hash length to be extremely long.&lt;/p>
&lt;h2 id="6-conclusion-the-limits-of-human-intuition">6. Conclusion: The Limits of Human Intuition
&lt;/h2>&lt;p>The Birthday Paradox is a perfect example of &lt;strong>how fragile human intuition is against &amp;ldquo;exponential growth&amp;rdquo; and &amp;ldquo;combinatorial explosions.&amp;rdquo;&lt;/strong>&lt;/p>
&lt;p>We are strong at linear (additive) growth, but we cannot simulate in our brains a phenomenon where the number of pairs explodes at a pace of $n^2$.
Behind the intuition that &amp;ldquo;the number 23 is too small compared to the large number 365,&amp;rdquo; there are &lt;strong>&amp;ldquo;253 invisible threads (pairs)&amp;rdquo;&lt;/strong> woven by 23 people.&lt;/p>
&lt;p>Next time you go to a place where people gather, try to imagine not just the visible &amp;ldquo;number of people,&amp;rdquo; but the &amp;ldquo;threads of combinations&amp;rdquo; that exist innumerably among them. The way you view the world should change just a little bit mathematically.&lt;/p></description></item></channel></rss>