Diary

Welcome back to my blog! Today, we're diving into KL divergence (Kullback-Leibler divergence), a fascinating concept used to measure how much two probability distributions differ. Instead of jumping straight into formulas, we'll build our understanding from the ground up, using an intuitive example. Let's make this fun and insightful!

Imagine we're looking at the subjects chosen by students in two different years at a school. We'll represent these choices as probability distributions to see how preferences shift over time. So lets say we have a 2 graphs each displaying the subjects chosen by students in each year

Year 2024: Distribution Q(x)

In 2024, students chose subjects as follows:

• English: 50% (0.5)
• Spanish: 40% (0.4)
• Japanese: 10% (0.1)

Let's call this distribution Q(x)

Year 2025: Distribution P(x)

In 2025, the preferences shifted:

• English: 50% (0.5)
• Spanish: 10% (0.1)
• Japanese: 40% (0.4)

We'll call this distribution P(x). Notice that English stays the same, but Spanish and Japanese swap their proportions!

We want to quantify how different these two distributions (P(x) and Q(x)) are. You might think, "Let's compare the distributions by dividing their probabilities and summing the ratios."

For each subject:

• English: P(x)/Q(x) = 0.5/0.5 = 1
• Spanish: P(x)/Q(x) = 0.1/0.4 = 0.25
• Japanese: P(x)/Q(x) = 0.4/0.1 = 4

BUT? is this correct way to find difference in distribution? While we can see in graph for both years that Spanish and Japanese have swapped in terms of popularity, these raw ratios don't capture that symmetry. One is 0.25, the other is 4. There's no clear sense of balance or symmetry in these numbers.

In fact, they're misleading. They suggest one subject changed drastically (Japanese), while the other barely did (Spanish), which isn't true. They both changed by the same amount, just in opposite directions.

So What's a Better Way?

So what we want is a method that reflects that balance like something that treats one going up the same way as the other going down.

What we need is a function that we apply to the ratio and it should give us a positive value when something goes up, and a negative value when something goes down by the same amount. So what we actually need is a function f(x) that takes in this ratio and returns a value that behaves like this:

• If we put in 4, it gives some number (say +y)
• If we put in 1/4, it gives -y
• If we put in 1, it gives 0 (because no change happened)

This way, we treat up and down changes equally but with opposite signs, which makes sense when values just swap (like Spanish and Japanese did).

This gives us two important points we can imagine plotting on a graph:

• When the ratio is $\frac{1}{4}$, the function should return $-y$
• When the ratio is $4$, the function should return $+y$

So we get the points:

• $(\frac{1}{4}, -y)$
• $(4, y)$

Similarly, we can also consider more points to draw graph

• $(\frac{1}{2}, -z)$
• $(2, z)$

for 1 , since both values will be same:

• $(\frac{1}{1}, w)$
• $(1, w)$ where w will come out to be 0 because that is only possible value;

plotting this on a graph we will get something like:

Even without diving into the math yet, this shape should feel familiar. We're essentially looking for a curve that reflects change around the point x=1, with equal response on both sides. It turns out, there's a function that behaves exactly this way and its graph is very similar to the one we need and that function is the logarithmic function.

If you recall from math, the graph of the logarithmic function has a shape that is very similar to what we need. Here is a resource in case you want to refresh your memory on Logarithmic Fn

Looks like log is the function which we were looking for ,The logarithmic function gives us exactly what we want:

• It's zero when nothing changes
• It gives equal and opposite values when something goes up or down by the same factor
• It grows gradually, so small changes don't create huge spikes

So if we put our values $\frac{1}{4}$ and $4$ into the expression, we get:

Using base-10 logarithm:

• For $\frac{1}{4}$: $\log_{10}\left(\frac{1}{4}\right) \approx -0.602$

• For $4$: $\log_{10}(4) \approx 0.602$

So, in both cases, the values are exactly (or nearly) equal in magnitude but opposite in sign. This is exactly what we want to reflect in subject preference shifts.So instead of just using $P(x)/Q(x)$, we use: $\log\left(\frac{P(x)}{Q(x)}\right)$

Now that we've identified the logarithmic function as the ideal way to quantify these changes, we can start to build a formula that actually measures the overall difference between the two distributions but we still need one more step to make this a weighted average of the differences.

Why? Because not every subject is equally important in the new distribution. For example:

• Japanese changed a lot and now takes 40% of the student choices so it should have a bigger impact on the divergence.
• Spanish dropped to just 10% so it should have a smaller impact.

So we should weigh these changes according to how likely each outcome is in the new distribution P(x)

That gives us the full formula for Kullback-Leibler divergence, which is:

$D_{KL}(P || Q) = \sum_{x} P(x) \cdot \log_{10}\left(\frac{P(x)}{Q(x)}\right)$

This might look a bit mathematical, but it's really just:

• Take each value of x (each subject),
• Compute the ratio $P(x)/Q(x)$,
• Take the log of that ratio,
• Multiply by the corresponding $P(x)$,
• Then sum over all subjects.

Let's plug in our actual subject distributions.

KL Divergence for Our Example

Let's compute $D_{KL}(P || Q)$ for the subjects:

Subject	P(x) 2025	Q(x) 2024	P/Q	log10(P/Q)	P * log10(P/Q)
English	0.5	0.5	1	0	0
Spanish	0.1	0.4	0.25	-0.602	0.1 × -0.602 = -0.0602
Japanese	0.4	0.1	4	0.602	0.4 × 0.602 = 0.2408

Now summing these:

$Divergence(P || Q) = 0 + (-0.0602) + 0.2408 = 0.1806$

KL Divergence ≈ 0.1806 (base-10)

This value tells us that there is a moderate divergence between the preferences of 2024 and 2025, mostly due to the shifts in Spanish and Japanese.

Outro for Dummies

KL divergence is more than just a math formula taught in books, I hope this blog helped you understand the fundamentals behind it. It's a way to measure change, surprise, and uncertainty between two sets of beliefs or realities.

In our case, it helped us capture how student preferences for subjects evolved between 2024 and 2025. And even though raw ratios were misleading, the log-weighted divergence gave us a fair, and powerful measure of difference. If you found this explanation helpful, please share it with your friends and colleagues! If you have any questions or suggestions for future topics, feel free to ping me on Twitter / X or LinkedIn