Module 4. Jensen's inequality: when averaging meets nonlinearity

Let

$$P(X=0)=P(X=2)=\frac12.$$

Then

$$\mathbb E X=1.$$

1. The square

For $\phi(x)=x^2$,

$$\phi(\mathbb E X)=1^2=1,$$

while

$$\mathbb E\phi(X) = \frac12\cdot0^2+\frac12\cdot2^2 =2.$$

Hence

$$\mathbb E[X^2]-(\mathbb E X)^2=2-1=1.$$

2. Relation to variance

By definition,

$$\operatorname{Var}(X) = \mathbb E[X^2]-(\mathbb E X)^2 =1.$$

Thus the Jensen gap for $x^2$ is exactly the variance.

3. The exponential

For $\phi(x)=e^x$,

$$\phi(\mathbb E X)=e\approx2.71828,$$

whereas

$$\mathbb E[e^X] = \frac{1+e^2}{2} \approx4.19453.$$

The gap is approximately

$$4.19453-2.71828 \approx1.47625.$$

4. Why both inequalities are strict

Both $x^2$ on $\mathbb R$ and $e^x$ are strictly convex. Since $X$ is not almost surely constant, equality in Jensen is impossible.

We have

$$q_1=(0.7,0.2,0.1), \qquad q_2=(0.2,0.3,0.5),$$

equal weights, and target class $y=1$.

1. Average member NLL

$$L_1=-\ln0.7\approx0.356675,$$ $$L_2=-\ln0.2\approx1.609438.$$

Therefore

$$\bar L = \frac{L_1+L_2}{2} \approx0.983056.$$

2. Arithmetic probability mixture

$$q_{\mathrm{mix}} = \frac{q_1+q_2}{2} = (0.45,0.25,0.30).$$

For the target class,

$$L_{\mathrm{mix}} = -\ln0.45 \approx0.798508,$$

so

$$0.798508<0.983056.$$

3. Normalized geometric mean

The unnormalized weights are

$$r_k=\sqrt{q_1(k)q_2(k)}.$$

Thus

$$r = (\sqrt{0.14},\sqrt{0.06},\sqrt{0.05}) \approx (0.374166,0.244949,0.223607).$$

After normalization,

$$q_{\mathrm{geom}} \approx (0.443997,0.290664,0.265339).$$

Its NLL is

$$L_{\mathrm{geom}} = -\ln0.443997 \approx0.811938,$$

again below the average member NLL:

$$0.811938<0.983056.$$

4. What has been verified

For probability averaging, Jensen is applied to $-\log$:

$$-\log\frac{q_1(y)+q_2(y)}2 \le \frac{-\log q_1(y)-\log q_2(y)}2.$$

For logit averaging, we use convexity of softmax cross-entropy in the logits.

5. Why dataset accuracy does not follow

The inequalities concern NLL of the observed class. Accuracy depends only on whether the argmax equals the label, while NLL uses the entire assigned probability. An ensemble may improve NLL without changing accuracy, and it may move some argmax decisions in either direction.

The model is

$$z\sim\mathcal N(0,1), \qquad x\mid z\sim\mathcal N(z,1).$$

1. Marginal distribution

Write

$$x=z+\varepsilon, \qquad \varepsilon\sim\mathcal N(0,1),$$

with independent $z$ and $\varepsilon$. Hence

$$p(x)=\mathcal N(0,2).$$

At $x=1$,

$$\log p(1) = -\frac12\log(4\pi)-\frac14 \approx-1.515512.$$

2. Exact posterior

The posterior precision is

$$1+1=2,$$

so its variance is $1/2$. Its mean is

$$\frac{1\cdot x}{1+1} = \frac12.$$

Therefore

$$p(z\mid x=1) = \mathcal N\left(\frac12,\frac12\right).$$

3. KL for $q=\mathcal N(0,1)$

For univariate Gaussians,

$$D_{\mathrm{KL}} \bigl( \mathcal N(\mu_q,\sigma_q^2) \| \mathcal N(\mu_p,\sigma_p^2) \bigr) = \frac12 \left[ \frac{\sigma_q^2}{\sigma_p^2} + \frac{(\mu_p-\mu_q)^2}{\sigma_p^2} -1 + \ln\frac{\sigma_p^2}{\sigma_q^2} \right].$$

Substituting

$$\mu_q=0,\quad \sigma_q^2=1,\quad \mu_p=\frac12,\quad \sigma_p^2=\frac12$$

gives

$$\begin{aligned} D_{\mathrm{KL}}(q\|p) &= \frac12 \left[ 2+\frac{(1/2)^2}{1/2}-1+\ln\frac12 \right]\\ &= \frac12 \left[ \frac32-\ln2 \right]\\ &\approx0.403426. \end{aligned}$$

4. ELBO

Using the exact gap,

$$\log p(x)-\operatorname{ELBO}(x) = D_{\mathrm{KL}}(q\|p(z\mid x)).$$

Therefore

$$\operatorname{ELBO}(1) = -1.515512-0.403426 \approx-1.918939.$$

5. Tightness

Choose

$$q(z\mid x) = p(z\mid x) = \mathcal N\left(\frac12,\frac12\right).$$

Then the KL gap is zero.

1. KL before grouping

$$\begin{aligned} D(P\|Q) &= 0.4\ln\frac{0.4}{0.1} + 0.3\ln\frac{0.3}{0.2}\\ &\quad+ 0.2\ln\frac{0.2}{0.3} + 0.1\ln\frac{0.1}{0.4}\\ &\approx0.456435\ \text{nats}. \end{aligned}$$

2. KL after grouping

The grouped distributions are

$$\bar P=(0.7,0.3), \qquad \bar Q=(0.3,0.7).$$

Thus

$$\begin{aligned} D(\bar P\|\bar Q) &= 0.7\ln\frac{0.7}{0.3} + 0.3\ln\frac{0.3}{0.7}\\ &\approx0.338919\ \text{nats}. \end{aligned}$$

Hence

$$D(\bar P\|\bar Q) < D(P\|Q).$$

3. Two log-sum applications

For the first group,

$$0.4\ln\frac{0.4}{0.1} + 0.3\ln\frac{0.3}{0.2} \ge 0.7\ln\frac{0.7}{0.3}.$$

For the second,

$$0.2\ln\frac{0.2}{0.3} + 0.1\ln\frac{0.1}{0.4} \ge 0.3\ln\frac{0.3}{0.7}.$$

Adding the inequalities proves the result.

4. Equality condition

Within each group, the conditional distributions under $P$ and $Q$ must match:

$$P(X=i\mid X\in G) = Q(X=i\mid X\in G)$$

for every relevant $i\in G$.

Thus, once the group is known, the finer state must provide no additional ability to distinguish $P$ from $Q$.

Let

$$m=\mathbb E X.$$

By definition,

$$D_\phi(X,m) = \phi(X)-\phi(m)-\nabla\phi(m)^\top(X-m).$$

Taking expectations,

$$\begin{aligned} \mathbb E D_\phi(X,m) &= \mathbb E\phi(X)-\phi(m) -\nabla\phi(m)^\top(\mathbb E X-m)\\ &= \mathbb E\phi(X)-\phi(m). \end{aligned}$$

Since $m=\mathbb E X$,

$$\mathbb E D_\phi(X,\mathbb E X) = \mathbb E\phi(X)-\phi(\mathbb E X).$$

For $\phi(x)=x^2/2$,

$$D_\phi(x,m) = \frac{x^2}{2}-\frac{m^2}{2}-m(x-m) = \frac12(x-m)^2.$$

Therefore

$$\mathbb E D_\phi(X,\mathbb E X) = \frac12\operatorname{Var}(X),$$

which matches the Jensen gap for $x^2/2$.