Xrfgtjtk

While studying a slide deck, I encountered the following exercise:

We are to give a bias variance decomposition when the prediction is given as a probability distribution over $C$ classes.
Let $P = [P_1, . . . , P_C ]$ be the ground truth class distribution associated to a particular input pattern. Assume the random estimator of class probabilities $$bar{{P}} = [bar{{P}}_1, . . . , bar{{P}}_C ] $$ for the same input pattern. The error function is given by
the KL-divergence between the ground truth and the estimated probability distribution:
$$text{Error} = E[D_{KL}(P||bar{{P}})]$$

First, we would like to determine the mean of the class distribution estimator $bar{{P}}$. We define the mean as the distribution
that minimizes its expected KL divergence from the class distribution estimator, that is, the distribution
$R$ that optimizes
$$begin{matrix}min\R end{matrix} space E[ D_{KL}(R||bar{{P}})]$$

I have found a way to proof that this is:

$$ R = [R_1, . . . , R_C ] $$
$$ text{where} spacespace R_i = frac{{expspace E[log bar{P_i} ]}}{{sum_jexpspace E[log bar{P_j}]}} space space space ∀ space 1 ≤ i ≤ C.$$

We are now asked to proof that:

$$ Error(hat{P}) = Bias(hat{P}) + Var(hat{P}) $$

where

$$Error(hat{P}) = E[D_{KL}(P || hat{P})$$

$$Bias(hat{P}) = D_{KL}(P || R)$$
$$Var(hat{P}) = E[D_{KL}(R || hat{P})$$

I started by writing out the KL Divergence:
$$Bias(hat{P}) + Var(hat{P}) = sum_{i=1}^{C} left (P_i log left (frac{P_i}{R_i} right ) right) + E left [ sum_{i=1}^{c} left (R_i log left (frac{R_i}{hat{P}_i} right ) right ) right ]$$

But I don't know how to continue from here on.

We want to proof following statement:
$$Error(hat{P}) = Bias(hat{P}) + Var(hat{P}) = E[D_{KL}(P || hat{P})].$$

$$Error(hat{P}) = E[D_{KL}(P || hat{P})]$$
$$= E[sum_{i=1}^{N}P_{i} log(frac{P_{i}}{hat{P_{i}}})] \$$
$$= E[E[log(P) - log(hat{P})]] \$$
$$= E[E[log(P) - log(R) + log(R) - log(hat{P})]] \$$
$$= E[E[log(P) - log(R)] + E[log(R) - log(hat{P})]] \$$
$$= E[log(P) - log(R)] + E[E[log(R) - log(hat{P})]] \$$
$$= sum_{i=1}^{N}P_{i} log(frac{P_{i}}{R_{i}}) + E[sum_{i=1}^{N}R_{i}
log(frac{R_{i}}{hat{P_{i}}})]$$
$$= D_{KL}(P || R) + E[D_{KL}(R || hat{P})] \
= Bias(hat{P}) + Var(hat{P}) $$