Xrfgtjtk

I know about the Kullback Leibler divergence and that it can be used to measure the difference between two probability distributions.

But it is not very symmetric. For example watching P from Q, if $q$ ever becomes zero:

$$p(x)logleft(frac{p(x)}{q(x)}right), q(x)=0, p(x)neq 0$$

Will of course be infinite. This is reasonable in the sense that if an event is impossible in $q$ but not in $p$ then it is impossible to "repair" in some sense. But this never happens from the other "view":

$$q(x)logleft(frac{q(x)}{p(x)}right), q(x)=0,p(x)neq 0$$

We can convince ourselves (probably by investigating some limit) this should not be considered infinite.

So how can we build a less unsymmetric distance measure?

There are a lot of these, you can see some examples here.

The ones I feel like come up the most often are:

• Hellinger Distance


• Total Variation Distance


• Wasserstein Distance

Though, there are plenty of them.

There's a lot of ways one could approach the idea of a "distance on probability distributions". For instance, the Total Variation Distance defined on $mathcal{P}times mathcal{P}$, the product space of probability distributions defined on the same measurable space $(Omega,mathcal{B})$, is denoted as: $TV(p_1,p_2) = sup_{B in mathcal{B}}|p_1[B] - p_2[B]|$ the largest gap in probabilities assigned to sets in the shared $sigma-$algebra.

That is kind of an abstracted notion of distance, however, if $p_1$ and $p_2$ have densities $f_1$ and $f_2$, then:
begin{equation}
TV(p_1,p_2) = ||f_1 - f_2||_{L_1} = int_{xinmathcal{X}}|f_1(x) - f_2(x)|dx
end{equation}

Which is usually more convenient to work with.

Similarly, the Wasserstein distance turns out to be useful for different settings as well. You can read more about it in that link, but generally Wasserstein distance on continuous spaces is kind of abstracted and unwieldy. However, computing Wasserstein distance on discrete spaces reduces to solving an integer/linear program. A lot of research (especially in computer imaging) goes into framing these problems into programs or other optimization problems.

See here, here. It's also used to show consistency in convergence of probability measures in statistical applications, for example here.

I guess a last point would be that just because some Divergences are not symmetric does not mean that they can't sometimes be stronger than symmetric distance metrics. For example if $KL(P_n||Q)overset{nrightarrowinfty}{rightarrow}0$ then $P_noverset{T.V.}{rightarrow}Q$