Bound on KL-divergence-like quantity (with squared logarithms)
$begingroup$
Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following expression:
$$
left|sum_{i=1}^n p_i (log^2 p_i - log^2 q_i) right|.
$$
This is very similar to the KL-divergence, with the exception that the logs are squared. This innocent change, however, seems to destroy the standard ways of proving the properties of the KL-divergence via Log Sum Inequality, Gibb's inequality, etc. and I haven't been able to figure out a clever workaround. Note that I'm only interested in bounding the magnitude of the expression but I'm not even sure it's always positive.
For my purposes, it would be good enough to have an upper bound like $leq c(n) D(p|q)^2$, where $D(p|q)$ denotes the KL-divergence and $c(n)$ is a constant that may depend on $n$.
Finally, I should also note that I need to know the answer only for the special case in which $p$ is a joint distribution over two random variables and $q$ is the product of its marginals (i.e. the special case in which the KL-divergence gives the mutual information of $p$), maybe that special case is easier to handle.
probability functions logarithms information-theory entropy
asked Jan 6 at 12:31
PaulPaul
134
$endgroup$
add a comment |
$begingroup$
Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following expression:
$$
left|sum_{i=1}^n p_i (log^2 p_i - log^2 q_i) right|.
$$
This is very similar to the KL-divergence, with the exception that the logs are squared. This innocent change, however, seems to destroy the standard ways of proving the properties of the KL-divergence via Log Sum Inequality, Gibb's inequality, etc. and I haven't been able to figure out a clever workaround. Note that I'm only interested in bounding the magnitude of the expression but I'm not even sure it's always positive.
For my purposes, it would be good enough to have an upper bound like $leq c(n) D(p|q)^2$, where $D(p|q)$ denotes the KL-divergence and $c(n)$ is a constant that may depend on $n$.
Finally, I should also note that I need to know the answer only for the special case in which $p$ is a joint distribution over two random variables and $q$ is the product of its marginals (i.e. the special case in which the KL-divergence gives the mutual information of $p$), maybe that special case is easier to handle.
probability functions logarithms information-theory entropy
asked Jan 6 at 12:31
PaulPaul
134
$endgroup$
add a comment |
$begingroup$
Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following expression:
$$
left|sum_{i=1}^n p_i (log^2 p_i - log^2 q_i) right|.
$$
This is very similar to the KL-divergence, with the exception that the logs are squared. This innocent change, however, seems to destroy the standard ways of proving the properties of the KL-divergence via Log Sum Inequality, Gibb's inequality, etc. and I haven't been able to figure out a clever workaround. Note that I'm only interested in bounding the magnitude of the expression but I'm not even sure it's always positive.
For my purposes, it would be good enough to have an upper bound like $leq c(n) D(p|q)^2$, where $D(p|q)$ denotes the KL-divergence and $c(n)$ is a constant that may depend on $n$.
Finally, I should also note that I need to know the answer only for the special case in which $p$ is a joint distribution over two random variables and $q$ is the product of its marginals (i.e. the special case in which the KL-divergence gives the mutual information of $p$), maybe that special case is easier to handle.
probability functions logarithms information-theory entropy
asked Jan 6 at 12:31
PaulPaul
134
$endgroup$
Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following expression:
$$
left|sum_{i=1}^n p_i (log^2 p_i - log^2 q_i) right|.
$$
This is very similar to the KL-divergence, with the exception that the logs are squared. This innocent change, however, seems to destroy the standard ways of proving the properties of the KL-divergence via Log Sum Inequality, Gibb's inequality, etc. and I haven't been able to figure out a clever workaround. Note that I'm only interested in bounding the magnitude of the expression but I'm not even sure it's always positive.
For my purposes, it would be good enough to have an upper bound like $leq c(n) D(p|q)^2$, where $D(p|q)$ denotes the KL-divergence and $c(n)$ is a constant that may depend on $n$.
Finally, I should also note that I need to know the answer only for the special case in which $p$ is a joint distribution over two random variables and $q$ is the product of its marginals (i.e. the special case in which the KL-divergence gives the mutual information of $p$), maybe that special case is easier to handle.
probability functions logarithms information-theory entropy
probability functions logarithms information-theory entropy
asked Jan 6 at 12:31
PaulPaul
134
asked Jan 6 at 12:31
PaulPaul
134
asked Jan 6 at 12:31
PaulPaul
134
asked Jan 6 at 12:31
PaulPaul
134
asked Jan 6 at 12:31
PaulPaul
134
134
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3063802%2fbound-on-kl-divergence-like-quantity-with-squared-logarithms%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3063802%2fbound-on-kl-divergence-like-quantity-with-squared-logarithms%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
