Generalization of an absorbing Markov Chain
A process moves on the integers 1, 2, 3, 4, and 5. It starts at 1 and, on each
successive step, moves to an integer greater than its present position, moving
with equal probability to each of the remaining larger integers. State five is
an absorbing state. Find the expected number of steps to reach state five.
I solved using matrices.
If there are $r$ absorbing states and $t$ transient states,
the transition matrix will have the following canonical form
$P$ = $begin{bmatrix}Q&R\O&Iend{bmatrix}$
Here all the submatrices are transition matrices. $I$ is an $r$-by-$r$ identity matrix, $O$ is an $r$-by-$t$ zero matrix, $R$ is a nonzero $t$-by-$r$ matrix, and $Q$ is an $t$-by-$t$ matrix.
If $N = I + Q + Q^2 + ... = (I-Q)^{-1}$
If we add all the entries in the $i^{th}$ row of $N$, we will have the expected
number of times in any of the transient states for a given starting state $s_{i}$, that is, the expected time required before being absorbed.
Now let $t_{i}$ be the expected number of steps before the chain is absorbed,
given that the chain starts in state $s_{i}$, and let $t$ be the column vector whose $i^{th}$ entry is $t_{i}$. Then
$t = Nc$ ,
where $c$ is a column vector all of whose entries are $1$.
So the answer here is the first element of $t$ or the sum of all elements of first row in $N$.
But how to generalize and get the answer for $n$ numbers? Please help. I think there should be another easier approach other than matrices.
Question and Solution Reference: Grinstead, C.M. and Snell, J.L., 2012. Introduction to probability. American Mathematical Soc.
Edit:
I am trying to solve this using recursion.
$E(i)$ denotes the expected no. of steps to reach $n$ starting from number $i$.
So,
$E(i) = 1+frac{1}{n-i}sum_{k=i+1}^{n-1}E(k)$
and base cases: $E(n) = 0$ and $E(n-1) = 1$.
Using these base values, I manually solved recursion for smaller numbers by substitution and found:
$E(i) = sum_{k=1}^{n-i}frac{1}{k}$
But now how to prove this relation for all numbers?
probability markov-chains expected-value
asked Dec 12 '18 at 15:29
noobnoob
183
|
show 3 more comments
A process moves on the integers 1, 2, 3, 4, and 5. It starts at 1 and, on each
successive step, moves to an integer greater than its present position, moving
with equal probability to each of the remaining larger integers. State five is
an absorbing state. Find the expected number of steps to reach state five.
I solved using matrices.
If there are $r$ absorbing states and $t$ transient states,
the transition matrix will have the following canonical form
$P$ = $begin{bmatrix}Q&R\O&Iend{bmatrix}$
Here all the submatrices are transition matrices. $I$ is an $r$-by-$r$ identity matrix, $O$ is an $r$-by-$t$ zero matrix, $R$ is a nonzero $t$-by-$r$ matrix, and $Q$ is an $t$-by-$t$ matrix.
If $N = I + Q + Q^2 + ... = (I-Q)^{-1}$
If we add all the entries in the $i^{th}$ row of $N$, we will have the expected
number of times in any of the transient states for a given starting state $s_{i}$, that is, the expected time required before being absorbed.
Now let $t_{i}$ be the expected number of steps before the chain is absorbed,
given that the chain starts in state $s_{i}$, and let $t$ be the column vector whose $i^{th}$ entry is $t_{i}$. Then
$t = Nc$ ,
where $c$ is a column vector all of whose entries are $1$.
So the answer here is the first element of $t$ or the sum of all elements of first row in $N$.
But how to generalize and get the answer for $n$ numbers? Please help. I think there should be another easier approach other than matrices.
Question and Solution Reference: Grinstead, C.M. and Snell, J.L., 2012. Introduction to probability. American Mathematical Soc.
Edit:
I am trying to solve this using recursion.
$E(i)$ denotes the expected no. of steps to reach $n$ starting from number $i$.
So,
$E(i) = 1+frac{1}{n-i}sum_{k=i+1}^{n-1}E(k)$
and base cases: $E(n) = 0$ and $E(n-1) = 1$.
Using these base values, I manually solved recursion for smaller numbers by substitution and found:
$E(i) = sum_{k=1}^{n-i}frac{1}{k}$
But now how to prove this relation for all numbers?
probability markov-chains expected-value
asked Dec 12 '18 at 15:29
noobnoob
183
3
All this can be bypassed using the classical one-step Markov chain decomposition, to the effect that, if $t_n$ denotes the mean number of steps when there are $n$ remining spots, then $t_1=1$ and $$t_{n+1}=1+frac1nsum_{k=1}^nt_k$$ which is readily solved by $$t_n=sum_{k=1}^nfrac1k$$ and your answer is $t_4$.
– Did
Dec 12 '18 at 15:50
Nice. How did you come up with this formula?
– noob
Dec 12 '18 at 16:12
Quote: "using the classical one-step Markov chain decomposition".
– Did
Dec 12 '18 at 16:48
Sorry, I didn't understand chain decomposition. I could come up with a very similar recursion equation but cannot come to your second solution equation.
– noob
Dec 12 '18 at 16:55
Then show what you can do and explain why the steps which escape you, are a problem.
– Did
Dec 12 '18 at 16:56
|
show 3 more comments
A process moves on the integers 1, 2, 3, 4, and 5. It starts at 1 and, on each
successive step, moves to an integer greater than its present position, moving
with equal probability to each of the remaining larger integers. State five is
an absorbing state. Find the expected number of steps to reach state five.
I solved using matrices.
If there are $r$ absorbing states and $t$ transient states,
the transition matrix will have the following canonical form
$P$ = $begin{bmatrix}Q&R\O&Iend{bmatrix}$
Here all the submatrices are transition matrices. $I$ is an $r$-by-$r$ identity matrix, $O$ is an $r$-by-$t$ zero matrix, $R$ is a nonzero $t$-by-$r$ matrix, and $Q$ is an $t$-by-$t$ matrix.
If $N = I + Q + Q^2 + ... = (I-Q)^{-1}$
If we add all the entries in the $i^{th}$ row of $N$, we will have the expected
number of times in any of the transient states for a given starting state $s_{i}$, that is, the expected time required before being absorbed.
Now let $t_{i}$ be the expected number of steps before the chain is absorbed,
given that the chain starts in state $s_{i}$, and let $t$ be the column vector whose $i^{th}$ entry is $t_{i}$. Then
$t = Nc$ ,
where $c$ is a column vector all of whose entries are $1$.
So the answer here is the first element of $t$ or the sum of all elements of first row in $N$.
But how to generalize and get the answer for $n$ numbers? Please help. I think there should be another easier approach other than matrices.
Question and Solution Reference: Grinstead, C.M. and Snell, J.L., 2012. Introduction to probability. American Mathematical Soc.
Edit:
I am trying to solve this using recursion.
$E(i)$ denotes the expected no. of steps to reach $n$ starting from number $i$.
So,
$E(i) = 1+frac{1}{n-i}sum_{k=i+1}^{n-1}E(k)$
and base cases: $E(n) = 0$ and $E(n-1) = 1$.
Using these base values, I manually solved recursion for smaller numbers by substitution and found:
$E(i) = sum_{k=1}^{n-i}frac{1}{k}$
But now how to prove this relation for all numbers?
probability markov-chains expected-value
asked Dec 12 '18 at 15:29
noobnoob
183
A process moves on the integers 1, 2, 3, 4, and 5. It starts at 1 and, on each
successive step, moves to an integer greater than its present position, moving
with equal probability to each of the remaining larger integers. State five is
an absorbing state. Find the expected number of steps to reach state five.
I solved using matrices.
If there are $r$ absorbing states and $t$ transient states,
the transition matrix will have the following canonical form
$P$ = $begin{bmatrix}Q&R\O&Iend{bmatrix}$
Here all the submatrices are transition matrices. $I$ is an $r$-by-$r$ identity matrix, $O$ is an $r$-by-$t$ zero matrix, $R$ is a nonzero $t$-by-$r$ matrix, and $Q$ is an $t$-by-$t$ matrix.
If $N = I + Q + Q^2 + ... = (I-Q)^{-1}$
If we add all the entries in the $i^{th}$ row of $N$, we will have the expected
number of times in any of the transient states for a given starting state $s_{i}$, that is, the expected time required before being absorbed.
Now let $t_{i}$ be the expected number of steps before the chain is absorbed,
given that the chain starts in state $s_{i}$, and let $t$ be the column vector whose $i^{th}$ entry is $t_{i}$. Then
$t = Nc$ ,
where $c$ is a column vector all of whose entries are $1$.
So the answer here is the first element of $t$ or the sum of all elements of first row in $N$.
But how to generalize and get the answer for $n$ numbers? Please help. I think there should be another easier approach other than matrices.
Question and Solution Reference: Grinstead, C.M. and Snell, J.L., 2012. Introduction to probability. American Mathematical Soc.
Edit:
I am trying to solve this using recursion.
$E(i)$ denotes the expected no. of steps to reach $n$ starting from number $i$.
So,
$E(i) = 1+frac{1}{n-i}sum_{k=i+1}^{n-1}E(k)$
and base cases: $E(n) = 0$ and $E(n-1) = 1$.
Using these base values, I manually solved recursion for smaller numbers by substitution and found:
$E(i) = sum_{k=1}^{n-i}frac{1}{k}$
But now how to prove this relation for all numbers?
probability markov-chains expected-value
probability markov-chains expected-value
asked Dec 12 '18 at 15:29
noobnoob
183
asked Dec 12 '18 at 15:29
noobnoob
183
edited Dec 12 '18 at 17:17
noob
asked Dec 12 '18 at 15:29
noobnoob
183
asked Dec 12 '18 at 15:29
noobnoob
183
asked Dec 12 '18 at 15:29
noobnoob
183
183
3
All this can be bypassed using the classical one-step Markov chain decomposition, to the effect that, if $t_n$ denotes the mean number of steps when there are $n$ remining spots, then $t_1=1$ and $$t_{n+1}=1+frac1nsum_{k=1}^nt_k$$ which is readily solved by $$t_n=sum_{k=1}^nfrac1k$$ and your answer is $t_4$.
– Did
Dec 12 '18 at 15:50
Nice. How did you come up with this formula?
– noob
Dec 12 '18 at 16:12
Quote: "using the classical one-step Markov chain decomposition".
– Did
Dec 12 '18 at 16:48
Sorry, I didn't understand chain decomposition. I could come up with a very similar recursion equation but cannot come to your second solution equation.
– noob
Dec 12 '18 at 16:55
Then show what you can do and explain why the steps which escape you, are a problem.
– Did
Dec 12 '18 at 16:56
|
show 3 more comments
3
All this can be bypassed using the classical one-step Markov chain decomposition, to the effect that, if $t_n$ denotes the mean number of steps when there are $n$ remining spots, then $t_1=1$ and $$t_{n+1}=1+frac1nsum_{k=1}^nt_k$$ which is readily solved by $$t_n=sum_{k=1}^nfrac1k$$ and your answer is $t_4$.
– Did
Dec 12 '18 at 15:50
Nice. How did you come up with this formula?
– noob
Dec 12 '18 at 16:12
Quote: "using the classical one-step Markov chain decomposition".
– Did
Dec 12 '18 at 16:48
Sorry, I didn't understand chain decomposition. I could come up with a very similar recursion equation but cannot come to your second solution equation.
– noob
Dec 12 '18 at 16:55
Then show what you can do and explain why the steps which escape you, are a problem.
– Did
Dec 12 '18 at 16:56
3
3
All this can be bypassed using the classical one-step Markov chain decomposition, to the effect that, if $t_n$ denotes the mean number of steps when there are $n$ remining spots, then $t_1=1$ and $$t_{n+1}=1+frac1nsum_{k=1}^nt_k$$ which is readily solved by $$t_n=sum_{k=1}^nfrac1k$$ and your answer is $t_4$.
– Did
Dec 12 '18 at 15:50
All this can be bypassed using the classical one-step Markov chain decomposition, to the effect that, if $t_n$ denotes the mean number of steps when there are $n$ remining spots, then $t_1=1$ and $$t_{n+1}=1+frac1nsum_{k=1}^nt_k$$ which is readily solved by $$t_n=sum_{k=1}^nfrac1k$$ and your answer is $t_4$.
– Did
Dec 12 '18 at 15:50
Nice. How did you come up with this formula?
– noob
Dec 12 '18 at 16:12
Nice. How did you come up with this formula?
– noob
Dec 12 '18 at 16:12
Quote: "using the classical one-step Markov chain decomposition".
– Did
Dec 12 '18 at 16:48
Quote: "using the classical one-step Markov chain decomposition".
– Did
Dec 12 '18 at 16:48
Sorry, I didn't understand chain decomposition. I could come up with a very similar recursion equation but cannot come to your second solution equation.
– noob
Dec 12 '18 at 16:55
Sorry, I didn't understand chain decomposition. I could come up with a very similar recursion equation but cannot come to your second solution equation.
– noob
Dec 12 '18 at 16:55
Then show what you can do and explain why the steps which escape you, are a problem.
– Did
Dec 12 '18 at 16:56
Then show what you can do and explain why the steps which escape you, are a problem.
– Did
Dec 12 '18 at 16:56
|
show 3 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3036818%2fgeneralization-of-an-absorbing-markov-chain%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3036818%2fgeneralization-of-an-absorbing-markov-chain%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
All this can be bypassed using the classical one-step Markov chain decomposition, to the effect that, if $t_n$ denotes the mean number of steps when there are $n$ remining spots, then $t_1=1$ and $$t_{n+1}=1+frac1nsum_{k=1}^nt_k$$ which is readily solved by $$t_n=sum_{k=1}^nfrac1k$$ and your answer is $t_4$.
– Did
Dec 12 '18 at 15:50
Nice. How did you come up with this formula?
– noob
Dec 12 '18 at 16:12
Quote: "using the classical one-step Markov chain decomposition".
– Did
Dec 12 '18 at 16:48
Sorry, I didn't understand chain decomposition. I could come up with a very similar recursion equation but cannot come to your second solution equation.
– noob
Dec 12 '18 at 16:55
Then show what you can do and explain why the steps which escape you, are a problem.
– Did
Dec 12 '18 at 16:56