Steady State Probability

Topics: Regular Markov Chain - Markov Chain

---

In a regular Markov chain, it happens that, regardless of the initial state, the probability of reaching a given state is a constant when the number of steps is sufficiently high.

(fundamental theorem)

For every regular Markov chain, the following limit (n step transition probability) exists and it’s the same one regardless of $i$:

$${\pi }_j=\underset{n\to \infty }{\lim }{p}_{ij}^{(n)}$$

Furthermore, we have that ${\pi }_j\ge 0$ and the following equations are satisfied:

1. ${\pi }_j={\sum }_{i=0}^m{\pi }_i{p}_{ij}$, for $j=0,1,2,\dots ,m$
2. ${\sum }_{j=0}^m{\pi }_j=1$

The values ${\pi }_j$ are called steady state probabilities, while the vector that they form is called the stationary distribution vector.

In other words, the probability of reaching a state $j$ tends to ${\pi }_j$ as $n\to \infty$, regardless of the initial state.

Calculation of Steady State Probabilities §

(observation)

We can calculate all of the ${\pi }_j$ in a given chain by formulating all corresponding equations then solving the resulting equation system.

Notice that this resulting equation system has $m+2$ equations but only $m+1$ unknowns. This system has a unique solution, so one of these equations will be redundant.

Thus, we may find it easier to solve the system that consists of all equations but one (the redundant one). Note that the equation that establishes that all ${\pi }_j$ must add up to $1$ is never redundant, so it can never be discarded.

Transitory States Tend to 0 §

(corollary)

If $j$ is a transitory state, then:

$$\underset{n\to \infty }{\lim }{p}_{ij}^{(n)}={\pi }_j=0$$

…since, the more steps we take, the more likely we are to step out of its containing (transitory) set and thus, stop being able to return to it.

Limit of the Transition Matrix §

(corollary)

From the previous theorem, it’s also possible to observe that, given $P^{(n)}$ the n step transition matrix of the regular Markov chain:

$$\underset{n\to \infty }{\lim }{P}^{(n)}=\left(\begin{array}{cccc}{\pi }_0& {\pi }_1& \dots & {\pi }_m\\ {\pi }_0& {\pi }_1& \dots & {\pi }_m\\ ⋮& ⋮& \ddots & ⋮\\ {\pi }_0& {\pi }_1& \dots & {\pi }_m\end{array}\right)$$