Lecture 13: Markov Chains: Ergodicity

Author

Junaid Hasan

Published

July 21, 2024

Markov Chains: Ergodicity

Recall last time we discussed two kinds of states:
Recurrent states.
Transient states.
In simple language a state $s$ is recurrent if with probability 0 you do not escape $s$, once you start at $s$.
Similarly a state $s$ is transient if you can escape $s$ (never return to $s$) after starting from $s$ with a non-zero probability.
Today we will attempt to prove a few observations from last time.
Let us start with the notion of ergodicity.

Suppose $P$ is a transition matrix for a markov chain $M$.
Furthermore, let $P^n = (P^{(n)}_{ij})$ record the $n$-step transition probabilities.
In other words, probability of going from state $i$ to $j$ after $n$-steps.
Suppose $P^n$ is special: the entries $P^{(n)}_{ij}$ are all non-zero.
In other words, $P$ allows movement from any state $i$ to any state $j$ after $n$ steps.
Then $M$ is said to be Ergodic, if there exists an $n$ such that $P^n$ is special.

In simpler terms, $M$ is ergodic if some power its transition matrix contains no zeros.
Note that if $P$ itself contains no zeros (for example the weather example from last time) \[P = \begin{pmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix},\] then it is ergodic trivially.
However, $P$ may have zeros, but some power of $P$ may still have all non-zero entries.
Consider $P = \begin{pmatrix}0.2 & 0.8 & 0 \\ 0 & 0.6 & 0.4 \\ 0.7 & 0 & 0.3 \end{pmatrix}$ then $P$ has zero entries, but
$P^2 = \begin{pmatrix}0.04 & 0.64 & 0.32 \\ 0.28 & 0.36 & 0.36 \\ 0.35 & 0.56 & 0.09 \end{pmatrix}$ has non-zero entries, making $P$ ergodic.

Ergodicity is helpful because it gives a number $n$ such that after $n$ steps it is possible to go from any state to any other state.
We saw the weather example from last time was ergodic for $n = 1$
However, the random walk on $\{ -3, -2, -1, 0, 1, 2, 3 \}$ is not ergodic, because,
for any power $n$ there are zero entries.
For example if $n$ is even then cannot go from 1 to 2 (because 1 is odd, and 2 is even).
Similarly, if $n$ is odd then cannot go from 0 to 2 (because 0 and 2 are both even and $n$ is odd).

You may ask why do I need ergodicity.
Answer:
Theorem: For ergodic Markov chains we have a limiting distribution $\pi$, and every row of the matrix $P^n$ converges to the limiting distribution.

Let us revisit the weather example.
The matrix $P = \begin{pmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix}$ had the property that for higher values of $n$
\[P^n \approx P^{10} = \begin{pmatrix} 0.28395062 & 0.44444444 & 0.27160494 \\ 0.28395062 & 0.44444444 & 0.27160494 \\ 0.28395062 & 0.44444444 & 0.27160494 \end{pmatrix}\]
In other words the limiting distribution in this case is $\pi = [0.28395062, 0.44444444 , 0.27160494 ]$.
This says that in the long run a typical day has 28% chance of being Sunny, 44% chance of being Cloudy and 27% chance of being Rainy.

Suppose we are provided with a ergodic markov chain with transition matrix $P = \begin{pmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix}$.
And we must find the limit matrix \[\lim_{n \to \infty} P^n = \begin{pmatrix}\pi \\ \pi \\ \vdots \\ \pi \end{pmatrix}\]
For example in the weather case we saw the answer is \[P^n \approx P^{10} = \begin{pmatrix} 0.28395062 & 0.44444444 & 0.27160494 \\ 0.28395062 & 0.44444444 & 0.27160494 \\ 0.28395062 & 0.44444444 & 0.27160494 \end{pmatrix}\].

Let $\overline{P} = \lim_{n \to \infty} P^n = \begin{pmatrix}\pi \\ \pi \\ \vdots \\ \pi \end{pmatrix}$ be the limit matrix.
Then for any initial state $v$ the final state is $\pi$ \[v \overline{P} = \pi.\]
This means $$\begin{aligned}\pi = v \lim_{n \to \infty} P^n = v \lim_{n+1 \to \infty} P^{n+1} = v \lim_{n \to \infty}P^n P = v \overline{P} P = \pi P \end{aligned}$
In other words, we arrive at the surprising fact! that
$\pi$ is an eigenvector of $P$ with eigenvalue 1.

Now let us solve for the eigenvector for $P = \begin{pmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix}$.
We want \[ \pi P = \pi\]
In other words if $\pi = [\pi_1 \cdots \pi_k]$ then \[\begin{bmatrix} \pi_1 & \cdots & \pi_k \end{bmatrix} \cdot \begin{bmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{bmatrix} = \begin{bmatrix} \pi_1 & \cdots & \pi_k \end{bmatrix}.\]
In our case \[\begin{aligned}0.4 \pi_1 + 0.2 \pi_2 + 0.3 \pi_3 &= \pi_1 \\ 0.4 \pi_1 + 0.5 \pi_2 + 0.4 \pi_3 &= \pi_2 \\0.2 \pi_1 + 0.3 \pi_2 + 0.3 \pi_3 &= \pi_3 \end{aligned}\]

\[\begin{aligned}0.4 \pi_1 + 0.2 \pi_2 + 0.3 \pi_3 &= \pi_1 \\ 0.4 \pi_1 + 0.5 \pi_2 + 0.4 \pi_3 &= \pi_2 \\0.2 \pi_1 + 0.3 \pi_2 + 0.3 \pi_3 &= \pi_3 \end{aligned}\]
After solving we get \[\begin{aligned} \pi_1 &= \frac{23}{81} &\approx 0.28395 \\ \pi_2 &= \frac{36}{81} &\approx 0.44444 \\ \pi_3 &= \frac{22}{81} &\approx 0.2716 \end{aligned}\]
which agrees with our initial guess!

One can use the sympy library to perform symbolic computations.
We enter rational number for example $\frac{4}{10}$ by sympy.Rational(4,10)
We create a matrix $P$ by P = sympy.Matrix()
Eigenvalues and eigenvectors can be computed by P.eigenvals() and P.eigenvects() respectively.
Refer to jupyter notebook “Eigenvector Calculation” for full details.