Lecture 12: Markov Chains

Author

Junaid Hasan

Published

July 19, 2021

Markov Chains: Introduction

Let us start with an example.
Suppose we want to model the weather in Seattle
Say there are three possibilities
S: Sunny
C: Cloudy
R: Rainy

Weather Example

Suppose we look at the historic data and make the following observation:
Probability of sunny day following a rainy day is 0.3.
Probability of cloudy day following a rainy day is 0.4.
Probability of rainy day following a rainy day is 0.3.
We can get this probability by taking a note of data for say the past 100 years.

Example contd..

Suppose we collect all the data in a matrix:
\[\begin{array}{c|c c c} & S & C & R \\\hline S & 0.4 & 0.4 & 0.2 \\ C & 0.2 & 0.5 & 0.3 \\ R & 0.3 & 0.4 & 0.3 \end{array}\]
On the left column is the current state
On the top row is the next state
The matrix gives the probability from current state to next state.

Stochastic Matrix

The matrix \[\begin{pmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix}\]
is called a stochastic matrix.
Notice the \((i,j)\)th entry gives the probability to go from \(i\) to \(j\).
The rows sum to \(1\) (since we must go from current state to some other state).
The columns may not sum to 1 because some states may be more likely

Assumptions

Notice that the matrix being finite and having constant numbers makes two assumptions
The number of states is finite.
The probability of going from one state to another does not depend on the entire history, it only depends on the current state.

State Diagram

Instead of a matrix we can associate a state diagram
State Diagram
Note that it is a directed graph, with weights and self-loops.

More info

Suppose we ask the question:
What is the chance it is sunny two days from now, given it is rainy today? Or maybe ten days from now ?
A more general question is what is the probability any given day is rainy, sunny, cloudy? If at all it can be determined.

Question

Lets address the question:
chance it is sunny two days from now, given it is rainy today?
It can be sunny two days from now given its rainy today in the possible ways:
R -> S -> S
R -> C -> S
R -> R -> S

Computation

From the matrix \[\begin{pmatrix} 0.4 & 0.5 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix}\]
R -> S -> S has probability \(0.3 \cdot 0.4 = 0.12\)
R -> C -> S has probability \(0.4 \cdot 0.2 = 0.08\)
R -> R -> S has probability \(0.3 \cdot 0.3 = 0.09\)
These events are disjoint so the probability that it is sunny two days from now given rainy today is by
adding we get \(0.12+0.08+0.09 = 0.29\)

Observation

Can we do it in a smarter way.
Notice that the previous calculation resembles Matrix multiplication.
Let us look at the matrix more carefully \[A = \begin{pmatrix} 0.4 & 0.4 & 0.2 \\ 0.2 & 0.5 & 0.3 \\ 0.3 & 0.4 & 0.3 \end{pmatrix}\]
What will \(A^2\) tell us?
It tells us given a current state what will be the next state after 2 days.

Powers of Transition Matrix

\[A^2 = \begin{pmatrix} 0.3 & 0.44 & 0.26 \\ 0.27 & 0.45 & 0.28 \\ 0.29 & 0.44 & 0.27 \end{pmatrix}\]
One way to manipulate matrices in Python is using numpy library.
import numpy
A = numpy.matrix([[0.4, 0.4, 0.2],[0.2, 0.5, 0.3],[0.3, 0.4, 0.3]])
Then \(A^2\) is given by
A*A or A2 = numpy.multiply(A,A)
Similarly the probability after n steps is given by \(A^n\).

Some more powers

For this we use An= numpy.linalg.matrix_power(A,n)
\[A^3 = \begin{pmatrix} 0.286 & 0.444 & 0.27 \\ 0.282 & 0.445 & 0.273 \\ 0.285 & 0.444 & 0.271 \end{pmatrix}\]
\[A^4 = \begin{pmatrix} 0.2842 & 0.4444 & 0.2714 \\ 0.2837 & 0.4445 & 0.2718 \\ 0.2841 & 0.4444 & 0.2715 \end{pmatrix}\]
One can see that the values seem to stabilize
Do you notice anything else?
The rows seem to be almost the same.

contd..

For example
\[A^{10} = \begin{pmatrix} 0.28395062 & 0.44444444 & 0.27160494 \\ 0.28395062 & 0.44444444 & 0.27160494 \\ 0.28395062 & 0.44444444 & 0.27160494 \end{pmatrix}\]
Do you notice something interesting?
The rows are identical!
Therefore it does not depend on what the initial weather is.
It seems pretty reasonable that weather 10 days from now does not depend much on what it is today.
Therefore any given day it is 28% Sunny, 44% Cloudy , 27% Rainy.
This state is known as stationary state.
It does not happen always!

Markov Chain

The weather example was an example of a Markov Chain
There are three states: Sunny, Cloudy and Rainy.
At each step (day) the chain is exactly one of these states.
The matrix \(A = (a_{ij})\) is an example of a transition matrix: entry \(a_{ij}\) of \(A\) gives the probability of transitioning from state \(i\) to state \(j\).

Definition of Markov Chain

Imagine a system S which can be, at any time, in one of a finite set of states \(s_1, s_2, \ldots, s_k\) and which can change its state only at a discrete set of times (each second, each day, each coin flip, each doctor visit etc.)
The system \(S\) is a Markov Chain if the probability of transition into a state \(s_i\) depends only on the current state and is unaffected by the states at earlier times.
An example was weather.

Another example: Random Walk on Integers

Say a particle sits on the origin at \(t = 0\).
Each second, the particle moves one step to the right or left determined by a coin flip.
Moreover, to make the system finite, suppose that the particle stops if it reach 3 or -3.
The states of the chain are \(\{-3, -2, -1, 0, 1, 2, 3 \}\)

Random Walk contd..

Transition Probabilities
\[A = \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.5 & 0 & 0.5 & 0 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0.5 & 0 & 0 & 0 \\ 0 & 0 & 0.5 & 0 & 0.5 & 0 & 0 \\ 0 & 0 & 0 & 0.5 & 0 & 0.5 & 0 \\ 0 & 0 & 0 & 0 & 0.5 & 0 & 0.5 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}\]

Powers

\[A^3 = \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0.25 & 0 & 0.125 & 0 & 0 \\ 0.25 & 0.25 & 0 & 0.375 & 0 & 0.125 & 0 \\ 0.125 & 0 & 0.375 & 0 & 0.375 & 0 & 0.125 \\ 0 & 0.125 & 0 & 0.375 & 0 & 0.25 & 0.25 \\ 0 & 0 & 0.125 & 0 & 0.25 & 0 & 0.625 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}\]
For example starting at -2, in three steps the probability to go to -3 is 0.625

Powers

Let us investigate higher powers
\[A^{30} = \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.829 & 0.002 & 0 & 0.004 & 0 & 0.002 & 0.162 \\ 0.660 & 0 & 0.007 & 0 & 0.007 & 0 & 0.327 \\ 0.491 & 0.004 & 0 & 0.008 & 0 & 0.004 & 0.491 \\ 0.327 & 0 & 0.007 & 0 & 0.007 & 0 & 0.660 \\ 0.162 & 0.002 & 0 & 0.004 & 0 & 0.002 & 0.829 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}\]
\[A^{31} = \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.829 & 0 & 0.003 & 0 & 0.003 & 0 & 0.163 \\ 0.660 & 0.003 & 0 & 0.007 & 0 & 0.003 & 0.327 \\ 0.493 & 0 & 0.007 & 0 & 0.007 & 0 & 0.493 \\ 0.327 & 0.003 & 0 & 0.007 & 0 & 0.003 & 0.660 \\ 0.163 & 0 & 0.003 & 0 & 0.003 & 0 & 0.830 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}\]

Powers contd..

Note that certain states cannot be reached from certain other stages in odd/even steps, hence the difference between \(A^{30}\) and \(A^{31}\) for example.
The chain is an example of a
discrete
one-dimensional
unbiased
absorbing
random-walk.
States -3,3 are absorbing.

Recurrent and Transient states

A state \(i\) is recurrent if starting from a state \(i\), eventual return to state \(i\) is certain.
A state is transient if it is not recurrent, there is positive chance that never come back.
Let \(X_0\) denote initial state.
Then a state is recurrent if \[\Pr(\text{ever reenter }i | X_0 = i) = 1\]
A state \(i\) is transient if \[\Pr(\text{ever reenter }i | X_0 = i) < 1\]
Equivalently a state \(i\) is transient if \[\Pr(\text{never enter }i | X_0 = i) > 0.\]

Example

State-Diagram-2
States 1,2,3 are recurrent.
Probability that starting at 1,2,3 you never come back to it is 0.
However, state 0 is transient, starting at 0 there is 0.8 probability that you never come back.

contd.

State-Diagram-3
Now we added state 4.
Now states 1,2,3 are transient as well, because
there is positive chance that you go to state 4 and never come back.
State 4 is absorbing, once entered cannot leave.
Therefore 4 is recurrent.

Summary

In the weather case all states are recurrent.
In the random walk on [-3, -2, -1, 0, 1, 2, 3] only 3, -3 are recurrent, all other are transient.