Let us consider a Markov chain with state space a b c d e an

Let us consider a Markov chain with state space {a, b, c, d, e} and the transition probabilities Pa, d = 1, P_b, a = P_c,a = 1/2, P_c, b = 1/2, P_d, a = 1/3, P_d,b = 1/3 Compute the invariant distribution. Let us define tau := min{n 1 : X_n = b or c} and a reward function f( ) with f(a) = 2, f(d) = 3, f(e) = 1. Compute the cumulative rewards before the time tau: E[summation n = 0 tau - 1 f(X_n)|X_0 = a].

Solution

The transistion matrix is = 0 0 0 1 0 1 0 0 0 0 1/2 1/2 0 0 0 1/3 1/3 1/3 0 0 1/4 1/4 1/4 1/4 0 the invariant distribution is (1/5,1/5,1/5,1/5,1/5). Cumulative rewards before time is (1/2,1/2)