3 Let X 0 be a Markov chain with state space D a b c d e a

(3) Let (X) > 0 be a Markov chain with state space D = {a, b, c, d, e} and transition matrix (c) Define the reward function g by Compute

Solution

Solution :

recall what u have studied from weak law of large numbers :

Given X₁, X₂, ... an infinite sequence of i.i.d. random variables with finite expected value E(X₁) = E(X₂) = ... =