Homework SourseThe capacity of a water reservoir is C unit C is a positive
The capacity of a water reservoir is C unit, C is a positive integer. The amount of water inow an
each day can be considered as independent identically distributed random variables In with PMF
qj = P [In = j], j = 0,1,2,... . A unit amount of water is drained each day from the reservoir unless
it is empty or overowed. In the later case the reservoir is full at the end of the day.
Denote by Xn the amount of water in the reservoir at the end of day n.
(a) Describe the evolution of (Xn)n>0, that is, express Xn+1 as a function of Xn and In+1.
(b) Find the state space and the transition probability matrix of this Markov chain.
Solution
Answer
Any iid sequence forms a Markov chain d for if {Xn} is iid , then {Xn+1, Xn+2......} (the future) is independent of { X0, ....Xn-1} (the past) , the given Xn (the present).Infact { Xn+1, Xn+2,...} is independent of { X0 , .... Xn) (the past and the present). For an iid sequence, the future is independent of the past and the present state. Let p(j) = P(X =j) denote the common probability mass function (pmf) of the function Xn. Then Pij = P(X1 = j / X0 = i = P(X1 = j) = p(j) because of the independence of X0 and X1. Pij does not depend on i: Each row of P is the same, namely the pmf(p(j)).
An iid sequence is a very special kind of Markov chain ; Whereas a Markov chain\'s future is allowed (but not required)to depend on the present state , an iid sequence\'s future does not depend on the present state at all.
We must specify what happens when either state 0 or 3 is hit. Let\'s assume that P0,0 = 1 = P3,3. meaning that both states 0 and 3 are absorbing. Then the transition matrix is
P = 1 0 0 0
1-p 0 p 0
0 1-p 0 p
0 0 0 1
