Markov Chains and Stochastic Processes Questions and Answers

Answer: B

Both (I) and (II) are true.

A persistent (recurrent) state is null persistent when its mean recurrence time \(\mu_{i}\) is infinite; that is the definition, so (II) is true. For such a state \(p_{ii}^{(n)} \to 0\), so \(\limsup p_{ii}^{(n)} = 0\) and (I) is true.

A positive persistent state has finite mean recurrence time and, if aperiodic, \(p_{ii}^{(n)} \to 1/\mu_{i} \gt 0\). So \(\limsup p_{ii}^{(n)} \gt 0\) characterises positive persistence, and (III) is false for a null persistent state.

Answer: D

\(\beta\) is 0.4.

In a doubly stochastic matrix every row and every column sums to 1. Work through the entries that are forced:

\[ \begin{aligned} \text{column 1: } 0.3 + 0.4 + \delta = 1 &\Rightarrow \delta = 0.3 \\ \text{row 3: } 0.3 + 0.2 + \varepsilon = 1 &\Rightarrow \varepsilon = 0.5 \\ \text{column 3: } \beta + 0.1 + 0.5 = 1 &\Rightarrow \beta = 0.4 \end{aligned} \]

As a check, row 1 then gives \(\alpha = 0.3\) and column 2 gives \(\gamma = 0.5\), with row 2 summing to 1. So \(\beta\) can be computed and equals exactly 0.4.

Answer: A

The realized values are X₁ = 2 and X₂ = 2.

To simulate a step, compare the uniform number with the cumulative probabilities of the current row.

From \(X_{0} = 3\), row 3 is \((\frac{1}{6}, \frac{1}{3}, \frac{1}{2})\), with cumulative values 0.1667, 0.5, 1. Since \(0.1667 \lt 0.36 \le 0.5\), \(X_{1} = 2\).

From state 2, row 2 is \((\frac{3}{8}, \frac{5}{8}, 0)\), with cumulative values 0.375, 1, 1. Since \(0.375 \lt 0.64 \le 1\), \(X_{2} = 2\).

Reading 0.36 against row 1 or 0.64 against row 3 gives the other combinations, which use the wrong rows.

Answer: A

States 1 and 3 are essential states.

From state 1 the chain moves only to 1 or 3, and from 3 only to 1 or 3. So {1, 3} is a closed class: once entered it is never left, and 1 and 3 communicate. A state is essential if every state it leads to leads back to it, which holds for 1 and 3.

A finite closed communicating class is positive persistent, so 1 and 3 are not transient or inessential, and not null persistent (a finite chain has no null persistent states). The same holds for the separate closed class {2, 4}.

Answer: B

The probability is e⁻¹⁰.

From 10:00 am to 12:00 noon is 2 hours. In a Poisson process with rate 5 per hour, the number of messages in 2 hours is Poisson with mean \(5 \times 2 = 10\), so

\[ P(N = 0) = e^{-10} \]

\(1 - e^{-10}\) is the probability of at least one message, \(2e^{-5}\) uses a one-hour mean and doubles it incorrectly, and \(e^{-2/5}\) divides the time by the rate instead of multiplying.

Answer: C

Only (I) and (III) are true.

A pure birth process with constant rate \(\lambda\) starting at 0 is the Poisson process. It has stationary and independent increments (I), and X(t) is Poisson with mean \(\lambda t\) for each t (III).

(II) is false: the mean is \(E(X(t)) = \lambda t\), not \(e^{\lambda t}\). The exponential mean belongs to the Yule process, where the birth rate is proportional to the population, \(\lambda_{i} = i\lambda\). (IV) is false: X(0) = 0 is the usual starting point of the Poisson process.

Answer: B

The chains with stationary distribution \(\left(\frac{1}{2}, \frac{1}{2}\right)\) are \(P_{1}\) and \(P_{2}\).

Check \(\pi P = \pi\) with \(\pi = \left(\frac{1}{2}, \frac{1}{2}\right)\). For the identity \(P_{1}\), every distribution is stationary. \(P_{2}\) swaps the two states, so it maps \(\left(\frac{1}{2}, \frac{1}{2}\right)\) to itself; in general a doubly stochastic matrix has the uniform distribution stationary.

\[ \begin{aligned} \pi P_{3} &= \left(\tfrac{1}{4} + \tfrac{1}{3}, \tfrac{1}{4} + \tfrac{1}{6}\right) = \left(\tfrac{7}{12}, \tfrac{5}{12}\right) \\ \pi P_{4} &= \left(\tfrac{1}{4}, \tfrac{1}{4} + \tfrac{1}{2}\right) = \left(\tfrac{1}{4}, \tfrac{3}{4}\right) \end{aligned} \]

Neither is \(\left(\frac{1}{2}, \frac{1}{2}\right)\), so every option containing \(P_{3}\) or \(P_{4}\) is wrong. (\(P_{4}\)'s stationary distribution is (0, 1), since state 2 is absorbing.)

Answer: D

Only (I), (II) and (III) are true.

From the matrix, state 1 moves only to 1 or 3, and state 3 only to 1 or 3; state 2 moves only to 2 or 4, and state 4 only to 2 or 4. So the chain splits into two closed classes, {1, 3} and {2, 4}.

(I) Starting in {1, 3}, the chain can never reach 2, so \(f_{12} = 0\). (II) and (III) A finite closed irreducible class is recurrent, so from any state of the class every other state is reached with probability 1: \(f_{24} = 1\) and \(f_{13} = 1\).

(IV) State 3 is recurrent, so \(f_{33} = 1\), not less than 1. That leaves exactly (I), (II) and (III).

Answer: A

The true option is \(f_{22} \lt 1\).

A state is recurrent exactly when \(\sum_{n} p_{22}^{(n)} = \infty\). Here the sum is the finite number 2.45, so state 2 is transient and \(f_{22} \lt 1\). In fact the sum equals \(\frac{f_{22}}{1 - f_{22}}\), so

\[ f_{22} = \frac{2.45}{3.45} \approx 0.71 \]

A transient state has \(f_{22} \lt 1\) (so option B fails) and an infinite mean recurrence time (option C fails), and since the series of \(p_{22}^{(n)}\) converges, its terms tend to 0 (option D fails).

Answer: D

All three are true.

The first-passage decomposition gives \(p_{ii}^{(n)} = \sum_{k=1}^{n} f_{ii}^{(k)} p_{ii}^{(n-k)}\), with \(p_{ii}^{(0)} = 1\).

For \(n = 2\): \(p_{ii}^{(2)} = f_{ii}^{(2)} + f_{ii}^{(1)}p_{ii}^{(1)}\), which rearranges to (I). This holds in any chain.

For \(n = 4\):

\[ \begin{aligned} p_{ii}^{(4)} = &f_{ii}^{(4)} + f_{ii}^{(3)}p_{ii}^{(1)} \\ &+ f_{ii}^{(2)}p_{ii}^{(2)} + f_{ii}^{(1)}p_{ii}^{(3)} \end{aligned} \]

With period 2, returns are possible only at even times, so \(p_{ii}^{(1)} = p_{ii}^{(3)} = 0\) and \(f_{ii}^{(1)} = f_{ii}^{(3)} = 0\), leaving (II).

For the same reason a first return at the odd time 7 is impossible, so \(f_{ii}^{(7)} = 0\), which is (III).