Markov Chain: Reward

Let's say the player gets +1 for winning, -1 for losing, and for a draw. Given that the dealer has a sum of d, what is the player's expected winnings for stopping on p?

(Ignoring aces for the moment.)

• If $p = \mbox{bust}$ r^p(d) = -1.
• Else if $d = \mbox{bust}$, r^p(d) = +1.
• Else if $d \ge 17$, d>y, r^p(d) = -1.
• Else if $d \ge 17$, d=y, r^p(d) = 0.
• Else if $d \ge 17$, d<y, r^p(d) = +1.
• Else if d < 17, $r^p(d) = \sum_{d'} \Pr(d'\vert d) r^p(d')$.

Here, d' denotes the next card value for the dealer. For example,

• $\Pr(d'=15\vert d=12) = 1/13$.
• $\Pr(d'=17\vert d=7) = 4/13$.