Chapter3 Markov Decision Processes(MDP)

参考了《Reinforcement Learning: An Introduction》和
David Silver强化学习公开课，
这一章主要来自David Silver的ppt，建议直接看ppt，我只把容易犯错的地方点出来了

马尔科夫过程是强化学习的基础

Finite Markov Decision Processes

Markov property

A state $S_t$ is Markov if and only of

$P [S_{t + 1} | S_{t}] = P [S_{t + 1} | S_{1}, \dots, S_{t}]$ $\mathbb{P}[S_{t+1}|S_t]=\mathbb{P}[S_{t+1}|S_1,\cdots,S_t]$

The state captures all relevant information from the history

Once the state is know, the history may be thrown away

i.e. The state is a sufficient statistic of the future

A Markov process is a memoryless random process, i.e. a sequence of random states $S_1,S_2,\cdots$ with the Markov property.
Markov Process

A Markov Process (or Markov Chain) is a tuple $\langle S,P \rangle$

S is a (finite) set of states

P is a state transition probability matrix, $P_{ss^\prime}=\mathbb{P}[S_{t+1}=s^\prime|S_t=s]$

A Markov reward process is a Markov chain with values.
Markov Reward Process

A Markov Process (or Markov Chain) is a tuple $\langle S,P,\color{red}{R,\gamma} \rangle$

S is a (finite) set of states

P is a state transition probability matrix, $P_{ss^\prime}=\mathbb{P}[S_{t+1}=s^\prime|S_t=s]$

$\color{red}{\text{R is a reward function, $R_s=E[R_{t+1}|S_t=s]$}}$

$\color{red}{\gamma \text{ is a discount factor, }\gamma \in [0,1]}$

注意这里 $P_{ss^\prime}$ 的定义，是指从状态 $s$ 到 $s^\prime$ 的概率

后面常因为名字(return)忘记这个的定义，跟上面的单个Reward不一样
Return

The return $G_t$ is the total discounted reward from time-step t.

$G_{t} = R_{t + 1} + γ R_{t + 2} + \dots = \sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1}$ $G_t=R_{t+1}+\gamma R_{t+2}+\cdots=\sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$

The discount $\gamma \in [0,1]$ is the present value of future rewards

The value of receiving reward R after k+1 time-steps is $\gamma^k R$

$\gamma$ close to 0 leads to “myopic(近视)” evaluation

$\gamma$ close to 1 leads to “far-sighted(远见)” evaluation
后面提到的很多方法都是看的很远(远见)的

Value Function

The state value function v(s) of an $\color{red}{\text{MRP}}$ is the expected return starting form state s

$v (s) = E [G_{t} | S_{t} = s]$ $v(s)=\mathbb{E}[G_t|S_t=s]$

确实有必要看一下MRP的Bellman Equation，并与MDP对比。在MRP中没有考虑任何关于action的事情。因为MDP才是强化学习的主角，所以不看David Silver的ppt中的MRP实例了，容易对后面MDP的理解造成误解。
简单看一下Bellman Equation

\begin{aligned} v (s) & = E [G_{t} | S_{t} = s] \\ = E [R_{t + 1} + γ v (S_{t + 1}) | S_{t} = s] \end{aligned}

$\begin{align*} v(s) & = \mathbb{E} [G_t|S_t=s] \\ & = \mathbb{E} [R_{t+1}+\gamma v(S_{t+1})|S_t=s] \end{align*}$
MRP的状态转换，没有任何action的影响，我们在后面MDP中会考虑actions的影响
MRP state transfer

v (s) = R_{s} + γ \sum_{s^{'} \in S} P_{s s^{'}} v (s^{'})

$v(s)=R_s+\gamma \sum_{s^\prime \in S} P_{ss^\prime} v(s^\prime)$
其实观察上式，上面计算的是动态规划，而注意到Bellman Equation又称为动态规划方程，上面的计算就很容易理解了

A Markov decision process (MDP) is a Markov reward process with decisions. It is an environment in which all states are Markov.
Markov Decision Process

A Markov Process (or Markov Chain) is a tuple $\langle S,\color{red}{A},P,R,\gamma \rangle$

S is a (finite) set of states

A is finite set of actions

P is a state transition probability matrix, $P^{\color{red}{\text{a}}}_{ss^\prime}=\mathbb{P}[S_{t+1}=s^\prime|S_t=s, A_t=\color{red}{\text{a}}]$

R is a reward function, $R^\color{red}{\text{a}}_s=E[R_{t+1}|S_t=s, A_t=\color{red}{\text{a}}]$

$\gamma$ is a discount factor, $\gamma \in [0,1]$

Student example for MDP
注意与上面MRP的区别，这里的黑点是执行一个action之后到达的中间状态，后面用 $q(s,a)$ 来定义此状态，黑点到达后面的状态 $s^\prime$ 的概率就是上面MDP中定义的那个 $P^a_{ss^\prime}=\mathbb{P}[S_{t+1}=s^\prime|S_t=s, A_t=a]$

Policy

A policy $\pi$ is a distribution over actions given states,

$π (a | s) = P [A_{t} = a | S_{t} = s]$ $\pi(a|s)=\mathbb{P}[A_t=a|S_t=s]$

A policy fully defines the behaviour of an agent

MDP policies depend on the current state (not the history)

i.e. Policies are stationary (time-independent), $A_t \sim \pi(\cdot |S_t), \forall t \gt 0$

Given an MDP $M=\langle S,A,P,R,\gamma \rangle$ and a policy $\pi$

The state sequence $S_1,S_2,\cdots$ is a Markov reward process $\langle S,P^{\pi} \rangle$

The state and reward sequence $S_1,R_2,S_2,\cdots$ is a Markov reward process $\langle S,P^{\pi},R^{\pi},\gamma \rangle$

where
$P_{s, s^{'}}^{π} = \sum_{a \in A} π (a | s) P_{s s^{'}}^{a} R_{s}^{π} = \sum_{a \in A} π (a | s) R_{s}^{a}$ $P^\pi_{s,s^\prime}=\sum_{a \in A} \pi(a|s)P^a_{ss^\prime}\\ R^\pi_s=\sum_{a \in A} \pi(a|s)R^a_s$

要特别注意policy的distribution的定义，因为在后面讲的off-policy方法的概念中，生成样本的policy和目标policy是不同的

Value Function这个是针对MDP的

The state-value function $v_{\pi}(s)$ of an MDP is the expected return starting from state $s$ , and then following policy $\pi$

$v_{π} (s) = E_{π} [G_{t} | S_{t} = s]$ $v_{\pi}(s)=\mathbb{E}_{\pi}[G_t|S_t=s]$

The action-value function $q_{\pi}(s,a)$ is the expected return
starting from state $s$ , taking action $a$ , and then following policy $\pi$

$q_{π} (s | a) = E_{π} [G_{t} | S_{t} = s, A_{t} = a]$ $q_{\pi}(s|a)=\mathbb{E}_{\pi}[G_t|S_t=s,A_t=a]$

Bellman Expectation Equation for $V^{\pi}$
$Bellman Expectation Equation for $V^{\pi}$$

v_{π} (s) = \sum_{a \in A} π (a | s) q_{π} (s, a)

$v_{\pi}(s)=\sum_{a \in A} \pi(a|s)q_{\pi}(s,a)$
Bellman Expectation Equation for

Q^{π}

$Q^{\pi}$
$Bellman Expectation Equation for $Q^{\pi}$$

q_{π} (s, a) = R_{s}^{a} + γ \sum_{s^{'} \in S} P_{s s^{'}}^{a} v_{π} (s^{'})

$q_{\pi}(s,a)=R^a_s+\gamma \sum_{s^\prime \in S}P^a_{ss^\prime} v_{\pi}(s^\prime)$
$Bellman Expectation Equation for $v_{\pi} 2$$

v_{π} (s) = \sum_{a \in A} π (a | s) (R_{s}^{a} + γ \sum_{s^{'} \in S} P_{s s^{'}}^{a} v_{π} (s^{'}))

$v_{\pi}(s)=\sum_{a \in A} \pi(a|s)(R^a_s+\gamma\sum_{s^\prime \in S} P^a_{ss^\prime} v_{\pi}(s^\prime))$
$Bellman Expectation Equation for $q_{\pi} 2$$

q_{π} (s, a) = R_{s}^{a} + γ \sum_{s^{'} \in S} P_{s s^{'}}^{a} \sum_{a^{'} \in A} π (a^{'} | s^{'}) q_{π} (s^{'}, a^{'})

$q_{\pi}(s,a)=R^a_s+\gamma \sum_{s^\prime \in S}P^a_{ss^\prime} \sum_{a^\prime \in A} \pi(a^\prime|s^\prime)q_{\pi}(s^\prime,a^\prime)$

Optimal Value Function

The optimal state-value function $v_*(s)$ is the maximum value function over all policies

$v_{*} (s) = max_{π} v_{π} (s)$ $v_*(s)=\underset{\pi}{\max} v_{\pi}(s)$

The optimal action-value function $q_*(s,a)$ is the maximum action-value function over all policies

$q_{*} (s, a) = max_{π} q_{π} (s, a)$ $q_*(s,a)=\underset{\pi}{\max} q_{\pi}(s,a)$

只要知道了 $q_*$ 问题就解决了，比知道 $v_*$ 更方便。还有注意的是，上面是在所有的 $\pi$ (policy)中选择使得 $q$ 最大的 $\pi$ (policy)，这就是值给出了最佳policy的概念，当然是没有很直接的办法得到结果的，后面将针对上述问题介绍各种逼近的方法

Optimal Policy
De ne a partial ordering over policies

π \geq π^{'} if v_{π} (s) \geq v_{π^{'}} (s), \forall s

$\pi \geq \pi^\prime \ \text{if}\ v_{\pi}(s) \geq v_{\pi^\prime}(s), \forall s$

Finding an Optimal Policy
An optimal policy can be found by maximising over $q_*(s,a)$ ,

π_{*} (a | s) = {\begin{cases} 1 & if a = \underset{a \in A}{a r g max} q_{*} (s, a) \\ 0 & otherwise \end{cases}

$\pi_*(a|s)= \begin{cases} 1 & \text{if a = }\underset{a\in A}{arg\max} q_*(s,a)\\ 0 & \text{otherwise} \end{cases}$
如果我们知道了

q_{*} (s, a)

$q_*(s,a)$ ，那么我就可以马上得到optimal policy

Optimal Bellman Expectation Equation

\begin{aligned} v_{π} (s) & ≐ E_{π} [G_{t} | S_{t} = s] \\ = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} | S_{t} = s] \\ = \sum_{a} π (a | s) \sum_{s^{'}} \sum_{r} p (s^{'}, r | s, a) [r + γ E_{π} [G_{t + 1} | S_{t + 1} = s^{'}]] \\ = \sum_{a} π (a | s) \sum_{s^{'}, r} p (s^{'}, r | s, a) [r + γ v_{π} (s^{'})], for all s \in S \end{aligned}

$\begin{align*} v_{\pi}(s) & \doteq \mathbb{E}_{\pi}[G_t|S_t=s] \\ & = \mathbb{E}_{\pi}[\sum_{k=0}^\infty{\gamma^k R_{t+k+1}|S_t=s}] \\ & = \sum_a \pi(a|s) \sum_{s^\prime}\sum_r p(s^\prime ,r|s,a)[r+\gamma \mathbb{E}_{\pi}[G_{t+1}|S_{t+1}=s^\prime]] \\ & = \sum_a \pi(a|s) \sum_{s^\prime, r} p(s^\prime ,r|s,a)[r+\gamma v_{\pi}(s^\prime)], \text{ for all $s \in S$} \end{align*}$

The Agent-Environment Interface

The learner and decision maker is called the agent.
The thing it interacts with, comprising everything outside the agent, is called the environment.

MDP和agent一起生成的sequence或者trajectory

S_{0}, A_{0}, R_{1}, S_{1}, A_{1}, R_{2}, S_{2}, A_{2}, R_{3}, \dots

$S_0, A_0, R_1, S_1, A_1, R_2, S_2, A_2, R_3,\cdots$

以下函数定义了MDP的动态性，agent处于某个状态s，在该状态下采取行动a，然后到达状态 $s^\prime$ ，并获得奖励r。这个公式是MDP的关键。这个四参数的函数可以推导出任何东西

p (s^{'}, r | s, a) ≐ Pr {S_{t} = s^{'}, R_{t} = r | S_{t - 1} = s, A_{t - 1} = a}

$p(s^\prime,r|s,a) \doteq \Pr\{S_t=s^\prime,R_t=r|S_{t-1}=s,A_{t-1}=a\}$

The agent-environment interaction in a Markov decision process
for all $s^\prime$ , $s\in S$ , $r\in R$ , and $a\in A(s)$

其中有

\sum_{s' \in S} \sum_{r \in R} p (s^{'}, r | s, a) = 1, for all s \in S, a \in A (s)

$\sum_{s\prime \in S}\sum_{r\in R}p(s^\prime,r|s,a)=1, \text{ for all $s\in S$, $a \in A(s)$}$

3.2 Goals and Rewards

agent的目的就是最大化它收到的全部rewards

3.5 Policies and Value Functions

state-value function for policy $\pi$

\begin{aligned} v_{π} (s) & ≐ E_{π} [G_{t} | S_{t} = s] \\ = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} | S_{t} = s] \\ = \sum_{a} π (a | s) \sum_{s^{'}} \sum_{r} p (s^{'}, r | s, a) [r + γ E_{π} [G_{t + 1} | S_{t + 1} = s^{'}]] \\ = \sum_{a} π (a | s) \sum_{s^{'}, r} p (s^{'}, r | s, a) [r + γ v_{π} (s^{'})], for all s \in S \end{aligned}

$\begin{align*} v_{\pi}(s) & \doteq E_{\pi}[G_t|S_t=s] \\ & = E_{\pi}[\sum_{k=0}^\infty{\gamma^k R_{t+k+1}|S_t=s}] \\ & = \sum_a \pi(a|s) \sum_{s^\prime}\sum_r p(s^\prime ,r|s,a)[r+\gamma E_{\pi}[G_{t+1}|S_{t+1}=s^\prime]] \\ & = \sum_a \pi(a|s) \sum_{s^\prime, r} p(s^\prime ,r|s,a)[r+\gamma v_{\pi}(s^\prime)], \text{ for all $s \in S$} \end{align*}$

action-value function for policy $\pi$

q_{π} (s, a) ≐ E_{π} [G_{t} | S_{t} = s, A_{t} = a] = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} | S_{t} = s, A_{t} = a]

$q_{\pi}(s,a) \doteq E_{\pi}[G_t|S_t=s,A_t=a]=E_{\pi}[\sum_{k=0}^\infty{\gamma^k R_{t+k+1}|S_t=s,A_t=a}]$

对于任何policy $\pi$ 和任何状态 $s$ ，state-value和其可能的后继状态的state-value之间存在以下一致性条件

3.6 Optimal Policies and Optimal Value Functions

optimal state-value function

v_{*} (s) ≐ max_{π} v_{π} (s)

$v_*(s) \doteq \underset{\pi}{\max} v_{\pi}(s)$
optimal action-value function

q_{*} (s, a) ≐ max_{π} q_{π} (s, a)

$q_*(s,a) \doteq \underset{\pi}{\max} q_{\pi}(s,a)$

写出关于 $v_*$ 的 $q_*$

q_{*} (s, a) = E [R_{t + 1} + γ v_{π} (S_{t + 1}) | S_{t} = s, A_{t} = a]

$q_*(s,a) = E[R_{t+1}+\gamma v_{\pi}(S_{t+1})|S_t=s,A_t=a]$

Bellman optimality equation

v_{*} (s) = max_{a \in A (s)} q_{π_{*}}

$v_*(s) = \underset{a \in A(s)}{\max} q_{\pi_*}$

Bellman Optimality Equation for $V^*$

\begin{aligned} v_{*} (s) & = max_{a \in A (s)} q_{π_{*}} (s, a) \\ = max_{a} E_{π_{*}} [G_{t} | S_{t} = s, A_{t} = a] \\ = max_{a} E_{π_{*}} [R_{t + 1} + γ G_{t + 1} | S_{t} = s, A_{t} = a] \\ = max_{a} E [R_{t + 1} + γ v_{*} (S_{t + 1}) | S_{t} = s, A_{t} = a] \\ = max_{a} \sum_{s^{'}, r} p (s^{'}, r | s, a) [r + γ v_{*} (s^{'})] \end{aligned}

$\begin{align*} v_*(s) & = \underset{a \in A(s)}{\max} q_{\pi_*}(s,a) \\ & = \underset{a}{\max} E_{\pi_*}[G_t|S_t=s,A_t=a] \\ & = \underset{a}{\max} E_{\pi_*}[R_{t+1}+\gamma G_{t+1}|S_t=s,A_t=a] \\ & = \underset{a}{\max} E[R_{t+1}+\gamma v_*(S_{t+1})|S_t=s,A_t=a] \\ & = \underset{a}{\max} \sum_{s^\prime ,r} p(s^\prime,r|s,a)[r+\gamma v_*(s^\prime)] \end{align*}$

Bellman Optimality Equation for $Q^*$

\begin{aligned} q_{*} (s, a) & = E [R_{t + 1} + γ max_{a^{'}} q_{*} (S_{t + 1}, a^{'}) | S_{t} = s, A_{t} = a] \\ = \sum_{s^{'}, r} p (s^{'}, r | s, a) [r + γ max_{a^{'}} q_{*} (s^{'}, a^{'})] \end{aligned}

$\begin{align*} q_*(s,a) & = E[R_{t+1}+\gamma \underset{a^\prime}{\max} q_*(S_{t+1},a^\prime)|S_t=s,A_t=a] \\ & = \sum_{s^\prime ,r} p(s^\prime,r|s,a)[r+\gamma \underset{a^\prime}{\max} q_*(s^\prime, a^\prime)] \end{align*}$