update

Author: cer

book/bookdraft2018.pdf

@@ -25,7 +25,7 @@
     "- [](#)\n",
     "- [](#)\n",
     "- [](#)\n",
-    "- [](#)\n",
+    "- [13. Policy Gradient Methods](#13.-Policy-Gradient-Methods)\n",
     "- [](#)\n",
     "- [](#)\n",
     "- [](#)\n",
@@ -314,10 +314,10 @@
     "### on-policy vs off-policy\n",
     "- on-policy只有一套policy，更简单，是首选。\n",
     "- off-policy使用两套policy，更复杂、更难收敛；但也更通用、更强大。\n",
+    "- on-policy和off-policy本质依然是Exploit vs Explore的权衡。\n",
     "\n",
     "### on-policy\n",
     "- 去评估和提高生成episode时采用的policy。**全过程只有一种策略**，MC ES属于on-policy。\n",
-    "- ![](https://github.com/applenob/rl_learn/raw/master/res/on_policy_fv_mc_control.png)\n",
     "\n",
     "### off-policy\n",
     "- 所有的MC控制方法都面临一个**困境**：它们都想找到一个最优的策略，但却必须采用非最优的策略去尽可能多地探索（explore）数据。\n",
@@ -345,7 +345,6 @@
     "- 上面的评估使用了采样权重增量式的方法。\n",
     "- 控制：\n",
     "- ![](https://github.com/applenob/rl_learn/raw/master/res/off_policy_mc_control.png)\n",
-    "\n",
     "\n"
    ]
   },
@@ -356,49 +355,67 @@
     "## 6. Temporal-Difference Learning\n",
     "\n",
     "\n",
-    "****：\n",
-    "![](https://github.com/applenob/rl_learn/raw/master/res/td0_est.png)\n",
-    "\n",
-    "$V(S)\\leftarrow V(S)+\\alpha[R+\\gamma V(S')-V(S)]$\n",
-    "![](https://github.com/applenob/rl_learn/raw/master/res/sarsa_2.png)\n",
-    "![](https://github.com/applenob/rl_learn/raw/master/res/sarsa_est.png)\n",
-    "$Q(S,A)\\leftarrow Q(S,A)+\\alpha[R+\\gamma Q(S',A')-Q(S,A)]$\n",
-    "![](https://github.com/applenob/rl_learn/raw/master/res/q_learn_backup.png)\n",
-    "\n",
-    "![](https://github.com/applenob/rl_learn/raw/master/res/q_learn.png)\n",
-    "\n",
-    "\n",
-    "****：价值函数更新：$Q(S,A)\\leftarrow Q(S,A)+\\alpha[R+\\gamma Q(S',A')-Q(S, A)]$\n",
-    "\n",
-    "****：\n",
-    "\n",
-    "****：\n",
-    "\n",
-    "****：\n",
-    "\n",
-    "****：\n",
-    "\n",
-    "****：\n",
-    "\n",
-    "****：\n",
-    "\n",
-    "****："
+    "### 时序差分（Temporal-Difference）简介\n",
+    "- 时序差分是强化学习的核心观点。\n",
+    "- 时序差分是DP和MC方法的结合。\n",
+    "- 时序差分不需要像MC一样，要等一个完整的序列结束；相反，每经历一步，都会更新价值函数。\n",
+    "- TD往往比MC高效\n",
+    "\n",
+    "### 什么是stationary？\n",
+    "- stationary：环境不随时间变化而变化；\n",
+    "- non-stationary：环境会随时间变化而变化。\n",
+    "\n",
+    "### TD(0)\n",
+    "- $V(S_t)\\leftarrow V(S_t)+\\alpha[R_{t+1}+\\gamma V(S_{t+1})-V(S_t)]$\n",
+    "- 因为直接使用现有的估计取更新估计，因此这种方法被称为**自举（bootstrap）**。\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/td0_est.png)\n",
+    "- **TD error**：$\\delta_t = R_{t+1}+\\gamma V(S_{t+1})-V(S_t)$\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/td0.png)\n",
+    "\n",
+    "### Sarsa\n",
+    "- 一种on-policy的TD控制。\n",
+    "- $Q(S_t,A_t)\\leftarrow Q(S_t,A_t)+\\alpha[R_{t+1}+\\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)]$\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/sarsa_est.png)\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/sarsa_backup.png)\n",
+    "\n",
+    "### Q-learning\n",
+    "- 一种off-policy的TD控制。\n",
+    "- 早期强化学习的一个突破。\n",
+    "- $Q(S_t,A_t)\\leftarrow Q(S_t,A_t)+\\alpha[R_{t+1}+\\gamma \\underset{a}{max}Q(S_{t+1},a)-Q(S_t,A_t)]$\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/q_learn.png)\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/q_learn_backup.png)\n",
+    "\n",
+    "### Expected Sarsa\n",
+    "- 一种off-policy的TD控制。\n",
+    "- $Q(S_t,A_t)\\leftarrow Q(S_t,A_t) + \\alpha[R_{t+1} + \\gamma\\sum_a\\pi(a|S_{t+1})Q(S_{t+1}, a)-Q(S_t,A_t)]$\n",
+    "\n",
+    "### Double Learning\n",
+    "- 解决Q-learning的**最大化偏差（maximization bias）**问题\n",
+    "- 2011年提出。\n",
+    "- ![](https://github.com/applenob/rl_learn/raw/master/res/double_q_learn.png)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": []
+   "source": [
+    "## 7. n-step Bootstrapping\n",
+    "\n"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": []
+   "source": [
+    "## 8. Planning and Learning with Tabular Methods\n"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": []
+   "source": [
+    "## 9. On-policy Prediction with Approximation"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -418,7 +435,9 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": []
+   "source": [
+    "## 13. Policy Gradient Methods"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -461,21 +480,21 @@
  "metadata": {
   "anaconda-cloud": {},
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 2",
    "language": "python",
-   "name": "python3"
+   "name": "python2"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 3
+    "version": 2
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.4"
+   "pygments_lexer": "ipython2",
+   "version": "2.7.14"
   }
  },
  "nbformat": 4,