Reinforcement learning: Alberta

optimistic initial values

set an initial value which is larger than the max value
This heuristic can only drive early exploration.
They are not suited for non-stationary problems.
We do not know the maximum value of each arm-bandit.
But it is still a good heuristic to combine with other methods.

explore探索起点

探索起点指的是初始环境各个状态概率都不为0的情况。但是很多情况无法满足这个初始条件。因为有的时候有些状态从未被访问过，所以概率是0。然而为了避免这样的情况，只能在过程中持续探索，所以出现了-soft 策略，就是在过程中持续探索。所以就可以丢掉探索开始这个初始条件了。

离策略学习

off-policy学习和软策略学习不同。它将目标策略和行为策略分开。目标策略是指要学习的策略，行为策略是指用于动作选择的策略。分开的好处是可以持续探索，而不要软策略那样的手段。因为软策略无论是目标策略还是行为策略都不是最优策略。

Discounting-aware importance sampling

Per-decision importance sampling

Advantages of TD learning

TD converges faster in a low learning rate than MC and has a lower final error.
TD does not require a model.
TD is fully online and incremental.

General policy iteration

contains: policy evaluation, policy improvement

backup diagram(备份图)

其中黑点是(s,a)对,白点是s。

policy具体指什么

policy的种类现在接触过的就是greedy policy, epsilon-greedy policy等类似的。也就是说如果target policy和behavior policy都是一样,那就是on-policy,否则就是off-policy。

Sarsa: policy iteration
Q-learning: value iteration

Q learning是off-policy的原因

因为Q-learning学习的是最有的value function,所以它的动作估计是under target policy的,也就是说它不需要纠正estimation,所以是off-policy的。

期望sarsa

has lower variance
期望sarsa既是on-policy,也是off-policy的,主要看策略的选取。

Planning

Models are used to improve policies through a process called planning.
Sample model就是做实验,distribution model给出具体分布和概率。