Reinforcement learning: Alberta
optimistic initial values
set an initial value which is larger than the max value
This heuristic can only drive early exploration.
They are not suited for non-stationary problems.
We do not know the maximum value of each arm-bandit.
But it is still a good heuristic to combine with other methods.
explore探索起点
探索起点指的是初始环境各个状态概率都不为0的情况。但是很多情况无法满足这个初始条件。因为有的时候有些状态从未被访问过，所以概率是0。然而为了避免这样的情况，只能在过程中持续探索，所以出现了-soft 策略，就是在过程中持续探索。所以就可以丢掉探索开始这个初始条件了。
离策略学习
off-policy学习和软策略学习不同。它将目标策略和行为策略分开。目标策略是指要学习的策略，行为策略是指用于动作选择的策略。分开的好处是可以持续探索，而不要软策略那样的手段。因为软策略无论是目标策略还是行为策略都不是最优策略。
Discounting-aware importance sampling
Per-decision importance sampling
Advantages of TD learning
TD converges faster in a low learning rate than MC and has a lower final error.
TD does not require a model.
TD is fully online and incremental.
General policy iteration
contains: policy evaluation, policy improvement
backup diagram(备份图)
其中黑点是(s,a)对,白点是s。
policy具体指什么
policy的种类现在接触过的就是greedy policy, epsilon-greedy policy等类似的。也就是说如果target policy和behavior policy都是一样,那就是on-policy,否则就是off-policy。
Sarsa: policy iteration
Q-learning: value iteration
Q learning是off-policy的原因
因为Q-learning学习的是最有的value function,所以它的动作估计是under target policy的,也就是说它不需要纠正estimation,所以是off-policy的。
期望sarsa
has lower variance
期望sarsa既是on-policy,也是off-policy的,主要看策略的选取。
Planning
Models are used to improve policies through a process called planning.
Sample model就是做实验,distribution model给出具体分布和概率。
