Learning Policy

\[R_\theta = \sum_\tau reward(\tau)p_\theta(\tau) \ \nabla R_\theta = \sum_\tau reward(\tau) \nabla p_\theta(\tau) \ = \sum_\tau reward(\tau) p_\theta(\tau) \frac{\nabla p_\theta(\tau)} {p_\theta(\tau)} \ = \sum_\tau reward(\tau) p_\theta(\tau) \nabla logp_\theta(\tau) \ = E_{\tau -p_\theta(\tau)}[reward(\tau)\nabla logp_\theta(\tau)] \ \approx \frac1n \sum_{\tau=1}^n reward(\tau)\nabla logp_\theta(\tau)\ = \frac1n \sum_{\tau=1}^n reward(\tau)\nabla log (p(s_{\tau1})p_\theta(a_{\tau1}|s_{\tau1})p(s_{\tau2}|s_{\tau1},a_{\tau1})p_\theta(a_{\tau2}|s_{\tau2})...)\ =\frac1n \sum_{\tau=1}^n reward(\tau) \sum_{t=1}^{T_\tau} \nabla logp_{\theta}(a_{\tau t}|s_{\tau t}) \ =\frac1n \sum_{\tau=1}^n \sum_{t=1}^{T_\tau} reward(\tau) \nabla logp_{\theta}(a_{\tau t}|s_{\tau t}) \ \] \[R_\theta = \s