WIXLIB

A Tour of Reinforcement LearningThe View from Continuous ControlBenjamin RechtUniversity of California, Berkeley

trustable, scalable, predictable

!Control TheoryReinforcement Learning is the study of how to use past datato enhance the future manipulation of a dynamical system

Disciplinary BiasesAE/CE/EE/MECSControl IEEE TransactionsControldiscretedataactionScience Magazine

Disciplinary BiasesAE/CE/EE/MECSReinforcementToday’stalk Theorywill try to unify these camps and pointControlLearningout how to merge their perspectives.RLcontinuousmodelactionIEEE TransactionsControldiscretedataactionScience Magazine

Main research challenge: What are thefundamental limits of learning systems thatinteract with the physical environment?How well must we understand a systemin order to control it? statistical learning theorytheoreticalfoundations robust control theory core optimization

sha1 base64 "3LLePIAX9QVYtt6zAX6FKosLpuU " 4oGhXistyd3E0GZ2eXmbuSsOTX Nr oP4bZ2MKJumFgcM592PuPXGupCXf/1vzlt4sv3238r6 Jx/b6wmIN4gD52HdSQog3LyQZj/skxPZ5kxj1NfMK rightXaUxi4zBRrYea0i/6d1C0q 1zFDOblMNCS5H1cI5VNCQDjrRIKUhdbVX lErxK9CWn8v gP6prm0lt05lX5LdP3fO6L2FZGdIMH/ 9v3TvwfniHL6lebVqzxWbCO3sGRfzH2A /latexit sha1 base64 "3LLePIAX9QVYtt6zAX6FKosLpuU " 4oGhXistyd3E0GZ2eXmbuSsOTX Nr oP4bZ2MKJumFgcM592PuPXGupCXf/1vzlt4sv3238r6 Jx/b6wmIN4gD52HdSQog3LyQZj/skxPZ5kxj1NfMK rightXaUxi4zBRrYea0i/6d1C0q 1zFDOblMNCS5H1cI5VNCQDjrRIKUhdbVX lErxK9CWn8v gP6prm0lt05lX5LdP3fO6L2FZGdIMH/ 9v3TvwfniHL6lebVqzxWbCO3sGRfzH2A /latexit latexitsha1 base64 "3LLePIAX9QVYtt6zAX6FKosLpuU " 4oGhXistyd3E0GZ2eXmbuSsOTX Nr oP4bZ2MKJumFgcM592PuPXGupCXf/1vzlt4sv3238r6 Jx/b6wmIN4gD52HdSQog3LyQZj/skxPZ5kxj1NfMK rightXaUxi4zBRrYea0i/6d1C0q 1zFDOblMNCS5H1cI5VNCQDjrRIKUhdbVX lErxK9CWn8v gP6prm0lt05lX5LdP3fO6L2FZGdIMH/ 9v3TvwfniHL6lebVqzxWbCO3sGRfzH2A /latexit latexitxtsha1 base64 "3LLePIAX9QVYtt6zAX6FKosLpuU " 4oGhXistyd3E0GZ2eXmbuSsOTX Nr oP4bZ2MKJumFgcM592PuPXGupCXf/1vzlt4sv3238r6 Jx/b6wmIN4gD52HdSQog3LyQZj/skxPZ5kxj1NfMK rightXaUxi4zBRrYea0i/6d1C0q 1zFDOblMNCS5H1cI5VNCQDjrRIKUhdbVX lErxK9CWn8v gP6prm0lt05lX5LdP3fO6L2FZGdIMH/ 9v3TvwfniHL6lebVqzxWbCO3sGRfzH2A /latexit latexity latexitControl theory is the study of dynamical systems with inputsuxt 1 f(xt , ut )xt is called the state, and the dimension of thestate is called the degree, d.ut is called the input, and the dimension is p.

yuxtsha1 base64 "czC6ncmDUgINlqXuL3YVJwrAP1A " AAACqHicbVFbaxNBFJ6sl9Z6S fNWf4L9xNo3QJB4Y Pi c5lzvtxK4TGO/zSiGzdv3d7ZvbN39979Bw b g3EKq2FSLieAMA5U1j2y7zCp8kSwuv1 OVG7KyjKPiw59dkzpdbHMkB4WGXZo1mzF3XgZdBskK9Aiq hl 410NDa8UKCRS GumSvV5RMeX9XOUhUzGc U2tJv nDQucvEkroW2BoPnVoEkhKRpan4qOhQOOch4A406Ev1I 9Me3oqpjP8p4a2tdx L6YC/eFpcE13tpKDIcnm bfB2ctuEneTT69aJ29X1uySx /wJ4d9RY /latexit sha1 base64 "czC6ncmDUgINlqXuL3YVJwrAP1A " AAACqHicbVFbaxNBFJ6sl9Z6S fNWf4L9xNo3QJB4Y Pi c5lzvtxK4TGO/zSiGzdv3d7ZvbN39979Bw b g3EKq2FSLieAMA5U1j2y7zCp8kSwuv1 OVG7KyjKPiw59dkzpdbHMkB4WGXZo1mzF3XgZdBskK9Aiq hl 410NDa8UKCRS GumSvV5RMeX9XOUhUzGc U2tJv nDQucvEkroW2BoPnVoEkhKRpan4qOhQOOch4A406Ev1I 9Me3oqpjP8p4a2tdx L6YC/eFpcE13tpKDIcnm bfB2ctuEneTT69aJ29X1uySx /wJ4d9RY /latexit latexitsha1 base64 "czC6ncmDUgINlqXuL3YVJwrAP1A " AAACqHicbVFbaxNBFJ6sl9Z6S fNWf4L9xNo3QJB4Y Pi c5lzvtxK4TGO/zSiGzdv3d7ZvbN39979Bw b g3EKq2FSLieAMA5U1j2y7zCp8kSwuv1 OVG7KyjKPiw59dkzpdbHMkB4WGXZo1mzF3XgZdBskK9Aiq hl 410NDa8UKCRS GumSvV5RMeX9XOUhUzGc U2tJv nDQucvEkroW2BoPnVoEkhKRpan4qOhQOOch4A406Ev1I 9Me3oqpjP8p4a2tdx L6YC/eFpcE13tpKDIcnm bfB2ctuEneTT69aJ29X1uySx /wJ4d9RY /latexit latexitsha1 base64 "czC6ncmDUgINlqXuL3YVJwrAP1A " AAACqHicbVFbaxNBFJ6sl9Z6S fNWf4L9xNo3QJB4Y Pi c5lzvtxK4TGO/zSiGzdv3d7ZvbN39979Bw b g3EKq2FSLieAMA5U1j2y7zCp8kSwuv1 OVG7KyjKPiw59dkzpdbHMkB4WGXZo1mzF3XgZdBskK9Aiq hl 410NDa8UKCRS GumSvV5RMeX9XOUhUzGc U2tJv nDQucvEkroW2BoPnVoEkhKRpan4qOhQOOch4A406Ev1I 9Me3oqpjP8p4a2tdx L6YC/eFpcE13tpKDIcnm bfB2ctuEneTT69aJ29X1uySx /wJ4d9RY /latexit latexitp(xt 1 past) p(xt 1 xt , ut ) latexitReinforcementLearningdiscreteControl theory is the study of dynamical systems with inputs Markov Decision Process (MDP)xt is the state, and it takes values in [d]ut is called the input, and takes values in [p].

sha1 base64 "DOx/ktybitgjChwuZWtodyh8jiA " QqJEu4O7lJLs7ObmfuFsPi7 hr/Vn 5K5nerKfWe G5Zh9m1FC6twoTXxkCjx0s1pFvqe1C YlvsiIaIxHdxJE7FuWgJJf6If K/eAyCoBF8C6KX0KA2ydkSUxb8fAb8qMVJ /latexit sha1 base64 "DOx/ktybitgjChwuZWtodyh8jiA " QqJEu4O7lJLs7ObmfuFsPi7 hr/Vn 5K5nerKfWe G5Zh9m1FC6twoTXxkCjx0s1pFvqe1C YlvsiIaIxHdxJE7FuWgJJf6If K/eAyCoBF8C6KX0KA2ydkSUxb8fAb8qMVJ /latexit latexitsha1 base64 "DOx/ktybitgjChwuZWtodyh8jiA " QqJEu4O7lJLs7ObmfuFsPi7 hr/Vn 5K5nerKfWe G5Zh9m1FC6twoTXxkCjx0s1pFvqe1C YlvsiIaIxHdxJE7FuWgJJf6If K/eAyCoBF8C6KX0KA2ydkSUxb8fAb8qMVJ /latexit latexitsha1 base64 "DOx/ktybitgjChwuZWtodyh8jiA " QqJEu4O7lJLs7ObmfuFsPi7 hr/Vn 5K5nerKfWe G5Zh9m1FC6twoTXxkCjx0s1pFvqe1C YlvsiIaIxHdxJE7FuWgJJf6If K/eAyCoBF8C6KX0KA2ydkSUxb8fAb8qMVJ /latexit latexit t ( t ) is the policy. This is the optimization decision variable. latexitsha1 base64 "oTGOPnlC3lpbuJxkZHAlqk3gehs " Z/JKSUdx/KcV3bp95 69vfv7Dx4 evykffD03BlvBQ6FUcZe5uBQSY1DkqTwsrIIZa7wIr961 Ta SZW exTfUPKPDrN2J /Eq P9pI0 r0rFBub1HOvpTAFbrGK5mQhkA6pBKmbreoPUin FbTjZ3I6oxs1tG3k7ns5lcH9s3AufbiTHA6SbNu/C86P kncTz6/7py uaZ2wjotFfdanQ g /latexit sha1 base64 "oTGOPnlC3lpbuJxkZHAlqk3gehs " Z/JKSUdx/KcV3bp95 69vfv7Dx4 evykffD03BlvBQ6FUcZe5uBQSY1DkqTwsrIIZa7wIr961 Ta SZW exTfUPKPDrN2J /Eq P9pI0 r0rFBub1HOvpTAFbrGK5mQhkA6pBKmbreoPUin FbTjZ3I6oxs1tG3k7ns5lcH9s3AufbiTHA6SbNu/C86P kncTz6/7py uaZ2wjotFfdanQ g /latexit latexitsha1 base64 "oTGOPnlC3lpbuJxkZHAlqk3gehs " Z/JKSUdx/KcV3bp95 69vfv7Dx4 evykffD03BlvBQ6FUcZe5uBQSY1DkqTwsrIIZa7wIr961 Ta SZW exTfUPKPDrN2J /Eq P9pI0 r0rFBub1HOvpTAFbrGK5mQhkA6pBKmbreoPUin FbTjZ3I6oxs1tG3k7ns5lcH9s3AufbiTHA6SbNu/C86P kncTz6/7py uaZ2wjotFfdanQ g /latexit latexitsha1 base64 "oTGOPnlC3lpbuJxkZHAlqk3gehs " Z/JKSUdx/KcV3bp95 69vfv7Dx4 evykffD03BlvBQ6FUcZe5uBQSY1DkqTwsrIIZa7wIr961 Ta SZW exTfUPKPDrN2J /Eq P9pI0 r0rFBub1HOvpTAFbrGK5mQhkA6pBKmbreoPUin FbTjZ3I6oxs1tG3k7ns5lcH9s3AufbiTHA6SbNu/C86P kncTz6/7py uaZ2wjotFfdanQ g /latexit latexit t ( u 1 , . . . , u t latexitsha1 base64 "Vs 14vGXEYCWQa4/aBIirWhHyZg " e/N2Z8ZpIYXFMPzlB1euXrt Y nuCgTHCPGjGdYUxpa2OHOGws5k79OKrJPsn zs JI4dMSJYb92kkS/TfioopaxcqdcpmMnada8D/ceMSs dxJXRRImh eVFWSoI5aZZDJsIAR7lwCeNGuLcSPmOGcXQrXLll6V0AX p9L/KeeQfeO /YG3nc3/Yj/4X/MvgSfAu Bz8upYHf1tz3ViL4 Rs0RP43 /latexit sha1 base64 "Vs 14vGXEYCWQa4/aBIirWhHyZg " e/N2Z8ZpIYXFMPzlB1euXrt Y nuCgTHCPGjGdYUxpa2OHOGws5k79OKrJPsn zs JI4dMSJYb92kkS/TfioopaxcqdcpmMnada8D/ceMSs dxJXRRImh eVFWSoI5aZZDJsIAR7lwCeNGuLcSPmOGcXQrXLll6V0AX p9L/KeeQfeO /YG3nc3/Yj/4X/MvgSfAu Bz8upYHf1tz3ViL4 Rs0RP43 /latexit latexitsha1 base64 "Vs 14vGXEYCWQa4/aBIirWhHyZg " e/N2Z8ZpIYXFMPzlB1euXrt Y nuCgTHCPGjGdYUxpa2OHOGws5k79OKrJPsn zs JI4dMSJYb92kkS/TfioopaxcqdcpmMnada8D/ceMSs dxJXRRImh eVFWSoI5aZZDJsIAR7lwCeNGuLcSPmOGcXQrXLll6V0AX p9L/KeeQfeO /YG3nc3/Yj/4X/MvgSfAu Bz8upYHf1tz3ViL4 Rs0RP43 /latexit latexitsha1 base64 "Vs 14vGXEYCWQa4/aBIirWhHyZg " e/N2Z8ZpIYXFMPzlB1euXrt Y nuCgTHCPGjGdYUxpa2OHOGws5k79OKrJPsn zs JI4dMSJYb92kkS/TfioopaxcqdcpmMnada8D/ceMSs dxJXRRImh eVFWSoI5aZZDJsIAR7lwCeNGuLcSPmOGcXQrXLll6V0AX p9L/KeeQfeO /YG3nc3/Yj/4X/MvgSfAu Bz8upYHf1tz3ViL4 Rs0RP43 /latexit latexit latexitOptimal controlminimizes.t.EeC t (x t , u t )xt 1 ft (xt , ut , et )ut t ( t )hPTt 11 , x0 , . . . , xt )ixuxteCt is the cost. If you maximize, it’s called a reward.et is a noise processft is the state-transition functionis an observed trajectory

sha1 base64 " ojsm2yurvuosb1Y5f0dyh5w9EA " WmlyHIa/OsGVq9eu39i5uXvr9p2797p797 pMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL BuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI 9kwmJ6X1z7Bcsv9GNFA4Ny9S7 9ay89gnBxQPuU/qk/byvvvKCd2zwb YubplrM/SLS5/m0wfNF/1Y8 TSNeisYh6INQuS31Rr01Q /latexit sha1 base64 " ojsm2yurvuosb1Y5f0dyh5w9EA " WmlyHIa/OsGVq9eu39i5uXvr9p2797p797 pMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL BuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI 9kwmJ6X1z7Bcsv9GNFA4Ny9S7 9ay89gnBxQPuU/qk/byvvvKCd2zwb YubplrM/SLS5/m0wfNF/1Y8 TSNeisYh6INQuS31Rr01Q /latexit latexitsha1 base64 " ojsm2yurvuosb1Y5f0dyh5w9EA " WmlyHIa/OsGVq9eu39i5uXvr9p2797p797 pMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL BuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI 9kwmJ6X1z7Bcsv9GNFA4Ny9S7 9ay89gnBxQPuU/qk/byvvvKCd2zwb YubplrM/SLS5/m0wfNF/1Y8 TSNeisYh6INQuS31Rr01Q /latexit latexitsha1 base64 " ojsm2yurvuosb1Y5f0dyh5w9EA " WmlyHIa/OsGVq9eu39i5uXvr9p2797p797 pMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL BuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI 9kwmJ6X1z7Bcsv9GNFA4Ny9S7 9ay89gnBxQPuU/qk/byvvvKCd2zwb YubplrM/SLS5/m0wfNF/1Y8 TSNeisYh6INQuS31Rr01Q /latexit latexitsha1 base64 "lJVoaMm0ZXUFsbpjH2gMNfJ3dfs " zr03L5T0FMd/GtGdjbv37m8 9r/eI7Oi tOadpgamGkZFDKYAClW03 ZIk6znUbaH1hRajQkFHjfS Egyer610F3v33YTs7eNDsfFpfZZC/YS9ZiCXvLOuyYnbIuE wH 8l sd/RbtSJPkXHN9aosch5zpYiOvsLgB7OUw /latexit sha1 base64 "lJVoaMm0ZXUFsbpjH2gMNfJ3dfs " zr03L5T0FMd/GtGdjbv37m8 9r/eI7Oi tOadpgamGkZFDKYAClW03 ZIk6znUbaH1hRajQkFHjfS Egyer610F3v33YTs7eNDsfFpfZZC/YS9ZiCXvLOuyYnbIuE wH 8l sd/RbtSJPkXHN9aosch5zpYiOvsLgB7OUw /latexit latexitsha1 base64 "lJVoaMm0ZXUFsbpjH2gMNfJ3dfs " zr03L5T0FMd/GtGdjbv37m8 9r/eI7Oi tOadpgamGkZFDKYAClW03 ZIk6znUbaH1hRajQkFHjfS Egyer610F3v33YTs7eNDsfFpfZZC/YS9ZiCXvLOuyYnbIuE wH 8l sd/RbtSJPkXHN9aosch5zpYiOvsLgB7OUw /latexit latexitsha1 base64 "lJVoaMm0ZXUFsbpjH2gMNfJ3dfs " zr03L5T0FMd/GtGdjbv37m8 9r/eI7Oi tOadpgamGkZFDKYAClW03 ZIk6znUbaH1hRajQkFHjfS Egyer610F3v33YTs7eNDsfFpfZZC/YS9ZiCXvLOuyYnbIuE wH 8l sd/RbtSJPkXHN9aosch5zpYiOvsLgB7OUw /latexit latexit latexitminimize latexitsha1 base64 "oI5Ov9KcOeHyn9bWcwwJat8Txu4 " GNM5BuQnX1TaeYquqm6rYkNP0BrsStn XOP3FpdRJlknih4HDOuY GPTUbN7jPWvqKJhkU0jSZ5mjlrI0YfUIZTZIoXgIw T xy2rpPo02k zCGVruSQeouTTdr5p3Uin6kRtHT2Qxxb qL9vJe29lIdE9O/HnNk 3zP4gbHP922D4fPBywD686B /Xl1mhzwkj8geYeSQHJP35JQMiSA/ye8gCMJQhF/Db H3pTUMVjkPyFqEP/4A9obmoA /latexit sha1 base64 "oI5Ov9KcOeHyn9bWcwwJat8Txu4 " GNM5BuQnX1TaeYquqm6rYkNP0BrsStn XOP3FpdRJlknih4HDOuY GPTUbN7jPWvqKJhkU0jSZ5mjlrI0YfUIZTZIoXgIw T xy2rpPo02k zCGVruSQeouTTdr5p3Uin6kRtHT2Qxxb qL9vJe29lIdE9O/HnNk 3zP4gbHP922D4fPBywD686B /Xl1mhzwkj8geYeSQHJP35JQMiSA/ye8gCMJQhF/Db H3pTUMVjkPyFqEP/4A9obmoA /latexit latexitsha1 base64 "oI5Ov9KcOeHyn9bWcwwJat8Txu4 " GNM5BuQnX1TaeYquqm6rYkNP0BrsStn XOP3FpdRJlknih4HDOuY GPTUbN7jPWvqKJhkU0jSZ5mjlrI0YfUIZTZIoXgIw T xy2rpPo02k zCGVruSQeouTTdr5p3Uin6kRtHT2Qxxb qL9vJe29lIdE9O/HnNk 3zP4gbHP922D4fPBywD686B /Xl1mhzwkj8geYeSQHJP35JQMiSA/ye8gCMJQhF/Db H3pTUMVjkPyFqEP/4A9obmoA /latexit latexitsha1 base64 "oI5Ov9KcOeHyn9bWcwwJat8Txu4 " GNM5BuQnX1TaeYquqm6rYkNP0BrsStn XOP3FpdRJlknih4HDOuY GPTUbN7jPWvqKJhkU0jSZ5mjlrI0YfUIZTZIoXgIw T xy2rpPo02k zCGVruSQeouTTdr5p3Uin6kRtHT2Qxxb qL9vJe29lIdE9O/HnNk 3zP4gbHP922D4fPBywD686B /Xl1mhzwkj8geYeSQHJP35JQMiSA/ye8gCMJQhF/Db H3pTUMVjkPyFqEP/4A9obmoA /latexit latexit latexitsha1 base64 "mr94ezQtH17vzwJopx3THjSdtck " WNLSRLmD2ZJEPPZTlntiQu SW 6o/y33g2jWASBwY vm/uU5SaAqfp71Zy5 fSWn4BG 2bvVZ3bFc/FCHIpMvBZn4pO4ED2hRCV iJ/iV7KTHCXHycltaNJa5TwTa5ac/gFfIcb8 /latexit sha1 base64 "mr94ezQtH17vzwJopx3THjSdtck " WNLSRLmD2ZJEPPZTlntiQu SW 6o/y33g2jWASBwY vm/uU5SaAqfp71Zy5 fSWn4BG 2bvVZ3bFc/FCHIpMvBZn4pO4ED2hRCV iJ/iV7KTHCXHycltaNJa5TwTa5ac/gFfIcb8 /latexit latexitsha1 base64 "mr94ezQtH17vzwJopx3THjSdtck " WNLSRLmD2ZJEPPZTlntiQu SW 6o/y33g2jWASBwY vm/uU5SaAqfp71Zy5 fSWn4BG 2bvVZ3bFc/FCHIpMvBZn4pO4ED2hRCV iJ/iV7KTHCXHycltaNJa5TwTa5ac/gFfIcb8 /latexit latexitsha1 base64 "mr94ezQtH17vzwJopx3THjSdtck " WNLSRLmD2ZJEPPZTlntiQu SW 6o/y33g2jWASBwY vm/uU5SaAqfp71Zy5 fSWn4BG 2bvVZ3bFc/FCHIpMvBZn4pO4ED2hRCV iJ/iV7KTHCXHycltaNJa5TwTa5ac/gFfIcb8 /latexit latexit latexitNewton’sLawssubject to xt 1TXt 01 0 ztxt vt 10xt ut11 /m zt 1 zt vtvt 1 vt atm at ut11 (xt )1