title
Reinforcement Learning: Machine Learning Meets Control Theory

description
Reinforcement learning is a powerful technique at the intersection of machine learning and control theory, and it is inspired by how biological systems learn to interact with their environment. In this video, we provide a high level overview of reinforcement learning, along with leading algorithms and impressive applications. Citable link for this video: https://doi.org/10.52843/cassyni.x2t0sp @eigensteve on Twitter eigensteve.com databookuw.com %%% CHAPTERS %%% 0:00 Introduction 3:34 Reinforcement Learning Overview 7:30 Mathematics of Reinforcement Learning 12:32 Markov Decision Process 13:33 Credit Assignment Problem 15:38 Optimization Techniques for RL 18:54 Examples of Reinforcement Learning 21:50 Q-Learning 23:53 Hindsight Replay

detail
{'title': 'Reinforcement Learning: Machine Learning Meets Control Theory', 'heatmap': [{'end': 476.586, 'start': 450.379, 'weight': 0.728}, {'end': 657.218, 'start': 591.965, 'weight': 0.749}, {'end': 774.555, 'start': 747.447, 'weight': 0.767}, {'end': 1330.925, 'start': 1312.535, 'weight': 0.728}], 'summary': 'Delves into reinforcement learning as a framework for learning control strategies, emphasizing deep reinforcement learning techniques and exploring applications in complex systems like a bipedal walker. it also covers the basics of reinforcement learning, its application in games and chess, concepts, and its integration with machine learning and robotics, showcasing examples of robots learning through imitation and trial and error.', 'chapters': [{'end': 214.789, 'segs': [{'end': 88.941, 'src': 'embed', 'start': 25.9, 'weight': 0, 'content': [{'end': 35.162, 'text': 'So reinforcement learning is essentially a branch of machine learning that deals with how to learn control strategies to interact with a complex environment.', 'start': 25.9, 'duration': 9.262}, {'end': 40.143, 'text': "And so one of the ways I think about this, the way I'm going to define this,", 'start': 36.783, 'duration': 3.36}, {'end': 45.565, 'text': 'is that reinforcement learning is a framework for learning how to interact with the environment from experience.', 'start': 40.143, 'duration': 5.422}, {'end': 50.107, 'text': 'This is a very biologically inspired idea.', 'start': 46.465, 'duration': 3.642}, {'end': 51.388, 'text': 'This is what animals do.', 'start': 50.127, 'duration': 1.261}, {'end': 59.814, 'text': 'So through trial and error, through experience, through positive and negative rewards and feedback, they learn how to interact with their environment.', 'start': 51.508, 'duration': 8.306}, {'end': 60.734, 'text': 'Okay, good.', 'start': 60.274, 'duration': 0.46}, {'end': 64.456, 'text': 'So before I jump in, I want to show some motivating videos.', 'start': 60.834, 'duration': 3.622}, {'end': 71.141, 'text': 'I really like this one where reinforcement learning is used to learn how to walk in this artificial environment.', 'start': 64.557, 'duration': 6.584}, {'end': 73.403, 'text': "And there's a lot of papers like this,", 'start': 71.681, 'duration': 1.722}, {'end': 82.314, 'text': 'where people use reinforcement learning as kind of an optimization framework to learn how to control a complex system, in this case a bipedal walker,', 'start': 73.403, 'duration': 8.911}, {'end': 83.895, 'text': 'often in a simulated environment.', 'start': 82.314, 'duration': 1.581}, {'end': 85.117, 'text': 'And this just looks really cool.', 'start': 83.915, 'duration': 1.202}, {'end': 86.859, 'text': "And it's a difficult control problem.", 'start': 85.277, 'duration': 1.582}, {'end': 88.941, 'text': 'This is a really hard nonlinear control problem.', 'start': 86.879, 'duration': 2.062}], 'summary': 'Reinforcement learning involves learning control strategies through trial and error, as seen in videos learning to walk in an artificial environment.', 'duration': 63.041, 'max_score': 25.9, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to25900.jpg'}, {'end': 214.789, 'src': 'embed', 'start': 177.108, 'weight': 1, 'content': [{'end': 178.488, 'text': 'And then, in a future video,', 'start': 177.108, 'duration': 1.38}, {'end': 189.213, 'text': "I'm going to talk about kind of deep reinforcement learning or reinforcement learning with modern techniques and deep neural networks and some of the incredible applications and performance that you can get out of those systems.", 'start': 178.488, 'duration': 10.725}, {'end': 196.757, 'text': "Good. So also, I'll point out, you can follow updates on these videos at eigen Steve, on Twitter.", 'start': 189.713, 'duration': 7.044}, {'end': 201.56, 'text': 'Please like, please subscribe, hit the bell, so you get notifications and comment below.', 'start': 196.757, 'duration': 4.803}, {'end': 203.001, 'text': 'Tell me what you want to see more of.', 'start': 201.56, 'duration': 1.441}, {'end': 205.283, 'text': "tell me what you like or don't like.", 'start': 203.001, 'duration': 2.282}, {'end': 211.727, 'text': 'oftentimes, people in the comments Provide a lot of really important, useful information that I might have left out of these videos.', 'start': 205.283, 'duration': 6.444}, {'end': 214.789, 'text': "So I think it's also a big service to other people watching these.", 'start': 211.727, 'duration': 3.062}], 'summary': "Future video will discuss deep reinforcement learning with modern techniques and deep neural networks, with updates available on eigen steve's twitter.", 'duration': 37.681, 'max_score': 177.108, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to177108.jpg'}], 'start': 7.52, 'title': 'Reinforcement learning framework', 'summary': 'Explores reinforcement learning as a framework to learn control strategies from experience, emphasizing positive reinforcement, disentanglement of learning to interact with the environment, and deep reinforcement learning techniques. it also provides examples of using reinforcement learning to control complex systems such as a bipedal walker in a simulated environment.', 'chapters': [{'end': 107.762, 'start': 7.52, 'title': 'Reinforcement learning: interacting with complex environments', 'summary': 'Discusses reinforcement learning as a framework to learn control strategies from experience, inspired by how animals learn to interact with the environment through trial and error, positive and negative rewards, and feedback, with examples of using reinforcement learning to control a complex system like a bipedal walker in a simulated environment.', 'duration': 100.242, 'highlights': ['Reinforcement learning is a framework for learning how to interact with the environment from experience, inspired by how animals learn through trial and error, positive and negative rewards, and feedback.', 'Reinforcement learning is used to learn how to control a complex system, such as a bipedal walker, in a simulated environment, demonstrating its application as an optimization framework.', 'Reinforcement learning aims to enable better robots and physical agents to interact with the world and learn how to learn, similar to humans and animals.']}, {'end': 214.789, 'start': 107.762, 'title': 'Reinforcement learning: framework and optimization', 'summary': "Discusses reinforcement learning and its framework, emphasizing positive reinforcement and the disentanglement of learning to interact with the environment and optimizing agent's actions, with a glimpse into deep reinforcement learning using modern techniques and deep neural networks.", 'duration': 107.027, 'highlights': ['The chapter emphasizes the concept of reinforcement learning and its framework, illustrating the use of positive reinforcement through training animals, with the example of teaching a dog to hold a treat on its nose until given permission to eat it.', "It highlights the disentanglement of the reinforcement learning framework for learning to interact with the environment and the optimization problem for optimizing the agent's actions or policies within that framework.", 'The speaker also mentions the future discussion of deep reinforcement learning with modern techniques and deep neural networks, emphasizing the incredible applications and performance achievable through these systems.', 'The chapter concludes with a call to action for audience engagement, encouraging viewers to follow updates on Twitter, like, subscribe, and provide feedback for future content improvement.']}], 'duration': 207.269, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to7520.jpg', 'highlights': ['Reinforcement learning aims to enable better robots and physical agents to interact with the world and learn how to learn, similar to humans and animals.', 'The chapter concludes with a call to action for audience engagement, encouraging viewers to follow updates on Twitter, like, subscribe, and provide feedback for future content improvement.', 'The speaker also mentions the future discussion of deep reinforcement learning with modern techniques and deep neural networks, emphasizing the incredible applications and performance achievable through these systems.', 'Reinforcement learning is used to learn how to control a complex system, such as a bipedal walker, in a simulated environment, demonstrating its application as an optimization framework.', 'The chapter emphasizes the concept of reinforcement learning and its framework, illustrating the use of positive reinforcement through training animals, with the example of teaching a dog to hold a treat on its nose until given permission to eat it.']}, {'end': 380.37, 'segs': [{'end': 260.476, 'src': 'embed', 'start': 233.279, 'weight': 1, 'content': [{'end': 238.162, 'text': "So in the first example, and I'm going to have a few examples, we're going to talk about a mouse in a maze.", 'start': 233.279, 'duration': 4.883}, {'end': 240.123, 'text': 'So the agent is a mouse.', 'start': 238.782, 'duration': 1.341}, {'end': 241.404, 'text': 'The environment is a maze.', 'start': 240.343, 'duration': 1.061}, {'end': 247.728, 'text': 'The mouse gets to measure its current state in the environment, so it measures its state S.', 'start': 242.444, 'duration': 5.284}, {'end': 250.49, 'text': "Notice that it doesn't measure the full state.", 'start': 247.728, 'duration': 2.762}, {'end': 253.232, 'text': 'The mouse does not have a top-down view of the whole maze.', 'start': 250.61, 'duration': 2.622}, {'end': 256.074, 'text': 'It just knows where it is right now and where it was in the past.', 'start': 253.312, 'duration': 2.762}, {'end': 260.476, 'text': 'And then the mouse gets to take some action A.', 'start': 257.053, 'duration': 3.423}], 'summary': 'Mouse in maze measures state s and takes action a.', 'duration': 27.197, 'max_score': 233.279, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to233279.jpg'}, {'end': 391.877, 'src': 'embed', 'start': 360.168, 'weight': 0, 'content': [{'end': 363.95, 'text': "but it's not nearly as much information as in classical supervised learning.", 'start': 360.168, 'duration': 3.782}, {'end': 374.863, 'text': "And that's one of the major challenges of reinforcement learning is that these labels are extremely rare and it's very hard to tell what actions gave rise to actually getting that reward.", 'start': 364.77, 'duration': 10.093}, {'end': 380.37, 'text': 'So this is a much harder optimization problem and oftentimes requires much more data and much more trial and error.', 'start': 374.883, 'duration': 5.487}, {'end': 381.372, 'text': "And I'm going to talk about that.", 'start': 380.47, 'duration': 0.902}, {'end': 391.877, 'text': 'Good. I also like to think about the game of chess or checkers or tic-tac-toe, basically games in general, where the agent,', 'start': 382.654, 'duration': 9.223}], 'summary': 'Reinforcement learning faces challenges due to rare labels, requiring more data and trial and error.', 'duration': 31.709, 'max_score': 360.168, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to360168.jpg'}], 'start': 215.169, 'title': 'Reinforcement learning basics', 'summary': 'Introduces the concept of reinforcement learning through the example of a mouse in a maze, emphasizing the interaction between the agent and the environment, sparse rewards, and the challenges of semi-supervised learning.', 'chapters': [{'end': 380.37, 'start': 215.169, 'title': 'Reinforcement learning basics', 'summary': 'Introduces the concept of reinforcement learning using the example of a mouse in a maze, highlighting the interaction between the agent and the environment, sparse rewards, and the challenges of semi-supervised learning.', 'duration': 165.201, 'highlights': ['The agent interacts with the environment by measuring its current state, taking actions, and receiving sparse rewards, such as a piece of cheese at the end of the maze.', 'Reinforcement learning involves semi-supervised learning, with time-delayed rewards, making it a harder optimization problem than classical supervised learning.', 'The environment provides sparse supervisory feedback, making it challenging to determine which actions lead to obtaining rewards.']}], 'duration': 165.201, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to215169.jpg', 'highlights': ['Reinforcement learning involves semi-supervised learning, with time-delayed rewards, making it a harder optimization problem than classical supervised learning.', 'The agent interacts with the environment by measuring its current state, taking actions, and receiving sparse rewards, such as a piece of cheese at the end of the maze.', 'The environment provides sparse supervisory feedback, making it challenging to determine which actions lead to obtaining rewards.']}, {'end': 678.454, 'segs': [{'end': 483.29, 'src': 'heatmap', 'start': 450.379, 'weight': 0, 'content': [{'end': 459.602, 'text': 'At the end of the day, the big challenge in reinforcement learning is to design a policy of what actions to take, given a state S,', 'start': 450.379, 'duration': 9.223}, {'end': 461.942, 'text': 'to maximize my chance of getting a future reward.', 'start': 459.602, 'duration': 2.34}, {'end': 465.363, 'text': "That's all that this agent can do is decide on a policy.", 'start': 462.182, 'duration': 3.181}, {'end': 476.586, 'text': "Now this is called a policy and not a control law, for a lot of reasons, partly because the environment is not deterministic, it's probabilistic,", 'start': 465.883, 'duration': 10.703}, {'end': 478.747, 'text': 'and so this policy is also gonna be probabilistic.', 'start': 476.586, 'duration': 2.161}, {'end': 483.29, 'text': 'Okay, so my policy pi, given a state and an action.', 'start': 479.207, 'duration': 4.083}], 'summary': 'Reinforcement learning aims to maximize future reward through a probabilistic policy design.', 'duration': 102.82, 'max_score': 450.379, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to450379.jpg'}, {'end': 657.218, 'src': 'heatmap', 'start': 591.965, 'weight': 0.749, 'content': [{'end': 595.669, 'text': 'Maybe I played a great game of chess and I made one mistake and I lose the game.', 'start': 591.965, 'duration': 3.704}, {'end': 599.232, 'text': 'Do I throw away that whole sequence of actions?', 'start': 596.349, 'duration': 2.883}, {'end': 602.234, 'text': 'How do you figure out what actions were good and what actions were bad?', 'start': 599.592, 'duration': 2.642}, {'end': 607.239, 'text': "That's a very, very hard optimization problem and that's at the absolute heart of reinforcement learning.", 'start': 602.274, 'duration': 4.965}, {'end': 616.829, 'text': 'Okay, so part of helping design a good policy is understanding what is the value of being in a certain state S given that policy pi.', 'start': 608.502, 'duration': 8.327}, {'end': 626.177, 'text': 'So, once I choose a policy, I can start to learn what is the value of each state, of the system, of each board position in chess, for example,', 'start': 617.37, 'duration': 8.807}, {'end': 633.203, 'text': 'based on what is the expected reward I will get in the future if I start at that state and I enact that policy.', 'start': 626.177, 'duration': 7.026}, {'end': 634.805, 'text': "I'm gonna say that again, that's a mouthful.", 'start': 633.564, 'duration': 1.241}, {'end': 647.03, 'text': "So the value of a state S given a policy pi is my expectation of how much reward I'll get in the future if I start in that state and I enact that policy.", 'start': 635.445, 'duration': 11.585}, {'end': 651.052, 'text': "And there's this gamma to the T, which is a discount rate.", 'start': 647.791, 'duration': 3.261}, {'end': 657.218, 'text': 'And so what this is saying is that I am slightly discounting my future rewards compared to my immediate rewards.', 'start': 651.212, 'duration': 6.006}], 'summary': 'Reinforcement learning involves optimizing actions and evaluating state values to maximize future rewards, using a discount rate.', 'duration': 65.253, 'max_score': 591.965, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to591965.jpg'}, {'end': 686.819, 'src': 'embed', 'start': 657.719, 'weight': 4, 'content': [{'end': 666.628, 'text': 'So gamma is a constant between zero and one that basically tells you how much you favor getting a reward right now versus far in the future.', 'start': 657.719, 'duration': 8.909}, {'end': 672.871, 'text': 'And this is intimately related to economic theory, psychology that generally,', 'start': 667.128, 'duration': 5.743}, {'end': 678.454, 'text': 'people are more eager to get a reward now than wait for a delayed reward much later.', 'start': 672.871, 'duration': 5.583}, {'end': 686.819, 'text': 'But the basic idea is that you can start to understand this policy and what policies are good or bad, based on what are good board positions,', 'start': 679.355, 'duration': 7.464}], 'summary': 'Gamma, a constant between 0 and 1, represents preference for immediate vs. delayed rewards, aiding policy evaluation.', 'duration': 29.1, 'max_score': 657.719, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to657719.jpg'}], 'start': 380.47, 'title': 'Reinforcement learning in games and chess', 'summary': 'Discusses reinforcement learning in games like chess, highlighting the interaction between the agent and the environment, and emphasizes the challenges and strategies in reinforcement learning using chess as an example, including the importance of designing a probabilistic policy and the impact of discount rate gamma on future rewards.', 'chapters': [{'end': 437.418, 'start': 380.47, 'title': 'Reinforcement learning in games', 'summary': 'Discusses the concept of reinforcement learning using examples from games such as chess, highlighting the interaction between the agent and the environment as well as the adversarial nature of some games.', 'duration': 56.948, 'highlights': ['Reinforcement learning involves the interaction between an agent and an environment with a finite set of actions, exemplified by games like chess, checkers, and tic-tac-toe.', 'In games like chess, the environment includes an adversarial opponent, adding complexity to the interaction as the agent attempts to beat the opponent while being pursued in return.', 'The concept of reinforcement learning is humorously related to the character Neo in the Matrix, highlighting the idea of an agent learning the rules of its environment.']}, {'end': 678.454, 'start': 438.499, 'title': 'Reinforcement learning in chess', 'summary': 'Explains the challenges of reinforcement learning using the game of chess as an example, emphasizing the importance of designing a probabilistic policy to maximize future rewards, measuring the value of being in a certain state, and the impact of discount rate gamma on future rewards.', 'duration': 239.955, 'highlights': ['The big challenge in reinforcement learning is to design a policy of what actions to take, given a state S, to maximize the chance of getting a future reward. Emphasizes the primary challenge in reinforcement learning - designing a policy to maximize future rewards based on the current state.', 'The policy is probabilistic, as the environment is not deterministic, and it tells the probability of taking action A given the current state S. Explains the probabilistic nature of the policy due to the probabilistic environment, emphasizing the importance of understanding the probability of taking an action in a given state.', 'The discount rate gamma determines how much future rewards are favored compared to immediate rewards, with a constant value between zero and one. Discusses the significance of the discount rate gamma in determining the preference for immediate rewards versus future rewards, highlighting its range and impact.']}], 'duration': 297.984, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to380470.jpg', 'highlights': ['The big challenge in reinforcement learning is to design a policy of what actions to take, given a state S, to maximize the chance of getting a future reward.', 'In games like chess, the environment includes an adversarial opponent, adding complexity to the interaction as the agent attempts to beat the opponent while being pursued in return.', 'The policy is probabilistic, as the environment is not deterministic, and it tells the probability of taking action A given the current state S.', 'Reinforcement learning involves the interaction between an agent and an environment with a finite set of actions, exemplified by games like chess, checkers, and tic-tac-toe.', 'The discount rate gamma determines how much future rewards are favored compared to immediate rewards, with a constant value between zero and one.', 'The concept of reinforcement learning is humorously related to the character Neo in the Matrix, highlighting the idea of an agent learning the rules of its environment.']}, {'end': 1039.326, 'segs': [{'end': 705.599, 'src': 'embed', 'start': 679.355, 'weight': 0, 'content': [{'end': 686.819, 'text': 'But the basic idea is that you can start to understand this policy and what policies are good or bad, based on what are good board positions,', 'start': 679.355, 'duration': 7.464}, {'end': 688.039, 'text': 'what are good value functions.', 'start': 686.819, 'duration': 1.22}, {'end': 690.281, 'text': 'And this kind of is how a human would play.', 'start': 688.58, 'duration': 1.701}, {'end': 698.651, 'text': 'is that you might, so the set of all states of a chess board is combinatorially large.', 'start': 691.161, 'duration': 7.49}, {'end': 699.732, 'text': "There's too many to count.", 'start': 698.671, 'duration': 1.061}, {'end': 701.314, 'text': 'You could never hold them all in your mind.', 'start': 699.832, 'duration': 1.482}, {'end': 705.599, 'text': 'But we start creating rules of thumb of what are good board positions.', 'start': 702.035, 'duration': 3.564}], 'summary': 'Understanding and evaluating policies based on good board positions and value functions, similar to how a human would play.', 'duration': 26.244, 'max_score': 679.355, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to679355.jpg'}, {'end': 774.555, 'src': 'heatmap', 'start': 747.447, 'weight': 0.767, 'content': [{'end': 750.409, 'text': "So at the end of the day, it's an optimization problem to solve for pi.", 'start': 747.447, 'duration': 2.962}, {'end': 759.712, 'text': 'So usually we think of our environment as not being fully deterministic like we do in classical mechanics and classical control systems often.', 'start': 752.35, 'duration': 7.362}, {'end': 764.993, 'text': "And instead we think of our environment as being somehow there's a random or a stochastic component.", 'start': 759.732, 'duration': 5.261}, {'end': 768.534, 'text': 'So these are called Markov decision processes, MDPs.', 'start': 765.113, 'duration': 3.421}, {'end': 774.555, 'text': 'And what that means is that if we are in a state S now and I take an action A now,', 'start': 768.854, 'duration': 5.701}], 'summary': 'Optimization problem to solve for pi in markov decision processes (mdps).', 'duration': 27.108, 'max_score': 747.447, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to747447.jpg'}, {'end': 852.239, 'src': 'embed', 'start': 827.607, 'weight': 1, 'content': [{'end': 838.033, 'text': "This issue was recognized as early as the 1960s by Minsky, It's the central challenge in reinforcement learning, and it has been for six decades.", 'start': 827.607, 'duration': 10.426}, {'end': 844.936, 'text': 'This is the problem that people are still working on today, is how to beat the credit assignment problem.', 'start': 838.133, 'duration': 6.803}, {'end': 849.818, 'text': 'A couple of keywords I think are important are dense versus sparse rewards.', 'start': 844.956, 'duration': 4.862}, {'end': 852.239, 'text': 'Again, the game of chess has very sparse rewards.', 'start': 850.238, 'duration': 2.001}], 'summary': 'Reinforcement learning faces credit assignment problem for 60 years, focusing on dense versus sparse rewards.', 'duration': 24.632, 'max_score': 827.607, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to827607.jpg'}, {'end': 929.08, 'src': 'embed', 'start': 896.395, 'weight': 2, 'content': [{'end': 901.64, 'text': "I'd have to have tons of examples to learn a good optimal policy given those sparse rewards.", 'start': 896.395, 'duration': 5.245}, {'end': 908.185, 'text': 'So sparse rewards and the credit assignment problem make it very hard to learn through optimization what the right policy is.', 'start': 902.2, 'duration': 5.985}, {'end': 910.708, 'text': "And that's related to sample efficiency.", 'start': 908.205, 'duration': 2.503}, {'end': 920.754, 'text': 'So, in general, what we do in a lot of systems is called reward shaping, where, even if you get an infrequent reward,', 'start': 911.548, 'duration': 9.206}, {'end': 929.08, 'text': 'an expert human might build a proxy reward so that you get more dense intermediate rewards on the way to this final reward.', 'start': 920.754, 'duration': 8.326}], 'summary': 'Sparse rewards and credit assignment make learning difficult. reward shaping increases intermediate rewards.', 'duration': 32.685, 'max_score': 896.395, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to896395.jpg'}], 'start': 679.355, 'title': 'Reinforcement learning concepts', 'summary': 'Explores learning policy through board positions and reinforcement learning overview, emphasizing the challenges of sparse rewards and optimization strategies like differential programming, monte carlo, and temporal difference balancing exploration and exploitation.', 'chapters': [{'end': 720.249, 'start': 679.355, 'title': 'Learning policy through board positions', 'summary': 'Explores the concept of learning policy through understanding good board positions and value functions, akin to how a human makes decisions, in the context of a combinatorially large set of states in a chess board and creating rules of thumb, such as counting points, to evaluate the value of a given state.', 'duration': 40.894, 'highlights': ['Understanding policies based on good board positions and value functions is akin to how a human plays chess, such as evaluating the expected chance of winning by counting points on the board.', 'Creating rules of thumb, like counting points on the board, helps in evaluating the value of a given state.', 'The set of all states of a chess board is combinatorially large, making it impossible to hold all of them in mind.']}, {'end': 874.926, 'start': 720.269, 'title': 'Reinforcement learning overview', 'summary': 'Discusses the optimization problem in reinforcement learning, focusing on the credit assignment problem, markov decision processes, and the challenge of sparse rewards, with a mention of the central challenge recognized since the 1960s.', 'duration': 154.657, 'highlights': ['The central challenge in reinforcement learning is the credit assignment problem, recognized since the 1960s and still being worked on today. This highlights the long-standing challenge in reinforcement learning and the ongoing efforts to address it.', 'Markov decision processes involve transitioning from a current state to a new state based on the probability of taking an action, posing challenges in optimizing policies. This explains the concept of Markov decision processes and the difficulty in optimizing policies due to their probabilistic nature.', 'Sparse rewards in games like chess make it challenging to determine the effectiveness of action sequences, in contrast to denser rewards that provide more concrete feedback. This emphasizes the difference between sparse and dense rewards and their impact on the effectiveness of action sequences.']}, {'end': 1039.326, 'start': 874.926, 'title': 'Optimizing reinforcement learning policies', 'summary': 'Discusses the challenges of sparse rewards in reinforcement learning, emphasizing the need for reward shaping and exploring optimization strategies like differential programming, monte carlo, and temporal difference balancing exploration and exploitation.', 'duration': 164.4, 'highlights': ['The challenge of sparse rewards in reinforcement learning makes it sample inefficient, requiring many examples to learn a good optimal policy.', 'Reward shaping involves an expert human providing more dense intermediate rewards to guide the learning process.', 'Reinforcement learning is an optimization problem, with strategies including differential programming, Monte Carlo, and temporal difference, which finds the balance between exploration and exploitation.']}], 'duration': 359.971, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to679355.jpg', 'highlights': ['Understanding policies based on good board positions and value functions is akin to how a human plays chess, such as evaluating the expected chance of winning by counting points on the board.', 'The central challenge in reinforcement learning is the credit assignment problem, recognized since the 1960s and still being worked on today. This highlights the long-standing challenge in reinforcement learning and the ongoing efforts to address it.', 'The challenge of sparse rewards in reinforcement learning makes it sample inefficient, requiring many examples to learn a good optimal policy.']}, {'end': 1316.956, 'segs': [{'end': 1121.995, 'src': 'embed', 'start': 1080.633, 'weight': 0, 'content': [{'end': 1086.168, 'text': 'A fundamental challenge in machine learning and control theory is this exploration-exploitation balance,', 'start': 1080.633, 'duration': 5.535}, {'end': 1088.595, 'text': "and it's a big problem in reinforcement learning also.", 'start': 1086.168, 'duration': 2.427}, {'end': 1095.596, 'text': 'Policy. iteration is basically you set up a dynamical system where, based on your rewards,', 'start': 1089.99, 'duration': 5.606}, {'end': 1103.503, 'text': 'you iteratively update the policy to make it better and better over time, based on new information, based on better information from new rewards.', 'start': 1095.596, 'duration': 7.907}, {'end': 1104.845, 'text': "That's policy iteration.", 'start': 1103.583, 'duration': 1.262}, {'end': 1107.367, 'text': 'There are lots of strategies to do this.', 'start': 1105.585, 'duration': 1.782}, {'end': 1109.89, 'text': "I'm just going to name a bunch of them.", 'start': 1107.808, 'duration': 2.082}, {'end': 1111.391, 'text': 'You can use simulated annealing.', 'start': 1109.93, 'duration': 1.461}, {'end': 1114.772, 'text': 'evolutionary optimization gradient descent,', 'start': 1111.711, 'duration': 3.061}, {'end': 1121.995, 'text': 'and you can use all of the modern tools in neural networks and machine learning stochastic gradient descent, atom optimization.', 'start': 1114.772, 'duration': 7.223}], 'summary': 'Balancing exploration-exploitation in reinforcement learning is addressed through policy iteration, utilizing strategies like simulated annealing, evolutionary optimization, and gradient descent.', 'duration': 41.362, 'max_score': 1080.633, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1080633.jpg'}, {'end': 1232.005, 'src': 'embed', 'start': 1189.591, 'weight': 2, 'content': [{'end': 1197.801, 'text': 'And finally, after 100 trials, this system has actually learned the rules of the game, the physics of how to get that ball in the cup.', 'start': 1189.591, 'duration': 8.21}, {'end': 1201.224, 'text': "Very simple robotic example, but it's also pretty interesting.", 'start': 1198.501, 'duration': 2.723}, {'end': 1208.813, 'text': 'I mean, this video is not that recent, but very interesting to show that it is possible to learn a real physical system.', 'start': 1201.244, 'duration': 7.569}, {'end': 1211.41, 'text': 'This is another example I love.', 'start': 1210.229, 'duration': 1.181}, {'end': 1212.731, 'text': 'This is called the Pilko learner.', 'start': 1211.45, 'duration': 1.281}, {'end': 1214.672, 'text': 'I encourage you to go read all about Pilko.', 'start': 1212.851, 'duration': 1.821}, {'end': 1221.117, 'text': "In this case, they're learning kind of how to swing up and stabilize a pendulum on a cart.", 'start': 1215.373, 'duration': 5.744}, {'end': 1232.005, 'text': 'And again, they are using some combination of trial and error and a physical model to learn how to do this very efficiently with very few samples.', 'start': 1221.938, 'duration': 10.067}], 'summary': 'After 100 trials, the system learns the physics of a robotic game efficiently with very few samples.', 'duration': 42.414, 'max_score': 1189.591, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1189591.jpg'}, {'end': 1285.755, 'src': 'embed', 'start': 1256.372, 'weight': 4, 'content': [{'end': 1261.877, 'text': 'to learn the random control signal that gets you near the upright position where you can start to stabilize it.', 'start': 1256.372, 'duration': 5.505}, {'end': 1264.799, 'text': "So a lot of trial and error if you don't have a model.", 'start': 1261.917, 'duration': 2.882}, {'end': 1269.983, 'text': 'So the PILCO learner, in some sense is leveraging the fact that there is physics.', 'start': 1265.279, 'duration': 4.704}, {'end': 1271.004, 'text': 'we do know physics.', 'start': 1269.983, 'duration': 1.021}, {'end': 1276.709, 'text': 'we do have models to learn this much, much faster, much more efficiently, many fewer samples.', 'start': 1271.004, 'duration': 5.705}, {'end': 1285.755, 'text': "And I forget which trial we're on, but after trial five or six or seven, it actually does learn how to get this thing up and stabilize.", 'start': 1277.629, 'duration': 8.126}], 'summary': 'Pilco learner leverages physics to learn control signal, achieving stabilization in 5-7 trials.', 'duration': 29.383, 'max_score': 1256.372, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1256372.jpg'}], 'start': 1039.986, 'title': 'Machine learning and robotics', 'summary': "Discusses the exploration-exploitation balance in policy iteration for reinforcement learning and examples of robots learning through imitation, trial and error, showcasing a robot's learning achievement after 100 trials and pilco learner's ability to stabilize a pendulum in just a few trials.", 'chapters': [{'end': 1121.995, 'start': 1039.986, 'title': 'Exploration-exploitation balance in machine learning', 'summary': 'Discusses the exploration-exploitation balance in policy iteration for reinforcement learning, emphasizing the challenge of deciding how much effort to allocate to optimizing existing strategies versus exploring new, potentially rewarding, but untested strategies.', 'duration': 82.009, 'highlights': ['Policy iteration involves iteratively updating the policy based on rewards to improve it over time, utilizing strategies such as simulated annealing, evolutionary optimization, gradient descent, and modern tools in neural networks and machine learning.', 'The fundamental challenge in machine learning and control theory is the exploration-exploitation balance, which involves deciding how much effort to allocate to exploring new strategies versus exploiting existing ones, posing a significant problem in reinforcement learning.']}, {'end': 1316.956, 'start': 1122.535, 'title': 'Learning with deep learning in robotics', 'summary': 'Discusses examples of robots learning through imitation, trial and error, and leveraging physical models, showcasing how a robot learned to catch a ball in a cup after 100 trials and how the pilco learner efficiently stabilized a pendulum on a cart in just a few trials.', 'duration': 194.421, 'highlights': ['Robotic example of learning to catch a ball in a cup after 100 trials, showcasing the use of visual information and trial and error learning.', "PILCO learner efficiently stabilizing a pendulum on a cart in just a few trials by leveraging physical models and Newton's laws, contrasting it with the inefficiency of trial and error without a model.", 'Discussion about the sample inefficiency of trial and error learning without a model, highlighting the importance of leveraging knowledge of physics for faster and more efficient learning.']}], 'duration': 276.97, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1039986.jpg', 'highlights': ['Policy iteration involves iteratively updating the policy based on rewards to improve it over time, utilizing strategies such as simulated annealing, evolutionary optimization, gradient descent, and modern tools in neural networks and machine learning.', 'The fundamental challenge in machine learning and control theory is the exploration-exploitation balance, which involves deciding how much effort to allocate to exploring new strategies versus exploiting existing ones, posing a significant problem in reinforcement learning.', 'Robotic example of learning to catch a ball in a cup after 100 trials, showcasing the use of visual information and trial and error learning.', "PILCO learner efficiently stabilizing a pendulum on a cart in just a few trials by leveraging physical models and Newton's laws, contrasting it with the inefficiency of trial and error without a model.", 'Discussion about the sample inefficiency of trial and error learning without a model, highlighting the importance of leveraging knowledge of physics for faster and more efficient learning.']}, {'end': 1560.684, 'segs': [{'end': 1360.487, 'src': 'embed', 'start': 1317.657, 'weight': 0, 'content': [{'end': 1324.959, 'text': 'So instead of just learning the policy and the value function separately, in queue learning, you can kind of learn them both at the same time.', 'start': 1317.657, 'duration': 7.302}, {'end': 1327.201, 'text': "So there's this Q function.", 'start': 1325.579, 'duration': 1.622}, {'end': 1330.925, 'text': "that's not just a function of the state S.", 'start': 1327.201, 'duration': 3.724}, {'end': 1337.511, 'text': "it's a function of the state and the action, and it tells you what is the quality of being in that state and taking that action.", 'start': 1330.925, 'duration': 6.586}, {'end': 1340.133, 'text': 'So it kind of combines the value and the policy.', 'start': 1337.551, 'duration': 2.582}, {'end': 1348.001, 'text': 'You can almost think of it as like a value function of the state and the action, assuming I do the smartest thing in the future,', 'start': 1341.375, 'duration': 6.626}, {'end': 1349.643, 'text': 'that I can the best thing in the future.', 'start': 1348.001, 'duration': 1.642}, {'end': 1352.963, 'text': "And so I'll walk you through what this could look like.", 'start': 1350.842, 'duration': 2.121}, {'end': 1360.487, 'text': 'So the way that you update this quality function is you take your old quality function, and then when you get a reward, you basically update.', 'start': 1353.043, 'duration': 7.444}], 'summary': 'In q-learning, the q function combines value and policy, updating with rewards.', 'duration': 42.83, 'max_score': 1317.657, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1317657.jpg'}, {'end': 1427.742, 'src': 'embed', 'start': 1402.967, 'weight': 2, 'content': [{'end': 1408.29, 'text': 'and this is really nice, because if you actually know the quality function, then once i find myself in a state s,', 'start': 1402.967, 'duration': 5.323}, {'end': 1413.334, 'text': 'i just have to look across all of the out, all of the actions a, and pick the one with the best quality.', 'start': 1408.29, 'duration': 5.044}, {'end': 1415.955, 'text': "so it's a really nice way of choosing an action.", 'start': 1413.334, 'duration': 2.621}, {'end': 1422.74, 'text': 'given this quality function, when i find myself in state s, i just pick the action that gives me the best quality and i enact that action.', 'start': 1415.955, 'duration': 6.785}, {'end': 1427.742, 'text': 'And if I do that in the future, I will maximize my value and that gives me a policy.', 'start': 1423.28, 'duration': 4.462}], 'summary': 'Select action with best quality based on state to maximize value and form a policy.', 'duration': 24.775, 'max_score': 1402.967, 'thumbnail': ''}, {'end': 1521.565, 'src': 'embed', 'start': 1499.679, 'weight': 3, 'content': [{'end': 1510.002, 'text': 'a lot more artificial rewards and you learn more about the physics and the dynamics of the system, about this kind of enhanced value.', 'start': 1499.679, 'duration': 10.323}, {'end': 1511.342, 'text': 'And so high insight.', 'start': 1510.562, 'duration': 0.78}, {'end': 1519.464, 'text': 'replay has been an absolutely critical advance in making these more data efficient and learning harder tasks that involve a more complex state space.', 'start': 1511.342, 'duration': 8.122}, {'end': 1521.565, 'text': "It's much more what a human would do.", 'start': 1519.804, 'duration': 1.761}], 'summary': 'Replay improves data efficiency, enabling learning of harder tasks in a more complex state space.', 'duration': 21.886, 'max_score': 1499.679, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1499679.jpg'}], 'start': 1317.657, 'title': 'Q-learning and hindsight replay', 'summary': 'Discusses q-learning, a method that combines the policy and value function into one quality function, and the concept of hindsight replay in reinforcement learning, leading to more data efficient learning and enhanced value.', 'chapters': [{'end': 1436.005, 'start': 1317.657, 'title': 'Q-learning and quality function', 'summary': 'Discusses q-learning, a method that combines the policy and value function into one quality function, allowing for simultaneous learning. it explains the update process of the quality function and its application in choosing the best action based on the current state, contributing to maximizing the policy and value.', 'duration': 118.348, 'highlights': ['The quality function in Q-learning combines the value and policy functions, allowing for simultaneous learning and decision-making based on current state and action.', 'The update process of the quality function involves utilizing the learning rate and discount rate, along with assuming the best possible action in the future to determine the quality of the current state and action.', 'The quality function enables the selection of the best action based on the current state, contributing to maximizing the policy and value, ultimately leading to efficient decision-making and value maximization.']}, {'end': 1560.684, 'start': 1436.285, 'title': 'Hindsight replay in reinforcement learning', 'summary': 'Discusses the concept of hindsight replay in reinforcement learning, which allows the system to learn from non-rewarding experiences by encoding them as potential actions for different reward structures, leading to more data efficient learning and enhanced value. it also mentions the application of this concept in neural networks for interacting with the environment.', 'duration': 124.399, 'highlights': ['Hindsight replay in reinforcement learning enables encoding non-rewarding experiences for potential actions for different reward structures, leading to more data efficient learning and enhanced value.', 'This concept has been critical in making the learning of harder tasks involving a more complex state space more data efficient.', 'The application of hindsight replay in neural networks for learning to interact with the environment is an exciting advance in the field.']}], 'duration': 243.027, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/0MNVhXEX9to/pics/0MNVhXEX9to1317657.jpg', 'highlights': ['The quality function in Q-learning combines the value and policy functions, allowing for simultaneous learning and decision-making based on current state and action.', 'The update process of the quality function involves utilizing the learning rate and discount rate, along with assuming the best possible action in the future to determine the quality of the current state and action.', 'The quality function enables the selection of the best action based on the current state, contributing to maximizing the policy and value, ultimately leading to efficient decision-making and value maximization.', 'Hindsight replay in reinforcement learning enables encoding non-rewarding experiences for potential actions for different reward structures, leading to more data efficient learning and enhanced value.', 'This concept has been critical in making the learning of harder tasks involving a more complex state space more data efficient.', 'The application of hindsight replay in neural networks for learning to interact with the environment is an exciting advance in the field.']}], 'highlights': ['Reinforcement learning aims to enable better robots and physical agents to interact with the world and learn how to learn, similar to humans and animals.', 'Reinforcement learning is used to learn how to control a complex system, such as a bipedal walker, in a simulated environment, demonstrating its application as an optimization framework.', 'The big challenge in reinforcement learning is to design a policy of what actions to take, given a state S, to maximize the chance of getting a future reward.', 'Understanding policies based on good board positions and value functions is akin to how a human plays chess, such as evaluating the expected chance of winning by counting points on the board.', 'Policy iteration involves iteratively updating the policy based on rewards to improve it over time, utilizing strategies such as simulated annealing, evolutionary optimization, gradient descent, and modern tools in neural networks and machine learning.', 'The quality function in Q-learning combines the value and policy functions, allowing for simultaneous learning and decision-making based on current state and action.']}