title
Drago Anguelov (Waymo) - MIT Self-Driving Cars

description
Drago Anguelov is a Principal Scientist at Waymo, developing and applying machine learning methods for autonomous vehicle perception and, more generally, in computer vision and robotics. For more lecture videos on deep learning, reinforcement learning (RL), artificial intelligence (AI & AGI), and podcast conversations, visit our website or follow TensorFlow code tutorials on our GitHub repo. INFO: Website: https://deeplearning.mit.edu GitHub: https://github.com/lexfridman/mit-deep-learning Playlist: http://bit.ly/2S1MVdy OUTLINE: 0:00 - Introduction 0:47 - Background 1:31 - Waymo story (2009 to today) 4:31 - Long tail of events 8:55 - Perception, prediction, and planning 14:54 - Machine learning at scale 26:43 - Addressing the limits of machine learning 29:38 - Large-scale testing 50:51 - Scaling to dozens and hundreds of cities 54:35 - Q&A CONNECT: - If you enjoyed this video, please subscribe to this channel. - Twitter: https://twitter.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Facebook: https://www.facebook.com/lexfridman - Instagram: https://www.instagram.com/lexfridman

detail
{'title': 'Drago Anguelov (Waymo) - MIT Self-Driving Cars', 'heatmap': [{'end': 3877.051, 'start': 3833.989, 'weight': 1}], 'summary': "Waymo has achieved 10+ million miles autonomously, launched a commercial service, and drago anguelov's expertise in perception and robotics is highlighted. challenges in self-driving perception, machine learning implementation at waymo, ai model development, trajectory optimization, scalability, and future prospects of self-driving cars are also discussed.", 'chapters': [{'end': 508.874, 'segs': [{'end': 32.362, 'src': 'embed', 'start': 0.109, 'weight': 2, 'content': [{'end': 4.372, 'text': 'All right, welcome back to 6S094, Deep Learning for Self-Driving Cars.', 'start': 0.109, 'duration': 4.263}, {'end': 9.315, 'text': 'Today we have Drago Anguielov, principal scientist at Waymo.', 'start': 4.972, 'duration': 4.343}, {'end': 13.897, 'text': 'Aside from having the coolest name in autonomous driving,', 'start': 10.175, 'duration': 3.722}, {'end': 21.022, 'text': 'Drago has done a lot of excellent work in developing and applying machine learning methods to autonomous vehicle perception and, more generally,', 'start': 13.897, 'duration': 7.125}, {'end': 22.222, 'text': 'in computer vision and robotics.', 'start': 21.022, 'duration': 1.2}, {'end': 32.362, 'text': "He's now helping Waymo lead the world in autonomous driving 10 plus million miles achieved autonomously to date,", 'start': 22.623, 'duration': 9.739}], 'summary': 'Waymo has achieved over 10 million miles autonomously with drago anguielov leading in autonomous driving.', 'duration': 32.253, 'max_score': 0.109, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY109.jpg'}, {'end': 125.773, 'src': 'embed', 'start': 94.881, 'weight': 0, 'content': [{'end': 98.803, 'text': 'Waymo actually this month has its 10-year anniversary.', 'start': 94.881, 'duration': 3.922}, {'end': 105.285, 'text': 'It started when Sebastian Thrun convinced the Google leadership to try an exciting new moonshot.', 'start': 100.223, 'duration': 5.062}, {'end': 113.089, 'text': 'And the goal that they set for themselves was to drive 10 different segments that were 100 miles long.', 'start': 107.266, 'duration': 5.823}, {'end': 118.711, 'text': 'And later that year, they succeeded and drove an order of magnitude more than anyone has ever driven.', 'start': 114.389, 'duration': 4.322}, {'end': 125.773, 'text': 'In 2015, we brought this car to the road.', 'start': 121.812, 'duration': 3.961}], 'summary': 'Waymo celebrates 10-year anniversary, achieved goal of driving 10 segments 100 miles long, an order of magnitude more than anyone else, and brought the car to the road in 2015.', 'duration': 30.892, 'max_score': 94.881, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY94881.jpg'}, {'end': 224.13, 'src': 'embed', 'start': 156.551, 'weight': 1, 'content': [{'end': 165.216, 'text': 'We worked hard and in 2017, we launched a fleet of fully self-driving vehicles on the streets in Phoenix metro area.', 'start': 156.551, 'duration': 8.665}, {'end': 172.68, 'text': 'And we have been doing fully driverless operations ever since.', 'start': 167.237, 'duration': 5.443}, {'end': 179.584, 'text': 'So I wanted to give you a feel for what fully driverless experience is like.', 'start': 175.842, 'duration': 3.742}, {'end': 219.048, 'text': 'And so we continued.', 'start': 218.047, 'duration': 1.001}, {'end': 224.13, 'text': 'Last year, we launched our first commercial service in the metro area of Phoenix.', 'start': 219.968, 'duration': 4.162}], 'summary': 'In 2017, a fleet of fully self-driving vehicles was launched in phoenix, and in the following year, the first commercial service was introduced.', 'duration': 67.579, 'max_score': 156.551, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY156551.jpg'}], 'start': 0.109, 'title': "Waymo's autonomous driving journey", 'summary': "Discusses waymo's 10-year journey, achieving 10+ million miles autonomously, launching a commercial service, and drago anguielov's expertise in perception and robotics.", 'chapters': [{'end': 94.301, 'start': 0.109, 'title': 'Taming the long tail of autonomous driving challenges', 'summary': "Discusses drago anguielov's work at waymo, highlighting their achievement of 10+ million miles achieved autonomously and his extensive experience in perception and robotics.", 'duration': 94.192, 'highlights': ['Drago Anguielov has led the research team at Waymo, achieving over 10 million miles autonomously, showcasing significant accomplishment in autonomous driving.', 'He has a background in perception and robotics, having worked with pioneers like Daphne Koller and Professor Sebastian Thrun, and contributed to developing deep models for detection neural net architectures.', "Drago's talk is focused on 'Taming the Long Tail of Autonomous Driving Challenges', emphasizing the complexity of self-driving problems and the solutions developed by Waymo.", 'He has also worked at Google and Zoox, where he was involved in research on perception, development of deep models, and heading the 3D perception team for autonomous driving.', "The chapter highlights Drago Anguielov's extensive experience in developing and applying machine learning methods to autonomous vehicle perception, computer vision, and robotics."]}, {'end': 508.874, 'start': 94.881, 'title': "Waymo's 10-year journey to fully driverless mobility", 'summary': "Highlights waymo's 10-year journey, including driving 10 million miles, launching a commercial service, and addressing diverse scenarios to enable a truly self-driverless future.", 'duration': 413.993, 'highlights': ["Waymo has driven 10 million miles on public roads, both driverlessly and with human drivers to collect data. Driving 10 million miles demonstrates extensive testing and data collection, showcasing Waymo's commitment to developing self-driving technology.", "Waymo launched its first commercial service in the metro area of Phoenix, allowing people to call for rides, run errands, and go to school. The launch of the commercial service signifies a significant milestone, demonstrating Waymo's progress in bringing fully driverless mobility to the public.", "Addressing diverse scenarios such as encountering unusual situations on the road, understanding sirens of special vehicles, and preventing issues caused by others not following traffic rules. The focus on diverse scenarios highlights Waymo's efforts to handle rare situations and ensure the safety and efficiency of self-driving vehicles in various real-world conditions."]}], 'duration': 508.765, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY109.jpg', 'highlights': ['Waymo has driven 10 million miles on public roads, showcasing extensive testing and data collection.', 'Waymo launched its first commercial service in Phoenix, marking a significant milestone in bringing fully driverless mobility to the public.', 'Drago Anguielov has extensive experience in developing and applying machine learning methods to autonomous vehicle perception, computer vision, and robotics.', "Drago Anguielov's talk emphasizes the complexity of self-driving problems and the solutions developed by Waymo.", 'Drago Anguielov has worked at Google and Zoox, contributing to research on perception and development of deep models for autonomous driving.']}, {'end': 892.911, 'segs': [{'end': 594.43, 'src': 'embed', 'start': 548.69, 'weight': 0, 'content': [{'end': 552.793, 'text': "There's others, we can talk about others as well in a little bit, but let's focus on these first.", 'start': 548.69, 'duration': 4.103}, {'end': 559.137, 'text': 'So perception is mapping from sensory inputs and potentially prior knowledge of the environment to a scene representation.', 'start': 553.233, 'duration': 5.904}, {'end': 567.063, 'text': 'And that scene representation can contain objects, it can contain scene semantics, potentially you construct a map,', 'start': 559.417, 'duration': 7.646}, {'end': 569.204, 'text': 'you can learn about object relationships, and so on.', 'start': 567.063, 'duration': 2.141}, {'end': 576.031, 'text': 'And Perception, the space of things you need to handle in perception is fairly hard.', 'start': 570.945, 'duration': 5.086}, {'end': 576.971, 'text': "It's a complex mapping.", 'start': 576.091, 'duration': 0.88}, {'end': 586.258, 'text': 'So you have sensors, the pixels come, lighter points come, or radar scans come, and you have multiple axis of variability in the environment.', 'start': 577.892, 'duration': 8.366}, {'end': 587.699, 'text': "So obviously there's a lot of objects.", 'start': 586.298, 'duration': 1.401}, {'end': 591.149, 'text': 'They have different types, appearance, pose.', 'start': 588.807, 'duration': 2.342}, {'end': 594.43, 'text': "I don't know if you see this well.", 'start': 592.349, 'duration': 2.081}], 'summary': 'Perception involves mapping sensory inputs to scene representation, handling complex mappings with multiple axis of variability.', 'duration': 45.74, 'max_score': 548.69, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY548690.jpg'}, {'end': 853.972, 'src': 'embed', 'start': 824.62, 'weight': 3, 'content': [{'end': 831.908, 'text': 'It produces vehicle behavior, typically ends up in control commands to the vehicle, accelerate, slow down, steer the wheel.', 'start': 824.62, 'duration': 7.288}, {'end': 836.473, 'text': 'And you need to generate behavior that ultimately has several properties to it.', 'start': 833.029, 'duration': 3.444}, {'end': 839.115, 'text': "And it's important to think of them, which is safe.", 'start': 836.493, 'duration': 2.622}, {'end': 840.216, 'text': 'Safety comes first.', 'start': 839.416, 'duration': 0.8}, {'end': 844.381, 'text': 'Comfortable for the passengers.', 'start': 842.279, 'duration': 2.102}, {'end': 853.972, 'text': 'and also sends the right signals to the other traffic participants, because they can interact with you and they will react to your actions.', 'start': 845.806, 'duration': 8.166}], 'summary': 'Vehicle behavior must prioritize safety, comfort, and signaling for traffic interaction.', 'duration': 29.352, 'max_score': 824.62, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY824620.jpg'}, {'end': 897.594, 'src': 'embed', 'start': 867.639, 'weight': 5, 'content': [{'end': 868.7, 'text': "I'll show you just one scene.", 'start': 867.639, 'duration': 1.061}, {'end': 873.323, 'text': 'This is a complex, I think, school gathering.', 'start': 870.501, 'duration': 2.822}, {'end': 879.228, 'text': "There's bicyclists trailing us, vehicles really closely hemmed within us, a bunch of pedestrians, and we need to make progress.", 'start': 873.363, 'duration': 5.865}, {'end': 885.233, 'text': "And here is us, we're driving and reasonably well in crowded scenes.", 'start': 879.769, 'duration': 5.464}, {'end': 892.911, 'text': 'And that is part of the prerequisite of bringing this technology to the dense urban environments, being able to do this.', 'start': 885.653, 'duration': 7.258}, {'end': 895.292, 'text': 'So how are we going to do it? Well, I gave it up.', 'start': 893.511, 'duration': 1.781}, {'end': 897.594, 'text': "I'm a machine learning person.", 'start': 896.553, 'duration': 1.041}], 'summary': 'Navigating crowded urban environments with vehicles and pedestrians using machine learning technology.', 'duration': 29.955, 'max_score': 867.639, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY867639.jpg'}], 'start': 509.374, 'title': 'Autonomous vehicle challenges', 'summary': 'Explores challenges in self-driving perception, emphasizing the complexity of mapping sensory inputs, handling object variability, and predicting agent behavior for autonomous vehicles.', 'chapters': [{'end': 661.128, 'start': 509.374, 'title': 'Challenges in self-driving perception', 'summary': 'Delves into the diverse and challenging situations faced in self-driving technology, highlighting the complexity of perception tasks such as mapping sensory inputs to scene representations and handling the variability of objects and environments.', 'duration': 151.754, 'highlights': ['Perception involves complex mapping of sensory inputs to scene representations, encompassing objects, scene semantics, and environmental knowledge. Perception tasks involve mapping sensory inputs and prior knowledge of the environment to scene representations, which can include objects, scene semantics, and potentially constructing a map, as well as learning about object relationships.', 'The space of things to handle in perception is fairly hard due to the variability in objects, appearances, poses, and environmental conditions. The challenges in perception include handling the variability of objects, appearances, poses, and environmental conditions, such as different types of objects, appearances, poses, and environments, including occlusions, reflections, and diverse object relationships.', 'Different environments and objects present a wide range of configurations, adding complexity to the perception tasks. Perception tasks involve handling different environments and objects in various configurations, including different times of day, seasons, and environments, as well as different object relationships and configurations, such as occlusions, reflections, and diverse object placements.']}, {'end': 892.911, 'start': 661.128, 'title': 'Autonomous driving: predicting and planning', 'summary': 'Discusses the importance of predicting agent behavior for autonomous vehicles, emphasizing the need to anticipate, predict, and plan vehicle behavior based on sensor inputs and environmental cues.', 'duration': 231.783, 'highlights': ['Autonomous vehicles need to anticipate and predict the behavior of other agents in the environment, such as pedestrians and cyclists, to ensure safe and comfortable driving. The chapter emphasizes the necessity for autonomous vehicles to anticipate and predict the behavior of other agents, such as pedestrians and cyclists, in order to ensure safe and comfortable driving.', 'The decision-making process for autonomous vehicles involves generating behavior that is safe, comfortable for passengers, and sends the right signals to other traffic participants. The decision-making process for autonomous vehicles involves generating behavior that prioritizes safety, passenger comfort, and appropriate signaling to other traffic participants.', 'The ability to reason and make decisions in complex environments, such as crowded urban scenes, is crucial for autonomous vehicles to navigate effectively and deliver passengers safely. The chapter underscores the significance of the ability to reason and make decisions in complex environments, particularly in crowded urban scenes, as crucial for the effective navigation and safe delivery of passengers by autonomous vehicles.']}], 'duration': 383.537, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY509374.jpg', 'highlights': ['Perception tasks involve mapping sensory inputs and prior knowledge of the environment to scene representations, encompassing objects, scene semantics, and environmental knowledge.', 'The challenges in perception include handling the variability of objects, appearances, poses, and environmental conditions, such as different types of objects, appearances, poses, and environments, including occlusions, reflections, and diverse object relationships.', 'Perception tasks involve handling different environments and objects in various configurations, including different times of day, seasons, and environments, as well as different object relationships and configurations, such as occlusions, reflections, and diverse object placements.', 'The chapter emphasizes the necessity for autonomous vehicles to anticipate and predict the behavior of other agents, such as pedestrians and cyclists, in order to ensure safe and comfortable driving.', 'The decision-making process for autonomous vehicles involves generating behavior that prioritizes safety, passenger comfort, and appropriate signaling to other traffic participants.', 'The chapter underscores the significance of the ability to reason and make decisions in complex environments, particularly in crowded urban scenes, as crucial for the effective navigation and safe delivery of passengers by autonomous vehicles.']}, {'end': 1229.684, 'segs': [{'end': 945.562, 'src': 'embed', 'start': 917.234, 'weight': 0, 'content': [{'end': 919.755, 'text': 'Obviously, this is now a machine learning revolution.', 'start': 917.234, 'duration': 2.521}, {'end': 924.296, 'text': 'And machine learning is permeating all parts of the Waymo stack.', 'start': 920.515, 'duration': 3.781}, {'end': 928.057, 'text': "All of these systems that I'm talking about, it helps us perceive the world.", 'start': 924.876, 'duration': 3.181}, {'end': 931.858, 'text': 'It helps us make decisions about what others are going to do.', 'start': 928.137, 'duration': 3.721}, {'end': 933.218, 'text': 'It helps us make our own decisions.', 'start': 931.918, 'duration': 1.3}, {'end': 937.439, 'text': 'And machine learning is a tool to handle the long tail.', 'start': 935.259, 'duration': 2.18}, {'end': 940.5, 'text': "And I'll tell you a little more on this how.", 'start': 938.8, 'duration': 1.7}, {'end': 945.562, 'text': 'So I have this allegory about machine learning that I like to think about.', 'start': 942.421, 'duration': 3.141}], 'summary': 'Machine learning revolutionizing waymo, aiding perception, decision-making, and addressing long-tail with an allegory.', 'duration': 28.328, 'max_score': 917.234, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY917234.jpg'}, {'end': 1036.529, 'src': 'embed', 'start': 1002.547, 'weight': 1, 'content': [{'end': 1003.507, 'text': 'You need to build a factory.', 'start': 1002.547, 'duration': 0.96}, {'end': 1007.148, 'text': 'Once you do it, now you can iterate.', 'start': 1005.468, 'duration': 1.68}, {'end': 1008.169, 'text': "It's scalable.", 'start': 1007.508, 'duration': 0.661}, {'end': 1012.61, 'text': 'Just keep the right data, keep feeding the machine, keeps giving you good models.', 'start': 1009.189, 'duration': 3.421}, {'end': 1022.421, 'text': 'So what is an ML factory for self-driving models? Well, roughly it goes like this.', 'start': 1015.851, 'duration': 6.57}, {'end': 1027.084, 'text': "We have a software release, we put it on the vehicle, we're able to drive.", 'start': 1023.742, 'duration': 3.342}, {'end': 1031.226, 'text': 'We drive, we collect data, we collect it, and we store it.', 'start': 1027.284, 'duration': 3.942}, {'end': 1036.529, 'text': 'Then we select some parts of this data and we send it to labelers.', 'start': 1033.547, 'duration': 2.982}], 'summary': 'Build a scalable factory for ml self-driving models to iterate and generate good models from collected data.', 'duration': 33.982, 'max_score': 1002.547, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1002547.jpg'}, {'end': 1128.051, 'src': 'embed', 'start': 1101.414, 'weight': 3, 'content': [{'end': 1109.619, 'text': "So ingredient one is computing software infrastructure and we're part of Alphabet, Google and we are able to first of all leverage TensorFlow,", 'start': 1101.414, 'duration': 8.205}, {'end': 1110.72, 'text': 'the deep learning framework.', 'start': 1109.619, 'duration': 1.101}, {'end': 1114.903, 'text': 'We have access to the experts that wrote TensorFlow and know it in depth.', 'start': 1111.32, 'duration': 3.583}, {'end': 1119.786, 'text': 'We have data centers to run large-scale parallel compute and also train models.', 'start': 1115.703, 'duration': 4.083}, {'end': 1128.051, 'text': 'We have specialized hardware for training models, which make it cheaper and more affordable and faster so you can iterate better.', 'start': 1120.746, 'duration': 7.305}], 'summary': "Google's infrastructure leverages tensorflow for faster, more affordable model training.", 'duration': 26.637, 'max_score': 1101.414, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1101414.jpg'}, {'end': 1229.684, 'src': 'embed', 'start': 1183.619, 'weight': 4, 'content': [{'end': 1191.741, 'text': 'We want to select data for examples that are interesting in some way and complement capture these long tail cases that we potentially may not be doing so well on.', 'start': 1183.619, 'duration': 8.122}, {'end': 1198.682, 'text': 'And so for this, we have active learning and data mining pipelines.', 'start': 1192.461, 'duration': 6.221}, {'end': 1201.503, 'text': 'Given exemplars, find the rare examples.', 'start': 1199.662, 'duration': 1.841}, {'end': 1208.304, 'text': 'Look for parts of your system which are uncertain or inconsistent over time and go and label those cases.', 'start': 1202.003, 'duration': 6.301}, {'end': 1212.08, 'text': 'Last but not least, we also produce auto labels.', 'start': 1209.98, 'duration': 2.1}, {'end': 1218.622, 'text': 'So how can you do that? Well, when you collect data, you also see the future for many of the objects, what they did.', 'start': 1212.58, 'duration': 6.042}, {'end': 1222.922, 'text': 'And so, because of that, now, knowing the past and the future,', 'start': 1219.502, 'duration': 3.42}, {'end': 1229.684, 'text': 'you can annotate your data better and then go back to your model that does not know the future and try to replicate that with that model.', 'start': 1222.922, 'duration': 6.762}], 'summary': 'Use active learning and data mining to select interesting and rare examples, and produce auto labels for better data annotation and model replication.', 'duration': 46.065, 'max_score': 1183.619, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1183619.jpg'}], 'start': 893.511, 'title': 'Machine learning for self-driving models and infrastructure at waymo', 'summary': "Delves into the machine learning revolution and its implementation in waymo's self-driving models, highlighting scalability and iterative processes. it also explores the machine learning infrastructure at waymo, encompassing tensorflow utilization, data handling, and model training enhancement through active learning and auto labeling.", 'chapters': [{'end': 1076.35, 'start': 893.511, 'title': 'Machine learning for self-driving models', 'summary': 'Discusses the machine learning revolution, its permeation in the waymo stack, and the comparison of classical system with machine learning system for self-driving models, emphasizing the scalability and iterative process of modern machine learning.', 'duration': 182.839, 'highlights': ['Machine learning is permeating all parts of the Waymo stack, helping in perceiving the world, making decisions, and handling the long tail. Machine learning is integrated into various systems in Waymo, aiding in perception, decision-making, and handling complex scenarios.', 'Modern machine learning is likened to a factory, providing scalability and iterative processes for self-driving models. Modern machine learning functions like a factory, enabling scalability and iterative processes for self-driving models, leading to continuous improvement.', 'The process of building an ML factory for self-driving models involves data collection, labeling, training, testing, validation, and iterative improvement. The process of creating an ML factory for self-driving models involves various steps such as data collection, labeling, training, testing, validation, and iterative improvement, ensuring continuous enhancement of the models.']}, {'end': 1229.684, 'start': 1076.971, 'title': 'Machine learning infrastructure at waymo', 'summary': 'Discusses the machine learning infrastructure at waymo, including leveraging tensorflow, data collection and storage, data annotation, active learning, and auto labeling to improve model training and iteration.', 'duration': 152.713, 'highlights': ['Waymo leverages TensorFlow for deep learning and has access to experts for in-depth knowledge, data centers for large-scale parallel compute, and specialized hardware for training models. Waymo has the advantage of leveraging TensorFlow for deep learning and has access to experts and specialized hardware, enabling large-scale parallel compute and faster model training.', 'Waymo collects and stores high-quality labeled data at scale, utilizing active learning and data mining pipelines to find rare examples and complement capture long tail cases. Waymo collects and stores high-quality labeled data at scale and uses active learning and data mining pipelines to find rare examples and complement capturing long tail cases, improving the quality of the training data.', 'Waymo utilizes auto labeling by annotating data based on the future actions of objects, allowing for better data annotation and model replication. Waymo uses auto labeling by annotating data based on the future actions of objects, enhancing data annotation and model replication capabilities.']}], 'duration': 336.173, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY893511.jpg', 'highlights': ['Machine learning is integrated into various systems in Waymo, aiding in perception, decision-making, and handling complex scenarios.', 'Modern machine learning functions like a factory, enabling scalability and iterative processes for self-driving models, leading to continuous improvement.', 'The process of creating an ML factory for self-driving models involves various steps such as data collection, labeling, training, testing, validation, and iterative improvement, ensuring continuous enhancement of the models.', 'Waymo has the advantage of leveraging TensorFlow for deep learning and has access to experts and specialized hardware, enabling large-scale parallel compute and faster model training.', 'Waymo collects and stores high-quality labeled data at scale and uses active learning and data mining pipelines to find rare examples and complement capturing long tail cases, improving the quality of the training data.', 'Waymo uses auto labeling by annotating data based on the future actions of objects, enhancing data annotation and model replication capabilities.']}, {'end': 1650.282, 'segs': [{'end': 1292.174, 'src': 'embed', 'start': 1252.58, 'weight': 0, 'content': [{'end': 1260.063, 'text': 'It was 2013 when I got on to do deep learning and a lot of things were not understood and we were there working on it earlier than most people.', 'start': 1252.58, 'duration': 7.483}, {'end': 1264.905, 'text': 'And so through that we had the opportunity and the chance to develop some of the.', 'start': 1260.123, 'duration': 4.782}, {'end': 1272.188, 'text': 'in my time, the team I managed, we invented neural net architecture like Inception, which became popular later.', 'start': 1264.905, 'duration': 7.283}, {'end': 1276.81, 'text': 'We invented at the time the state of the art object detection, fast object detector called SSD.', 'start': 1272.248, 'duration': 4.562}, {'end': 1280.391, 'text': 'And we won the ImageNet 2014.', 'start': 1278.15, 'duration': 2.241}, {'end': 1285.932, 'text': 'And now if you go to the conferences, Google and DeepMind are leaders in perception and reinforcement learning and smart agents.', 'start': 1280.391, 'duration': 5.541}, {'end': 1292.174, 'text': 'And there is state of the art, say, semantic segmentation networks, pose estimation, and so on.', 'start': 1286.613, 'duration': 5.561}], 'summary': 'Pioneered deep learning, invented inception, ssd, won imagenet 2014, and contributed to state-of-the-art advancements in perception and reinforcement learning.', 'duration': 39.594, 'max_score': 1252.58, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1252580.jpg'}, {'end': 1340.174, 'src': 'embed', 'start': 1312.339, 'weight': 3, 'content': [{'end': 1324.768, 'text': 'This is a project we did recently and today we put online in our blog about automatic machine learning for tuning and adjusting architectures of neural networks.', 'start': 1312.339, 'duration': 12.429}, {'end': 1333.409, 'text': 'So what did we do? So there is a team at Google working on AutoML, automatic machine learning.', 'start': 1325.549, 'duration': 7.86}, {'end': 1337.252, 'text': 'And usually networks themselves have complex architecture.', 'start': 1334.31, 'duration': 2.942}, {'end': 1340.174, 'text': "They're crafted by practitioners, two artisans of networks in some way.", 'start': 1337.292, 'duration': 2.882}], 'summary': 'Project involved implementing automatic machine learning for tuning and adjusting neural network architectures.', 'duration': 27.835, 'max_score': 1312.339, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1312339.jpg'}, {'end': 1481.041, 'src': 'embed', 'start': 1452.007, 'weight': 6, 'content': [{'end': 1457.309, 'text': 'And we saw one of two things happened for the various versions that we found.', 'start': 1452.007, 'duration': 5.302}, {'end': 1463.81, 'text': 'One of them is we can find models with similar quality but much lower latency and less compute.', 'start': 1457.449, 'duration': 6.361}, {'end': 1468.491, 'text': 'And then there is models of a bit higher quality at the same latency.', 'start': 1464.27, 'duration': 4.221}, {'end': 1471.612, 'text': 'So essentially we found better models than the human engineers did.', 'start': 1468.571, 'duration': 3.041}, {'end': 1481.041, 'text': 'Similar results were obtained for other problems, lane detection as well with this transfer learning approach.', 'start': 1473.614, 'duration': 7.427}], 'summary': 'Improved models with lower latency and better quality found, outperforming human engineers in transfer learning approach.', 'duration': 29.034, 'max_score': 1452.007, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1452007.jpg'}, {'end': 1581.439, 'src': 'embed', 'start': 1543.28, 'weight': 4, 'content': [{'end': 1547.403, 'text': 'And so this way we can explore a much larger space of network architectures.', 'start': 1543.28, 'duration': 4.123}, {'end': 1554.968, 'text': 'So what happened? So on the left, this is 4, 000 different models spanning the scale and latency and quality.', 'start': 1547.843, 'duration': 7.125}, {'end': 1557.93, 'text': 'And in red was the transfer model.', 'start': 1555.889, 'duration': 2.041}, {'end': 1563.994, 'text': 'So after the first round of search, we actually did not produce a better model than the transfer, which already leveraged their insight.', 'start': 1558.21, 'duration': 5.784}, {'end': 1571.036, 'text': 'So then we took the learnings and the best models from this search and did the second round of search, which was in yellow,', 'start': 1564.875, 'duration': 6.161}, {'end': 1572.197, 'text': 'which allowed us to beat it.', 'start': 1571.036, 'duration': 1.161}, {'end': 1581.439, 'text': 'And third is we also executed reinforcement learning algorithm developed by the AI researchers on 6, 000 different architectures.', 'start': 1572.257, 'duration': 9.182}], 'summary': 'Explored 4000 models, beat transfer model, used reinforcement learning on 6000 architectures', 'duration': 38.159, 'max_score': 1543.28, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1543280.jpg'}, {'end': 1657.684, 'src': 'embed', 'start': 1631.733, 'weight': 7, 'content': [{'end': 1636.318, 'text': 'And how do you do this? So one part is, of course, you want redundant and complementary sensors.', 'start': 1631.733, 'duration': 4.585}, {'end': 1641.48, 'text': 'So we have given 360 degree field of view on our vehicles, both in camera, LiDAR, and radar.', 'start': 1637.159, 'duration': 4.321}, {'end': 1643.42, 'text': "And they're complementary modalities.", 'start': 1641.88, 'duration': 1.54}, {'end': 1646.681, 'text': 'First of all, an object is seen in all of them.', 'start': 1644.121, 'duration': 2.56}, {'end': 1650.282, 'text': 'Second of all, they all have different strengths and different modes of failure.', 'start': 1646.741, 'duration': 3.541}, {'end': 1654.103, 'text': 'And so whenever one of them tends to fail, the others usually work fine.', 'start': 1651.002, 'duration': 3.101}, {'end': 1656.283, 'text': 'And so that helps a lot.', 'start': 1654.583, 'duration': 1.7}, {'end': 1657.684, 'text': 'Make sure we do not miss anything.', 'start': 1656.324, 'duration': 1.36}], 'summary': 'Utilize redundant and complementary sensors with 360-degree field of view in camera, lidar, and radar to minimize failure and ensure comprehensive object detection.', 'duration': 25.951, 'max_score': 1631.733, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1631733.jpg'}], 'start': 1231.464, 'title': 'Ai model development, collaboration, and automation', 'summary': 'Covers the development of high-quality ai models in collaboration with google and deepmind, including winning imagenet 2014 and recent projects on automatic machine learning. it also discusses automating model search and improvement using transfer learning, reinforcement learning, and robustness measures, leading to significant quality improvements and latency reduction.', 'chapters': [{'end': 1450.367, 'start': 1231.464, 'title': 'Ai model development and collaboration', 'summary': 'Discusses the development of high-quality ai models, emphasizing the collaboration with google and deepmind, winning the imagenet 2014, and the recent project on automatic machine learning for tuning and adjusting architectures of neural networks.', 'duration': 218.903, 'highlights': ['Collaboration with Google and DeepMind on perception and reinforcement learning The collaboration with Google and DeepMind on perception and reinforcement learning has led to state-of-the-art advancements in semantic segmentation networks, pose estimation, and object detection.', 'Winning the ImageNet 2014 with state-of-the-art object detection The team managed to win the ImageNet 2014 by inventing state-of-the-art object detection, specifically a fast object detector called SSD.', 'Invention of neural net architecture like Inception The team managed to invent neural net architecture like Inception, which became popular later, showcasing their early involvement in the deep learning revolution.', 'Recent project on automatic machine learning for tuning and adjusting architectures of neural networks The team has recently put online a project about automatic machine learning for tuning and adjusting architectures of neural networks, collaborating with Google researchers to develop a system that searches the space of architectures and found a set of components of neural networks.']}, {'end': 1650.282, 'start': 1452.007, 'title': 'Automating model search and improvement', 'summary': 'Outlines a process of automating the search for better models by leveraging transfer learning, executing reinforcement learning algorithms, and ensuring robustness through redundant and complementary sensors, resulting in significant improvements in model quality and reduction in latency.', 'duration': 198.275, 'highlights': ['The search for better models involved executing reinforcement learning algorithms on 6,000 different architectures, resulting in significant improvement in model quality. The reinforcement learning algorithm developed by the AI researchers on 6,000 different architectures significantly improved model quality, outperforming the in-house algorithm.', 'Leveraging transfer learning and executing a second round of search allowed for the discovery of better models, surpassing the initial transfer model. By leveraging transfer learning and conducting a second round of search, better models were discovered, surpassing the initial transfer model.', 'The process involved exploring a much larger space of network architectures, resulting in the discovery of models with similar quality but much lower latency and less compute, outperforming the models designed by human engineers. Exploring a much larger space of network architectures led to the discovery of models with similar quality but lower latency and less compute, surpassing those designed by human engineers.', 'Ensuring robustness through redundant and complementary sensors, such as camera, LiDAR, and radar, with a 360-degree field of view, to handle uncertain or potentially mistaken model predictions. The system was designed to be robust by employing redundant and complementary sensors, such as camera, LiDAR, and radar, to handle uncertain or potentially mistaken model predictions.']}], 'duration': 418.818, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1231464.jpg', 'highlights': ['Collaboration with Google and DeepMind on perception and reinforcement learning has led to state-of-the-art advancements in semantic segmentation networks, pose estimation, and object detection.', 'The team managed to win the ImageNet 2014 by inventing state-of-the-art object detection, specifically a fast object detector called SSD.', 'The team managed to invent neural net architecture like Inception, which became popular later, showcasing their early involvement in the deep learning revolution.', 'The team has recently put online a project about automatic machine learning for tuning and adjusting architectures of neural networks, collaborating with Google researchers to develop a system that searches the space of architectures and found a set of components of neural networks.', 'The reinforcement learning algorithm developed by the AI researchers on 6,000 different architectures significantly improved model quality, outperforming the in-house algorithm.', 'By leveraging transfer learning and conducting a second round of search, better models were discovered, surpassing the initial transfer model.', 'Exploring a much larger space of network architectures led to the discovery of models with similar quality but lower latency and less compute, surpassing those designed by human engineers.', 'The system was designed to be robust by employing redundant and complementary sensors, such as camera, LiDAR, and radar, to handle uncertain or potentially mistaken model predictions.']}, {'end': 2668.074, 'segs': [{'end': 1678.446, 'src': 'embed', 'start': 1651.002, 'weight': 1, 'content': [{'end': 1654.103, 'text': 'And so whenever one of them tends to fail, the others usually work fine.', 'start': 1651.002, 'duration': 3.101}, {'end': 1656.283, 'text': 'And so that helps a lot.', 'start': 1654.583, 'duration': 1.7}, {'end': 1657.684, 'text': 'Make sure we do not miss anything.', 'start': 1656.324, 'duration': 1.36}, {'end': 1662.465, 'text': "Also, we've designed our system to be a hybrid system.", 'start': 1660.424, 'duration': 2.041}, {'end': 1664.971, 'text': 'And this is the point I want to make.', 'start': 1663.649, 'duration': 1.322}, {'end': 1673.18, 'text': 'So some of these mapping problems or problems in which we apply our models are very complicated.', 'start': 1665.291, 'duration': 7.889}, {'end': 1674.101, 'text': "They're high dimensional.", 'start': 1673.22, 'duration': 0.881}, {'end': 1675.443, 'text': 'The image has a lot of pixels.', 'start': 1674.121, 'duration': 1.322}, {'end': 1678.446, 'text': 'Lighter has a lot of lighter points.', 'start': 1676.124, 'duration': 2.322}], 'summary': 'Hybrid system handles complex, high-dimensional mapping problems effectively.', 'duration': 27.444, 'max_score': 1651.002, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1651002.jpg'}, {'end': 1744.955, 'src': 'embed', 'start': 1716.666, 'weight': 0, 'content': [{'end': 1724.09, 'text': 'And there is also, of course, experts can put in their knowledge in terms of designing the algorithm which incorporates it as well, right?', 'start': 1716.666, 'duration': 7.424}, {'end': 1725.872, 'text': 'And so our system is this hybrid.', 'start': 1724.531, 'duration': 1.341}, {'end': 1737.648, 'text': "And so an example of what that looks for perception is Well, no matter if there's cases where the machine learning system may be not confident,", 'start': 1727.432, 'duration': 10.216}, {'end': 1744.955, 'text': 'we still have tracks and obstacles from LiDAR and radar scans and we make sure that we drive relative to those safely.', 'start': 1737.648, 'duration': 7.307}], 'summary': 'Hybrid system incorporates expert knowledge for safe driving.', 'duration': 28.289, 'max_score': 1716.666, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1716666.jpg'}, {'end': 1816.936, 'src': 'embed', 'start': 1790.345, 'weight': 2, 'content': [{'end': 1794.047, 'text': 'So how do you normally develop a self-driving algorithm?', 'start': 1790.345, 'duration': 3.702}, {'end': 1802.75, 'text': "Well, the ideal thing you're gonna do is you make your algorithm change and you would put it on the vehicle and drive a bunch and say now it looks great.", 'start': 1794.067, 'duration': 8.683}, {'end': 1805.071, 'text': "All right, let's make the next one.", 'start': 1802.77, 'duration': 2.301}, {'end': 1813.695, 'text': 'The problem is, I mean, we have a big fleet, we have a lot of data, but some of the conditions and situations occur very, very rarely.', 'start': 1805.551, 'duration': 8.144}, {'end': 1816.936, 'text': "And so if you do this, you're gonna wait a long time.", 'start': 1814.875, 'duration': 2.061}], 'summary': 'Developing a self-driving algorithm involves testing it on a vehicle and gathering data, but rare conditions can lead to long wait times.', 'duration': 26.591, 'max_score': 1790.345, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1790345.jpg'}, {'end': 1903.351, 'src': 'embed', 'start': 1869.241, 'weight': 3, 'content': [{'end': 1882.851, 'text': 'So we simulate the equivalent of 25, 000 cars, virtual cars, driving, 10 million miles a day, and over seven billion miles simulated.', 'start': 1869.241, 'duration': 13.61}, {'end': 1886.174, 'text': "It's a key part of our release process.", 'start': 1884.712, 'duration': 1.462}, {'end': 1888.905, 'text': 'So why do you need to simulate this much?', 'start': 1887.304, 'duration': 1.601}, {'end': 1898.529, 'text': 'Well, hopefully I convinced you there is a variety of cases to worry about and that you need to test through so far.', 'start': 1890.505, 'duration': 8.024}, {'end': 1903.351, 'text': 'Furthermore, it goes all the way bottom-up.', 'start': 1900.409, 'duration': 2.942}], 'summary': 'Simulating 25,000 virtual cars driving 10 million miles a day helps test a variety of cases in the release process.', 'duration': 34.11, 'max_score': 1869.241, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1869241.jpg'}, {'end': 2059.257, 'src': 'embed', 'start': 2031.944, 'weight': 4, 'content': [{'end': 2036.989, 'text': 'So the insight is, in simulation, our actions affect the environment and it need to be accounted for.', 'start': 2031.944, 'duration': 5.045}, {'end': 2048.141, 'text': 'So what does that mean? If you want to have effective simulations on a large scale, you need to simulate realistic driver and pedestrian behavior.', 'start': 2038.671, 'duration': 9.47}, {'end': 2052.71, 'text': 'So you could think of a simple model.', 'start': 2049.847, 'duration': 2.863}, {'end': 2059.257, 'text': "Well, what is a good proxy or what's a good approximation of a realistic behavior? Well, you can do a break and swerve model.", 'start': 2052.75, 'duration': 6.507}], 'summary': 'In simulation, realistic driver and pedestrian behavior is crucial for effective large-scale simulations.', 'duration': 27.313, 'max_score': 2031.944, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2031944.jpg'}, {'end': 2195.527, 'src': 'embed', 'start': 2164.067, 'weight': 5, 'content': [{'end': 2167.39, 'text': 'so we put a paper on archive about a month ago, i believe on, uh,', 'start': 2164.067, 'duration': 3.323}, {'end': 2177.816, 'text': 'We took 60 hours of footage of driving and we tried to see how well we can imitate it using a deep neural network.', 'start': 2168.311, 'duration': 9.505}, {'end': 2184.18, 'text': 'And so one option is to do exactly the same to end-agent policy, but we wanted to make our task easier.', 'start': 2179.317, 'duration': 4.863}, {'end': 2195.527, 'text': "How?? Well, we have a good perception system at Waymo, so why don't we use its products? for that agent also can simplify the input representation a bit.", 'start': 2185.84, 'duration': 9.687}], 'summary': 'Paper on driving imitation using deep neural network with 60 hours of footage.', 'duration': 31.46, 'max_score': 2164.067, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2164067.jpg'}, {'end': 2358.723, 'src': 'embed', 'start': 2325.684, 'weight': 6, 'content': [{'end': 2326.984, 'text': 'And there is techniques to handle this.', 'start': 2325.684, 'duration': 1.3}, {'end': 2329.947, 'text': 'One thing we did was synthesize perturbations.', 'start': 2328.005, 'duration': 1.942}, {'end': 2337.949, 'text': 'So you have your trajectory, and we synthesize, deform the trajectory, and force the vehicle to learn to come back to the middle of the lane.', 'start': 2331.164, 'duration': 6.785}, {'end': 2339.83, 'text': "So that's something you can do.", 'start': 2338.889, 'duration': 0.941}, {'end': 2341.731, 'text': "That's reasonable.", 'start': 2340.39, 'duration': 1.341}, {'end': 2346.955, 'text': 'Now, you know, if you just have direct imitation based on supervision,', 'start': 2342.612, 'duration': 4.343}, {'end': 2351.538, 'text': "we are trying to pass a vehicle in the street and it's stopping and never continuing.", 'start': 2346.955, 'duration': 4.583}, {'end': 2358.723, 'text': 'So now we did perturbations, and well, it kind of ran through the vehicle.', 'start': 2351.758, 'duration': 6.965}], 'summary': 'Using synthesized perturbations to improve vehicle trajectory and lane control.', 'duration': 33.039, 'max_score': 2325.684, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2325684.jpg'}, {'end': 2446.055, 'src': 'embed', 'start': 2420.708, 'weight': 7, 'content': [{'end': 2426.333, 'text': 'So you add this structural knowledge, that adds a lot more constraints to the system as it trains.', 'start': 2420.708, 'duration': 5.625}, {'end': 2432.879, 'text': "So it's not just limited, but what it's explicitly seen, it allows it to reason about things it has not explicitly seen as well.", 'start': 2426.874, 'duration': 6.005}, {'end': 2437.424, 'text': "And so now, here's an example of us driving with this network.", 'start': 2434.561, 'duration': 2.863}, {'end': 2446.055, 'text': "And you can see that we're predicting the future with the yellow boxes and we're driving safely through intersections and complex scenarios.", 'start': 2438.672, 'duration': 7.383}], 'summary': 'Structural knowledge enhances system constraints, enabling safe driving predictions.', 'duration': 25.347, 'max_score': 2420.708, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2420708.jpg'}], 'start': 1651.002, 'title': 'Autonomous vehicle technology', 'summary': 'Discusses the benefits of a hybrid system for perception, challenges in self-driving algorithm development, and end-to-end agent learning for autonomous vehicles, including leveraging expert domain knowledge, large-scale testing needs, and training an end-to-end agent using 60 hours of driving footage.', 'chapters': [{'end': 1737.648, 'start': 1651.002, 'title': 'Hybrid system for perception', 'summary': 'Discusses the benefits of a hybrid system for perception, leveraging expert domain knowledge to improve input representations and learning models with fewer examples.', 'duration': 86.646, 'highlights': ['Humans can develop the right input representations and put in expert bias, making it easier to learn models with fewer examples.', 'Expert domain knowledge can help in designing algorithms that incorporate this knowledge, contributing to the hybrid system for perception.', 'The state of the art in models keeps improving, including zero-shot and one-shot learning, while leveraging expert domain knowledge.', 'Mapping problems and high-dimensional data make it challenging to train models with very few examples, highlighting the need for a hybrid system.']}, {'end': 2142.688, 'start': 1737.648, 'title': 'Challenges in self-driving algorithm development', 'summary': 'Discusses the challenges of self-driving algorithm development, including the need for large-scale testing, simulation of rare scenarios, and modeling realistic driver and pedestrian behavior.', 'duration': 405.04, 'highlights': ['Large-scale testing is a key problem in the pipeline and getting vehicles on the road Large-scale testing is crucial in ensuring the readiness of self-driving algorithms for deployment, addressing the challenge of testing under rare conditions and situations.', 'Simulating the equivalent of 25,000 virtual cars driving 10 million miles a day, and over seven billion miles simulated The scale of simulation, equivalent to 25,000 virtual cars driving 10 million miles a day and over seven billion miles simulated, illustrates the extensive testing required for self-driving algorithms.', 'Need to simulate realistic driver and pedestrian behavior for effective simulations on a large scale The importance of simulating realistic driver and pedestrian behavior is emphasized for creating effective simulations on a large scale, highlighting the complexity of human behavior in driving scenarios.']}, {'end': 2668.074, 'start': 2144.091, 'title': 'End-to-end agent learning for autonomous vehicles', 'summary': "Discusses training an end-to-end agent using 60 hours of driving footage to imitate driving behavior, incorporating perturbations, adding structural constraints to predict collisions and road adherence, and testing the model's performance in both simulation and real-world scenarios, highlighting its capabilities and limitations.", 'duration': 523.983, 'highlights': ['Training an end-to-end agent using 60 hours of driving footage The work involves using 60 hours of driving footage to train an end-to-end agent to imitate driving behavior.', 'Incorporating perturbations to handle small errors and deviations Synthesizing perturbations in the trajectory to train the vehicle to come back to the middle of the lane and prevent small errors or deviations.', 'Adding structural constraints to predict collisions and road adherence Augmenting the network to predict a mask for the road and incorporating constraints to avoid collisions and ensure adherence to the road, leading to a more comprehensive training approach.', "Testing the model's performance in simulation and real-world scenarios Evaluating the model's performance in both simulation and real-world scenarios, showcasing its capabilities, limitations, and the need to address long-tail testing situations."]}], 'duration': 1017.072, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY1651002.jpg', 'highlights': ['Expert domain knowledge contributes to the hybrid system for perception.', 'The need for a hybrid system due to mapping problems and high-dimensional data.', 'Large-scale testing is crucial for ensuring the readiness of self-driving algorithms.', 'The extensive scale of simulation required for testing self-driving algorithms.', 'Simulating realistic driver and pedestrian behavior is essential for effective large-scale simulations.', 'Training an end-to-end agent using 60 hours of driving footage.', 'Incorporating perturbations to handle small errors and deviations in training.', 'Adding structural constraints to predict collisions and road adherence in training.', "Evaluating the model's performance in both simulation and real-world scenarios."]}, {'end': 3112.462, 'segs': [{'end': 2723.01, 'src': 'embed', 'start': 2692.489, 'weight': 0, 'content': [{'end': 2693.79, 'text': 'And you could have many models.', 'start': 2692.489, 'duration': 1.301}, {'end': 2695.191, 'text': "I mean, there's not one.", 'start': 2693.89, 'duration': 1.301}, {'end': 2698.574, 'text': 'You could just tune to various aspects of this distribution.', 'start': 2695.271, 'duration': 3.303}, {'end': 2701.095, 'text': 'You can have little models for all the aspects you care about.', 'start': 2698.594, 'duration': 2.501}, {'end': 2702.296, 'text': 'You can mix and match.', 'start': 2701.135, 'duration': 1.161}, {'end': 2704.077, 'text': "So that's another way to do it.", 'start': 2702.857, 'duration': 1.22}, {'end': 2706.779, 'text': 'So let me tell you about one such a model.', 'start': 2705.238, 'duration': 1.541}, {'end': 2710.822, 'text': "It's a trajectory optimization agent.", 'start': 2709.061, 'duration': 1.761}, {'end': 2713.504, 'text': 'So we take inspiration from motion control theory.', 'start': 2710.882, 'duration': 2.622}, {'end': 2717.487, 'text': 'And we want to plan a good trajectory for the vehicle.', 'start': 2714.264, 'duration': 3.223}, {'end': 2723.01, 'text': 'the agent vehicle, and that satisfies a bunch of constraints and preferences.', 'start': 2718.447, 'duration': 4.563}], 'summary': 'Multiple models can be used to optimize trajectories for a vehicle, based on various aspects and preferences.', 'duration': 30.521, 'max_score': 2692.489, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2692489.jpg'}, {'end': 2816.164, 'src': 'embed', 'start': 2784.002, 'weight': 1, 'content': [{'end': 2785.742, 'text': "Sometimes they're multidimensional.", 'start': 2784.002, 'duration': 1.74}, {'end': 2786.663, 'text': "There's a few parameters.", 'start': 2785.842, 'duration': 0.821}, {'end': 2790.403, 'text': "Typically, we're talking a few dozen parameters or less.", 'start': 2786.683, 'duration': 3.72}, {'end': 2792.964, 'text': 'And you can learn them too.', 'start': 2792.084, 'duration': 0.88}, {'end': 2796.745, 'text': 'So there is a technique called inverse reinforcement learning.', 'start': 2793.424, 'duration': 3.321}, {'end': 2803.854, 'text': "You want to learn these parameters that produce trajectories that come close to the trajectories you've observed in the real world.", 'start': 2798.53, 'duration': 5.324}, {'end': 2810.339, 'text': 'So if you pick a bunch of trajectories that represent certain type of behavior, you want to model the tune your parameters to behave like it.', 'start': 2803.894, 'duration': 6.445}, {'end': 2816.164, 'text': 'Then you want to generate reasonable trajectories, continuous, feasible, that satisfy this.', 'start': 2810.9, 'duration': 5.264}], 'summary': 'Inverse reinforcement learning tunes parameters to generate reasonable trajectories that match observed behavior.', 'duration': 32.162, 'max_score': 2784.002, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2784002.jpg'}, {'end': 2853.199, 'src': 'embed', 'start': 2824.771, 'weight': 3, 'content': [{'end': 2826.633, 'text': 'So this is a complex interactive scenario.', 'start': 2824.771, 'duration': 1.862}, {'end': 2837.229, 'text': "Two vehicles, but you can see on the left is, on the right is the aggressive guy, blue is the agent, red is our vehicle we're testing in simulation.", 'start': 2827.523, 'duration': 9.706}, {'end': 2841.472, 'text': 'And so let me play one more time once this ends.', 'start': 2838.17, 'duration': 3.302}, {'end': 2848.516, 'text': 'Essentially on the left is the conservative driver, on the right is the aggressive driver, and they pass us.', 'start': 2841.532, 'duration': 6.984}, {'end': 2853.199, 'text': 'And they induce very different reactions in our vehicle.', 'start': 2850.938, 'duration': 2.261}], 'summary': 'Simulation involves two vehicles with different driving behaviors inducing reactions in the tested vehicle.', 'duration': 28.428, 'max_score': 2824.771, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2824771.jpg'}, {'end': 3023.124, 'src': 'embed', 'start': 2987.536, 'weight': 5, 'content': [{'end': 2988.937, 'text': 'This is a very interesting space.', 'start': 2987.536, 'duration': 1.401}, {'end': 2996.663, 'text': "Ultimately, I wanted to show you there's many possible agents and they have different utility and they have different number of examples.", 'start': 2989.017, 'duration': 7.646}, {'end': 2997.664, 'text': 'you need to train them with.', 'start': 2996.663, 'duration': 1.001}, {'end': 3003.771, 'text': 'And so one other takeaway I wanted to tell you is smart agents are critical for autonomy at scale.', 'start': 2999.025, 'duration': 4.746}, {'end': 3006.975, 'text': 'This is something I truly believe working in the space.', 'start': 3004.592, 'duration': 2.383}, {'end': 3016.106, 'text': "And this line of direction is exciting and ultimately one of the exciting problems that there's still a lot of interesting progress to be made.", 'start': 3008.036, 'duration': 8.07}, {'end': 3023.124, 'text': 'And why? Well, you have accurate models of human behavior of drivers and pedestrians, and they help you achieve several things.', 'start': 3017.222, 'duration': 5.902}], 'summary': 'Smart agents are critical for autonomy at scale, with accurate models of human behavior enabling progress.', 'duration': 35.588, 'max_score': 2987.536, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2987536.jpg'}, {'end': 3055.341, 'src': 'embed', 'start': 3027.125, 'weight': 4, 'content': [{'end': 3030.186, 'text': "You'll be able to anticipate what others will do better, and that will be helpful.", 'start': 3027.125, 'duration': 3.061}, {'end': 3037.048, 'text': 'Second, you can develop a robust simulation environment with those insights, also very important.', 'start': 3031.086, 'duration': 5.962}, {'end': 3041.63, 'text': 'Third, well, our vehicle is also one more agent in the environment.', 'start': 3038.229, 'duration': 3.401}, {'end': 3045.511, 'text': "It's an agent we have more control than the others, but a lot of these insights apply.", 'start': 3041.67, 'duration': 3.841}, {'end': 3048.737, 'text': 'And so this is very exciting and interesting.', 'start': 3047.096, 'duration': 1.641}, {'end': 3055.341, 'text': 'So I wanted to finish the talk, just maybe as a mental exercise, right?', 'start': 3049.657, 'duration': 5.684}], 'summary': "Anticipate others' actions, develop simulation environment, apply insights to vehicle control.", 'duration': 28.216, 'max_score': 3027.125, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3027125.jpg'}], 'start': 2668.074, 'title': 'Trajectory optimization for autonomous vehicles', 'summary': 'Discusses utilizing trajectory optimization agents for planning vehicle trajectories, incorporating expert design input and inverse reinforcement learning. it also emphasizes the importance of testing autonomous vehicle agents with diverse scenarios to improve decision-making and robust simulation environments.', 'chapters': [{'end': 2824.371, 'start': 2668.074, 'title': 'Trajectory optimization agent', 'summary': 'Discusses using trajectory optimization agents, inspired by motion control theory, to plan vehicle trajectories that satisfy constraints and preferences, utilizing expert design input and inverse reinforcement learning to tune parameters and generate reasonable trajectories.', 'duration': 156.297, 'highlights': ['The chapter discusses using trajectory optimization agents, inspired by motion control theory, to plan vehicle trajectories that satisfy constraints and preferences. The trajectory optimization agent is used to plan a good trajectory for the vehicle that satisfies a bunch of constraints and preferences, taking inspiration from motion control theory.', 'Utilizing expert design input and inverse reinforcement learning to tune parameters and generate reasonable trajectories. Expert design input and inverse reinforcement learning are used to tune the parameters of the trajectory optimization agent to produce trajectories that come close to the observed real-world trajectories.', "The trajectory optimization agent has parameters, typically a few dozen or less, which can be learned using inverse reinforcement learning. The trajectory optimization agent's parameters, typically a few dozen or less, can be learned using inverse reinforcement learning to model and tune the behavior of the trajectories."]}, {'end': 3112.462, 'start': 2824.771, 'title': 'Testing autonomous vehicle agents', 'summary': "Discusses the importance of testing autonomous vehicle agents with different scenarios and behaviors, emphasizing the need for smart agents to achieve better decision-making, anticipation of others' actions, and robust simulation environments.", 'duration': 287.691, 'highlights': ["The importance of testing autonomous vehicle agents with different scenarios and behaviors is emphasized, showcasing the impact of aggressive and conservative drivers on the vehicle's reactions. Impact of aggressive and conservative drivers, testing different scenarios, reactions of the vehicle", 'The need for a menagerie of agents for testing autonomous vehicle systems is highlighted, emphasizing the complexity of modeling agent behavior and the benefits of learning from diverse illustrations. Importance of diverse agents, complexity of modeling agent behavior, benefits of learning from diverse illustrations', "The significance of smart agents for autonomy at scale is emphasized, with a focus on the ability to anticipate others' actions and the development of robust simulation environments. Significance of smart agents, ability to anticipate actions, development of robust simulation environments"]}], 'duration': 444.388, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY2668074.jpg', 'highlights': ['The trajectory optimization agent is used to plan a good trajectory for the vehicle that satisfies a bunch of constraints and preferences, taking inspiration from motion control theory.', 'Utilizing expert design input and inverse reinforcement learning to tune parameters and generate reasonable trajectories.', "The trajectory optimization agent's parameters, typically a few dozen or less, can be learned using inverse reinforcement learning to model and tune the behavior of the trajectories.", "The importance of testing autonomous vehicle agents with different scenarios and behaviors is emphasized, showcasing the impact of aggressive and conservative drivers on the vehicle's reactions.", 'The need for a menagerie of agents for testing autonomous vehicle systems is highlighted, emphasizing the complexity of modeling agent behavior and the benefits of learning from diverse illustrations.', "The significance of smart agents for autonomy at scale is emphasized, with a focus on the ability to anticipate others' actions and the development of robust simulation environments."]}, {'end': 3448.156, 'segs': [{'end': 3177.323, 'src': 'embed', 'start': 3129.73, 'weight': 0, 'content': [{'end': 3137.117, 'text': 'You take your vehicles, we put a bunch of Waymo cars and we drive a long time in that environment with drivers maybe 30 days, maybe more,', 'start': 3129.73, 'duration': 7.387}, {'end': 3137.778, 'text': 'at least that long.', 'start': 3137.117, 'duration': 0.661}, {'end': 3152.597, 'text': 'And you collect all the data, right? And then your system should be able to improve a lot on the data you have collected, right? So, Drive a bunch.', 'start': 3139.179, 'duration': 13.418}, {'end': 3157.118, 'text': "Obviously, you don't want to train the system too much in the real world while it's driving.", 'start': 3153.077, 'duration': 4.041}, {'end': 3161.819, 'text': "But you want to train it after you've collected data about the environment.", 'start': 3157.858, 'duration': 3.961}, {'end': 3164.22, 'text': 'So it needs to be trainable and collected data.', 'start': 3162.299, 'duration': 1.921}, {'end': 3174.422, 'text': "It's very important for a system to be able to quantify or have a notion to elicit from it whether it's incorrect or not confident.", 'start': 3165.76, 'duration': 8.662}, {'end': 3177.323, 'text': 'Because then you can take action.', 'start': 3176.143, 'duration': 1.18}], 'summary': "Waymo cars drive for at least 30 days to collect data, improving system's confidence.", 'duration': 47.593, 'max_score': 3129.73, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3129730.jpg'}, {'end': 3274.593, 'src': 'embed', 'start': 3248.374, 'weight': 2, 'content': [{'end': 3255.017, 'text': "And I think it's interesting to think of systems where you can do reasoning and the representations that these models need to have.", 'start': 3248.374, 'duration': 6.643}, {'end': 3261.309, 'text': 'And last but not least, We need scalable training and testing infrastructure.', 'start': 3256.918, 'duration': 4.391}, {'end': 3264.85, 'text': 'This is part of the fact that I was talking about.', 'start': 3262.709, 'duration': 2.141}, {'end': 3273.313, 'text': "I'm very lucky at Waymo to have wonderful infrastructure and it allows this virtuous cycle to happen.", 'start': 3264.87, 'duration': 8.443}, {'end': 3274.593, 'text': 'Thank you.', 'start': 3274.353, 'duration': 0.24}], 'summary': 'Systems need scalable infrastructure for reasoning and training. waymo has the necessary infrastructure.', 'duration': 26.219, 'max_score': 3248.374, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3248374.jpg'}, {'end': 3381.926, 'src': 'embed', 'start': 3327.241, 'weight': 3, 'content': [{'end': 3329.302, 'text': "There's many ways to do this ultimately.", 'start': 3327.241, 'duration': 2.061}, {'end': 3332.944, 'text': 'So achieving realism in simulator is an open research problem.', 'start': 3329.562, 'duration': 3.382}, {'end': 3343.188, 'text': 'I assume there is a lot of rules that you have to put into a system to be able to trust it.', 'start': 3334.622, 'duration': 8.566}, {'end': 3344.089, 'text': 'you know.', 'start': 3343.188, 'duration': 0.901}, {'end': 3353.015, 'text': "and so how you find the balance between this automatic models like neural network, when you're not quite sure what they would do,", 'start': 3344.089, 'duration': 8.926}, {'end': 3356.017, 'text': "and rules where you're sure, but it's not scalable?", 'start': 3353.015, 'duration': 3.002}, {'end': 3359.835, 'text': 'I mean through lots and lots of testing and analysis, right?', 'start': 3356.713, 'duration': 3.122}, {'end': 3367.178, 'text': 'So you keep keeping track of the performance of your models and you see where they come short, right?', 'start': 3359.935, 'duration': 7.243}, {'end': 3373.642, 'text': 'And then those are the areas you most need expert to complement, right?', 'start': 3367.238, 'duration': 6.404}, {'end': 3375.323, 'text': 'But the balance can change over time, right?', 'start': 3373.662, 'duration': 1.661}, {'end': 3378.804, 'text': "And it's a natural process of evolution, right?", 'start': 3375.363, 'duration': 3.441}, {'end': 3380.285, 'text': 'So, evolving your system as you go.', 'start': 3378.904, 'duration': 1.381}, {'end': 3381.926, 'text': 'I mean generally.', 'start': 3380.305, 'duration': 1.621}], 'summary': 'Balancing automatic models like neural networks with rules to achieve trust and scalability in simulators is an ongoing research problem, which requires continual testing and expert input to evolve the system over time.', 'duration': 54.685, 'max_score': 3327.241, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3327241.jpg'}, {'end': 3455.26, 'src': 'embed', 'start': 3425.461, 'weight': 5, 'content': [{'end': 3430.303, 'text': 'Either directly regress its uncertainty for certain products, or use ensembles of networks,', 'start': 3425.461, 'duration': 4.842}, {'end': 3436.15, 'text': 'or dropout or techniques like this that also provide measure of uncertainty.', 'start': 3430.303, 'duration': 5.847}, {'end': 3440.252, 'text': 'Another way of doing uncertainty is to leverage constraints in the environment.', 'start': 3436.81, 'duration': 3.442}, {'end': 3448.156, 'text': "So if you have temporal sequences, you don't want, for example, objects to appear or disappear,", 'start': 3440.292, 'duration': 7.864}, {'end': 3455.26, 'text': 'or generally unreasonable changes in the environment or inconsistent prediction in your models are good areas to look.', 'start': 3448.156, 'duration': 7.104}], 'summary': 'Manage uncertainty in models using ensembles, dropout, and environmental constraints.', 'duration': 29.799, 'max_score': 3425.461, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3425461.jpg'}], 'start': 3112.922, 'title': 'Scalable self-improvement and training simulator data', 'summary': 'Emphasizes the importance of scalable processes for self-driving systems at waymo, and discusses challenges in achieving realism in simulator training, evolving systems, and quantifying uncertainty in neural networks for training simulator scenarios.', 'chapters': [{'end': 3298.992, 'start': 3112.922, 'title': 'Scalable self-improvement process for systems', 'summary': 'Discusses the importance of scalable processes for self-driving systems, emphasizing the collection of data, the need for trainable and self-updating systems, and the requirement for scalable training and testing infrastructure at waymo.', 'duration': 186.07, 'highlights': ["The importance of scalable processes for self-driving systems is emphasized, particularly in the collection of data to improve the system, with a suggested duration of at least 30 days for data collection. This highlights the critical role of data in system improvement (e.g., Waymo's approach to collecting and utilizing data for system enhancement).", 'The need for systems to be trainable and capable of self-improvement based on collected data is highlighted, along with the significance of being able to quantify and elicit incorrect or not confident data to take necessary actions. This emphasizes the requirement for systems to be adaptable and capable of self-improvement (e.g., discussing the importance of trainable and self-updating systems in the context of self-driving technology).', 'The discussion of scalable training and testing infrastructure, including the emphasis on the virtuous cycle enabled by such infrastructure at Waymo, is highlighted. This points to the importance of scalable infrastructure in facilitating system improvement and development (e.g., the significance of scalable training and testing infrastructure for system enhancement and continuous improvement).']}, {'end': 3448.156, 'start': 3299.292, 'title': 'Training simulator data for real scenarios', 'summary': 'Discusses the challenges of achieving realism in simulator training, the balance between automatic models and rules, evolving systems, and quantifying uncertainty in neural networks for training simulator scenarios.', 'duration': 148.864, 'highlights': ['The challenges of achieving realism in simulator training is an open research problem, with the need for a balance between automatic models like neural networks and rules to be able to trust the system. challenges in achieving realism in simulator training, balance between automatic models and rules', 'Evolving the system over time through testing and analysis to complement areas where models fall short is a natural process, with the system growing as the capabilities in the data sets grow. evolving the system, testing and analysis, system growth with data set capabilities', 'Importance of quantifying uncertainty in predictions made by models, and techniques such as leveraging neural nets to predict their own uncertainty, using ensembles of networks, dropout, or leveraging constraints in the environment to provide measures of uncertainty. importance of quantifying uncertainty, techniques for predicting uncertainty in neural nets']}], 'duration': 335.234, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3112922.jpg', 'highlights': ['The importance of scalable processes for self-driving systems, particularly in data collection for system improvement, with a suggested duration of at least 30 days for data collection.', 'The need for systems to be trainable and capable of self-improvement based on collected data, along with the significance of being able to quantify and elicit incorrect or not confident data to take necessary actions.', 'The discussion of scalable training and testing infrastructure, including the emphasis on the virtuous cycle enabled by such infrastructure at Waymo, highlighting the importance of scalable infrastructure in facilitating system improvement and development.', 'The challenges of achieving realism in simulator training is an open research problem, with the need for a balance between automatic models like neural networks and rules to be able to trust the system.', 'Evolving the system over time through testing and analysis to complement areas where models fall short is a natural process, with the system growing as the capabilities in the data sets grow.', 'Importance of quantifying uncertainty in predictions made by models, and techniques such as leveraging neural nets to predict their own uncertainty, using ensembles of networks, dropout, or leveraging constraints in the environment to provide measures of uncertainty.']}, {'end': 3888.261, 'segs': [{'end': 3475.126, 'src': 'embed', 'start': 3448.156, 'weight': 0, 'content': [{'end': 3455.26, 'text': 'or generally unreasonable changes in the environment or inconsistent prediction in your models are good areas to look.', 'start': 3448.156, 'duration': 7.104}, {'end': 3463.504, 'text': "I'm just wondering do you guys train and deploy different models depending on where the car is driving, like what city??", 'start': 3456.041, 'duration': 7.463}, {'end': 3469.547, 'text': 'Or do you train and deploy a single model that adapts to most scenarios?', 'start': 3464.185, 'duration': 5.362}, {'end': 3475.126, 'text': 'Well, ideally you would have one model that adapts to most scenarios.', 'start': 3471.645, 'duration': 3.481}], 'summary': 'Adapt one model to most scenarios for car driving.', 'duration': 26.97, 'max_score': 3448.156, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3448156.jpg'}, {'end': 3533.31, 'src': 'embed', 'start': 3495.93, 'weight': 1, 'content': [{'end': 3504.083, 'text': "And I imagine I mean there's an advantage in that you're sensing from a vehicle and you kind of know Your sensors are like first person from a vehicle but not from a pedestrian.", 'start': 3495.93, 'duration': 8.153}, {'end': 3505.405, 'text': "And that's correct.", 'start': 3504.784, 'duration': 0.621}, {'end': 3510.453, 'text': 'I mean so if you want to simulate pedestrians far away in an environment right?', 'start': 3505.485, 'duration': 4.968}, {'end': 3518.539, 'text': "and you want to simulate them at very high resolution and you've collected log data, you may not have the detailed data on that pedestrian.", 'start': 3511.593, 'duration': 6.946}, {'end': 3525.304, 'text': 'At the same time, the subtle cues for that pedestrian matter less at that distance as well,', 'start': 3519.639, 'duration': 5.665}, {'end': 3528.867, 'text': "because it's not like you observed them or reacted to them in the first place.", 'start': 3525.304, 'duration': 3.563}, {'end': 3533.31, 'text': 'So there is an interesting question at what fidelity do you need to simulate things?', 'start': 3529.427, 'duration': 3.883}], 'summary': 'Simulating pedestrians at distance may require lower fidelity due to less noticeable cues.', 'duration': 37.38, 'max_score': 3495.93, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3495930.jpg'}, {'end': 3615.387, 'src': 'embed', 'start': 3583.419, 'weight': 3, 'content': [{'end': 3587.24, 'text': "I think I mean, I'm not completely sure.", 'start': 3583.419, 'duration': 3.821}, {'end': 3593.682, 'text': 'I think one thing I would say is it will take a while for self-driving cars to roll out at scale right?', 'start': 3588.2, 'duration': 5.482}, {'end': 3599.043, 'text': 'So this is not a technology that just turn a crank and appears everywhere right?', 'start': 3594.662, 'duration': 4.381}, {'end': 3605.705, 'text': "There's logistics and algorithms and all this tuning and testing needed to make sure it's really safe in the various environments.", 'start': 3599.644, 'duration': 6.061}, {'end': 3607.266, 'text': 'So it will take some time.', 'start': 3606.386, 'duration': 0.88}, {'end': 3615.387, 'text': 'When you were talking about prediction, you mentioned looking at a context and saying if a person or if someone is looking at us,', 'start': 3608.602, 'duration': 6.785}], 'summary': 'Self-driving cars rollout will take time due to logistics, algorithms, and testing for safety.', 'duration': 31.968, 'max_score': 3583.419, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3583419.jpg'}, {'end': 3708.899, 'src': 'embed', 'start': 3676.43, 'weight': 6, 'content': [{'end': 3688.194, 'text': "So if you read Tversky-Kahneman, type one, type two reasoning, we're really good at the instinctive mapping type of tasks, right?", 'start': 3676.43, 'duration': 11.764}, {'end': 3698.338, 'text': 'So like some low to mid to maybe high level perception up to a point, but the reasoning part with neural networks, right?', 'start': 3688.234, 'duration': 10.104}, {'end': 3707.053, 'text': "And generally with models that's a bit less explored and i think it's long term it's fruitful.", 'start': 3698.398, 'duration': 8.655}, {'end': 3708.899, 'text': "that's my personal opinion, right,", 'start': 3707.053, 'duration': 1.846}], 'summary': "Tversky-kahneman's type one, type two reasoning is fruitful with neural networks for instinctive mapping tasks.", 'duration': 32.469, 'max_score': 3676.43, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3676430.jpg'}, {'end': 3784.202, 'src': 'embed', 'start': 3755.914, 'weight': 7, 'content': [{'end': 3761.081, 'text': 'There is a very strong constraint that if you predict the depth and you predict the motion correctly,', 'start': 3755.914, 'duration': 5.167}, {'end': 3763.604, 'text': 'then you can project certain things and they will look good.', 'start': 3761.081, 'duration': 2.523}, {'end': 3765.847, 'text': "And that's a very strong constraint.", 'start': 3764.245, 'duration': 1.602}, {'end': 3766.688, 'text': "That's a consistency.", 'start': 3765.867, 'duration': 0.821}, {'end': 3768.61, 'text': 'You know this about the environment, you expect it.', 'start': 3766.708, 'duration': 1.902}, {'end': 3770.152, 'text': 'This can help train your model.', 'start': 3769.071, 'duration': 1.081}, {'end': 3773.717, 'text': 'And so more of this type of reasoning may be interesting.', 'start': 3771.414, 'duration': 2.303}, {'end': 3784.202, 'text': "You mentioned expert design algorithms and I was wondering from your perspective, also from Waymo's perspective, how important are those, say,", 'start': 3774.329, 'duration': 9.873}], 'summary': "Accurate depth and motion prediction lead to consistency in projections, aiding in model training. expert design algorithms' importance questioned.", 'duration': 28.288, 'max_score': 3755.914, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3755914.jpg'}, {'end': 3823.866, 'src': 'embed', 'start': 3797.674, 'weight': 4, 'content': [{'end': 3804.916, 'text': 'Every now and then you just sprinkle in like here we can try expert-designed algorithms because we actually understand some parts of the problem.', 'start': 3797.674, 'duration': 7.242}, {'end': 3812.219, 'text': 'and I was wondering like, what is really important for the challenges in autonomous driving outside of the field of machine learning?', 'start': 3804.916, 'duration': 7.303}, {'end': 3817.764, 'text': 'I mean generally the problem is you want to be safe in the environment.', 'start': 3813.103, 'duration': 4.661}, {'end': 3823.866, 'text': "That makes it such that you don't want to make errors in perception, prediction and planning right?", 'start': 3818.284, 'duration': 5.582}], 'summary': 'Key challenges in autonomous driving include safety and minimizing errors in perception, prediction, and planning.', 'duration': 26.192, 'max_score': 3797.674, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3797674.jpg'}, {'end': 3877.051, 'src': 'heatmap', 'start': 3833.989, 'weight': 1, 'content': [{'end': 3838.73, 'text': 'And so throughout your stack, with the current state of machine learning, it needs to be complemented.', 'start': 3833.989, 'duration': 4.741}, {'end': 3847.3, 'text': "And so we've carefully done it and I think machine learning as it improves, I think there'll be less and less need to do it.", 'start': 3840.238, 'duration': 7.062}, {'end': 3854.642, 'text': "It's somewhat effort intensive bringing, especially in an evolving system to do that, to have a hybrid system.", 'start': 3848.32, 'duration': 6.322}, {'end': 3863.944, 'text': 'But right now I think this is the main thing that keeps you able to do complex behaviors in some cases,', 'start': 3855.162, 'duration': 8.782}, {'end': 3866.764, 'text': "for which it's very hard to collect data and you still need to handle.", 'start': 3863.944, 'duration': 2.82}, {'end': 3868.925, 'text': "Then it's the right thing to do.", 'start': 3867.144, 'duration': 1.781}, {'end': 3872.307, 'text': "So the way I view it, I'm a machine learning person.", 'start': 3870.005, 'duration': 2.302}, {'end': 3874.049, 'text': 'I like to doing better and better.', 'start': 3872.528, 'duration': 1.521}, {'end': 3877.051, 'text': 'That said, we are not religious and should not be.', 'start': 3874.109, 'duration': 2.942}], 'summary': 'Machine learning complements current stack, reducing need for hybrid systems and enabling complex behaviors.', 'duration': 43.062, 'max_score': 3833.989, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3833989.jpg'}, {'end': 3863.944, 'src': 'embed', 'start': 3840.238, 'weight': 5, 'content': [{'end': 3847.3, 'text': "And so we've carefully done it and I think machine learning as it improves, I think there'll be less and less need to do it.", 'start': 3840.238, 'duration': 7.062}, {'end': 3854.642, 'text': "It's somewhat effort intensive bringing, especially in an evolving system to do that, to have a hybrid system.", 'start': 3848.32, 'duration': 6.322}, {'end': 3863.944, 'text': 'But right now I think this is the main thing that keeps you able to do complex behaviors in some cases,', 'start': 3855.162, 'duration': 8.782}], 'summary': 'Machine learning will reduce the need for effort-intensive hybrid systems.', 'duration': 23.706, 'max_score': 3840.238, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3840238.jpg'}], 'start': 3448.156, 'title': 'Challenges and future prospects of self-driving cars', 'summary': 'Discusses challenges of simulating pedestrians for autonomous vehicles, need for single adaptable model, importance of fidelity in simulation, time required for widespread adoption of self-driving cars, importance of reasoning in deep learning, role of expert-designed algorithms, and necessity of a hybrid system for complex behaviors.', 'chapters': [{'end': 3558.112, 'start': 3448.156, 'title': 'Simulation for autonomous vehicles', 'summary': 'Discusses the challenges of simulating pedestrians for autonomous vehicles and the need for a single adaptable model for different driving scenarios, while also addressing the importance of fidelity in simulation.', 'duration': 109.956, 'highlights': ['The importance of having a single model that adapts to most scenarios for autonomous vehicles.', 'The challenges of simulating pedestrians due to their less constrained behavior and the need to consider the fidelity of simulation in parallel with the attention of models.', "The impact of sensing from a vehicle compared to a pedestrian and the level of realism in simulation needed to parallel the models' attention."]}, {'end': 3888.261, 'start': 3558.893, 'title': 'Future of self-driving cars', 'summary': 'Discusses the challenges and future prospects of self-driving cars, emphasizing the time required for their widespread adoption, the importance of reasoning in deep learning, the role of expert-designed algorithms in complementing machine learning, and the necessity of a hybrid system for complex behaviors.', 'duration': 329.368, 'highlights': ['The widespread adoption of self-driving cars will take time due to the logistics, algorithms, tuning, and testing required to ensure safety in various environments.', 'The concept of reasoning in deep learning is underexplored, and there is potential for long-term fruitful exploration in neural networks and models.', 'Expert-designed algorithms play a significant role in complementing machine learning for autonomous driving, particularly in handling complex behaviors and environments.', 'Weekly supervised learning and the use of consistency and reasoning constraints, such as ego motion versus depth estimation, can help train models for autonomous driving.', 'The necessity of a hybrid system is emphasized to handle complex behaviors in cases where data collection is challenging, until machine learning reaches a state where it minimizes errors across the scope of autonomous driving challenges.']}], 'duration': 440.105, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/Q0nGo2-y0xY/pics/Q0nGo2-y0xY3448156.jpg', 'highlights': ['The importance of having a single model that adapts to most scenarios for autonomous vehicles.', 'The challenges of simulating pedestrians due to their less constrained behavior and the need to consider the fidelity of simulation in parallel with the attention of models.', "The impact of sensing from a vehicle compared to a pedestrian and the level of realism in simulation needed to parallel the models' attention.", 'The widespread adoption of self-driving cars will take time due to the logistics, algorithms, tuning, and testing required to ensure safety in various environments.', 'Expert-designed algorithms play a significant role in complementing machine learning for autonomous driving, particularly in handling complex behaviors and environments.', 'The necessity of a hybrid system is emphasized to handle complex behaviors in cases where data collection is challenging, until machine learning reaches a state where it minimizes errors across the scope of autonomous driving challenges.', 'The concept of reasoning in deep learning is underexplored, and there is potential for long-term fruitful exploration in neural networks and models.', 'Weekly supervised learning and the use of consistency and reasoning constraints, such as ego motion versus depth estimation, can help train models for autonomous driving.']}], 'highlights': ["Waymo has achieved 10+ million miles autonomously, launched a commercial service, and Drago Anguelov's expertise in perception and robotics is highlighted.", 'Waymo has driven 10 million miles on public roads, showcasing extensive testing and data collection.', 'Waymo launched its first commercial service in Phoenix, marking a significant milestone in bringing fully driverless mobility to the public.', 'Drago Anguielov has extensive experience in developing and applying machine learning methods to autonomous vehicle perception, computer vision, and robotics.', 'Perception tasks involve mapping sensory inputs and prior knowledge of the environment to scene representations, encompassing objects, scene semantics, and environmental knowledge.', 'Machine learning is integrated into various systems in Waymo, aiding in perception, decision-making, and handling complex scenarios.', 'Collaboration with Google and DeepMind on perception and reinforcement learning has led to state-of-the-art advancements in semantic segmentation networks, pose estimation, and object detection.', 'The trajectory optimization agent is used to plan a good trajectory for the vehicle that satisfies a bunch of constraints and preferences, taking inspiration from motion control theory.', 'The importance of scalable processes for self-driving systems, particularly in data collection for system improvement, with a suggested duration of at least 30 days for data collection.', 'The importance of having a single model that adapts to most scenarios for autonomous vehicles.', 'The challenges of simulating pedestrians due to their less constrained behavior and the need to consider the fidelity of simulation in parallel with the attention of models.']}