title
Ishan Misra: Self-Supervised Deep Learning in Computer Vision | Lex Fridman Podcast #206

description
Ishan Misra is a research scientist at FAIR working on self-supervised visual learning. Please support this podcast by checking out our sponsors: - Onnit: https://lexfridman.com/onnit to get up to 10% off - The Information: https://theinformation.com/lex to get 75% off first month - Grammarly: https://grammarly.com/lex to get 20% off premium - Athletic Greens: https://athleticgreens.com/lex and use code LEX to get 1 month of fish oil EPISODE LINKS: Ishan's twitter: https://twitter.com/imisra_ Ishan's website: https://imisra.github.io Ishan's FAIR page: https://ai.facebook.com/people/ishan-misra/ PODCAST INFO: Podcast website: https://lexfridman.com/podcast Apple Podcasts: https://apple.co/2lwqZIr Spotify: https://spoti.fi/2nEwCF8 RSS: https://lexfridman.com/feed/podcast/ Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4 Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41 OUTLINE: 0:00 - Introduction 2:27 - Self-supervised learning 11:02 - Self-supervised learning is the dark matter of intelligence 14:54 - Categorization 23:28 - Is computer vision still really hard? 27:12 - Understanding Language 36:51 - Harder to solve: vision or language 43:36 - Contrastive learning & energy-based models 47:37 - Data augmentation 51:57 - Fixed audio spike by lowering sound with pen tool 1:00:10 - Real data vs. augmented data 1:03:54 - Non-contrastive learning energy based self supervised learning methods 1:07:32 - Unsupervised learning (SwAV) 1:10:14 - Self-supervised Pretraining (SEER) 1:15:21 - Self-supervised learning (SSL) architectures 1:21:21 - VISSL pytorch-based SSL library 1:24:15 - Multi-modal 1:31:43 - Active learning 1:37:22 - Autonomous driving 1:48:49 - Limits of deep learning 1:52:57 - Difference between learning and reasoning 1:58:03 - Building super-human AI 2:05:51 - Most beautiful idea in self-supervised learning 2:09:40 - Simulation for training AI 2:13:04 - Video games replacing reality 2:14:18 - How to write a good research paper 2:18:45 - Best programming language for beginners 2:19:39 - PyTorch vs TensorFlow 2:23:03 - Advice for getting into machine learning 2:25:09 - Advice for young people 2:27:35 - Meaning of life SOCIAL: - Twitter: https://twitter.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Facebook: https://www.facebook.com/lexfridman - Instagram: https://www.instagram.com/lexfridman - Medium: https://medium.com/@lexfridman - Reddit: https://reddit.com/r/lexfridman - Support on Patreon: https://www.patreon.com/lexfridman

detail
{'title': 'Ishan Misra: Self-Supervised Deep Learning in Computer Vision | Lex Fridman Podcast #206', 'heatmap': [], 'summary': "Ishan mizra's work focuses on self-supervised machine learning in computer vision, aiming for gpt-3-like success in language models and leaving a robot to become smarter after watching youtube videos. the discussions cover the emergence of self-supervised learning, challenges in vision and language learning, breakthroughs in dialogue understanding, non-contrastive learning methods, self-supervised learning advancements, nlp, ai, autonomous driving, ai integration challenges and limits, and writing good papers, phd research, pytorch vs tensorflow, and embracing failure.", 'chapters': [{'end': 127.327, 'segs': [{'end': 43.07, 'src': 'embed', 'start': 0.129, 'weight': 0, 'content': [{'end': 5.917, 'text': 'The following is a conversation with Ishan Mizra, research scientist at Facebook AI Research,', 'start': 0.129, 'duration': 5.788}, {'end': 12.226, 'text': 'who works on self-supervised machine learning in the domain of computer vision, or in other words,', 'start': 5.917, 'duration': 6.309}, {'end': 17.513, 'text': 'making AI systems understand the visual world with minimal help from us humans.', 'start': 12.226, 'duration': 5.287}, {'end': 28.14, 'text': "Transformers and self-attention has been successfully used by OpenAI's GPT-3 and other language models to do self-supervised learning in the domain of language.", 'start': 18.113, 'duration': 10.027}, {'end': 35.825, 'text': 'Ishan, together with Yann LeCun and others, is trying to achieve the same success in the domain of images and video.', 'start': 28.76, 'duration': 7.065}, {'end': 43.07, 'text': 'The goal is to leave a robot watching YouTube videos all night and in the morning come back to a much smarter robot.', 'start': 36.425, 'duration': 6.645}], 'summary': 'Ishan mizra is working on self-supervised machine learning in computer vision to make ai systems understand the visual world with minimal human help, aiming to achieve success in images and video like gpt-3 in language models.', 'duration': 42.941, 'max_score': 0.129, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI129.jpg'}], 'start': 0.129, 'title': 'Self-supervised learning in computer vision', 'summary': "Discusses ishan mizra's work on self-supervised machine learning in computer vision, aiming to achieve success similar to gpt-3 in language models and leaving a robot to become smarter after watching youtube videos.", 'chapters': [{'end': 127.327, 'start': 0.129, 'title': 'Self-supervised learning in computer vision', 'summary': "Discusses ishan mizra's work on self-supervised machine learning in computer vision, aiming to achieve success similar to gpt-3 in language models and leaving a robot to become smarter after watching youtube videos.", 'duration': 127.198, 'highlights': ['Ishan Mizra, along with Yann LeCun and others, aims to achieve success in self-supervised learning for images and video using transformers and self-attention, similar to GPT-3 in language models.', 'The goal is to leave a robot watching YouTube videos all night and come back to a much smarter robot in the morning.', 'The podcast aims to have conversations with world-class researchers in AI, math, physics, biology, and other sciences, as well as historians, musicians, athletes, and comedians.', 'The podcast used to be called Artificial Intelligence Podcast and is now trying to have more fun by increasing the frequency to three times a week.', 'The chapter mentions quick sponsors such as Onnit, The Information, Grammarly, and Athletic Greens.', "The conversation includes a challenge to the listener to count the number of times the word 'banana' is mentioned."]}], 'duration': 127.198, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI129.jpg', 'highlights': ['Ishan Mizra, along with Yann LeCun, aims to achieve success in self-supervised learning for images and video using transformers and self-attention, similar to GPT-3 in language models.', 'The goal is to leave a robot watching YouTube videos all night and come back to a much smarter robot in the morning.', 'The podcast aims to have conversations with world-class researchers in AI, math, physics, biology, and other sciences, as well as historians, musicians, athletes, and comedians.', 'The podcast used to be called Artificial Intelligence Podcast and is now trying to have more fun by increasing the frequency to three times a week.', 'The chapter mentions quick sponsors such as Onnit, The Information, Grammarly, and Athletic Greens.', "The conversation includes a challenge to the listener to count the number of times the word 'banana' is mentioned."]}, {'end': 1271.484, 'segs': [{'end': 222.981, 'src': 'embed', 'start': 194.85, 'weight': 6, 'content': [{'end': 197.892, 'text': "So it's taking this input of the data and then trying to mimic the output.", 'start': 194.85, 'duration': 3.042}, {'end': 201.995, 'text': 'So it looks at an image and the human has tagged that this image contains a banana.', 'start': 198.432, 'duration': 3.563}, {'end': 204.657, 'text': 'And now the system is basically trying to mimic that.', 'start': 202.455, 'duration': 2.202}, {'end': 205.918, 'text': "So that's its learning signal.", 'start': 204.677, 'duration': 1.241}, {'end': 213.299, 'text': 'And so for supervised learning, we try to gather lots of such data and we train these machine learning models to imitate the input output.', 'start': 206.738, 'duration': 6.561}, {'end': 221.021, 'text': 'And the hope is basically by doing so now on unseen or like new kinds of data, this model can automatically learn to predict these concepts.', 'start': 213.499, 'duration': 7.522}, {'end': 222.981, 'text': 'So this is a standard sort of supervised setting.', 'start': 221.321, 'duration': 1.66}], 'summary': 'Supervised learning trains models to imitate input-output data for predicting concepts.', 'duration': 28.131, 'max_score': 194.85, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI194850.jpg'}, {'end': 292.47, 'src': 'embed', 'start': 253.835, 'weight': 5, 'content': [{'end': 257.315, 'text': "So these concepts are basically just nouns and they're annotated on images.", 'start': 253.835, 'duration': 3.48}, {'end': 260.798, 'text': 'This entire dataset was a mammoth data collection effort.', 'start': 258.497, 'duration': 2.301}, {'end': 263.739, 'text': 'It actually gave rise to a lot of powerful learning algorithms.', 'start': 260.838, 'duration': 2.901}, {'end': 266.46, 'text': "It's credited with the rise of deep learning as well.", 'start': 263.819, 'duration': 2.641}, {'end': 271.181, 'text': 'But this dataset took about 22 human years to collect, to annotate.', 'start': 267.24, 'duration': 3.941}, {'end': 273.442, 'text': "It's not even that many concepts.", 'start': 272.061, 'duration': 1.381}, {'end': 274.502, 'text': "It's not even that many images.", 'start': 273.482, 'duration': 1.02}, {'end': 275.902, 'text': '14 million is nothing really.', 'start': 274.842, 'duration': 1.06}, {'end': 283.165, 'text': 'You have about I think 400 million images or so or even more than that uploaded to most of the popular social media websites today.', 'start': 277.023, 'duration': 6.142}, {'end': 286.306, 'text': "So now supervised learning just doesn't scale.", 'start': 284.225, 'duration': 2.081}, {'end': 292.47, 'text': "If I want to now annotate more concepts, if I want to have various types of fine-grained concepts, then it won't really scale.", 'start': 286.446, 'duration': 6.024}], 'summary': 'Mammoth dataset, 14m images, credited with rise of deep learning, but not scalable for fine-grained concepts.', 'duration': 38.635, 'max_score': 253.835, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI253835.jpg'}, {'end': 670.48, 'src': 'embed', 'start': 645.688, 'weight': 8, 'content': [{'end': 652.993, 'text': "And then in the language domain or anything that has sequences like language or something that's like a time series,", 'start': 645.688, 'duration': 7.305}, {'end': 655.094, 'text': 'then you can chop up parts in time.', 'start': 652.993, 'duration': 2.101}, {'end': 661.418, 'text': "It's similar to the story of RNNs and CNNs, of RNNs and covenants.", 'start': 655.494, 'duration': 5.924}, {'end': 670.48, 'text': 'You and Jan LeCun wrote the blog post in March 2021 titled Self-Supervised Learning, The Dark Matter of Intelligence.', 'start': 662.398, 'duration': 8.082}], 'summary': 'Discussion on sequence data analysis and reference to a blog post by yann lecun on self-supervised learning in march 2021.', 'duration': 24.792, 'max_score': 645.688, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI645688.jpg'}, {'end': 859.286, 'src': 'embed', 'start': 834.924, 'weight': 0, 'content': [{'end': 843.291, 'text': 'So let me reframe it then that human supervision cannot be at large scale, the source of the solution to intelligence.', 'start': 834.924, 'duration': 8.367}, {'end': 849.757, 'text': 'Right So there has, we, the machines have to discover the supervision in the natural signal of the world.', 'start': 843.431, 'duration': 6.326}, {'end': 854.101, 'text': 'Right I mean, the other thing is also that humans are not particularly good labellers.', 'start': 849.837, 'duration': 4.264}, {'end': 855.182, 'text': "They're not very consistent.", 'start': 854.121, 'duration': 1.061}, {'end': 859.286, 'text': "For example, like what's the difference between a dining table and a table??", 'start': 856.023, 'duration': 3.263}], 'summary': 'Human supervision at large scale is not source of intelligence; machines must discover it in natural world signals.', 'duration': 24.362, 'max_score': 834.924, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI834924.jpg'}], 'start': 128.067, 'title': 'Self-supervised learning and challenges in machine learning', 'summary': 'Discusses the emergence of self-supervised learning as a better alternative to supervised learning in computer vision and nlp, highlighting its scalability and importance in learning common sense. it also addresses the challenges in machine learning, including the limitations of human supervision and the significance of similarity in understanding.', 'chapters': [{'end': 541.423, 'start': 128.067, 'title': 'Self-supervised learning in computer vision', 'summary': 'Discusses the limitations of supervised learning in computer vision, the emergence of self-supervised learning as a better alternative, and provides insights into self-supervised learning with examples from various domains.', 'duration': 413.356, 'highlights': ['Self-supervised learning uses the data itself as a source of supervision, making it a more scalable and efficient approach compared to supervised learning, which relies on human-labeled data.', 'Supervised learning, although effective, does not scale well, as evidenced by the massive effort needed to collect and annotate the ImageNet dataset, highlighting the need for alternative learning paradigms like semi-supervised and self-supervised learning.', 'Examples of self-supervised learning applications include NLP models predicting masked out words in sentences and video prediction models learning about the structure of the world without the need for explicit human annotations.', "The term 'self-supervised' is preferred over 'unsupervised' to explicitly indicate that the data itself serves as the source of supervision, a concept endorsed by prominent figures in the field such as Yann LeCun, Virginia de Sars, and Jitendra Malik."]}, {'end': 768.714, 'start': 542.044, 'title': 'Self-supervised learning', 'summary': 'Discusses the concept of self-supervised learning, with a focus on its applications in nlp and computer vision, emphasizing the importance of learning common sense about the world and its scalability compared to supervised learning.', 'duration': 226.67, 'highlights': ['Self-supervised learning is a powerful way to learn common sense about the world or stuff that is hard to label, as supervised learning is not scalable for tasks such as determining the weight of an object. The concept of self-supervised learning is highlighted as a powerful approach to learn common sense about the world and handle tasks that are hard to label, emphasizing the scalability issue of supervised learning for tasks like determining the weight of an object.', 'The widely used tricks for self-supervised learning include using consistency inherent to physical reality, such as in NLP and computer vision, where sequences and image crops are leveraged to generate self-supervision signals. The discussion emphasizes the widely used tricks for self-supervised learning, including leveraging the consistency inherent to physical reality in NLP and computer vision by using sequences and image crops to generate self-supervision signals.', "The blog post 'Self-Supervised Learning, The Dark Matter of Intelligence' highlights the potential importance of self-supervised learning for future machine learning algorithms and its role in developing intelligence. The significance of self-supervised learning for future machine learning algorithms and its role in developing intelligence is highlighted, as discussed in the blog post titled 'Self-Supervised Learning, The Dark Matter of Intelligence.'"]}, {'end': 993.215, 'start': 768.714, 'title': 'Challenges in machine learning', 'summary': 'Discusses the challenges in machine learning, including the need for self-supervised learning, human supervision limitations, and the difficulty in creating a perfect taxonomy for objects, highlighting the questions surrounding interactivity, reasoning, and categorization in machine learning.', 'duration': 224.501, 'highlights': ['The need for self-supervised learning and the questions regarding interactivity, reward signals, and reasoning in machine learning. The chapter raises questions about the self-supervised learning process, including the level of interactivity, reward signals, and reasoning involved, highlighting the open challenges in these areas.', 'Limitations of human supervision and the inconsistency in labeling objects, leading to the need for machines to discover supervision in natural signals and the potential confusion in specifying end goals. It discusses the limitations of human supervision, emphasizing the inconsistency in labeling objects and the challenge of specifying end goals without confusing the machine, highlighting the need for machines to discover supervision in natural signals.', 'The difficulty in creating a perfect taxonomy for objects and the compositional nature of categorization, leading to the impossibility of constructing a flawless taxonomy. It explores the challenges in creating a perfect taxonomy for objects, emphasizing the compositional nature of categorization and the impossibility of constructing a flawless taxonomy, highlighting the limitations in this pursuit.']}, {'end': 1271.484, 'start': 993.956, 'title': 'The power of similarity in understanding', 'summary': 'Explores the concept of similarity between objects and how it can lead to understanding, emphasizing the importance of similarity in learning and reasoning, and questioning the limitations of categorization and supervised learning.', 'duration': 277.528, 'highlights': ['The concept of similarity between objects is crucial in learning and reasoning, enabling individuals to relate new experiences to familiar ones and understand how to use them effectively. Importance of similarity in learning and reasoning', 'Questioning the limitations of categorization and supervised learning, emphasizing the practical challenges and inefficiency of manual annotation in the context of object recognition tasks. Challenges and inefficiency of manual annotation', 'Highlighting the difficulty in discrete categorization due to the presence of both similarity and dissimilarity among objects, and its impact on understanding various concepts. Difficulty in discrete categorization']}], 'duration': 1143.417, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI128067.jpg', 'highlights': ['Self-supervised learning uses the data itself as a source of supervision, making it a more scalable and efficient approach compared to supervised learning.', "The term 'self-supervised' is preferred over 'unsupervised' to explicitly indicate that the data itself serves as the source of supervision.", 'Examples of self-supervised learning applications include NLP models predicting masked out words in sentences and video prediction models learning about the structure of the world without the need for explicit human annotations.', 'The widely used tricks for self-supervised learning include using consistency inherent to physical reality, such as in NLP and computer vision, where sequences and image crops are leveraged to generate self-supervision signals.', "The blog post 'Self-Supervised Learning, The Dark Matter of Intelligence' highlights the potential importance of self-supervised learning for future machine learning algorithms and its role in developing intelligence.", 'The need for self-supervised learning and the questions regarding interactivity, reward signals, and reasoning in machine learning.', 'Limitations of human supervision and the inconsistency in labeling objects, leading to the need for machines to discover supervision in natural signals and the potential confusion in specifying end goals.', 'The concept of similarity between objects is crucial in learning and reasoning, enabling individuals to relate new experiences to familiar ones and understand how to use them effectively.', 'Questioning the limitations of categorization and supervised learning, emphasizing the practical challenges and inefficiency of manual annotation in the context of object recognition tasks.', 'Highlighting the difficulty in discrete categorization due to the presence of both similarity and dissimilarity among objects, and its impact on understanding various concepts.']}, {'end': 2596.279, 'segs': [{'end': 2000.978, 'src': 'embed', 'start': 1973.659, 'weight': 1, 'content': [{'end': 1976.94, 'text': 'how poor we are at predicting how successful a particular technique is going to be.', 'start': 1973.659, 'duration': 3.281}, {'end': 1982.222, 'text': 'So I think I can say something now, but like 10 years from now, I look completely stupid basically predicting this.', 'start': 1977.381, 'duration': 4.841}, {'end': 1984.263, 'text': 'In the language domain.', 'start': 1982.902, 'duration': 1.361}, {'end': 1994.006, 'text': 'is there something in your work that you find useful and insightful and transferable to computer vision, but also just,', 'start': 1984.263, 'duration': 9.743}, {'end': 1997.567, 'text': "I don't know beautiful and profound that I think carries through to the vision domain?", 'start': 1994.006, 'duration': 3.561}, {'end': 2000.978, 'text': 'The idea of masking has been very powerful.', 'start': 1998.457, 'duration': 2.521}], 'summary': 'Predicting technique success is challenging; masking is a powerful idea.', 'duration': 27.319, 'max_score': 1973.659, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI1973659.jpg'}, {'end': 2183.859, 'src': 'embed', 'start': 2158.913, 'weight': 0, 'content': [{'end': 2164.217, 'text': 'which I looked at when I was a PhD student, where he would basically have a blob of pixels and he would ask you hey, what is this??', 'start': 2158.913, 'duration': 5.304}, {'end': 2168.821, 'text': 'And it looked basically like a shoe, or like it could look like a TV remote.', 'start': 2164.978, 'duration': 3.843}, {'end': 2169.761, 'text': 'It could look like anything.', 'start': 2168.861, 'duration': 0.9}, {'end': 2171.383, 'text': 'And it turns out it was a beer bottle.', 'start': 2170.062, 'duration': 1.321}, {'end': 2177.017, 'text': "but I'm not sure it was one of these three things, but basically he showed you the full picture and then it was very obvious what it was.", 'start': 2172.295, 'duration': 4.722}, {'end': 2181.258, 'text': "But the point is just by looking at that particular local window, you couldn't figure it out.", 'start': 2177.517, 'duration': 3.741}, {'end': 2183.859, 'text': 'Because of resolution, because of other things.', 'start': 2181.839, 'duration': 2.02}], 'summary': "An example from a phd student's research shows how a pixel blob of a beer bottle could be mistaken for other objects due to resolution and other factors.", 'duration': 24.946, 'max_score': 2158.913, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI2158913.jpg'}, {'end': 2386.724, 'src': 'embed', 'start': 2362.827, 'weight': 4, 'content': [{'end': 2370.273, 'text': 'And so the thing is for NLP, it has been really successful because we are very good at predicting, doing this distribution over a finite set.', 'start': 2362.827, 'duration': 7.446}, {'end': 2375.076, 'text': 'And the problem is, when this set becomes really large, we are going to become really,', 'start': 2370.993, 'duration': 4.083}, {'end': 2380.26, 'text': 'really bad at making these predictions and at solving basically this particular set of problems.', 'start': 2375.076, 'duration': 5.184}, {'end': 2386.724, 'text': 'So if you were to do it exactly in the same way as NLP for vision, there is very limited success.', 'start': 2381.101, 'duration': 5.623}], 'summary': 'Nlp successful for finite set, limited for large set like vision.', 'duration': 23.897, 'max_score': 2362.827, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI2362827.jpg'}, {'end': 2536.347, 'src': 'embed', 'start': 2512.836, 'weight': 3, 'content': [{'end': 2522.201, 'text': "So that's, it's similar, but different than the computer vision problem is in the 2D plane is a projection with three-dimensional world.", 'start': 2512.836, 'duration': 9.365}, {'end': 2525.423, 'text': 'So perhaps there are similar problems there.', 'start': 2522.721, 'duration': 2.702}, {'end': 2528.464, 'text': "Maybe this- I mean, I think what I'm saying is NLP is not easy.", 'start': 2525.683, 'duration': 2.781}, {'end': 2529.485, 'text': "Of course, don't get me wrong.", 'start': 2528.624, 'duration': 0.861}, {'end': 2536.347, 'text': 'abstract thought expressed in knowledge, or knowledge basically expressed in language, is really hard to understand, right?', 'start': 2530.485, 'duration': 5.862}], 'summary': 'Nlp and computer vision present complex challenges in understanding abstract knowledge and 3d projections.', 'duration': 23.511, 'max_score': 2512.836, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI2512836.jpg'}], 'start': 1272.785, 'title': 'Challenges in vision and language learning', 'summary': 'Explores challenges in computer vision and self-supervised learning, emphasizing the significance of human understanding in machine communication. it also delves into the power of masking in language and vision, highlighting its impact on the development of nlp techniques and its application in computer vision.', 'chapters': [{'end': 1407.839, 'start': 1272.785, 'title': 'Challenges in self-supervised learning', 'summary': 'Discusses the challenges and philosophical questions related to annotation and self-supervised learning in computer vision, questioning the value and purpose of drawing precise boundaries around objects and the potential limitations of self-supervised learning.', 'duration': 135.054, 'highlights': ['The philosophical questions and challenges of annotation and self-supervised learning are discussed, raising doubts about the value and purpose of drawing precise boundaries around objects (e.g., semantic segmentation) and the potential limitations of self-supervised learning.', 'The speaker questions the fundamental concepts of what constitutes an object in computer vision, highlighting the existential crisis faced when trying to understand and interpret 2D representations of 3D objects.', 'The importance of understanding the foundational concepts and common sense base in order to reason and interpret the three-dimensional and four-dimensional world is emphasized, suggesting the need for a deeper understanding beyond self-supervised learning.']}, {'end': 1745.155, 'start': 1408.579, 'title': 'Challenges in computer vision and self-supervised learning', 'summary': 'Discusses the challenges in computer vision, the potential role of self-supervised learning, and the significance of human understanding in machine communication, emphasizing the difficulty of achieving human-level understanding in computer vision.', 'duration': 336.576, 'highlights': ['The difficulty of achieving human-level understanding in computer vision The chapter emphasizes the difficulty of achieving human-level understanding in computer vision', 'The potential role of self-supervised learning in machine communication and understanding The chapter discusses the potential role of self-supervised learning in machine communication and understanding', 'The challenges in computer vision and the need for human understanding in machine communication The chapter discusses the challenges in computer vision and emphasizes the need for human understanding in machine communication', 'The concept of distributional hypothesis in natural language processing and its application in understanding word relationships The chapter explains the distributional hypothesis in NLP and its application in understanding word relationships', 'The history of success of self-supervised learning in natural language processing and language modeling The chapter briefly touches on the history of success of self-supervised learning in NLP and language modeling']}, {'end': 2188.841, 'start': 1745.815, 'title': 'The power of masking in language and vision', 'summary': 'Discusses the effectiveness of the masking process in language and vision, highlighting its role in self-supervised learning, its impact on the development of nlp techniques like word2vec and bert, and its application in computer vision through the transformer model and its self-attention mechanism.', 'duration': 443.026, 'highlights': ['The masking process in language has been integral to the development of NLP techniques like Word2vec and BERT, enabling tasks such as entailment prediction and similarity assessment. Impact on NLP techniques, entailment prediction, and similarity assessment', 'The concept of masking has also been successfully applied in computer vision through the transformer model and its self-attention mechanism, allowing for a broader understanding of context and patterns within images. Application in computer vision, transformer model, self-attention mechanism', 'The potential for leveraging additional self-supervised signals and tricks beyond the masking process in both language and vision domains is a key area of interest for further advancements. Exploration of additional self-supervised signals and tricks, potential for further advancements']}, {'end': 2596.279, 'start': 2189.682, 'title': 'Vision vs language intelligence', 'summary': 'Discusses the challenges in computer vision and natural language processing, highlighting the fundamental differences in structure, success, and prediction problems between the two domains, concluding that achieving impressive performance in language tasks is slightly easier than in computer vision.', 'duration': 406.597, 'highlights': ['The success in natural language processing (NLP) has been much higher than in computer vision due to the structured nature of language and the ease of predicting over a finite vocabulary. Higher success in NLP compared to computer vision.', 'The challenge in computer vision lies in the combinatorially large prediction problems, making it intractable to predict and solve compared to NLP, which excels in predicting over a finite set. Challenges in computer vision due to combinatorially large prediction problems.', 'The distributional hypothesis in NLP, where the context supplies meaning to the word, is much more structured and easier to comprehend compared to the variability and complexity of capturing images in computer vision. Difference in structured meaning between language and vision.', 'Achieving impressive performance in language tasks is slightly easier than in computer vision due to the fundamental differences in challenges and structure between the two domains. Impressive performance easier to achieve in language tasks compared to computer vision.']}], 'duration': 1323.494, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI1272785.jpg', 'highlights': ['The importance of understanding foundational concepts and common sense base to reason and interpret the three-dimensional and four-dimensional world is emphasized, suggesting the need for a deeper understanding beyond self-supervised learning.', 'The concept of distributional hypothesis in natural language processing and its application in understanding word relationships is explained.', 'The masking process in language has been integral to the development of NLP techniques like Word2vec and BERT, enabling tasks such as entailment prediction and similarity assessment.', 'The challenges in computer vision and the need for human understanding in machine communication are discussed.', 'The potential for leveraging additional self-supervised signals and tricks beyond the masking process in both language and vision domains is a key area of interest for further advancements.', 'The difficulty of achieving human-level understanding in computer vision is emphasized.', 'The success in natural language processing (NLP) has been much higher than in computer vision due to the structured nature of language and the ease of predicting over a finite vocabulary.', 'The challenge in computer vision lies in the combinatorially large prediction problems, making it intractable to predict and solve compared to NLP, which excels in predicting over a finite set.']}, {'end': 3817.377, 'segs': [{'end': 2875.745, 'src': 'embed', 'start': 2847.058, 'weight': 0, 'content': [{'end': 2848.759, 'text': "It's also used in supervised learning.", 'start': 2847.058, 'duration': 1.701}, {'end': 2854.603, 'text': 'The thing is because we are not really using labels to get these positive or negative pairs.', 'start': 2849.82, 'duration': 4.783}, {'end': 2856.845, 'text': 'it can basically also be used for self-supervised learning.', 'start': 2854.603, 'duration': 2.242}, {'end': 2864.17, 'text': 'You mentioned one of the ideas in the vision context that works is to have different crops.', 'start': 2857.686, 'duration': 6.484}, {'end': 2872.244, 'text': 'So you could think of that as a way to sort of manipulating the data to generate examples that are similar.', 'start': 2865.302, 'duration': 6.942}, {'end': 2875.745, 'text': "Obviously there's a bunch of other techniques.", 'start': 2873.425, 'duration': 2.32}], 'summary': 'Unsupervised learning can be used for self-supervised learning by manipulating data to generate similar examples.', 'duration': 28.687, 'max_score': 2847.058, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI2847058.jpg'}, {'end': 3085.018, 'src': 'embed', 'start': 3063.227, 'weight': 1, 'content': [{'end': 3072.492, 'text': 'Like. in order for us to place a concept in its proper place, We have to basically crop it in all kinds of ways,', 'start': 3063.227, 'duration': 9.265}, {'end': 3077.374, 'text': 'do basic data augmentation on it in whatever very clever ways that the brain likes to do.', 'start': 3072.492, 'duration': 4.882}, {'end': 3082.757, 'text': 'Right Like spinning around in our mind somehow that is very effective.', 'start': 3077.654, 'duration': 5.103}, {'end': 3085.018, 'text': 'So I think for some of them, we like need to do it.', 'start': 3083.097, 'duration': 1.921}], 'summary': 'Concepts need to be processed and augmented for effective understanding and retention by the brain.', 'duration': 21.791, 'max_score': 3063.227, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI3063227.jpg'}], 'start': 2596.979, 'title': 'Breakthroughs in dialogue understanding, contrastive learning, and data augmentation in visual intelligence', 'summary': 'Emphasizes the necessity of breakthroughs in dialogue understanding and computer vision, discusses contrastive learning in self-supervised learning, and highlights the importance of imaginative and diverse data augmentation in visual intelligence, with a focus on utilizing energy-based models and parameterized techniques for achieving big improvements in machine learning.', 'chapters': [{'end': 2642.379, 'start': 2596.979, 'title': 'Breakthroughs in dialogue understanding', 'summary': 'Discusses the necessity of breakthroughs in dialogue understanding and computer vision for achieving impressive performance, as well as the concepts of contrastive learning and energy-based models.', 'duration': 45.4, 'highlights': ['The necessity of breakthroughs in dialogue understanding and computer vision for achieving impressive performance', 'Explanation of contrastive learning and the concept of learning an embedding space by contrasting samples']}, {'end': 3038.019, 'start': 2642.899, 'title': 'Contrastive learning for self-supervised learning', 'summary': 'Discusses the concept of contrastive learning in self-supervised learning, emphasizing the use of energy-based models and data augmentation for creating positive and negative pairs to enforce similarity and dissimilarity between images, thereby revealing the commonalities between different modern learning methods.', 'duration': 395.12, 'highlights': ['Energy-based models and their role in explaining modern learning methods like contrastive models, GANs, and VAEs. Jan explains the concept of energy-based models, which can be used to explain contrastive models, GANs, and VAEs, providing a common language for understanding different modern learning methods.', 'The use of contrastive learning and the concept of positive and negative pairs to enforce similarity and dissimilarity between images. Contrastive learning is used to enforce similarity between positive pairs of images while pushing them away from negative pairs, thereby revealing the commonalities between different modern learning methods.', 'The fundamental role of data augmentation in creating perturbations of images to enforce similarity between different crops. Data augmentation plays a fundamental role in creating perturbations of images to enforce similarity between different crops, thereby contributing to the effectiveness of contrastive learning in self-supervised learning.']}, {'end': 3303.799, 'start': 3038.973, 'title': 'Imaginative data augmentation for visual intelligence', 'summary': 'Discusses the importance of imaginative and diverse data augmentation in visual intelligence, questioning the potential benefits of wild imagination in data augmentation and the need for more intelligent and parameterized data augmentation techniques to achieve big improvements in machine learning.', 'duration': 264.826, 'highlights': ['The importance of imaginative and diverse data augmentation Imagination and diverse perspectives play a crucial role in understanding concepts, prompting the need for imaginative and diverse data augmentation techniques.', 'Questioning the potential benefits of wild imagination in data augmentation The discussion raises the question of whether a wild imagination could be beneficial for networks and data augmentation, emphasizing the need for imaginative and diverse approaches in the process.', 'The need for more intelligent and parameterized data augmentation techniques The chapter emphasizes the necessity for more intelligent and parameterized data augmentation techniques to achieve significant improvements in machine learning, highlighting the potential for learning-based data augmentation.']}, {'end': 3817.377, 'start': 3305.36, 'title': 'Data augmentation in self-supervised learning', 'summary': 'Discusses the importance of data augmentation in self-supervised learning, emphasizing the need for realistic and domain-specific augmentation, potential integration of augmentation into the learning process, and the reliance on good augmentation algorithms over large natural datasets for effective learning.', 'duration': 512.017, 'highlights': ['The reliance on good data augmentation algorithms is emphasized over large natural datasets for effective learning. The speaker highlights the importance of good data augmentation algorithms, stating that even with an infinite source of image data, a good data augmentation algorithm would be more useful for learning.', 'The speaker emphasizes the need for realistic and domain-specific data augmentation techniques. The speaker stresses the importance of realistic and domain-specific data augmentation, stating that subtle but realistic augmentation can provide significant benefits, especially in specific domains like medical imaging.', 'The potential integration of data augmentation into the learning process is discussed, leading to a semi-supervised learning setting. The potential integration of data augmentation into the learning process is discussed, highlighting the shift towards a semi-supervised learning setting, where understanding the end goal and incorporating relevant augmentation could significantly improve learning outcomes.']}], 'duration': 1220.398, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI2596979.jpg', 'highlights': ['The necessity of breakthroughs in dialogue understanding and computer vision for achieving impressive performance', 'Energy-based models and their role in explaining modern learning methods like contrastive models, GANs, and VAEs', 'The importance of imaginative and diverse data augmentation', 'The reliance on good data augmentation algorithms is emphasized over large natural datasets for effective learning']}, {'end': 4502.319, 'segs': [{'end': 3988.027, 'src': 'embed', 'start': 3959.971, 'weight': 1, 'content': [{'end': 3965.554, 'text': "And now all you're doing is basically saying that the features produced by the teacher network and the student network should be very similar.", 'start': 3959.971, 'duration': 5.583}, {'end': 3966.435, 'text': "That's it.", 'start': 3966.155, 'duration': 0.28}, {'end': 3968.656, 'text': 'There is no notion of a negative anymore.', 'start': 3966.675, 'duration': 1.981}, {'end': 3969.657, 'text': "And that's it.", 'start': 3969.256, 'duration': 0.401}, {'end': 3972.799, 'text': "So it's all about similarity maximization between these two features.", 'start': 3969.677, 'duration': 3.122}, {'end': 3980.202, 'text': 'And so all I need to now do is figure out how to have these two sorts of parallel networks, a student network and a teacher network.', 'start': 3973.737, 'duration': 6.465}, {'end': 3984.184, 'text': 'And basically researchers have figured out very cheap methods to do this.', 'start': 3980.702, 'duration': 3.482}, {'end': 3988.027, 'text': 'So you can actually have for free, really two types of neural networks.', 'start': 3984.204, 'duration': 3.823}], 'summary': 'Using cheap methods, researchers create similar features in teacher and student networks for free.', 'duration': 28.056, 'max_score': 3959.971, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI3959971.jpg'}, {'end': 4023.278, 'src': 'embed', 'start': 3994.051, 'weight': 2, 'content': [{'end': 3997.213, 'text': 'So you can ensure that they always remain different enough.', 'start': 3994.051, 'duration': 3.162}, {'end': 4000.756, 'text': "So the thing doesn't collapse into something boring.", 'start': 3998.234, 'duration': 2.522}, {'end': 4007.082, 'text': 'Exactly So the main sort of enemy of self-supervised learning, any kind of similarity maximization technique is collapse.', 'start': 4001.095, 'duration': 5.987}, {'end': 4014.41, 'text': 'So collapse means that you learn the same feature representation for all demons in the world, which is completely useless.', 'start': 4007.703, 'duration': 6.707}, {'end': 4015.592, 'text': 'Everything is a banana.', 'start': 4014.731, 'duration': 0.861}, {'end': 4016.553, 'text': 'Everything is a banana.', 'start': 4015.732, 'duration': 0.821}, {'end': 4017.294, 'text': 'Everything is a cat.', 'start': 4016.593, 'duration': 0.701}, {'end': 4018.054, 'text': 'Everything is a car.', 'start': 4017.334, 'duration': 0.72}, {'end': 4023.278, 'text': 'Yeah And so all we need to do is basically come up with ways to prevent collapse.', 'start': 4018.295, 'duration': 4.983}], 'summary': 'Preventing collapse in self-supervised learning is crucial to maintain distinct feature representations for different items.', 'duration': 29.227, 'max_score': 3994.051, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI3994051.jpg'}, {'end': 4072.205, 'src': 'embed', 'start': 4036.786, 'weight': 6, 'content': [{'end': 4039.928, 'text': "So that's inspired a little bit by like Horace Barlow's neuroscience principles.", 'start': 4036.786, 'duration': 3.142}, {'end': 4048.972, 'text': 'By the way, I should comment that whoever counts the number of times the word banana, apple, cat and dog we use in this conversation,', 'start': 4040.785, 'duration': 8.187}, {'end': 4049.793, 'text': 'wins the internet.', 'start': 4048.972, 'duration': 0.821}, {'end': 4050.833, 'text': 'I wish you luck.', 'start': 4050.213, 'duration': 0.62}, {'end': 4062.243, 'text': 'What is Suave and the main improvement proposed in the paper on supervised learning of visual features by contrasting cluster assignments?', 'start': 4053.175, 'duration': 9.068}, {'end': 4072.205, 'text': 'Suave, basically, is a clustering based technique which is for again the same thing for self supervised learning in vision, where we have two crops.', 'start': 4063.052, 'duration': 9.153}], 'summary': 'Suave is a clustering-based technique for self-supervised learning in vision.', 'duration': 35.419, 'max_score': 4036.786, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI4036786.jpg'}, {'end': 4109.566, 'src': 'embed', 'start': 4083.976, 'weight': 4, 'content': [{'end': 4088.998, 'text': 'Now, typically, if you were to do this clustering, you would perform clustering offline.', 'start': 4083.976, 'duration': 5.022}, {'end': 4097.56, 'text': 'What that means is, if you have a dataset of n examples, you would run over all of these n examples, get features for them, perform clustering.', 'start': 4089.518, 'duration': 8.042}, {'end': 4101.683, 'text': 'so basically, get some clusters and then repeat the process again.', 'start': 4097.56, 'duration': 4.123}, {'end': 4106.425, 'text': 'So this is offline basically because I need to do one pass through the data to compute its clusters.', 'start': 4102.023, 'duration': 4.402}, {'end': 4109.566, 'text': 'Suave is basically just a simple way of doing this online.', 'start': 4107.265, 'duration': 2.301}], 'summary': 'Suave enables online clustering, avoiding offline processing and repetitive passes through data.', 'duration': 25.59, 'max_score': 4083.976, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI4083976.jpg'}, {'end': 4195.304, 'src': 'embed', 'start': 4166.55, 'weight': 0, 'content': [{'end': 4170.651, 'text': 'So collapse can be viewed as a way in which all samples belong to one cluster, right?', 'start': 4166.55, 'duration': 4.101}, {'end': 4175.133, 'text': 'So all this, if all features become the same, then you have basically just one mega cluster.', 'start': 4170.671, 'duration': 4.462}, {'end': 4177.394, 'text': "You don't even have like 10 clusters or 3000 clusters.", 'start': 4175.153, 'duration': 2.241}, {'end': 4183.877, 'text': 'So swap basically ensures that at each point, all these 3000 clusters are being used in the clustering process.', 'start': 4178.194, 'duration': 5.683}, {'end': 4188.299, 'text': "And that's it basically just to figure out how to do this online.", 'start': 4185.157, 'duration': 3.142}, {'end': 4195.304, 'text': "And again, basically just make sure that two crops from the same image belong to the same cluster and others don't.", 'start': 4188.639, 'duration': 6.665}], 'summary': 'Collapse results in one mega cluster when all features become the same, instead of 10 or 3000 clusters, while swap ensures usage of 3000 clusters. the goal is to cluster two crops from the same image together.', 'duration': 28.754, 'max_score': 4166.55, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI4166550.jpg'}, {'end': 4271.613, 'src': 'embed', 'start': 4247.217, 'weight': 3, 'content': [{'end': 4252.999, 'text': 'So we take ImageNet, which of course I talked about as having lots of labels, and then we throw away the labels,', 'start': 4247.217, 'duration': 5.782}, {'end': 4256.46, 'text': 'like throw away all the hard work that went behind basically the labeling process.', 'start': 4252.999, 'duration': 3.461}, {'end': 4259.281, 'text': 'And we pretend that it is like unsupervised.', 'start': 4256.86, 'duration': 2.421}, {'end': 4269.412, 'text': 'But the problem here is that we have, when we collected these images, the ImageNet dataset has a particular distribution of concepts right?', 'start': 4260.267, 'duration': 9.145}, {'end': 4271.613, 'text': 'So these images are very curated.', 'start': 4270.032, 'duration': 1.581}], 'summary': 'Imagenet dataset labels discarded for unsupervised approach, but dataset is highly curated.', 'duration': 24.396, 'max_score': 4247.217, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI4247217.jpg'}], 'start': 3817.397, 'title': 'Non-contrastive learning methods and preventing collapse in self-supervised learning', 'summary': 'Discusses non-contrastive learning methods such as clustering and self-distillation, and techniques like suave and seer to prevent feature collapse in self-supervised learning. it highlights the challenges with access to negatives, the success of self-supervised learning on imagenet and uncurated internet images, and the use of about a billion images to train a large convolutional model.', 'chapters': [{'end': 3993.791, 'start': 3817.397, 'title': 'Non-contrastive learning methods', 'summary': 'Discusses non-contrastive, energy-based, self-supervised learning methods, such as clustering and self-distillation, as alternatives to contrastive learning, highlighting the challenges with access to negatives and the promise of these methods in scaling learning algorithms.', 'duration': 176.394, 'highlights': ['Non-contrastive methods like clustering and self-distillation are promising alternatives to contrastive learning, addressing challenges with access to negatives and offering scalability in learning algorithms. Non-contrastive methods, such as clustering and self-distillation, offer promising alternatives to contrastive learning by addressing challenges with access to negatives and providing scalability in learning algorithms.', 'Self-distillation involves maximizing similarity between features produced by a teacher network and a student network, eliminating the notion of negatives in the learning process. Self-distillation involves maximizing similarity between features produced by a teacher network and a student network, eliminating the notion of negatives in the learning process.', 'Clustering as a non-contrastive method involves grouping similar samples into clusters, offering a different approach to learning without the need for explicit negatives. Clustering as a non-contrastive method involves grouping similar samples into clusters, offering a different approach to learning without the need for explicit negatives.', 'Contrastive learning requires access to numerous negatives for effective learning, posing scalability challenges for learning algorithms. Contrastive learning requires access to numerous negatives for effective learning, posing scalability challenges for learning algorithms.']}, {'end': 4502.319, 'start': 3994.051, 'title': 'Preventing collapse in self-supervised learning', 'summary': 'Discusses the importance of preventing feature collapse in self-supervised learning, introducing techniques such as suave and seer, which are demonstrated on imagenet and uncurated internet images, respectively, to show the success of self-supervised learning and the need to move away from dataset biases, with the latter using about a billion images to train a large convolutional model.', 'duration': 508.268, 'highlights': ["SEER: Demonstrating the success of self-supervised learning on uncurated internet images SEER is a self-supervised pre-training method using about a billion uncurated internet images, representing a significant scale shift from the curated ImageNet dataset, to investigate if self-supervised learning is overfit to ImageNet and determine the model's capabilities to learn different types of objects.", 'Suave: Online clustering technique for self-supervised learning in vision Suave is a clustering-based technique for self-supervised learning in vision, aiming to prevent feature collapse by ensuring that features from two crops of an image lie in the same cluster, with a fixed number of clusters (k) and an equipartition constraint to equally partition the entire set of samples into k clusters, thus preventing collapse and allowing robust clustering.', "Importance of moving away from dataset biases and filtering for self-supervised learning The chapter emphasizes the need to move away from dataset biases and filtering, as demonstrated by SEER's use of uncurated internet images, indicating the importance of training self-supervised models on a diverse range of data to avoid overfitting to specific dataset biases."]}], 'duration': 684.922, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI3817397.jpg', 'highlights': ['Non-contrastive methods like clustering and self-distillation offer promising alternatives to contrastive learning by addressing challenges with access to negatives and providing scalability in learning algorithms.', 'Self-distillation involves maximizing similarity between features produced by a teacher network and a student network, eliminating the notion of negatives in the learning process.', 'Clustering as a non-contrastive method involves grouping similar samples into clusters, offering a different approach to learning without the need for explicit negatives.', 'Contrastive learning requires access to numerous negatives for effective learning, posing scalability challenges for learning algorithms.', 'SEER is a self-supervised pre-training method using about a billion uncurated internet images, representing a significant scale shift from the curated ImageNet dataset, demonstrating the success of self-supervised learning on uncurated internet images.', 'Suave is a clustering-based technique for self-supervised learning in vision, aiming to prevent feature collapse by ensuring that features from two crops of an image lie in the same cluster, with a fixed number of clusters (k) and an equipartition constraint to equally partition the entire set of samples into k clusters, thus preventing collapse and allowing robust clustering.', "The chapter emphasizes the need to move away from dataset biases and filtering, as demonstrated by SEER's use of uncurated internet images, indicating the importance of training self-supervised models on a diverse range of data to avoid overfitting to specific dataset biases."]}, {'end': 5437.716, 'segs': [{'end': 5403.552, 'src': 'embed', 'start': 5348.857, 'weight': 0, 'content': [{'end': 5352.779, 'text': "Exactly So you don't need a lot of videos of humans doing actions annotated.", 'start': 5348.857, 'duration': 3.922}, {'end': 5355.781, 'text': 'You can just use a few of them to basically get your recognition.', 'start': 5352.819, 'duration': 2.962}, {'end': 5362.228, 'text': 'How much insight do you draw from the fact that he can figure out where the sound is coming from?', 'start': 5356.141, 'duration': 6.087}, {'end': 5367.465, 'text': "I'm trying to see so that's kind of very, it's very CVPR.", 'start': 5363.624, 'duration': 3.841}, {'end': 5368.565, 'text': 'beautiful, right?', 'start': 5367.465, 'duration': 1.1}, {'end': 5369.985, 'text': "It's a cool little insight.", 'start': 5368.605, 'duration': 1.38}, {'end': 5372.586, 'text': 'I wonder how profound that is.', 'start': 5370.045, 'duration': 2.541}, {'end': 5383.627, 'text': 'You know, does it speak to the idea that multiple modalities are somehow much bigger than the sum of their parts??', 'start': 5373.786, 'duration': 9.841}, {'end': 5398.39, 'text': "Or is it really really useful to have multiple modalities? Or is it just a cool thing that there's parts of our world that are can be revealed like effectively through multiple modalities?", 'start': 5384.448, 'duration': 13.942}, {'end': 5403.552, 'text': 'but most of it is really all about vision or about one of the modalities.', 'start': 5398.39, 'duration': 5.162}], 'summary': 'Using few videos for recognition, exploring the significance of multiple modalities.', 'duration': 54.695, 'max_score': 5348.857, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI5348857.jpg'}], 'start': 4502.679, 'title': 'Self-supervised learning advancements', 'summary': 'Covers advancements in self-supervised learning, including scaling with regnet models, introduction of vissel ssl library for pytorch, and the concept of multimodal learning for video and audio, emphasizing efficient neural network architectures and the promise of multimodal learning in self-supervised video networks.', 'chapters': [{'end': 4870.283, 'start': 4502.679, 'title': 'Scaling self-supervised learning with regnet', 'summary': 'Discusses the biases in datasets, the effectiveness of using regnet models, and the importance of efficient neural network architectures for self-supervised learning, emphasizing the need for network families that are efficient in both flops and memory space and the challenges of scaling large neural networks.', 'duration': 367.604, 'highlights': ['RegNet models are efficient in terms of compute and memory, making them effective for scaling self-supervised learning. RegNet models, particularly the RegNet architecture family, are known for their efficiency in terms of compute versus accuracy, making them suitable for scaling self-supervised learning. This is due to their efficient use of both flops and memory space, allowing for the creation of large neural networks that can fit well on GPU memory.', 'The biases in datasets and user base affect the availability of images from different parts of the world, impacting the training of neural networks. The biases in datasets and user base lead to limited availability of images from different parts of the world, affecting the training of neural networks. This is due to certain regions having lower internet accessibility and fewer uploads, resulting in a bias in the dataset and user base.', 'The importance of designing efficient neural network architectures that optimize for both flops and memory space in self-supervised learning. The chapter emphasizes the need to design network families or neural network architectures that are efficient in both flops and memory space for self-supervised learning. This is crucial as it allows for the creation of very large networks while optimizing for efficiency and fitting well on GPU memory.']}, {'end': 5102.367, 'start': 4870.283, 'title': 'Vissel: self-supervised learning library', 'summary': 'Discusses vissel, a pytorch-based self-supervised learning (ssl) library, its use cases, and the challenges of small-scale experimentation, highlighting the need for standardized benchmarking and the promise of multimodal learning.', 'duration': 232.084, 'highlights': ['Vissel is a PyTorch-based self-supervised learning (SSL) library designed for evaluating, training, and developing new self-supervised techniques in vision, with a standardized benchmark of tasks for evaluating representations. Vissel serves as a common framework for implementing self-supervised learning methods in vision, providing a benchmark of tasks for evaluating representations.', "Vissel aims to standardize the experimental setup for self-supervised learning models, addressing the challenge of inconsistent experimental setups and varying interpretations of reported accuracy. Vissel's goal is to standardize the experimental setup for self-supervised learning models, ensuring consistent evaluation and interpretation of model performance.", 'The chapter emphasizes the challenges of small-scale experimentation in self-supervised learning, noting the difficulty of translating empirical observations from small-scale setups to larger datasets like ImageNet. The difficulty of translating empirical observations from small-scale setups to larger datasets is highlighted, underscoring the challenges of small-scale experimentation in self-supervised learning.', "The discussion touches on the potential of multimodal learning, citing the success of the paper 'Audiovisual Instance Discrimination with Cross-Modal Agreement' and its surprising effectiveness in practice. The potential of multimodal learning is discussed, emphasizing the surprising effectiveness of 'Audiovisual Instance Discrimination with Cross-Modal Agreement' in practice."]}, {'end': 5437.716, 'start': 5102.467, 'title': 'Multimodal learning for video and audio', 'summary': 'Discusses the concept of multimodal learning, where both audio and video signals are used to learn a common feature space, showcasing the ability to recognize human actions, different types of sounds, and objects using a self-supervised video network.', 'duration': 335.249, 'highlights': ['The chapter discusses the concept of multimodal learning, where both audio and video signals are used to learn a common feature space, showcasing the ability to recognize human actions, different types of sounds, and objects using a self-supervised video network.', 'The research involved training two different neural networks, one on the video signal and one on the audio signal, aiming for similar features from both networks, leading to the discovery of very powerful feature representations for video and audio.', "The self-supervised video network can be utilized to recognize various human actions, with the example of the Kinetics dataset containing 400 different types of human actions, showcasing the network's ability to learn and recognize actions without explicit annotations.", "The network's capability to visualize and determine the source of sound from a given video, such as focusing on the guitar when provided with the sound of a guitar being strummed, demonstrates its ability to naturally associate sounds with objects and distinguish different voices without explicit annotation.", 'The discussion also explores the idea of the profound impact of having multiple modalities, indicating that while most information can be inferred from one modality, having an extra modality can significantly enhance understanding, as exemplified by the ease of identifying objects based on characteristic sounds.']}], 'duration': 935.037, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI4502679.jpg', 'highlights': ['RegNet models are efficient in terms of compute and memory, making them effective for scaling self-supervised learning.', 'Vissel is a PyTorch-based self-supervised learning (SSL) library designed for evaluating, training, and developing new self-supervised techniques in vision, with a standardized benchmark of tasks for evaluating representations.', 'The chapter discusses the concept of multimodal learning, where both audio and video signals are used to learn a common feature space, showcasing the ability to recognize human actions, different types of sounds, and objects using a self-supervised video network.']}, {'end': 6444.375, 'segs': [{'end': 5769.122, 'src': 'embed', 'start': 5735.687, 'weight': 1, 'content': [{'end': 5739.17, 'text': "And that's going to tell you a lot more about what this concept of a banana is.", 'start': 5735.687, 'duration': 3.483}, {'end': 5741.772, 'text': "So that that's kind of a heuristic.", 'start': 5739.57, 'duration': 2.202}, {'end': 5752.957, 'text': "I wonder if it's possible to also learn learn ways to discover the most likely, the most beneficial image.", 'start': 5741.892, 'duration': 11.065}, {'end': 5760.019, 'text': "So like so not just looking a thing that's somewhat similar to a banana but not exactly similar,", 'start': 5752.977, 'duration': 7.042}, {'end': 5769.122, 'text': 'but have some kind of more complicated learning system, like learned discovery mechanism that tells you what image to look for.', 'start': 5760.019, 'duration': 9.103}], 'summary': 'Exploring the possibility of developing a more sophisticated learning system to identify the most beneficial image related to the concept of a banana.', 'duration': 33.435, 'max_score': 5735.687, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI5735687.jpg'}, {'end': 5897.295, 'src': 'embed', 'start': 5875.92, 'weight': 3, 'content': [{'end': 5885.507, 'text': 'So because you have this nice sequence of data that is coming in a video stream of it, associated, of course, with the actions that say the car took,', 'start': 5875.92, 'duration': 9.587}, {'end': 5888.309, 'text': "you can form a very nice predictive model of what's happening.", 'start': 5885.507, 'duration': 2.802}, {'end': 5892.572, 'text': 'So, for example, like all the way, like one way, possibly,', 'start': 5888.329, 'duration': 4.243}, {'end': 5897.295, 'text': "in which how they're figuring out what data to get labeled is basically through prediction uncertainty, right?, Mm-hmm.", 'start': 5892.572, 'duration': 4.723}], 'summary': 'Using video stream data, predictive models can be formed to analyze car actions.', 'duration': 21.375, 'max_score': 5875.92, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI5875920.jpg'}, {'end': 6065.618, 'src': 'embed', 'start': 6042.462, 'weight': 2, 'content': [{'end': 6049.964, 'text': "And it's, let's say significantly better, like say five times less accidents than humans.", 'start': 6042.462, 'duration': 7.502}, {'end': 6059.526, 'text': 'Sufficiently safer, such that the public feels like that transition is, you know, enticing, beneficial both for our safety and financially,', 'start': 6050.684, 'duration': 8.842}, {'end': 6060.366, 'text': 'and all those kinds of things.', 'start': 6059.526, 'duration': 0.84}, {'end': 6063.955, 'text': "Okay, so first disclaimer, I'm not an expert in autonomous driving.", 'start': 6061.129, 'duration': 2.826}, {'end': 6065.618, 'text': 'So let me put it out there.', 'start': 6064.276, 'duration': 1.342}], 'summary': 'Autonomous driving is five times safer than humans, enticing and beneficial for safety and finances.', 'duration': 23.156, 'max_score': 6042.462, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI6042462.jpg'}, {'end': 6200.105, 'src': 'embed', 'start': 6168.935, 'weight': 4, 'content': [{'end': 6170.617, 'text': 'I was going to call it self-supervised driving, but..', 'start': 6168.935, 'duration': 1.682}, {'end': 6173.539, 'text': 'Vision-based autonomous driving.', 'start': 6171.978, 'duration': 1.561}, {'end': 6178.901, 'text': "that's the reason I'm quite optimistic about it, because I think there are going to be lots of these advances on the sensor side itself.", 'start': 6173.539, 'duration': 5.362}, {'end': 6182.463, 'text': "So acquiring this data, we're actually going to get much better about it.", 'start': 6179.001, 'duration': 3.462}, {'end': 6188.766, 'text': "And then, of course, once we're able to scale out and get all of these edge cases in, as like Andre described,", 'start': 6182.663, 'duration': 6.103}, {'end': 6190.927, 'text': "I think that's going to make us go very far away.", 'start': 6188.766, 'duration': 2.161}, {'end': 6193.622, 'text': "Yeah, so it's funny.", 'start': 6191.761, 'duration': 1.861}, {'end': 6200.105, 'text': "I'm very much with you on the five to 10 years, maybe 10 years, but you made it.", 'start': 6193.622, 'duration': 6.483}], 'summary': 'Optimistic about vision-based autonomous driving and sensor advancements leading to significant progress in 5-10 years.', 'duration': 31.17, 'max_score': 6168.935, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI6168935.jpg'}, {'end': 6368.354, 'src': 'embed', 'start': 6339.757, 'weight': 0, 'content': [{'end': 6349.923, 'text': "makes me think that it's now starting to be coupled to this self-supervised learning vision, which is like if that's going to work.", 'start': 6339.757, 'duration': 10.166}, {'end': 6354.485, 'text': 'if through purely this process you can get really far, then maybe you can solve driving that way.', 'start': 6349.923, 'duration': 4.562}, {'end': 6355.385, 'text': "I don't know.", 'start': 6354.945, 'duration': 0.44}, {'end': 6360.268, 'text': "I tend to believe we don't give enough credit to the..", 'start': 6355.405, 'duration': 4.863}, {'end': 6368.354, 'text': 'how amazing humans are both at driving and at supervising autonomous systems.', 'start': 6362.989, 'duration': 5.365}], 'summary': 'Coupling self-supervised learning with driving for autonomous systems.', 'duration': 28.597, 'max_score': 6339.757, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI6339757.jpg'}], 'start': 5438.356, 'title': 'Nlp, ai, and autonomous driving', 'summary': 'Highlights the importance of modalities and context in nlp, discusses the significance of smell in ai and active learning with emphasis on language and vision, and examines the feasibility of vision-based autonomous driving, estimating it to be 5-10 years away.', 'chapters': [{'end': 5472.606, 'start': 5438.356, 'title': 'Modalities and context in nlp', 'summary': 'Discusses the importance of modalities and context in nlp, highlighting how the distributional hypothesis applies to sound context in videos and its impact on identifying common concepts.', 'duration': 34.25, 'highlights': ['The distributional hypothesis in NLP is related to how context gives meaning to a word, and it may also apply to sound context in videos, aiding in identifying common concepts and patterns.', 'Observing the same sound across multiple videos helps in identifying common factors and concepts, making life easier when accessing different modalities.', "Having access to different modalities can significantly simplify one's life by aiding in the observation of common concepts and patterns."]}, {'end': 5995.158, 'start': 5473.587, 'title': 'Importance of smell in ai and active learning', 'summary': 'Discusses the importance of smell in ai, active learning, and its application in autonomous driving, emphasizing the power of language and vision over smell, the efficiency and power of active learning, and the use of self-supervised learning for predictive models in autonomous driving.', 'duration': 521.571, 'highlights': ['Active learning is efficient and powerful, potentially surpassing other learning techniques. Active learning can enable learning in fewer samples, making it more efficient at using data, potentially surpassing other learning techniques.', 'Active learning involves understanding and discovering new concepts, which makes it a powerful technique. Active learning involves understanding and discovering new concepts, which makes it a powerful technique for increasing knowledge and asking valuable questions.', 'The use of active learning in data labeling, particularly in self-supervised models, is highly beneficial. Active learning is highly beneficial in data labeling, especially in self-supervised models, for predicting similarities and dissimilarities between data, leading to effective data labeling.', 'Self-supervised learning can be used in autonomous driving for forming predictive models based on prediction uncertainty. Self-supervised learning can be used in autonomous driving to form predictive models based on prediction uncertainty, improving the understanding of edge cases and potential failures.']}, {'end': 6444.375, 'start': 5995.399, 'title': 'Vision-based autonomous driving', 'summary': 'Discusses the feasibility and timeline for vision-based autonomous driving, estimating it to be 5-10 years away, with recent advancements in camera and sensor technology proving the potential of this approach, challenging the conventional belief in lidar-based systems.', 'duration': 448.976, 'highlights': ['Recent advancements in camera and sensor technology have proven the potential of vision-based autonomous driving, challenging the conventional belief in LiDAR-based systems. The chapter highlights recent advancements in camera and sensor technology, proving the potential of vision-based autonomous driving, challenging the conventional belief in LiDAR-based systems. This suggests a shift towards a completely vision-based system and a more optimistic outlook for the future of autonomous driving.', 'The estimated timeline for vision-based autonomous driving is 5-10 years, with a potential for significant advancements in the sensor technology that could accelerate the progress. The chapter estimates the timeline for vision-based autonomous driving to be 5-10 years, with a potential for significant advancements in the sensor technology that could accelerate the progress. This timeline is based on recent developments and the potential for innovations in sensor technology, indicating a more optimistic view on the feasibility of this approach.', 'The skepticism towards the capability of self-supervised learning and human behavior modeling in autonomous driving, along with the need for deeper consideration of human factors and driver supervision, presents challenges to the widespread implementation of autonomous driving. The chapter presents skepticism towards the capability of self-supervised learning and human behavior modeling in autonomous driving, along with the need for deeper consideration of human factors and driver supervision, which presents challenges to the widespread implementation of autonomous driving. This highlights the complexity of human-robot interaction and the potential hindrances in achieving widespread autonomous driving.']}], 'duration': 1006.019, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI5438356.jpg', 'highlights': ['Access to different modalities simplifies life by observing common concepts and patterns.', 'Active learning is efficient, powerful, and beneficial in data labeling for self-supervised models.', 'Vision-based autonomous driving challenges LiDAR-based systems, with an estimated timeline of 5-10 years.', 'Recent advancements in camera and sensor technology prove the potential of vision-based autonomous driving.', 'Skepticism towards self-supervised learning and human behavior modeling presents challenges to autonomous driving.']}, {'end': 8043.874, 'segs': [{'end': 6727.508, 'src': 'embed', 'start': 6694.135, 'weight': 6, 'content': [{'end': 6695.215, 'text': 'There are no guarantees for it.', 'start': 6694.135, 'duration': 1.08}, {'end': 6698.056, 'text': "Now you can argue that humans also don't have any guarantees.", 'start': 6695.875, 'duration': 2.181}, {'end': 6702.137, 'text': 'Like there is no guarantee that I can recognize a cat in every scenario.', 'start': 6698.176, 'duration': 3.961}, {'end': 6707.418, 'text': "I'm sure there are going to be lots of cats that I don't recognize, lots of scenarios in which I don't recognize cats in general.", 'start': 6702.277, 'duration': 5.141}, {'end': 6716.743, 'text': 'But I think from just a sort of application perspective, you do need guarantees, right? We call these things algorithms.', 'start': 6708.078, 'duration': 8.665}, {'end': 6719.764, 'text': 'Now algorithms, like traditional CS algorithms, have guarantees.', 'start': 6716.983, 'duration': 2.781}, {'end': 6721.005, 'text': 'Sorting is a guarantee.', 'start': 6720.024, 'duration': 0.981}, {'end': 6727.508, 'text': "If you were to call sort on a particular array of numbers, you are guaranteed that it's going to be sorted.", 'start': 6721.525, 'duration': 5.983}], 'summary': 'Algorithms provide guarantees, like sorting an array with a guarantee of being sorted.', 'duration': 33.373, 'max_score': 6694.135, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI6694135.jpg'}, {'end': 6913.648, 'src': 'embed', 'start': 6873.244, 'weight': 1, 'content': [{'end': 6879.408, 'text': "Can that be learnable? What's your intuition there? Or like, I guess, similar set of techniques.", 'start': 6873.244, 'duration': 6.164}, {'end': 6881.51, 'text': 'Do you think that would be applicable?', 'start': 6879.448, 'duration': 2.062}, {'end': 6883.311, 'text': 'So I think it is.', 'start': 6882.05, 'duration': 1.261}, {'end': 6887.654, 'text': 'of course it is learnable because, like we, are prime examples of machines that have like,', 'start': 6883.311, 'duration': 4.343}, {'end': 6890.636, 'text': 'or individuals that have learned this right? Like humans have learned this.', 'start': 6887.654, 'duration': 2.982}, {'end': 6893.898, 'text': 'So it is, of course, it is a technique that is very easy to learn.', 'start': 6891.116, 'duration': 2.782}, {'end': 6900.653, 'text': 'I think where we are kind of hitting a wall, basically with like current machine learning,', 'start': 6895.942, 'duration': 4.711}, {'end': 6904.662, 'text': 'is the fact that When the network learns all of this information,', 'start': 6900.653, 'duration': 4.009}, {'end': 6909.826, 'text': "we basically are not able to figure out how well it's going to generalize to an unseen thing.", 'start': 6904.662, 'duration': 5.164}, {'end': 6913.648, 'text': 'Yeah And we have no, like a priori, no way of characterizing that.', 'start': 6910.146, 'duration': 3.502}], 'summary': 'Discussion on learnability and applicability of techniques in machine learning, with emphasis on the challenge of generalization.', 'duration': 40.404, 'max_score': 6873.244, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI6873244.jpg'}, {'end': 7168.717, 'src': 'embed', 'start': 7145.391, 'weight': 0, 'content': [{'end': 7153.277, 'text': 'and so This basically indicates that I already have a predictive model in my head and something that I predicted or something that I thought was likely to happen.', 'start': 7145.391, 'duration': 7.886}, {'end': 7157.538, 'text': 'And then there was something that I observed that happened that there was a disconnect between these two things.', 'start': 7153.777, 'duration': 3.761}, {'end': 7164.021, 'text': 'And that basically is like, maybe one of the reasons I like you have a lot of emotions.', 'start': 7158.339, 'duration': 5.682}, {'end': 7168.717, 'text': 'Yeah, I think, so I talk to people a lot about it more like Lisa Feldman Barrett.', 'start': 7164.335, 'duration': 4.382}], 'summary': 'Speaker explains how predictive models fail to align with observations, leading to emotional complexity.', 'duration': 23.326, 'max_score': 7145.391, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI7145391.jpg'}, {'end': 7306.249, 'src': 'embed', 'start': 7275.859, 'weight': 3, 'content': [{'end': 7278.52, 'text': 'where I just like sit in my armchair and look at videos and learn.', 'start': 7275.859, 'duration': 2.661}, {'end': 7286.041, 'text': 'I do think that we will need to have some kind of embodiment or some kind of interaction to figure out things about the world.', 'start': 7279.241, 'duration': 6.8}, {'end': 7287.938, 'text': 'What about consciousness?', 'start': 7286.957, 'duration': 0.981}, {'end': 7293.381, 'text': 'How often do you think about consciousness when you think about your work?', 'start': 7290.539, 'duration': 2.842}, {'end': 7306.249, 'text': 'You could think of it as the more simple thing of self-awareness, of being aware that you are a perceiving, sensing, acting thing in this world.', 'start': 7294.482, 'duration': 11.767}], 'summary': 'Consideration of embodiment and interaction for learning and understanding consciousness in work.', 'duration': 30.39, 'max_score': 7275.859, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI7275859.jpg'}, {'end': 8026.833, 'src': 'embed', 'start': 7992.872, 'weight': 5, 'content': [{'end': 8000.058, 'text': "Forget, like ultra realistic, to where you can't tell the difference, but like it's so nice that you just want to stay there.", 'start': 7992.872, 'duration': 7.186}, {'end': 8003.961, 'text': "You just want to stay there and you don't want to come back.", 'start': 8001.059, 'duration': 2.902}, {'end': 8011.862, 'text': "Do you think that's doable within our lifetime? Within our lifetime? Probably, yeah.", 'start': 8004.957, 'duration': 6.905}, {'end': 8013.283, 'text': 'I eat healthy, I live long.', 'start': 8012.182, 'duration': 1.101}, {'end': 8026.833, 'text': "Does that make you sad that there'll be like population of kids that basically spend 95% 99% of their time in a virtual world?", 'start': 8015.845, 'duration': 10.988}], 'summary': 'Achieving ultra-realistic virtual world experience, possibly in our lifetime, may lead to a population spending majority of time in virtual reality.', 'duration': 33.961, 'max_score': 7992.872, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI7992872.jpg'}], 'start': 6444.415, 'title': 'Ai integration challenges and limits', 'summary': 'Discusses challenges of integrating ai into society, the limits of deep learning, and the need for consciousness in ai, emphasizing societal impact, reasoning capabilities, and self-awareness, and highlighting the emergence of object concepts in self-supervised learning.', 'chapters': [{'end': 6529.181, 'start': 6444.415, 'title': 'Challenges of integrating ai into society', 'summary': 'Discusses the necessity of integrating ai systems into human society and the challenges associated with it, particularly in the context of autonomous driving, emphasizing the impact on human life and the need to account for societal forces.', 'duration': 84.766, 'highlights': ['The integration of AI systems into society, particularly in the context of autonomous driving, is discussed. The discussion explores the impact of integrating AI into human society, with a specific focus on autonomous driving.', 'The challenges of accounting for societal forces and human nature in the development and integration of AI systems are highlighted. The chapter emphasizes the need to consider societal forces, including politicians, public opinion, and journalists, in the development and integration of AI systems, particularly in the context of autonomous driving.', 'The impact of AI system success on its integration into society and the associated complexities are examined. The chapter delves into how the success of AI systems leads to their increased integration into society, resulting in complexities related to navigating human nature and societal forces.']}, {'end': 7258.374, 'start': 6530.12, 'title': 'Limits of deep learning', 'summary': 'Explores the limits of deep learning, emphasizing challenges such as data efficiency, guarantees, reasoning capabilities, and the need for richer communication in ai. it discusses the difficulty in generalizing from a single sample, the lack of guarantees in machine learning, the distinction between learning and reasoning, and the challenge of retaining knowledge for future tasks. additionally, it delves into the importance of emotion and rich communication in building systems with human-level general intelligence.', 'duration': 728.254, 'highlights': ["The challenge of generalizing from a single sample It's difficult for deep learning methods to generalize from just one or two samples, posing a significant challenge in data efficiency.", 'Lack of guarantees in machine learning Unlike traditional CS algorithms, machine learning lacks clear guarantees, making it challenging to characterize the notion of correctness and transferability of learned knowledge.', 'Distinction between learning and reasoning Neural networks excel at recognition but struggle with reasoning, particularly in unfamiliar and complex scenarios, indicating the limitations of current machine learning capabilities.', 'The importance of emotion and rich communication in AI Emotion and rich communication are highlighted as essential elements for building systems with human-level general intelligence, emphasizing the need to go beyond traditional NLP and vision in AI development.']}, {'end': 7582.584, 'start': 7258.394, 'title': 'Need for consciousness in ai', 'summary': 'Discusses the necessity of self-awareness and consciousness in ai, emphasizing the importance of displaying elements of consciousness to create meaningful interactions with humans and the potential for faking consciousness in ai.', 'duration': 324.19, 'highlights': ['The importance of self-awareness and consciousness in creating AGI, contextualizing its role and interactions with the world. ', 'The belief that displaying consciousness is crucial for AGI to connect with humans and other living entities. ', 'The consideration of faking consciousness in AI as a way to form close connections with humans, potentially through the effective display of human-like elements. ']}, {'end': 8043.874, 'start': 7583.72, 'title': 'Emergence of object concepts in self-supervised learning', 'summary': 'Discusses the emergence of object boundaries and concepts like object permanence and symmetry in self-supervised learning, highlighting the surprising capabilities of models and questioning the effectiveness of simulation in machine learning.', 'duration': 460.154, 'highlights': ['The network automatically figures out object boundaries and concepts without explicit training, demonstrating the power of self-supervised learning. The network autonomously identifies object boundaries, such as those around a dog, without specific training, showcasing the effectiveness of self-supervised learning.', 'The discussion explores the potential emergence of concepts like object permanence, symmetry, and counting through self-supervised learning on billions of images. The potential emergence of concepts like object permanence, symmetry, and counting through self-supervised learning on a large dataset is contemplated, challenging traditional notions of concept acquisition.', 'The skepticism towards the effectiveness of simulation in machine learning, particularly in the context of autonomous driving, is brought to light, emphasizing the challenges in accurately simulating visual and behavioral aspects. The limitations and challenges of using simulation in machine learning, particularly in the context of autonomous driving, are highlighted, questioning the feasibility of accurately simulating visual and behavioral aspects.']}], 'duration': 1599.459, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI6444415.jpg', 'highlights': ['The integration of AI systems into society, particularly in the context of autonomous driving, is discussed.', 'The challenges of accounting for societal forces and human nature in the development and integration of AI systems are highlighted.', 'The impact of AI system success on its integration into society and the associated complexities are examined.', "It's difficult for deep learning methods to generalize from just one or two samples, posing a significant challenge in data efficiency.", 'Neural networks excel at recognition but struggle with reasoning, particularly in unfamiliar and complex scenarios, indicating the limitations of current machine learning capabilities.', 'Emotion and rich communication are highlighted as essential elements for building systems with human-level general intelligence, emphasizing the need to go beyond traditional NLP and vision in AI development.', 'The importance of self-awareness and consciousness in creating AGI, contextualizing its role and interactions with the world.', 'The network automatically figures out object boundaries and concepts without explicit training, demonstrating the power of self-supervised learning.', 'The discussion explores the potential emergence of concepts like object permanence, symmetry, and counting through self-supervised learning on billions of images, challenging traditional notions of concept acquisition.', 'The limitations and challenges of using simulation in machine learning, particularly in the context of autonomous driving, are highlighted, questioning the feasibility of accurately simulating visual and behavioral aspects.']}, {'end': 9020.714, 'segs': [{'end': 8150.442, 'src': 'embed', 'start': 8118.864, 'weight': 13, 'content': [{'end': 8120.887, 'text': 'If you are optimistic about it, then go ahead.', 'start': 8118.864, 'duration': 2.023}, {'end': 8122.028, 'text': "That's why I started this podcast.", 'start': 8120.907, 'duration': 1.121}, {'end': 8123.99, 'text': 'I keep asking people about the meaning of life.', 'start': 8122.108, 'duration': 1.882}, {'end': 8127.213, 'text': "I'm hoping by episode 220, I'll figure it out.", 'start': 8124.05, 'duration': 3.163}, {'end': 8130.517, 'text': 'Not too many episodes to go.', 'start': 8128.915, 'duration': 1.602}, {'end': 8132.339, 'text': 'Maybe today.', 'start': 8131.758, 'duration': 0.581}, {'end': 8132.719, 'text': "I don't know.", 'start': 8132.379, 'duration': 0.34}, {'end': 8133.76, 'text': "But you're right.", 'start': 8133.08, 'duration': 0.68}, {'end': 8136.123, 'text': 'So that seems intractable at the moment.', 'start': 8133.8, 'duration': 2.323}, {'end': 8142.972, 'text': "Right. So I think it's just the fact of like, if you're starting a PhD, for example, what is one problem that you want to focus on?", 'start': 8136.303, 'duration': 6.669}, {'end': 8150.442, 'text': "that you do think is interesting enough and you will be able to make a reasonable amount of headway into it that you think you'll be doing a PhD for?", 'start': 8142.972, 'duration': 7.47}], 'summary': 'Podcast host aims to discover meaning of life by episode 220.', 'duration': 31.578, 'max_score': 8118.864, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8118864.jpg'}, {'end': 8258.165, 'src': 'embed', 'start': 8228.709, 'weight': 0, 'content': [{'end': 8233.232, 'text': 'because I used to always believe that doing the experiments was actually the bigger part of research than writing.', 'start': 8228.709, 'duration': 4.523}, {'end': 8236.574, 'text': 'And my advisor always told me that you should start writing very early on.', 'start': 8233.892, 'duration': 2.682}, {'end': 8237.834, 'text': "And I thought, oh, it doesn't matter.", 'start': 8236.634, 'duration': 1.2}, {'end': 8238.896, 'text': "I don't know what he's talking about.", 'start': 8237.875, 'duration': 1.021}, {'end': 8241.697, 'text': "But I think more and more I realize that's the case.", 'start': 8239.755, 'duration': 1.942}, {'end': 8245.739, 'text': "Like whenever I write something that I'm doing, I actually think much better about it.", 'start': 8241.777, 'duration': 3.962}, {'end': 8251.2, 'text': 'And so if you start writing earlier, early on, you actually, I think, get better ideas,', 'start': 8246.439, 'duration': 4.761}, {'end': 8258.165, 'text': 'or at least you figure out like holes in your theory or like particular experiments that you should run to plug those holes, and so on.', 'start': 8251.2, 'duration': 6.965}], 'summary': 'Early writing in research leads to better ideas and identifies gaps for experiments.', 'duration': 29.456, 'max_score': 8228.709, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8228709.jpg'}, {'end': 8507.759, 'src': 'embed', 'start': 8470.224, 'weight': 4, 'content': [{'end': 8472.325, 'text': 'And LuaTorch was much more friendly for all of these things.', 'start': 8470.224, 'duration': 2.101}, {'end': 8480.148, 'text': "Okay, so in terms of PyTorch and TensorFlow, my personal bias is PyTorch just because I've been using it longer and I'm more familiar with it.", 'start': 8473.625, 'duration': 6.523}, {'end': 8488.211, 'text': "And also that PyTorch is much easier to debug is what I find because it's imperative in nature compared to like TensorFlow, which is not imperative.", 'start': 8480.848, 'duration': 7.363}, {'end': 8495.274, 'text': "But that's telling you a lot that basically the imperative design is sort of a way in which a lot of people are taught programming,", 'start': 8488.671, 'duration': 6.603}, {'end': 8498.075, 'text': "and that's what actually makes debugging easier for them.", 'start': 8495.274, 'duration': 2.801}, {'end': 8502.937, 'text': 'So like I learned programming in C++, and so for me, imperative way of programming is more natural.', 'start': 8498.175, 'duration': 4.762}, {'end': 8507.759, 'text': "Do you think it's good to have kind of these two communities, this kind of competition?", 'start': 8504.056, 'duration': 3.703}], 'summary': 'Pytorch is easier to debug and more natural for imperative programming compared to tensorflow.', 'duration': 37.535, 'max_score': 8470.224, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8470224.jpg'}, {'end': 8620.939, 'src': 'embed', 'start': 8588.585, 'weight': 1, 'content': [{'end': 8593.947, 'text': "You know, uh, maybe just started or haven't even started, but are curious about it and who want to get in the field?", 'start': 8588.585, 'duration': 5.362}, {'end': 8596.434, 'text': "Don't be afraid to get your hands dirty.", 'start': 8594.912, 'duration': 1.522}, {'end': 8597.595, 'text': "I think that's the main thing.", 'start': 8596.654, 'duration': 0.941}, {'end': 8601.919, 'text': "So if something doesn't work, like really drill into why things are not working.", 'start': 8597.615, 'duration': 4.304}, {'end': 8604.451, 'text': 'Can you elaborate what your hands dirty means??', 'start': 8602.21, 'duration': 2.241}, {'end': 8610.114, 'text': "Right so, for example, like if an algorithm, if you try to train a network and it's not converging whatever,", 'start': 8604.491, 'duration': 5.623}, {'end': 8616.337, 'text': 'rather than trying to like Google the answer, or trying to do something like really spend those like five, eight, 10, 15, 20,', 'start': 8610.114, 'duration': 6.223}, {'end': 8618.438, 'text': 'whatever number of hours really trying to figure it out yourself.', 'start': 8616.337, 'duration': 2.101}, {'end': 8620.939, 'text': "Because in that process, you'll actually learn a lot more.", 'start': 8618.998, 'duration': 1.941}], 'summary': "In data science, don't shy away from troubleshooting. spend time to learn and understand.", 'duration': 32.354, 'max_score': 8588.585, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8588585.jpg'}, {'end': 8753.643, 'src': 'embed', 'start': 8729.822, 'weight': 5, 'content': [{'end': 8738.446, 'text': "And I think, like I've been inspired by a lot of people who are just like driven and who really like go for what they want, no matter what, like,", 'start': 8729.822, 'duration': 8.624}, {'end': 8740.147, 'text': "you shouldn't want it, you should need it.", 'start': 8738.446, 'duration': 1.701}, {'end': 8743.868, 'text': 'So if you need something, you basically go towards the ends to make it work.', 'start': 8740.527, 'duration': 3.341}, {'end': 8753.643, 'text': "How do you know when you come across a thing that that's like you need? I think there's not going to be any single thing that you're going to need.", 'start': 8744.408, 'duration': 9.235}], 'summary': 'Driven individuals pursue needs, not wants, to achieve goals.', 'duration': 23.821, 'max_score': 8729.822, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8729822.jpg'}, {'end': 8855.113, 'src': 'embed', 'start': 8826.521, 'weight': 12, 'content': [{'end': 8829.062, 'text': "When you get through it is when you find the one thing that's actually working.", 'start': 8826.521, 'duration': 2.541}, {'end': 8832.664, 'text': 'Thomas Edison was one person like that.', 'start': 8829.583, 'duration': 3.081}, {'end': 8838.206, 'text': 'When I was a kid, I used to really read about how he found the filament, the light bulb filament.', 'start': 8833.104, 'duration': 5.102}, {'end': 8843.949, 'text': "And then I think his thing was he tried 990 things that didn't work or something of the sort.", 'start': 8838.747, 'duration': 5.202}, {'end': 8848.451, 'text': 'And then they asked him, so what did you learn? Because all of these were failed experiments.', 'start': 8844.389, 'duration': 4.062}, {'end': 8851.572, 'text': "And then he says, oh, these 990 things don't work.", 'start': 8848.511, 'duration': 3.061}, {'end': 8852.152, 'text': 'And I know that.', 'start': 8851.612, 'duration': 0.54}, {'end': 8855.113, 'text': "Did you know that? I mean, that's really inspiring.", 'start': 8852.192, 'duration': 2.921}], 'summary': "Thomas edison's perseverance: tried 990 things for light bulb filament, inspiring.", 'duration': 28.592, 'max_score': 8826.521, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8826521.jpg'}, {'end': 8890.435, 'src': 'embed', 'start': 8863.995, 'weight': 7, 'content': [{'end': 8868.717, 'text': "Have you figured out the meaning of life yet? I told you I'm doing this podcast to try to get the answer.", 'start': 8863.995, 'duration': 4.722}, {'end': 8870.597, 'text': "I'm hoping you could tell me.", 'start': 8869.197, 'duration': 1.4}, {'end': 8876.027, 'text': "What do you think the meaning of it all is? I don't think I figured this out, no.", 'start': 8870.757, 'duration': 5.27}, {'end': 8876.927, 'text': 'I have no idea.', 'start': 8876.287, 'duration': 0.64}, {'end': 8885.073, 'text': "Do you think AI will help us figure it out? Or do you think there's no answer? The whole point is to keep searching.", 'start': 8879.009, 'duration': 6.064}, {'end': 8888.574, 'text': "I think, yeah, I think it's an endless sort of quest for us.", 'start': 8885.532, 'duration': 3.042}, {'end': 8890.435, 'text': "I don't think AI will help us there.", 'start': 8888.794, 'duration': 1.641}], 'summary': 'Podcast explores the meaning of life; ai may not help.', 'duration': 26.44, 'max_score': 8863.995, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8863995.jpg'}, {'end': 9020.714, 'src': 'embed', 'start': 9005.308, 'weight': 6, 'content': [{'end': 9008.009, 'text': 'Check them out in the description to support this podcast.', 'start': 9005.308, 'duration': 2.701}, {'end': 9011.33, 'text': 'And now let me leave you with some words from Arthur C.', 'start': 9008.609, 'duration': 2.721}, {'end': 9017.152, 'text': 'Clarke Any sufficiently advanced technology is indistinguishable from magic.', 'start': 9011.33, 'duration': 5.822}, {'end': 9020.714, 'text': 'Thank you for listening and hope to see you next time.', 'start': 9018.173, 'duration': 2.541}], 'summary': 'Podcast outro with arthur c. clarke quote.', 'duration': 15.406, 'max_score': 9005.308, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI9005308.jpg'}], 'start': 8043.954, 'title': 'Writing good papers, phd research, pytorch vs tensorflow, and embracing failure', 'summary': 'Discusses the importance of problem selection, optimism in research, and challenges in writing papers, selecting feasible problems for phd research, focusing on simple ideas, pytorch vs tensorflow differences, and embracing failure for continuous learning.', 'chapters': [{'end': 8136.123, 'start': 8043.954, 'title': 'Writing good papers: advice and insights', 'summary': 'Discusses the importance of picking the right problem in research and the need for optimism in attempting to solve meaningful problems, as well as the challenges and uncertainties in writing good papers.', 'duration': 92.169, 'highlights': ['Picking the right problem in research is as important as finding the solution to it, as certain problems can be solved in a particular timeframe.', 'Optimism about making meaningful progress in solving a problem is crucial, as evidenced by the example of working on the meaning of life.', 'The uncertainty and challenges in attempting to solve complex problems are highlighted through the ongoing quest for the meaning of life in a podcast.']}, {'end': 8380.965, 'start': 8136.303, 'title': 'Phd research and writing advice', 'summary': 'Discusses the importance of selecting an interesting and feasible problem for a phd, the significance of genuine excitement, the value of focusing on one simple idea in writing papers, and the benefits of starting the writing process early on to enhance research ideas and identify potential gaps.', 'duration': 244.662, 'highlights': ['The importance of selecting an interesting and feasible problem for a PhD Selecting a problem that is interesting and feasible to make headway in during a PhD is crucial for success.', 'The significance of genuine excitement in research Genuine excitement is essential for grad students and researchers to stay passionate and committed to their work.', 'The value of focusing on one simple idea in writing papers Focusing on one simple idea in writing papers is more valuable as it allows the idea to stand out and encourages deeper thinking.', 'The benefits of starting the writing process early on to enhance research ideas Starting the writing process early helps in developing better research ideas and identifying potential gaps in theories or experiments.', 'The dominance of Python in machine learning programming Python is the recommended programming language for machine learning due to its ease of learning and widespread usage in the field.']}, {'end': 8800.929, 'start': 8381.887, 'title': 'Pytorch vs tensorflow: pros and cons', 'summary': 'Discusses the differences between pytorch and tensorflow, including their historical context, advantages, disadvantages, debugging ease, community competition, and advice for individuals new to machine learning.', 'duration': 419.042, 'highlights': ["PyTorch's imperative nature makes it easier to debug compared to TensorFlow, which is non-imperative. PyTorch's imperative nature makes debugging easier, particularly for those with a background in imperative programming like C++.", 'Competition between PyTorch and TensorFlow drives continuous improvement in both frameworks. Competition between PyTorch and TensorFlow motivates continuous improvement in both frameworks, benefiting the machine learning community.', 'The open-source community swiftly translates code between PyTorch and TensorFlow, bridging the gap for researchers and developers. The open-source community effectively translates code between PyTorch and TensorFlow, facilitating knowledge sharing and collaboration.', 'The chapter advises individuals new to machine learning to persevere through struggles and spend time debugging to gain a deeper understanding. The chapter advises newcomers to machine learning to persevere through debugging and struggles, emphasizing the value of independent problem-solving for deeper learning.', 'The importance of perseverance and embracing struggle is emphasized, correlating it to the development of valuable skills and attitudes. The chapter emphasizes the value of perseverance and struggle in developing essential skills and a resilient attitude, drawing parallels with personal development.']}, {'end': 9020.714, 'start': 8801.564, 'title': 'Embracing failure and the quest for meaning', 'summary': "Emphasizes the importance of embracing failure, citing examples like thomas edison's 990 failed experiments, and delves into the endless quest for the meaning of life and the unique human ability to continuously evolve and learn.", 'duration': 219.15, 'highlights': ['Failure is a common occurrence in research, with experiments failing almost every day, emphasizing the need to have a thick skin and persevere.', "Thomas Edison's relentless experimentation is illustrated with the example of trying 990 things before finding the working filament for the light bulb, highlighting the importance of perseverance.", 'The conversation delves into the endless quest for the meaning of life and the unique human ability to continuously evolve and learn, emphasizing the individuality and continuous evolution of human perspectives and objective functions.', 'The unique human ability to continuously evolve and learn is highlighted, showcasing the diversity of perspectives and the capacity for growth and improvement as a species.']}], 'duration': 976.76, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/FUS6ceIvUnI/pics/FUS6ceIvUnI8043954.jpg', 'highlights': ['Picking the right problem in research is crucial for timely solutions.', 'Optimism about making meaningful progress is crucial in research.', 'Selecting an interesting and feasible problem is crucial for success in PhD.', 'Genuine excitement is essential for grad students and researchers.', 'Focusing on one simple idea in writing papers encourages deeper thinking.', 'Starting the writing process early helps in developing better research ideas.', 'Python is the recommended programming language for machine learning.', "PyTorch's imperative nature makes debugging easier.", 'Competition between PyTorch and TensorFlow drives continuous improvement.', 'The open-source community effectively translates code between PyTorch and TensorFlow.', 'The chapter advises newcomers to machine learning to persevere through debugging and struggles.', 'The chapter emphasizes the value of perseverance and struggle in developing essential skills.', 'Failure is a common occurrence in research, emphasizing the need to persevere.', "Thomas Edison's relentless experimentation highlights the importance of perseverance.", 'The conversation delves into the endless quest for the meaning of life and continuous evolution.', 'The unique human ability to continuously evolve and learn is highlighted.']}], 'highlights': ['Ishan Mizra, along with Yann LeCun, aims to achieve success in self-supervised learning for images and video using transformers and self-attention, similar to GPT-3 in language models.', 'The goal is to leave a robot watching YouTube videos all night and come back to a much smarter robot in the morning.', 'Self-supervised learning uses the data itself as a source of supervision, making it a more scalable and efficient approach compared to supervised learning.', "The term 'self-supervised' is preferred over 'unsupervised' to explicitly indicate that the data itself serves as the source of supervision.", 'The importance of understanding foundational concepts and common sense base to reason and interpret the three-dimensional and four-dimensional world is emphasized, suggesting the need for a deeper understanding beyond self-supervised learning.', 'Non-contrastive methods like clustering and self-distillation offer promising alternatives to contrastive learning by addressing challenges with access to negatives and providing scalability in learning algorithms.', 'RegNet models are efficient in terms of compute and memory, making them effective for scaling self-supervised learning.', 'Access to different modalities simplifies life by observing common concepts and patterns.', 'The integration of AI systems into society, particularly in the context of autonomous driving, is discussed.', 'Picking the right problem in research is crucial for timely solutions.', 'Optimism about making meaningful progress is crucial in research.', 'Python is the recommended programming language for machine learning.', 'The open-source community effectively translates code between PyTorch and TensorFlow.']}