title
Jeremy Howard: fast.ai Deep Learning Courses and Research | Lex Fridman Podcast #35
description
detail
{'title': 'Jeremy Howard: fast.ai Deep Learning Courses and Research | Lex Fridman Podcast #35', 'heatmap': [{'end': 753.535, 'start': 687.479, 'weight': 0.706}, {'end': 2508.434, 'start': 2433.552, 'weight': 0.992}, {'end': 3882.091, 'start': 3810.923, 'weight': 1}, {'end': 4190.607, 'start': 3997.402, 'weight': 0.916}, {'end': 4379.234, 'start': 4312.108, 'weight': 0.763}, {'end': 4752.644, 'start': 4624.169, 'weight': 0.798}, {'end': 4944.089, 'start': 4868.13, 'weight': 0.892}, {'end': 5068.99, 'start': 5001.375, 'weight': 0.868}, {'end': 5318.753, 'start': 5244.721, 'weight': 0.858}, {'end': 5566.111, 'start': 5498.22, 'weight': 0.935}, {'end': 5755.416, 'start': 5684.137, 'weight': 0.956}, {'end': 5878.582, 'start': 5809.087, 'weight': 0.784}], 'summary': "Jeremy howard discusses fastai's practicality and lack of dilutive content, evolution of programming languages and their impact on productivity, python's limitations in data science, potential of deep learning in medicine, fast.ai's origin and practical impact, advancements in deep learning including super convergence allowing for 10 times faster training, evolution of deep learning frameworks, and the effectiveness of spaced repetition learning for continual improvement and memory retention.", 'chapters': [{'end': 60.949, 'segs': [{'end': 69.174, 'src': 'embed', 'start': 38.81, 'weight': 0, 'content': [{'end': 45.918, 'text': 'it has very little BS that can sometimes dilute the value of educational content on popular topics like deep learning.', 'start': 38.81, 'duration': 7.108}, {'end': 57.226, 'text': 'Fast AI has a focus on practical application of deep learning and hands-on exploration of the cutting edge that is incredibly both accessible to beginners and useful to experts.', 'start': 46.799, 'duration': 10.427}, {'end': 60.949, 'text': 'This is the Artificial Intelligence Podcast.', 'start': 58.067, 'duration': 2.882}, {'end': 69.174, 'text': 'If you enjoy it, subscribe on YouTube, give it five stars on iTunes, support it on Patreon or simply connect with me on Twitter.', 'start': 61.449, 'duration': 7.725}], 'summary': 'Fast ai focuses on practical application of deep learning, accessible to beginners and useful to experts.', 'duration': 30.364, 'max_score': 38.81, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk38810.jpg'}], 'start': 0.109, 'title': "Fastai's accessible deep learning", 'summary': 'Delves into fastai, highlighting its practicality and lack of dilutive content, making it a top resource for deep learning, as emphasized by jeremy howard, the founder.', 'chapters': [{'end': 60.949, 'start': 0.109, 'title': 'Accessible deep learning with fastai', 'summary': 'Features jeremy howard, founder of fastai, who emphasizes its accessibility, practical application, and lack of dilutive content, making it a top resource for deep learning enthusiasts.', 'duration': 60.84, 'highlights': ['FastAI is a top resource for deep learning beginners due to its free, easy, insightful, and accessible nature.', 'The focus on practical application and hands-on exploration makes FastAI useful to experts as well.', 'Jeremy Howard, the founder of FastAI, is a successful entrepreneur, educator, and researcher, contributing significantly to the AI community.']}], 'duration': 60.84, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk109.jpg', 'highlights': ['Jeremy Howard, the founder of FastAI, is a successful entrepreneur, educator, and researcher, contributing significantly to the AI community.', 'FastAI is a top resource for deep learning beginners due to its free, easy, insightful, and accessible nature.', 'The focus on practical application and hands-on exploration makes FastAI useful to experts as well.']}, {'end': 861.925, 'segs': [{'end': 351.574, 'src': 'embed', 'start': 324.983, 'weight': 7, 'content': [{'end': 330.344, 'text': "Then often if you want to get a nice programming model, you'll need to add an ORM on top.", 'start': 324.983, 'duration': 5.361}, {'end': 336.206, 'text': "And then, I don't know, there's all these pieces to tie together and it's just a lot more awkward than it should be.", 'start': 330.405, 'duration': 5.801}, {'end': 339.147, 'text': 'There are people that are trying to make it easier.', 'start': 337.007, 'duration': 2.14}, {'end': 344.909, 'text': 'So in particular, I think of F-Sharp Don Syme, who, him and his team,', 'start': 339.227, 'duration': 5.682}, {'end': 351.574, 'text': 'have done a great job of making something like a database appear in the type system.', 'start': 344.909, 'duration': 6.665}], 'summary': 'Developing a programming model with orm can be awkward. f-sharp team simplifies database integration.', 'duration': 26.591, 'max_score': 324.983, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk324983.jpg'}, {'end': 611.127, 'src': 'embed', 'start': 585.371, 'weight': 0, 'content': [{'end': 595.3, 'text': 'And the idea is that this math notation was much more flexible, expressive and also well-defined than traditional math notation,', 'start': 585.371, 'duration': 9.929}, {'end': 596.421, 'text': 'which is none of those things.', 'start': 595.3, 'duration': 1.121}, {'end': 597.402, 'text': 'Math notation is awful.', 'start': 596.501, 'duration': 0.901}, {'end': 606.705, 'text': 'And so he actually turned that into a programming language, because this was the late 50s, all the names were available.', 'start': 599.263, 'duration': 7.442}, {'end': 610.346, 'text': 'So he called his language a programming language, or APL.', 'start': 606.765, 'duration': 3.581}, {'end': 611.127, 'text': 'APL, wow.', 'start': 610.646, 'duration': 0.481}], 'summary': 'Apl, a programming language created in the late 50s, aimed to improve flexibility and expressiveness of math notation.', 'duration': 25.756, 'max_score': 585.371, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk585371.jpg'}, {'end': 753.535, 'src': 'heatmap', 'start': 642.444, 'weight': 2, 'content': [{'end': 646.207, 'text': 'Does it have object-oriented components? Does it have that kind of thing? Not really.', 'start': 642.444, 'duration': 3.763}, {'end': 648.189, 'text': "It's an array-oriented language.", 'start': 646.268, 'duration': 1.921}, {'end': 650.752, 'text': "It's the third path.", 'start': 648.23, 'duration': 2.522}, {'end': 653.595, 'text': 'Are you saying array? Array-oriented.', 'start': 650.872, 'duration': 2.723}, {'end': 655.636, 'text': 'What does it mean to be array-oriented?', 'start': 653.775, 'duration': 1.861}, {'end': 666.444, 'text': "Array-oriented means that you generally don't use any loops, but the whole thing is done with an extreme version of broadcasting.", 'start': 655.656, 'duration': 10.788}, {'end': 669.826, 'text': "if you're familiar with that, NumPy, slash Python concept.", 'start': 666.444, 'duration': 3.382}, {'end': 674.049, 'text': 'You do a lot with one line of code.', 'start': 672.368, 'duration': 1.681}, {'end': 679.733, 'text': 'It looks a lot like math notation, highly compact.', 'start': 674.689, 'duration': 5.044}, {'end': 687.479, 'text': 'The idea is that because you can do so much with one line of code, a single screen of code,', 'start': 680.514, 'duration': 6.965}, {'end': 690.701, 'text': 'you very rarely need more than that to express your program.', 'start': 687.479, 'duration': 3.222}, {'end': 695.425, 'text': 'You can keep it all in your head and you can clearly communicate it.', 'start': 691.382, 'duration': 4.043}, {'end': 700.609, 'text': "It's interesting that APL created two main branches, K and J.", 'start': 696.125, 'duration': 4.484}, {'end': 709.092, 'text': 'J is this kind of like open source niche community of crazy enthusiasts like me.', 'start': 701.649, 'duration': 7.443}, {'end': 712.113, 'text': 'And then the other path, K, was fascinating.', 'start': 709.452, 'duration': 2.661}, {'end': 722.237, 'text': "It's an astonishingly expensive programming language, which many of the world's most ludicrously rich hedge funds use.", 'start': 712.233, 'duration': 10.004}, {'end': 726.682, 'text': 'So the entire K machine is so small.', 'start': 722.937, 'duration': 3.745}, {'end': 735.773, 'text': "it sits inside level three cache on your CPU and it easily wins every benchmark I've ever seen in terms of data processing speed.", 'start': 726.682, 'duration': 9.091}, {'end': 741.682, 'text': "But you don't come across it very much because it's like a hundred thousand dollars per CPU to run it.", 'start': 735.793, 'duration': 5.889}, {'end': 753.535, 'text': "But it's like this path of programming languages is just so much more powerful in every way than the ones that almost anybody uses every day.", 'start': 742.703, 'duration': 10.832}], 'summary': 'J is an array-oriented language with extreme broadcasting, resembling math notation, and is part of an open source niche community. k, on the other hand, is an expensive language used by rich hedge funds, with astonishingly fast data processing capabilities, albeit at a high cost.', 'duration': 52.981, 'max_score': 642.444, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk642444.jpg'}], 'start': 61.449, 'title': 'Evolution of programming languages', 'summary': "Covers jeremy howard's early programming experience, evolution of programming languages like vba, visual basic, delphi, j, apl, k, perl, and python, emphasizing their impact on productivity and challenges in modern programming.", 'chapters': [{'end': 344.909, 'start': 61.449, 'title': 'Programming, music, and database', 'summary': "Discusses jeremy howard's early programming experience on a commodore 64, his interest in music, and his preference for programming in microsoft access, highlighting the challenges in modern programming with relational databases.", 'duration': 283.46, 'highlights': ["Jeremy Howard's early programming experience involved writing a program on a Commodore 64 in BASIC to search for better musical scales than the normal 12-tone, 12-interval scale.", "Jeremy Howard's interest in music led him to play various instruments, with saxophone being his favorite.", "Microsoft Access was Jeremy Howard's favorite programming environment in the earliest days, allowing the creation of user interfaces, tying data and actions to them, and creating reports.", 'The connection between Excel and Access is very close, with Access being the relational database equivalent and offering a richer programming model than Excel when combined with VBA.', 'Modern programming on top of a relational database is described as a lot more awkward than it should be, requiring the use of ORM and facing various challenges.']}, {'end': 666.444, 'start': 344.909, 'title': 'Evolution of programming languages', 'summary': "Discusses the speaker's journey with programming languages, including vba, visual basic, delphi, and j, highlighting their connection with data, the evolution of apl to j, and the unique features of j as an expressive and array-oriented language.", 'duration': 321.535, 'highlights': ["The speaker's journey with programming languages, including VBA, Visual Basic, Delphi, and J, highlighting their connection with data. VBA, Visual Basic, Delphi, and J, connection with data", 'The evolution of APL to J, and the unique features of J as an expressive and array-oriented language. Evolution of APL to J, unique features of J as an expressive and array-oriented language', 'The development of APL, its implementation as a tool of thought, and the creation of J as the most expressive and composable language. Development of APL, implementation as a tool of thought, creation of J as expressive and composable language']}, {'end': 861.925, 'start': 666.444, 'title': 'Evolution of programming languages', 'summary': 'Discusses the compactness and power of programming languages like apl, k, j, and the practicality of languages like perl and python, highlighting the computational focus and the impact on productivity.', 'duration': 195.481, 'highlights': ['The entire K machine sits inside level three cache on the CPU and easily wins every benchmark in terms of data processing speed. The K programming language is highly efficient, sitting inside level three cache on the CPU and outperforming every benchmark in terms of data processing speed.', "APL created two main branches, K and J, with K being an astonishingly expensive programming language used by the world's most rich hedge funds. APL branched into K and J, with K being an expensive language used by the world's richest hedge funds.", 'Perl was great for productivity and flexibility, used in creating an email company called Fastmail in the late 90s and early 2000s. Perl was valued for its productivity and flexibility, used in creating Fastmail in the late 90s and early 2000s.']}], 'duration': 800.476, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk61449.jpg', 'highlights': ["Jeremy Howard's early programming involved writing a program on a Commodore 64 in BASIC to search for better musical scales than the normal 12-tone, 12-interval scale.", "Microsoft Access was Jeremy Howard's favorite programming environment in the earliest days, allowing the creation of user interfaces, tying data and actions to them, and creating reports.", 'The connection between Excel and Access is very close, with Access being the relational database equivalent and offering a richer programming model than Excel when combined with VBA.', 'Modern programming on top of a relational database is described as a lot more awkward than it should be, requiring the use of ORM and facing various challenges.', "The speaker's journey with programming languages, including VBA, Visual Basic, Delphi, and J, highlighting their connection with data.", 'The evolution of APL to J, and the unique features of J as an expressive and array-oriented language.', 'The development of APL, its implementation as a tool of thought, and the creation of J as the most expressive and composable language.', 'The entire K machine sits inside level three cache on the CPU and easily wins every benchmark in terms of data processing speed.', "APL created two main branches, K and J, with K being an astonishingly expensive programming language used by the world's most rich hedge funds.", 'Perl was great for productivity and flexibility, used in creating an email company called Fastmail in the late 90s and early 2000s.']}, {'end': 1433.603, 'segs': [{'end': 1043.828, 'src': 'embed', 'start': 967.148, 'weight': 0, 'content': [{'end': 968.349, 'text': 'just creative idea?', 'start': 967.148, 'duration': 1.201}, {'end': 969.149, 'text': "It's everything.", 'start': 968.449, 'duration': 0.7}, {'end': 973.032, 'text': 'In the end, I want to be productive as a practitioner.', 'start': 969.209, 'duration': 3.823}, {'end': 979.915, 'text': 'At the moment, our understanding of deep learning is incredibly primitive.', 'start': 974.272, 'duration': 5.643}, {'end': 981.475, 'text': "There's very little we understand.", 'start': 980.055, 'duration': 1.42}, {'end': 985.477, 'text': "Most things don't work very well, even though it works better than anything else out there.", 'start': 981.515, 'duration': 3.962}, {'end': 988.298, 'text': "There's so many opportunities to make it better.", 'start': 986.198, 'duration': 2.1}, {'end': 990.579, 'text': 'You look at any domain area like.', 'start': 988.859, 'duration': 1.72}, {'end': 999.402, 'text': "I don't know speech recognition with deep learning or natural language processing classification with deep learning or whatever.", 'start': 992.34, 'duration': 7.062}, {'end': 1004.344, 'text': "Every time I look at an area with deep learning, I always see like, oh, it's terrible.", 'start': 999.442, 'duration': 4.902}, {'end': 1009.405, 'text': "There's lots and lots of obviously stupid ways to do things that need to be fixed.", 'start': 1004.464, 'duration': 4.941}, {'end': 1014.667, 'text': 'So then I want to be able to jump in there and quickly experiment and make them better.', 'start': 1010.246, 'duration': 4.421}, {'end': 1020.271, 'text': 'You think the programming language has a role in that? Huge role.', 'start': 1014.887, 'duration': 5.384}, {'end': 1034.182, 'text': 'Currently, Python has a big gap in terms of our ability to innovate, particularly around recurrent neural networks and natural language processing.', 'start': 1020.571, 'duration': 13.611}, {'end': 1043.828, 'text': "Because it's so slow, the actual loop where we actually loop through words, we have to do that whole thing in CUDA C.", 'start': 1034.962, 'duration': 8.866}], 'summary': "Deep learning has vast potential, but there are many areas for improvement, particularly in python's ability to innovate with recurrent neural networks and natural language processing.", 'duration': 76.68, 'max_score': 967.148, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk967148.jpg'}, {'end': 1207.135, 'src': 'embed', 'start': 1176.616, 'weight': 3, 'content': [{'end': 1183.48, 'text': "And that's just because, if you have to deal with the fact that I've got 10,000 threads and I have to synchronize between them all,", 'start': 1176.616, 'duration': 6.864}, {'end': 1190.723, 'text': "and I have to put my thing into grid blocks and think about warps and all this stuff, it's just so much boilerplate that to do that, well,", 'start': 1183.48, 'duration': 7.243}, {'end': 1192.084, 'text': 'you have to be a specialist at that.', 'start': 1190.723, 'duration': 1.361}, {'end': 1198.628, 'text': "And it's going to be a year's work to optimize that algorithm in that way.", 'start': 1192.224, 'duration': 6.404}, {'end': 1207.135, 'text': 'But with things like tensor comprehensions and TILE and MLIR and TVM.', 'start': 1199.708, 'duration': 7.427}], 'summary': "Optimizing algorithm for 10,000 threads requires specialist and a year's work, but tools like tensor comprehensions, tile, mlir, and tvm can help.", 'duration': 30.519, 'max_score': 1176.616, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk1176616.jpg'}], 'start': 862.606, 'title': 'Programming and deep learning challenges', 'summary': "Discusses python's limitations in data science, the potential of swift for addressing these limitations, challenges in deep learning programming, the need for higher-level programming languages, and the potential of mlir to optimize tensor computations for gpus.", 'chapters': [{'end': 967.148, 'start': 862.606, 'title': "Future of programming and swift's hackability", 'summary': "Discusses the shift to python for data science despite its limitations, and expresses hope for swift's success due to its hackability and potential to address python's shortcomings in data science and machine learning.", 'duration': 104.542, 'highlights': ["Python is used for data science despite its limitations The speaker acknowledges Python's less elegant nature but highlights its usage for data science due to its comprehensive libraries.", "Hope for Swift's success due to hackability and potential to address Python's limitations The speaker expresses hope for Swift's success, emphasizing its potential to be infinitely hackable and address the limitations of Python in data science and machine learning.", 'Desire for a hackable programming language like Swift for research and development The speaker desires a programming language that allows for comprehensive modification and accessibility, particularly for research and development purposes.', "Python's limitations in hackability due to slowness and lack of accessibility The speaker criticizes Python for being slow and extremely unhackable, particularly in comparison to the desired level of accessibility and modifiability."]}, {'end': 1433.603, 'start': 967.148, 'title': 'Deep learning programming challenges', 'summary': "Discusses the challenges of deep learning programming, including limitations in python for recurrent neural networks and natural language processing, the need for higher-level programming languages to enable easier experimentation, and the potential of mlir to optimize tensor computations for gpus and introduce competition to nvidia's dominance in the market.", 'duration': 466.455, 'highlights': ["MLIR's potential to optimize tensor computations for GPUs and introduce competition to NVIDIA's dominance in the market MLIR has the potential to optimize tensor computations for GPUs, leading to competition in the market and potentially reducing NVIDIA's dominance.", 'Limitations in Python for recurrent neural networks and natural language processing Python has limitations in innovating around recurrent neural networks and natural language processing due to its slow performance, particularly in the loop through words, requiring CUDA C for innovation.', 'Need for higher-level programming languages to enable easier experimentation in deep learning There is a need for higher-level programming languages like Swift to enable easier experimentation and creativity with recurrent neural networks and sparse convolutional networks in deep learning.']}], 'duration': 570.997, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk862606.jpg', 'highlights': ["MLIR's potential to optimize tensor computations for GPUs and introduce competition to NVIDIA's dominance in the market", 'Desire for a hackable programming language like Swift for research and development', "Hope for Swift's success due to hackability and potential to address Python's limitations", 'Need for higher-level programming languages to enable easier experimentation in deep learning', 'Python is used for data science despite its limitations']}, {'end': 2273.007, 'segs': [{'end': 1511.451, 'src': 'embed', 'start': 1482.402, 'weight': 0, 'content': [{'end': 1494.29, 'text': "It's very early but in terms of the opportunity, it's to take markets like India and China and Indonesia, which have big populations, Africa,", 'start': 1482.402, 'duration': 11.888}, {'end': 1505.226, 'text': 'small numbers of doctors, and provide diagnostic, particularly treatment planning and triage kind of on device,', 'start': 1494.29, 'duration': 10.936}, {'end': 1511.451, 'text': 'so that if you do a test for malaria or tuberculosis or whatever,', 'start': 1505.226, 'duration': 6.225}], 'summary': 'Opportunity to provide diagnostic and treatment planning in countries with large populations and small numbers of doctors.', 'duration': 29.049, 'max_score': 1482.402, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk1482402.jpg'}, {'end': 1571.938, 'src': 'embed', 'start': 1537.188, 'weight': 5, 'content': [{'end': 1544.012, 'text': "So if your kid is sick and they need something diagnosed through medical imaging, even if you're able to get medical imaging done,", 'start': 1537.188, 'duration': 6.824}, {'end': 1547.894, 'text': 'the person that looks at it will be a nurse at best.', 'start': 1544.012, 'duration': 3.882}, {'end': 1558.444, 'text': "But actually in India, for example, and China, almost no x-rays are read by anybody, by any trained professional, because they don't have enough.", 'start': 1548.915, 'duration': 9.529}, {'end': 1571.938, 'text': 'So if instead we had an algorithm that could take the most likely high risk 5% and say, triage basically, say, okay, someone needs to look at this.', 'start': 1559.345, 'duration': 12.593}], 'summary': 'Lack of trained professionals for medical imaging in india and china, using algorithm for triage could help prioritize high-risk cases.', 'duration': 34.75, 'max_score': 1537.188, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk1537188.jpg'}], 'start': 1433.603, 'title': 'Deep learning and ai in medicine', 'summary': 'Delves into the potential of deep learning to address the global shortage of doctors, especially in developing countries, and discusses the challenges in scaling ai in the medical field, from slow adoption to data privacy issues.', 'chapters': [{'end': 1655.151, 'start': 1433.603, 'title': 'Deep learning in medicine', 'summary': 'Discusses the potential of deep learning in addressing the global shortage of doctors, specifically in developing countries like india, china, and africa, where ai can assist in providing high-quality diagnostic and treatment planning services with limited human expertise.', 'duration': 221.548, 'highlights': ['Deep learning can address the 10x shortage of doctors in the world, particularly in developing countries like India and China, by providing diagnostic and treatment planning assistance with limited human expertise. 10x shortage of doctors, developing countries, diagnostic and treatment planning assistance, limited human expertise', 'AI can enable healthcare workers with minimal training to provide high-quality assessments for diseases such as malaria or tuberculosis, potentially reducing the need for highly trained doctors in certain cases. healthcare worker training, high-quality disease assessment, reduced need for highly trained doctors', 'In some countries, such as India and China, the shortage of trained professionals to read medical imaging results can be addressed by using AI algorithms to triage cases, significantly impacting the medical capabilities in the developing world. shortage of trained professionals, AI triage for medical imaging, impact on medical capabilities in developing countries']}, {'end': 2273.007, 'start': 1655.151, 'title': 'Ai in medical field challenges', 'summary': 'Discusses the challenges in scaling ai in the medical field, from the slow adoption due to regulatory and educational barriers to the issues of data privacy and the need for less data for impactful results.', 'duration': 617.856, 'highlights': ["The slow adoption of AI in the medical field is attributed to regulatory and educational barriers, with only a small number of deep learning practitioners and doctors currently emerging. The medical world's lack of awareness of AI opportunities and the regulatory and educational barriers have slowed down the adoption of AI in the medical field, with only a small number of deep learning practitioners and doctors currently emerging.", 'The need for less data for impactful results is emphasized, with transfer learning being a critical technique for requiring orders of magnitude less data, challenging the assumption of needing more data. The chapter emphasizes the importance of needing less data for impactful results, highlighting the critical technique of transfer learning, which challenges the assumption of needing more data for state-of-the-art results.', "The issue of data privacy and the balance between respecting individuals' privacy and leveraging data for technology advancements is explored, with an emphasis on controlling and sharing medical information at one's discretion. The chapter explores the issue of data privacy, emphasizing the balance between respecting individuals' privacy and leveraging data for technology advancements, with a focus on the control and sharing of medical information at one's discretion."]}], 'duration': 839.404, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk1433603.jpg', 'highlights': ['Deep learning can address the 10x shortage of doctors in the world, particularly in developing countries like India and China, by providing diagnostic and treatment planning assistance with limited human expertise.', 'AI can enable healthcare workers with minimal training to provide high-quality assessments for diseases such as malaria or tuberculosis, potentially reducing the need for highly trained doctors in certain cases.', 'In some countries, such as India and China, the shortage of trained professionals to read medical imaging results can be addressed by using AI algorithms to triage cases, significantly impacting the medical capabilities in the developing world.', 'The slow adoption of AI in the medical field is attributed to regulatory and educational barriers, with only a small number of deep learning practitioners and doctors currently emerging.', 'The need for less data for impactful results is emphasized, with transfer learning being a critical technique for requiring orders of magnitude less data, challenging the assumption of needing more data.', "The issue of data privacy and the balance between respecting individuals' privacy and leveraging data for technology advancements is explored, with an emphasis on controlling and sharing medical information at one's discretion."]}, {'end': 2691.238, 'segs': [{'end': 2301.061, 'src': 'embed', 'start': 2273.127, 'weight': 4, 'content': [{'end': 2279.329, 'text': 'But to return back to the origin story of Fast.ai.', 'start': 2273.127, 'duration': 6.202}, {'end': 2290.313, 'text': 'Right. So before I started Fast.ai, I spent a year researching where are the biggest opportunities for deep learning,', 'start': 2279.409, 'duration': 10.904}, {'end': 2296.898, 'text': 'because I knew from my time at Kaggle in particular, that deep learning had kind of hit this threshold point,', 'start': 2290.313, 'duration': 6.585}, {'end': 2301.061, 'text': 'where it was rapidly becoming the state-of-the-art approach in every area that looked at it.', 'start': 2296.898, 'duration': 4.163}], 'summary': 'Fast.ai founder researched opportunities for deep learning and found it rapidly becoming state-of-the-art in various areas.', 'duration': 27.934, 'max_score': 2273.127, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2273127.jpg'}, {'end': 2508.434, 'src': 'heatmap', 'start': 2433.552, 'weight': 0.992, 'content': [{'end': 2438.553, 'text': 'I think that would be a super powerful thing, you know, bigger societal impact I could have.', 'start': 2433.552, 'duration': 5.001}, {'end': 2441.394, 'text': 'So that, yeah, that was the thinking.', 'start': 2440.314, 'duration': 1.08}, {'end': 2452.238, 'text': 'So so much of fast AI students and researchers and the things you teach are, uh, pragmatically minded, practically minded, freaking,', 'start': 2441.815, 'duration': 10.423}, {'end': 2455.039, 'text': 'figuring out ways how to solve real problems, and fast.', 'start': 2452.238, 'duration': 2.801}, {'end': 2461.021, 'text': "Right. So, from your experience, what's the difference between theory and practice of deep learning??", 'start': 2455.239, 'duration': 5.782}, {'end': 2469.104, 'text': 'Well, most of the research in the deep learning world is a total waste of time.', 'start': 2463.76, 'duration': 5.344}, {'end': 2470.945, 'text': "Right That's what I was getting at.", 'start': 2469.925, 'duration': 1.02}, {'end': 2475.709, 'text': "Yeah It's a problem in science in general.", 'start': 2471.065, 'duration': 4.644}, {'end': 2479.691, 'text': 'Scientists need to be published,', 'start': 2476.369, 'duration': 3.322}, {'end': 2486.116, 'text': 'which means they need to work on things that their peers are extremely familiar with and can recognize and advance in that area.', 'start': 2479.691, 'duration': 6.425}, {'end': 2488.357, 'text': 'So that means that they all need to work on the same thing.', 'start': 2486.276, 'duration': 2.081}, {'end': 2498.301, 'text': "And the thing they work on, there's nothing to encourage them to work on things that are practically useful.", 'start': 2490.271, 'duration': 8.03}, {'end': 2508.434, 'text': "So you get just a whole lot of research, which is minor advances in stuff that's been very highly studied and has no significant practical impact.", 'start': 2498.982, 'duration': 9.452}], 'summary': 'Deep learning research lacks practical impact due to focus on familiar, non-practical areas.', 'duration': 74.882, 'max_score': 2433.552, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2433552.jpg'}, {'end': 2508.434, 'src': 'embed', 'start': 2479.691, 'weight': 0, 'content': [{'end': 2486.116, 'text': 'which means they need to work on things that their peers are extremely familiar with and can recognize and advance in that area.', 'start': 2479.691, 'duration': 6.425}, {'end': 2488.357, 'text': 'So that means that they all need to work on the same thing.', 'start': 2486.276, 'duration': 2.081}, {'end': 2498.301, 'text': "And the thing they work on, there's nothing to encourage them to work on things that are practically useful.", 'start': 2490.271, 'duration': 8.03}, {'end': 2508.434, 'text': "So you get just a whole lot of research, which is minor advances in stuff that's been very highly studied and has no significant practical impact.", 'start': 2498.982, 'duration': 9.452}], 'summary': 'Peers need to work on recognized and practical areas for significant impact.', 'duration': 28.743, 'max_score': 2479.691, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2479691.jpg'}, {'end': 2623.405, 'src': 'embed', 'start': 2576.949, 'weight': 2, 'content': [{'end': 2578.83, 'text': "Maybe I'll just start labeling those two classes.", 'start': 2576.949, 'duration': 1.881}, {'end': 2581.571, 'text': 'And then you start thinking Well, why did I do that manually?', 'start': 2578.87, 'duration': 2.701}, {'end': 2585.092, 'text': "Why can't I just get the system to tell me which things are gonna be hardest?", 'start': 2581.611, 'duration': 3.481}, {'end': 2591.534, 'text': "It's an obvious thing to do, but yeah, it's just like transfer learning.", 'start': 2585.173, 'duration': 6.361}, {'end': 2596.736, 'text': "it's understudied and the academic world just has no reason to care about practical results.", 'start': 2591.534, 'duration': 5.202}, {'end': 2599.897, 'text': "The funny thing is, I've only really ever written one paper.", 'start': 2597.516, 'duration': 2.381}, {'end': 2600.878, 'text': 'I hate writing papers.', 'start': 2600.037, 'duration': 0.841}, {'end': 2602.698, 'text': "I didn't even write it.", 'start': 2601.918, 'duration': 0.78}, {'end': 2605.32, 'text': 'It was my colleague, Sebastian Ruder, who actually wrote it.', 'start': 2602.838, 'duration': 2.482}, {'end': 2607.501, 'text': 'I just did the research for it.', 'start': 2605.58, 'duration': 1.921}, {'end': 2614.244, 'text': 'But it was basically introducing successful transfer learning to NLP for the first time.', 'start': 2608.101, 'duration': 6.143}, {'end': 2615.804, 'text': 'The algorithm is called ULM fit.', 'start': 2614.304, 'duration': 1.5}, {'end': 2623.405, 'text': 'I actually wrote it for the course, for the Fast AI course.', 'start': 2619.742, 'duration': 3.663}], 'summary': 'Introducing successful transfer learning to nlp for the first time using the algorithm ulm fit, presented in a paper written by sebastian ruder, with practical implications.', 'duration': 46.456, 'max_score': 2576.949, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2576949.jpg'}, {'end': 2691.238, 'src': 'embed', 'start': 2640.239, 'weight': 1, 'content': [{'end': 2645.744, 'text': 'it smashed the state of the art on one of the most important data sets in a field that I knew nothing about.', 'start': 2640.239, 'duration': 5.505}, {'end': 2650.447, 'text': 'And I just thought well, this is ridiculous.', 'start': 2646.785, 'duration': 3.662}, {'end': 2661.35, 'text': 'and so, um, I spoke to Sebastian about it and he kindly offered to write it up the results, and so it ended up being published in ACL,', 'start': 2650.447, 'duration': 10.903}, {'end': 2665.591, 'text': 'which is the top link with the computational linguistics conference.', 'start': 2661.35, 'duration': 4.241}, {'end': 2672.693, 'text': "so, like people do actually care once you do it, but I guess it's difficult for maybe like junior researchers.", 'start': 2665.591, 'duration': 7.102}, {'end': 2677.335, 'text': "I don't care whether I get citations or papers or whatever.", 'start': 2673.134, 'duration': 4.201}, {'end': 2682.516, 'text': "There's nothing in my life that makes that important, which is why I've never actually bothered to write a paper myself.", 'start': 2677.575, 'duration': 4.941}, {'end': 2691.238, 'text': 'But for people who do, I guess they have to pick the safe option, which is like yeah,', 'start': 2683.036, 'duration': 8.202}], 'summary': 'Achieved state-of-the-art results, published in top computational linguistics conference.', 'duration': 50.999, 'max_score': 2640.239, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2640239.jpg'}], 'start': 2273.127, 'title': "Fast.ai's origin and practical impact", 'summary': 'Explores the origin of fast.ai, emphasizing its goal to empower domain experts through accessible deep learning, and discusses the practical impact of deep learning, focusing on underemphasized transfer learning and active learning despite their potential impact.', 'chapters': [{'end': 2428.989, 'start': 2273.127, 'title': 'Origin story of fast.ai', 'summary': "Delves into the origin story of fast.ai, highlighting the deep learning opportunities, the author's frustration with limited capacity, and the goal of empowering domain experts by making deep learning accessible and effective.", 'duration': 155.862, 'highlights': ['The author spent a year researching the biggest opportunities for deep learning, recognizing its state-of-the-art impact across various domains.', 'The frustration of not being able to maximize the positive impact of deep learning led to the idea of empowering domain experts.', 'The goal of Fast.ai is to get deep learning into the hands of domain experts in a quick, easy, and effective way, aiming to empower them.', "The author's background in applied and industrial fields, including time at McKinsey & Company, contributes to the respect and appreciation for domain experts."]}, {'end': 2691.238, 'start': 2429.009, 'title': 'Practical impact of deep learning', 'summary': 'Discusses the practical impact of deep learning, highlighting the disparity between academic research and practical application, with a focus on the underemphasis of transfer learning and active learning despite their potential for significant practical impact.', 'duration': 262.229, 'highlights': ['The disparity between academic research and practical application is highlighted, with the underemphasis of practical impact in favor of familiar and recognized research areas.', 'Transfer learning and active learning are identified as underutilized areas with significant practical impact, despite being underrepresented in academic research.', "The speaker's experience with introducing successful transfer learning to NLP for the first time, leading to a significant advancement in the field, is discussed."]}], 'duration': 418.111, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2273127.jpg', 'highlights': ['The goal of Fast.ai is to get deep learning into the hands of domain experts in a quick, easy, and effective way, aiming to empower them.', 'The frustration of not being able to maximize the positive impact of deep learning led to the idea of empowering domain experts.', 'The author spent a year researching the biggest opportunities for deep learning, recognizing its state-of-the-art impact across various domains.', "The author's background in applied and industrial fields, including time at McKinsey & Company, contributes to the respect and appreciation for domain experts.", 'Transfer learning and active learning are identified as underutilized areas with significant practical impact, despite being underrepresented in academic research.', 'The disparity between academic research and practical application is highlighted, with the underemphasis of practical impact in favor of familiar and recognized research areas.', "The speaker's experience with introducing successful transfer learning to NLP for the first time, leading to a significant advancement in the field, is discussed."]}, {'end': 4149.188, 'segs': [{'end': 2722.142, 'src': 'embed', 'start': 2691.238, 'weight': 0, 'content': [{'end': 2693.899, 'text': "make a slight improvement on something that everybody's already working on.", 'start': 2691.238, 'duration': 2.661}, {'end': 2700.474, 'text': 'Yeah Nobody does anything interesting or succeeds in life with the safe option.', 'start': 2695.211, 'duration': 5.263}, {'end': 2710.939, 'text': "The nice thing is nowadays, everybody is now working on NLP transfer learning because since that time we've had GPT and GPT-2 and BERT.", 'start': 2701.834, 'duration': 9.105}, {'end': 2716.582, 'text': "Once you show that something's possible, everybody jumps in, I guess.", 'start': 2712.76, 'duration': 3.822}, {'end': 2722.142, 'text': 'I hope to be a part of, and I hope to see more innovation in active learning in the same way.', 'start': 2717.719, 'duration': 4.423}], 'summary': 'Nlp transfer learning gains popularity with gpt, gpt-2, and bert, spurring innovation in active learning.', 'duration': 30.904, 'max_score': 2691.238, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2691238.jpg'}, {'end': 3539.994, 'src': 'embed', 'start': 3515.943, 'weight': 1, 'content': [{'end': 3522.564, 'text': "So we're actually going to be releasing an audio library soon, which hopefully will encourage development of this because it's so underused.", 'start': 3515.943, 'duration': 6.621}, {'end': 3530.958, 'text': 'The basic approach we used for our super resolution, which Jason uses for Dealtify, of generating high-quality images.', 'start': 3523.185, 'duration': 7.773}, {'end': 3532.822, 'text': 'the exact same approach would work for audio.', 'start': 3530.958, 'duration': 1.864}, {'end': 3536.248, 'text': "No one's done it yet, but it would be a couple of months work.", 'start': 3533.503, 'duration': 2.745}, {'end': 3539.994, 'text': 'Okay, also learning rate in terms of Don Bench.', 'start': 3537.153, 'duration': 2.841}], 'summary': 'Releasing audio library to encourage development, using super resolution approach for high-quality audio, estimated couple of months work.', 'duration': 24.051, 'max_score': 3515.943, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk3515943.jpg'}, {'end': 3609.972, 'src': 'embed', 'start': 3579.486, 'weight': 3, 'content': [{'end': 3590.363, 'text': "Now, no one published that paper because it's not an area of active research in the academic world.", 'start': 3579.486, 'duration': 10.877}, {'end': 3592.424, 'text': 'No academics recognize that this is important.', 'start': 3590.463, 'duration': 1.961}, {'end': 3599.747, 'text': 'And also, deep learning in academia is not considered an experimental science.', 'start': 3592.844, 'duration': 6.903}, {'end': 3607.271, 'text': "So, unlike in physics, where you could say I just saw a subatomic particle do something which the theory doesn't explain,", 'start': 3599.987, 'duration': 7.284}, {'end': 3609.972, 'text': 'you could publish that without an explanation.', 'start': 3607.271, 'duration': 2.701}], 'summary': 'Deep learning is not recognized in academia, hindering its publication and acceptance.', 'duration': 30.486, 'max_score': 3579.486, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk3579486.jpg'}, {'end': 3786.926, 'src': 'embed', 'start': 3752.787, 'weight': 2, 'content': [{'end': 3755.848, 'text': 'they all work together in weird ways.', 'start': 3752.787, 'duration': 3.061}, {'end': 3765.651, 'text': "Different parts of the model this is another thing we've done a lot of work on is research into how different parts of the model should be trained at different rates in different ways.", 'start': 3757.048, 'duration': 8.603}, {'end': 3771.293, 'text': 'We do something we call discriminative learning rates, which is really important, particularly for transfer learning.', 'start': 3766.791, 'duration': 4.502}, {'end': 3777.338, 'text': 'So really, I think in the last 12 months, a lot of people have realized that all this stuff is important.', 'start': 3773.274, 'duration': 4.064}, {'end': 3786.926, 'text': "There's been a lot of great work coming out and we're starting to see algorithms appear which have very, very few dials, if any,", 'start': 3777.378, 'duration': 9.548}], 'summary': 'Research shows importance of discriminative learning rates for model training, leading to algorithms with very few dials.', 'duration': 34.139, 'max_score': 3752.787, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk3752787.jpg'}, {'end': 3882.091, 'src': 'heatmap', 'start': 3794.452, 'weight': 4, 'content': [{'end': 3806.14, 'text': "And instead it's just like we know enough about how to interpret the gradients and the change of gradients we see, to know how to set every parameter.", 'start': 3794.452, 'duration': 11.688}, {'end': 3810.923, 'text': 'So you see the future of deep learning.', 'start': 3806.36, 'duration': 4.563}, {'end': 3814.165, 'text': "where's the input of a human expert needed?", 'start': 3810.923, 'duration': 3.242}, {'end': 3820.209, 'text': 'Well, hopefully the input of the human expert will be almost entirely unneeded from the deep learning point of view.', 'start': 3814.645, 'duration': 5.564}, {'end': 3829.375, 'text': "So again, like Google's approach to this is to try and use thousands of times more compute to run lots and lots of models at the same time,", 'start': 3820.549, 'duration': 8.826}, {'end': 3830.416, 'text': 'and hope that one of them is good.', 'start': 3829.375, 'duration': 1.041}, {'end': 3834.158, 'text': 'AutoML kind of stuff, which I think is insane.', 'start': 3830.436, 'duration': 3.722}, {'end': 3841.723, 'text': 'When you better understand the mechanics of how models learn.', 'start': 3836.82, 'duration': 4.903}, {'end': 3845.566, 'text': "you don't have to try a thousand different models to find which one happens to work the best.", 'start': 3841.723, 'duration': 3.843}, {'end': 3850.549, 'text': "You can just jump straight to the best one, which means that it's more accessible in terms of compute.", 'start': 3845.646, 'duration': 4.903}, {'end': 3854.912, 'text': 'cheaper And also with less hyperparameters to set.', 'start': 3851.71, 'duration': 3.202}, {'end': 3862.256, 'text': "it means you don't need deep learning experts to train your deep learning model for you, which means that domain experts can do more of the work,", 'start': 3854.912, 'duration': 7.344}, {'end': 3870.56, 'text': 'which means that now you can focus the human time on the kind of interpretation, the data gathering, identifying model errors and stuff like that.', 'start': 3862.256, 'duration': 8.304}, {'end': 3872.384, 'text': 'Yeah, the data side.', 'start': 3871.343, 'duration': 1.041}, {'end': 3882.091, 'text': 'How often do you work with data these days in terms of the cleaning? Like Darwin looked at different species while traveling about.', 'start': 3872.844, 'duration': 9.247}], 'summary': 'Deep learning aims for minimal human expert input by using fewer models and less hyperparameters, making it more accessible and cost-effective.', 'duration': 25.757, 'max_score': 3794.452, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk3794452.jpg'}, {'end': 3872.384, 'src': 'embed', 'start': 3845.646, 'weight': 8, 'content': [{'end': 3850.549, 'text': "You can just jump straight to the best one, which means that it's more accessible in terms of compute.", 'start': 3845.646, 'duration': 4.903}, {'end': 3854.912, 'text': 'cheaper And also with less hyperparameters to set.', 'start': 3851.71, 'duration': 3.202}, {'end': 3862.256, 'text': "it means you don't need deep learning experts to train your deep learning model for you, which means that domain experts can do more of the work,", 'start': 3854.912, 'duration': 7.344}, {'end': 3870.56, 'text': 'which means that now you can focus the human time on the kind of interpretation, the data gathering, identifying model errors and stuff like that.', 'start': 3862.256, 'duration': 8.304}, {'end': 3872.384, 'text': 'Yeah, the data side.', 'start': 3871.343, 'duration': 1.041}], 'summary': 'Accessible, cheaper, and requires fewer hyperparameters, enabling domain experts to focus on interpretation and data gathering.', 'duration': 26.738, 'max_score': 3845.646, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk3845646.jpg'}, {'end': 3924.302, 'src': 'embed', 'start': 3902.326, 'weight': 5, 'content': [{'end': 3910.893, 'text': 'we learn how to analyze the results of the model by looking at examples of misclassified images and looking at a classification matrix,', 'start': 3902.326, 'duration': 8.567}, {'end': 3917.638, 'text': "and then doing research on Google to learn about the kinds of things that it's misclassifying.", 'start': 3910.893, 'duration': 6.745}, {'end': 3924.302, 'text': 'So, to me, one of the really cool things about machine learning models in general is that when you interpret them,', 'start': 3917.658, 'duration': 6.644}], 'summary': 'Learning to analyze model results and misclassified images.', 'duration': 21.976, 'max_score': 3902.326, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk3902326.jpg'}, {'end': 4160.433, 'src': 'embed', 'start': 4130.774, 'weight': 6, 'content': [{'end': 4135.377, 'text': "You just click on the link and you click start and you're going.", 'start': 4130.774, 'duration': 4.603}, {'end': 4136.237, 'text': "You'll go GCP.", 'start': 4135.397, 'duration': 0.84}, {'end': 4138.759, 'text': "I have to confess, I've never used the Google GCP.", 'start': 4136.296, 'duration': 2.463}, {'end': 4143.481, 'text': 'Yeah, GCP gives you $300 of compute for free, which is really nice.', 'start': 4138.799, 'duration': 4.682}, {'end': 4149.188, 'text': 'But as I say, Salamander and Paperspace are even easier still.', 'start': 4144.946, 'duration': 4.242}, {'end': 4160.433, 'text': 'Okay So, from the perspective of deep learning frameworks, you work with Fast.ai, if you go to this framework, and PyTorch and TensorFlow.', 'start': 4149.428, 'duration': 11.005}], 'summary': 'Google gcp offers $300 of free compute, but salamander and paperspace are easier. fast.ai uses pytorch and tensorflow.', 'duration': 29.659, 'max_score': 4130.774, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4130774.jpg'}], 'start': 2691.238, 'title': 'Advancements in deep learning', 'summary': "Discusses jeremy howard's dawn bench competition success, the advantages of single gpu training, computational photography, audio innovations, and the future of deep learning, highlighting advancements such as super convergence allowing for 10 times faster training with a 10 times higher learning rate, and the accessibility of cloud platforms like google gcp and aws.", 'chapters': [{'end': 3096.652, 'start': 2691.238, 'title': 'Dawn bench competition success', 'summary': "Discusses jeremy howard's success in the dawn bench competition, achieving top leaderboard positions for both time and cost in training models for cifar-10 and imagenet, demonstrating the effectiveness of using smaller image sizes and transfer learning in deep learning.", 'duration': 405.414, 'highlights': ['Jeremy Howard and his team achieved top leaderboard positions for both time and cost in training models for CIFAR-10 and ImageNet in the Dawn Bench competition They were able to achieve top positions in the competition, demonstrating their success in training models efficiently and cost-effectively.', 'Demonstrated the effectiveness of using smaller image sizes in training models By using 64 by 64 images for training, they were able to achieve excellent results and then further train the same model with larger images, showcasing the effectiveness of transfer learning and smaller image sizes in deep learning.', 'The chapter discusses the challenges and innovative strategies employed by Jeremy Howard and his team in the Dawn Bench competition The team faced challenges from competitors such as Google and Intel but employed innovative strategies, including using smaller image sizes and transfer learning, to achieve success in the competition.']}, {'end': 3385.43, 'start': 3098.453, 'title': 'Single gpu advantages', 'summary': 'Discusses the advantages of single gpu training for deep learning, highlighting how using smaller datasets and single gpus can yield comparable results to larger resources and multiple gpus, which can be detrimental to creativity and accessibility.', 'duration': 286.977, 'highlights': ['Using smaller datasets and single GPUs can yield comparable results to larger resources and multiple GPUs. The speaker mentions that training on smaller subsets of ImageNet can be completed on a single GPU in 10 minutes, and the results are directly transferable to ImageNet nearly all the time.', "The detrimental effect of large resources and multiple GPUs on creativity and accessibility in deep learning. The discussion emphasizes that the reliance on large resources can discourage creativity and accessibility, as people may abstain from learning deep learning due to the belief that it's only accessible to big companies like Google.", 'Major breakthroughs in AI can be achieved on a single GPU, contrary to the belief that multiple GPUs are necessary. The speaker asserts that all the major breakthroughs in AI in the next 20 years will be achievable on a single GPU, citing that none of the big breakthroughs of the last 20 years have required multiple GPUs.']}, {'end': 3672.327, 'start': 3385.43, 'title': 'Computational photography and audio innovations', 'summary': 'Discusses the potential of computational photography and audio innovations, highlighting the lack of progress in combining audio from multiple sources, and the overlooked concept of super convergence in deep learning, which allows for 10 times faster training with a 10 times higher learning rate.', 'duration': 286.897, 'highlights': ['The lack of progress in automatically combining audio from multiple sources to improve the combined audio is highlighted, indicating a learning problem and the need for innovation in this area.', 'The concept of super convergence in deep learning is discussed, emphasizing that certain networks, with specific settings of high parameters, could be trained 10 times faster by using a 10 times higher learning rate.', 'The potential of computational photography is emphasized, citing examples like the Google Pixel Nightlight and the use of computational techniques for background blurring, with the prediction that similar advancements will occur in the field of audio.', 'The upcoming release of an audio library is mentioned as a potential encouragement for the development of audio innovations, as the approach used for super resolution in images can be applied to audio with a couple of months of work.', 'The disinterest in publishing unexplained experimental results in the deep learning world is highlighted, pointing out the uniqueness of this disinterest compared to other scientific fields and the potential insights found in unpublished papers.']}, {'end': 4149.188, 'start': 3672.407, 'title': 'Future of deep learning: learning rate magic and cloud options', 'summary': 'Discusses the future of deep learning, focusing on learning rate magic and the cloud options for training networks, highlighting the potential disappearance of the learning rate concept and the accessibility of cloud platforms like google gcp and aws. it emphasizes the importance of interpreting machine learning models to understand data and the significant advancements in making cloud platforms more user-friendly.', 'duration': 476.781, 'highlights': ['Interpreting machine learning models is crucial for understanding data and becoming a domain expert, enabling focus on important features and addressing issues like data leakage.', 'The future of deep learning may see the disappearance of the learning rate concept, with advancements allowing the interpretation of gradients to set every parameter, reducing the need for human expertise.', 'Cloud platforms like Google GCP and AWS are becoming more accessible, with GCP offering $300 of free compute and user-friendly options like Salamander and Paperspace requiring minimal setup.', 'The combination of weight decay, learning rate, and discriminative learning rates plays a significant role in optimizing optimizers and training different parts of the model at varying rates.', 'Training models with super convergence using much higher learning rates than expected, along with discriminative learning rates, has been achieved, indicating substantial progress in understanding optimizers and model training.']}], 'duration': 1457.95, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk2691238.jpg', 'highlights': ['Jeremy Howard and team achieved top leaderboard positions in Dawn Bench competition for CIFAR-10 and ImageNet, demonstrating efficient and cost-effective model training.', 'Using smaller image sizes for training showcased the effectiveness of transfer learning and smaller image sizes in deep learning.', 'Training on smaller subsets of ImageNet can be completed on a single GPU in 10 minutes, yielding comparable results to larger resources and multiple GPUs.', 'Reliance on large resources can discourage creativity and accessibility in deep learning, emphasizing the importance of single GPU training.', 'Major breakthroughs in AI can be achieved on a single GPU, contrary to the belief that multiple GPUs are necessary.', 'The potential of super convergence in deep learning allows for 10 times faster training with a 10 times higher learning rate.', 'Computational photography advancements like Google Pixel Nightlight and background blurring techniques indicate potential similar advancements in audio innovations.', 'Interpreting machine learning models is crucial for understanding data and becoming a domain expert, enabling focus on important features and addressing issues like data leakage.', 'Cloud platforms like Google GCP and AWS are becoming more accessible, offering free compute and user-friendly options like Salamander and Paperspace.', 'Training models with super convergence using much higher learning rates than expected, along with discriminative learning rates, has been achieved, indicating substantial progress in understanding optimizers and model training.']}, {'end': 5540.42, 'segs': [{'end': 4379.234, 'src': 'heatmap', 'start': 4312.108, 'weight': 0.763, 'content': [{'end': 4318.573, 'text': 'and for lots of people that have won big learning competitions with it and written academic papers with it.', 'start': 4312.108, 'duration': 6.465}, {'end': 4319.474, 'text': "It's made a big difference.", 'start': 4318.673, 'duration': 0.801}, {'end': 4327.723, 'text': "We're still limited, though, by Python, and particularly this problem with things like recurrent neural nets, say,", 'start': 4320.655, 'duration': 7.068}, {'end': 4334.551, 'text': "where you just can't change things unless you accept it going so slowly that it's impractical.", 'start': 4327.723, 'duration': 6.828}, {'end': 4343.7, 'text': "So in the latest incarnation of the course and with some of the research we're now starting to do, we're starting to do some stuff in Swift.", 'start': 4335.732, 'duration': 7.968}, {'end': 4351.027, 'text': "I think we're three years away from that being super practical, but I'm in no hurry.", 'start': 4344.541, 'duration': 6.486}, {'end': 4353.89, 'text': "I'm very happy to invest the time to get there.", 'start': 4351.067, 'duration': 2.823}, {'end': 4363.789, 'text': 'But with that we actually already have a nascent version of the Fast.ai library for Vision running on Swift for TensorFlow,', 'start': 4355.526, 'duration': 8.263}, {'end': 4367.13, 'text': 'because Python for TensorFlow is not going to cut it.', 'start': 4363.789, 'duration': 3.341}, {'end': 4369.911, 'text': "It's just a disaster.", 'start': 4368.15, 'duration': 1.761}, {'end': 4379.234, 'text': 'What they did was they tried to replicate the bits that people were saying they like about PyTorch, this interactive computation,', 'start': 4369.971, 'duration': 9.263}], 'summary': 'Fast.ai is making a big difference, exploring swift for tensorflow due to limitations with python, aiming for a practical version in three years.', 'duration': 67.126, 'max_score': 4312.108, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4312108.jpg'}, {'end': 4433.75, 'src': 'embed', 'start': 4403.652, 'weight': 3, 'content': [{'end': 4406.334, 'text': "Yeah, I think it's probably very difficult to do that retooling.", 'start': 4403.652, 'duration': 2.682}, {'end': 4412.797, 'text': 'Yeah Well, particularly the way TensorFlow was written, it was written by a lot of people very quickly in a very disorganized way.', 'start': 4406.494, 'duration': 6.303}, {'end': 4420.461, 'text': "So when you actually look in the code, as I do often, I'm always just like, oh, God, what were they thinking? It's pretty awful.", 'start': 4413.438, 'duration': 7.023}, {'end': 4429.767, 'text': "So I'm really extremely negative about the potential future for Python for TensorFlow.", 'start': 4421.522, 'duration': 8.245}, {'end': 4433.75, 'text': 'Swift for TensorFlow can be a different beast altogether.', 'start': 4430.547, 'duration': 3.203}], 'summary': 'Retooling tensorflow in python is difficult due to disorganized code, leading to negative outlook. swift for tensorflow offers a different approach.', 'duration': 30.098, 'max_score': 4403.652, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4403652.jpg'}, {'end': 4752.644, 'src': 'heatmap', 'start': 4624.169, 'weight': 0.798, 'content': [{'end': 4633.075, 'text': "It'll be interesting to see what happens with Apple because Apple hasn't shown any sign of caring about numeric programming in Swift.", 'start': 4624.169, 'duration': 8.906}, {'end': 4635.077, 'text': "So hopefully they'll..", 'start': 4633.836, 'duration': 1.241}, {'end': 4645.028, 'text': 'get off their ass and start appreciating this because currently all of their low-level libraries are not written in Swift.', 'start': 4636.598, 'duration': 8.43}, {'end': 4647.351, 'text': "They're not particularly Swifty at all.", 'start': 4645.068, 'duration': 2.283}, {'end': 4650.295, 'text': "Stuff like CoreML, they're really pretty rubbish.", 'start': 4647.411, 'duration': 2.884}, {'end': 4652.998, 'text': "So.. Yeah, so there's a long way to go.", 'start': 4650.835, 'duration': 2.163}, {'end': 4660.066, 'text': 'But at least one nice thing is that Swift for TensorFlow can actually directly use Python code and Python libraries.', 'start': 4653.679, 'duration': 6.387}, {'end': 4668.216, 'text': 'Literally the entire Lesson 1 notebook of Fast.ai runs in Swift right now in Python mode.', 'start': 4661.588, 'duration': 6.628}, {'end': 4671.26, 'text': "So that's a nice intermediate thing.", 'start': 4668.657, 'duration': 2.603}, {'end': 4674.969, 'text': 'How long does it take?', 'start': 4671.687, 'duration': 3.282}, {'end': 4681.613, 'text': 'if you look at the two Fast.ai courses, how long does it take to get from point zero to completing both courses?', 'start': 4674.969, 'duration': 6.644}, {'end': 4684.095, 'text': 'It varies a lot.', 'start': 4683.314, 'duration': 0.781}, {'end': 4691.259, 'text': 'Somewhere between two months and two years, generally.', 'start': 4685.896, 'duration': 5.363}, {'end': 4708.444, 'text': 'So for two months, how many hours a day? So like somebody who is a very competent coder can do 70 hours per course and pick up 70.', 'start': 4693.148, 'duration': 15.296}, {'end': 4709.165, 'text': "That's it? Okay.", 'start': 4708.444, 'duration': 0.721}, {'end': 4722.729, 'text': 'Yep But a lot of people I know take a year off to study fast AI full time and say at the end of the year they feel pretty competent.', 'start': 4709.225, 'duration': 13.504}, {'end': 4725.549, 'text': "Because generally there's a lot of other things you do.", 'start': 4723.409, 'duration': 2.14}, {'end': 4728.51, 'text': "Generally they'll be entering Kaggle competitions.", 'start': 4725.849, 'duration': 2.661}, {'end': 4731.37, 'text': "They might be reading Ian Goodfellow's book.", 'start': 4728.67, 'duration': 2.7}, {'end': 4733.491, 'text': 'They might be doing a bunch of stuff.', 'start': 4731.45, 'duration': 2.041}, {'end': 4741.705, 'text': 'And often, particularly if they are a domain expert, their coding skills might be a little on the pedestrian side.', 'start': 4734.631, 'duration': 7.074}, {'end': 4744.411, 'text': "So part of it's just like doing a lot more writing.", 'start': 4741.766, 'duration': 2.645}, {'end': 4752.644, 'text': 'What do you find is the bottleneck for people, usually, except getting started and setting stuff up? I would say coding.', 'start': 4744.778, 'duration': 7.866}], 'summary': 'Apple needs to focus on swift for numeric programming; swift for tensorflow can directly use python code and libraries; fast.ai courses can take between two months and two years, with 70 hours per course for competent coders.', 'duration': 128.475, 'max_score': 4624.169, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4624169.jpg'}, {'end': 4691.259, 'src': 'embed', 'start': 4661.588, 'weight': 1, 'content': [{'end': 4668.216, 'text': 'Literally the entire Lesson 1 notebook of Fast.ai runs in Swift right now in Python mode.', 'start': 4661.588, 'duration': 6.628}, {'end': 4671.26, 'text': "So that's a nice intermediate thing.", 'start': 4668.657, 'duration': 2.603}, {'end': 4674.969, 'text': 'How long does it take?', 'start': 4671.687, 'duration': 3.282}, {'end': 4681.613, 'text': 'if you look at the two Fast.ai courses, how long does it take to get from point zero to completing both courses?', 'start': 4674.969, 'duration': 6.644}, {'end': 4684.095, 'text': 'It varies a lot.', 'start': 4683.314, 'duration': 0.781}, {'end': 4691.259, 'text': 'Somewhere between two months and two years, generally.', 'start': 4685.896, 'duration': 5.363}], 'summary': 'Fast.ai lesson 1 notebook runs in swift in python mode. completion time for both courses varies from 2 months to 2 years.', 'duration': 29.671, 'max_score': 4661.588, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4661588.jpg'}, {'end': 4844.709, 'src': 'embed', 'start': 4753.604, 'weight': 0, 'content': [{'end': 4757.507, 'text': 'I would say the people who are strong coders pick it up the best.', 'start': 4753.604, 'duration': 3.903}, {'end': 4760.729, 'text': 'Although another bottleneck is,', 'start': 4758.808, 'duration': 1.921}, {'end': 4770.836, 'text': "people who have a lot of experience of classic statistics can really struggle because the intuition is so the opposite of what they're used to.", 'start': 4760.729, 'duration': 10.107}, {'end': 4779.382, 'text': "They're very used to trying to reduce the number of parameters in their model and looking at individual coefficients and stuff like that.", 'start': 4770.896, 'duration': 8.486}, {'end': 4787.126, 'text': 'So I find people who have a lot of coding background and know nothing about statistics are generally going to be the best off.', 'start': 4779.402, 'duration': 7.724}, {'end': 4794.829, 'text': 'So you taught several courses on deep learning, and as Feynman says, the best way to understand something is to teach it.', 'start': 4788.627, 'duration': 6.202}, {'end': 4799.765, 'text': 'What have you learned about deep learning from teaching it? a lot.', 'start': 4795.89, 'duration': 3.875}, {'end': 4803.491, 'text': "It's a key reason for me to teach the courses.", 'start': 4800.626, 'duration': 2.865}, {'end': 4803.751, 'text': 'I mean.', 'start': 4803.551, 'duration': 0.2}, {'end': 4809.34, 'text': "obviously it's going to be necessary to achieve our goal of getting domain experts to be familiar with deep learning,", 'start': 4803.751, 'duration': 5.589}, {'end': 4814.007, 'text': 'but it was also necessary for me to achieve my goal of being really familiar with deep learning.', 'start': 4809.34, 'duration': 4.667}, {'end': 4825.663, 'text': 'I mean to see so many domain experts from so many different backgrounds.', 'start': 4817.16, 'duration': 8.503}, {'end': 4833.805, 'text': "it's definitely, I wouldn't say, taught me, but convinced me, something that I liked to believe was true, which was anyone can do it.", 'start': 4825.663, 'duration': 8.142}, {'end': 4840.247, 'text': "So there's a lot of kind of snobbishness out there about only certain people can learn to code.", 'start': 4834.926, 'duration': 5.321}, {'end': 4842.868, 'text': 'only certain people are going to be smart enough to do AI.', 'start': 4840.247, 'duration': 2.621}, {'end': 4844.709, 'text': "That's definitely bullshit.", 'start': 4843.148, 'duration': 1.561}], 'summary': 'Strong coders excel in deep learning; coding background trumps classic statistics experience, as anyone can learn deep learning.', 'duration': 91.105, 'max_score': 4753.604, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4753604.jpg'}, {'end': 4944.089, 'src': 'heatmap', 'start': 4868.13, 'weight': 0.892, 'content': [{'end': 4878.057, 'text': "And but if the ones who don't give up, pretty much everybody succeeds, you know, even if at first I'm just kind of like thinking like wow,", 'start': 4868.13, 'duration': 9.927}, {'end': 4880.499, 'text': "they really aren't quite getting it yet, are they?", 'start': 4878.057, 'duration': 2.442}, {'end': 4884.182, 'text': 'But eventually people get it and they succeed.', 'start': 4880.559, 'duration': 3.623}, {'end': 4885.723, 'text': "So I think that's been.", 'start': 4884.762, 'duration': 0.961}, {'end': 4891.767, 'text': "I think they're both things I liked to believe was true, but I don't feel like I really had strong evidence for them to be true.", 'start': 4885.723, 'duration': 6.044}, {'end': 4893.669, 'text': "but now I can say I've seen it again and again.", 'start': 4891.767, 'duration': 1.902}, {'end': 4903.413, 'text': 'What advice do you have for someone who wants to get started in deep learning? Train lots of models.', 'start': 4894.989, 'duration': 8.424}, {'end': 4907.415, 'text': "That's how you learn it.", 'start': 4904.494, 'duration': 2.921}, {'end': 4911.857, 'text': "It's not just me.", 'start': 4909.396, 'duration': 2.461}, {'end': 4915.539, 'text': "I think our course is very good, but also lots of people independently has said it's very good.", 'start': 4911.877, 'duration': 3.662}, {'end': 4919.561, 'text': 'It recently won the CogX award for AI courses as being the best in the world.', 'start': 4915.559, 'duration': 4.002}, {'end': 4922.102, 'text': "I'd say come to our course, course.fast.ai.", 'start': 4919.581, 'duration': 2.521}, {'end': 4931.065, 'text': 'And the thing I keep on harping on in my lessons is train models, print out the inputs to the models, print out to the outputs to the models.', 'start': 4923.042, 'duration': 8.023}, {'end': 4935.386, 'text': 'like study, you know, change the inputs a bit.', 'start': 4931.065, 'duration': 4.321}, {'end': 4937.347, 'text': 'look at how the outputs vary.', 'start': 4935.386, 'duration': 1.961}, {'end': 4944.089, 'text': "just run lots of experiments to get a, you know, an intuitive understanding of what's going on.", 'start': 4937.347, 'duration': 6.742}], 'summary': 'Training lots of models helps in deep learning. course.fast.ai won cogx award for best ai courses.', 'duration': 75.959, 'max_score': 4868.13, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4868130.jpg'}, {'end': 5068.99, 'src': 'heatmap', 'start': 5001.375, 'weight': 0.868, 'content': [{'end': 5005.756, 'text': "There's other widgets that help you study the results to see where the errors are happening.", 'start': 5001.375, 'duration': 4.381}, {'end': 5013.779, 'text': "And so now we've got over a thousand replies in our share your work here thread of students saying, here's the thing I built.", 'start': 5006.417, 'duration': 7.362}, {'end': 5017.32, 'text': "And so there's people who are like, And a lot of them are state of the art.", 'start': 5014.339, 'duration': 2.981}, {'end': 5021.121, 'text': "Like somebody said, oh, I tried looking at Devangari characters and I couldn't believe it.", 'start': 5017.6, 'duration': 3.521}, {'end': 5025.963, 'text': 'The thing that came out was more accurate than the best academic paper after lesson one.', 'start': 5021.141, 'duration': 4.822}, {'end': 5032.905, 'text': "And then there's others which are just more kind of fun, like somebody is doing Trinidad and Tobago hummingbirds.", 'start': 5026.683, 'duration': 6.222}, {'end': 5034.846, 'text': "She said that's kind of their national bird.", 'start': 5033.125, 'duration': 1.721}, {'end': 5038.367, 'text': "And she's got something that can now classify Trinidad and Tobago hummingbirds.", 'start': 5034.906, 'duration': 3.461}, {'end': 5044.149, 'text': 'So, yeah, train models, fine tune models with your data set and then study their inputs and outputs.', 'start': 5038.867, 'duration': 5.282}, {'end': 5047.397, 'text': 'How much is Fast.ai courses? Free.', 'start': 5045.236, 'duration': 2.161}, {'end': 5049.799, 'text': 'Everything we do is free.', 'start': 5048.998, 'duration': 0.801}, {'end': 5052.68, 'text': 'We have no revenue sources of any kind.', 'start': 5050.559, 'duration': 2.121}, {'end': 5054.361, 'text': "It's just a service to the community.", 'start': 5052.74, 'duration': 1.621}, {'end': 5056.323, 'text': "You're a saint.", 'start': 5055.622, 'duration': 0.701}, {'end': 5062.766, 'text': 'Okay, Once a person understands the basics, trains a bunch of models.', 'start': 5056.683, 'duration': 6.083}, {'end': 5068.99, 'text': 'if we look at the scale of years, what advice do you have for someone wanting to eventually become an expert?', 'start': 5062.766, 'duration': 6.224}], 'summary': 'Fast.ai offers free courses with over a thousand student replies and state-of-the-art results. the platform is a free service to the community.', 'duration': 67.615, 'max_score': 5001.375, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk5001375.jpg'}, {'end': 5318.753, 'src': 'heatmap', 'start': 5244.721, 'weight': 0.858, 'content': [{'end': 5253.647, 'text': 'but if you keep costs super low and try and save up some money beforehand so you can afford to have some time,', 'start': 5244.721, 'duration': 8.926}, {'end': 5257.417, 'text': 'then just sticking with it is one important thing.', 'start': 5255.535, 'duration': 1.882}, {'end': 5263.182, 'text': 'Doing something you understand and care about is important.', 'start': 5258.097, 'duration': 5.085}, {'end': 5271.99, 'text': 'The biggest problem I see with deep learning people is they do a PhD in deep learning and then they try and commercialize their PhD,', 'start': 5263.422, 'duration': 8.568}, {'end': 5275.633, 'text': "which is a waste of time, because that doesn't solve an actual problem.", 'start': 5271.99, 'duration': 3.643}, {'end': 5281.719, 'text': 'You picked your PhD topic because it was an interesting engineering or math or research exercise.', 'start': 5276.034, 'duration': 5.685}, {'end': 5290.741, 'text': "But yeah, if you've actually spent time as a recruiter and you know that most of your time was spent sifting through resumes,", 'start': 5282.539, 'duration': 8.202}, {'end': 5294.742, 'text': "and you know that most of the time you're just looking for certain kinds of things,", 'start': 5290.741, 'duration': 4.001}, {'end': 5303.784, 'text': "and you can try doing that with a model for a few minutes and see whether that's something which a model seems to be able to do as well as you could,", 'start': 5294.742, 'duration': 9.042}, {'end': 5306.604, 'text': "then you're on the right track to creating a startup.", 'start': 5303.784, 'duration': 2.82}, {'end': 5308.985, 'text': 'And then I think just, yeah, being..', 'start': 5307.644, 'duration': 1.341}, {'end': 5318.753, 'text': 'just be pragmatic and try and stay away from venture capital money as long as possible, preferably forever.', 'start': 5310.405, 'duration': 8.348}], 'summary': "Keep costs low, focus on real problems, test model's efficacy, avoid venture capital", 'duration': 74.032, 'max_score': 5244.721, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk5244721.jpg'}, {'end': 5345.251, 'src': 'embed', 'start': 5319.233, 'weight': 2, 'content': [{'end': 5323.377, 'text': 'So yeah, on that point do you venture capital?', 'start': 5319.233, 'duration': 4.144}, {'end': 5324.838, 'text': 'So did you.', 'start': 5323.457, 'duration': 1.381}, {'end': 5328.001, 'text': 'were you able to successfully run startups with self funded??', 'start': 5324.838, 'duration': 3.163}, {'end': 5330.143, 'text': 'Yeah, so, my first two were self funded.', 'start': 5328.282, 'duration': 1.861}, {'end': 5332.085, 'text': 'And that was the right way to do it.', 'start': 5330.223, 'duration': 1.862}, {'end': 5332.946, 'text': "That's scary.", 'start': 5332.385, 'duration': 0.561}, {'end': 5343.409, 'text': 'No, VC startups are much more scary because you have these people on your back, who do this all the time and who have done it for years,', 'start': 5334.381, 'duration': 9.028}, {'end': 5345.251, 'text': 'telling you grow, grow, grow, grow.', 'start': 5343.409, 'duration': 1.842}], 'summary': 'The speaker ran two successful self-funded startups, highlighting the advantages over vc-funded startups.', 'duration': 26.018, 'max_score': 5319.233, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk5319233.jpg'}], 'start': 4149.428, 'title': 'Deep learning and tensorflow evolution', 'summary': "Explores the strengths and limitations of deep learning frameworks, evolution from theano and tensorflow to pytorch and fastai, emphasizing pytorch's accessibility and flexibility, the multi-layered api of fast.ai, and potential of swift for tensorflow as a better alternative due to its utilization of mlir and llvm.", 'chapters': [{'end': 4353.89, 'start': 4149.428, 'title': 'Deep learning frameworks strengths', 'summary': 'Discusses the strengths and limitations of deep learning frameworks, highlighting the evolution from theano and tensorflow to pytorch and fastai, emphasizing the accessibility and flexibility of pytorch and the multi-layered api of fast.ai for efficient research and teaching.', 'duration': 204.462, 'highlights': ["PyTorch's flexibility and accessibility allowed for rapid research and experimentation, enabling the development of a multi-layered API for efficient research and teaching.", 'The Fast.ai library provides an API that allows training of state-of-the-art neural networks in three lines of code, contributing to winning learning competitions and writing academic papers.', 'The limitations of PyTorch for newcomers and researchers, requiring manual management of training loops and gradients, which hinders algorithmic focus and accessibility.', 'The evolution from Theano and TensorFlow to PyTorch and FastAI reflects the growth and development of the ecosystem of deep learning libraries.', 'Exploration of Swift for overcoming limitations in Python for tasks like recurrent neural nets is underway, foreseeing practicality in three years.']}, {'end': 4568.373, 'start': 4355.526, 'title': 'The future of tensorflow: python vs swift', 'summary': 'Discusses the limitations of python for tensorflow, emphasizing its slow runtime compared to pytorch and the inefficiency of tf.data, while highlighting the potential of swift for tensorflow as a better alternative due to its utilization of mlir and llvm.', 'duration': 212.847, 'highlights': ['The limitations of Python for TensorFlow are highlighted, with its runtime being 10 times slower than PyTorch, and its inefficiency in tf.data for data processing.', 'The potential of Swift for TensorFlow is discussed, emphasizing its capability to be a better alternative due to its use of MLIR and LLVM for efficient compilation.', 'The recommendation of fast AI and PyTorch for new students due to their ability to enable quicker understanding of concepts and learning of state-of-the-art techniques.']}, {'end': 4891.767, 'start': 4568.373, 'title': 'Swift for tensorflow and deep learning insights', 'summary': "Discusses the potential for swift to be used in data science, the challenges with apple's support for numeric programming in swift, the time required to complete fast.ai courses, common bottlenecks for learners, and key insights about deep learning from teaching it.", 'duration': 323.394, 'highlights': ["Swift for TensorFlow has potential but faces challenges The transcript discusses the potential for Swift to be used in data science, but highlights the challenges such as the lack of data science community libraries and tooling, as well as Apple's lack of support for numeric programming in Swift.", 'Time required to complete Fast.ai courses varies It is mentioned that the time to complete Fast.ai courses varies, with some individuals taking between two months and two years, and the possibility of completing both courses in 70 hours for a competent coder.', 'Bottlenecks for learners in deep learning The common bottlenecks for learners in deep learning are identified as coding proficiency and the struggle for individuals with classic statistics background, due to the contrasting intuition required in deep learning.', 'Insights about deep learning from teaching Teaching deep learning has provided insights that anyone can learn it, with the key differentiator being tenacity, debunking the notion that only certain people can succeed in coding and AI, highlighting the importance of perseverance.']}, {'end': 5294.742, 'start': 4891.767, 'title': 'Deep learning and fast.ai', 'summary': 'Highlights the importance of training lots of models, fine-tuning models with domain-specific data, and combining deep learning with domain expertise, emphasizing the free nature of fast.ai courses and the need for real-world problem-solving.', 'duration': 402.975, 'highlights': ['Fast.ai courses are free, with no revenue sources of any kind. The chapter mentions that Fast.ai courses are free, with no revenue sources, serving as a community service.', 'The importance of training lots of models and fine-tuning them with domain-specific data is emphasized. The chapter stresses the significance of training numerous models and fine-tuning them with domain-specific data, providing a practical approach to becoming an expert in a particular domain.', 'Emphasizing the need for real-world problem-solving and combining deep learning with domain expertise. The chapter highlights the importance of using deep learning to solve real-world problems and becoming an expert in a specific domain by combining deep learning with domain expertise.']}, {'end': 5540.42, 'start': 5294.742, 'title': 'Starting self-funded startups', 'summary': 'Discusses the benefits of self-funding startups over venture capital, sharing insights on successful self-funded startups, cost-cutting strategies, and the challenges of vc-backed startups.', 'duration': 245.678, 'highlights': ['Starting self-funded startups is recommended over venture capital, as it allows for a pragmatic approach and avoids the pressure to grow rapidly. The speaker suggests being pragmatic and staying away from venture capital money as long as possible, highlighting the benefits of self-funding and the avoidance of pressure to grow rapidly.', 'Successful self-funded startups were achieved with cost-cutting strategies, such as keeping server costs low and implementing payment models for additional services. The speaker shares experiences of self-funding startups, including keeping server costs low and implementing payment models for additional services, leading to profitable outcomes within six months.', "Challenges of VC-backed startups include the pressure to grow rapidly and the difficulty in prioritizing actions that align with the company's vision rather than pleasing the VC. The challenges of VC-backed startups are highlighted, including the pressure to grow rapidly and the difficulty in prioritizing actions that align with the company's vision rather than pleasing the VC."]}], 'duration': 1390.992, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk4149428.jpg', 'highlights': ["PyTorch's flexibility and accessibility enabled the development of a multi-layered API for efficient research and teaching.", 'The Fast.ai library provides an API that allows training of state-of-the-art neural networks in three lines of code.', 'The limitations of Python for TensorFlow are highlighted, with its runtime being 10 times slower than PyTorch.', 'The potential of Swift for TensorFlow is discussed, emphasizing its capability to be a better alternative due to its use of MLIR and LLVM for efficient compilation.', 'The recommendation of fast AI and PyTorch for new students due to their ability to enable quicker understanding of concepts and learning of state-of-the-art techniques.', 'The importance of training numerous models and fine-tuning them with domain-specific data is emphasized.', 'Emphasizing the need for real-world problem-solving and combining deep learning with domain expertise.', 'Starting self-funded startups is recommended over venture capital, as it allows for a pragmatic approach and avoids the pressure to grow rapidly.', "Challenges of VC-backed startups include the pressure to grow rapidly and the difficulty in prioritizing actions that align with the company's vision rather than pleasing the VC."]}, {'end': 6230.278, 'segs': [{'end': 5755.416, 'src': 'heatmap', 'start': 5684.137, 'weight': 0.956, 'content': [{'end': 5690.467, 'text': 'So you have to find ways to learn things productively and effectively, like treat your brain well.', 'start': 5684.137, 'duration': 6.33}, {'end': 5696.015, 'text': 'So using like mnemonics and stories and context and stuff like that.', 'start': 5690.607, 'duration': 5.408}, {'end': 5699.694, 'text': "Um, so yeah, it's, it's a super great technique.", 'start': 5696.813, 'duration': 2.881}, {'end': 5705.037, 'text': "It's like learning how to learn is something which everybody should learn before they actually learn anything.", 'start': 5699.734, 'duration': 5.303}, {'end': 5707.898, 'text': 'Um, but almost nobody does.', 'start': 5705.057, 'duration': 2.841}, {'end': 5709.139, 'text': 'So what have you?', 'start': 5707.918, 'duration': 1.221}, {'end': 5718.963, 'text': 'so certainly works well for learning new languages, for I mean for learning like small projects almost, but do you you know I started using it?', 'start': 5709.139, 'duration': 9.824}, {'end': 5721.244, 'text': 'for I forget who wrote a blog post about.', 'start': 5718.963, 'duration': 2.281}, {'end': 5722.064, 'text': 'this inspired me.', 'start': 5721.244, 'duration': 0.82}, {'end': 5724.546, 'text': "It might've been you, I'm not sure.", 'start': 5722.405, 'duration': 2.141}, {'end': 5731.407, 'text': "I started when I read papers, concepts and ideas, I'll put them.", 'start': 5726.186, 'duration': 5.221}, {'end': 5734.728, 'text': 'Was it Michael Nielsen? It was Michael Nielsen.', 'start': 5731.927, 'duration': 2.801}, {'end': 5741.689, 'text': 'Michael started doing this recently and has been writing about it.', 'start': 5734.748, 'duration': 6.941}, {'end': 5747.13, 'text': "Today's Ebbinghaus is a guy called Peter Wozniak, who developed a system called SuperMemo.", 'start': 5742.129, 'duration': 5.001}, {'end': 5755.416, 'text': "He's been basically trying to become like the world's greatest renaissance man over the last few decades.", 'start': 5748.211, 'duration': 7.205}], 'summary': "Learning how to learn effectively using techniques like mnemonics and stories can improve productivity and retention, as demonstrated by peter wozniak's supermemo system and michael nielsen's approach.", 'duration': 71.279, 'max_score': 5684.137, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk5684137.jpg'}, {'end': 5878.582, 'src': 'heatmap', 'start': 5809.087, 'weight': 0.784, 'content': [{'end': 5814.652, 'text': 'So really understand that concept deeply and study it carefully.', 'start': 5809.087, 'duration': 5.565}, {'end': 5826.982, 'text': "Well, decide if it really is important, if it is incorporated into our library, incorporated into how I do things, or decide it's not worth it.", 'start': 5814.672, 'duration': 12.31}, {'end': 5835.67, 'text': "So I find I find I then remember the things that I care about because I'm using it all the time.", 'start': 5828.082, 'duration': 7.588}, {'end': 5845.938, 'text': "So, for the last 25 years I've committed to spending at least half of every day learning or practicing something new,", 'start': 5835.75, 'duration': 10.188}, {'end': 5852.043, 'text': "which all my colleagues have always hated, because it always looks like I'm not working on what I'm meant to be working on,", 'start': 5845.938, 'duration': 6.105}, {'end': 5856.347, 'text': "but it always means I do everything faster because I've been practicing a lot of stuff.", 'start': 5852.043, 'duration': 4.304}, {'end': 5861.35, 'text': 'So I kind of give myself a lot of opportunity to practice new things.', 'start': 5856.987, 'duration': 4.363}, {'end': 5870.196, 'text': "And so I find now I don't, yeah, I don't often kind of find myself wishing I could remember something.", 'start': 5861.77, 'duration': 8.426}, {'end': 5873.398, 'text': "Cause if it's something that's useful, then I've been using it a lot.", 'start': 5870.216, 'duration': 3.182}, {'end': 5878.582, 'text': "It's easy enough to look it up on Google, but speaking Chinese, you can't look it up on Google.", 'start': 5873.939, 'duration': 4.643}], 'summary': 'Commit to half-day learning for 25 years, enhancing efficiency and memory retention.', 'duration': 69.495, 'max_score': 5809.087, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk5809087.jpg'}, {'end': 6091.735, 'src': 'embed', 'start': 6058.697, 'weight': 2, 'content': [{'end': 6064.36, 'text': "It's just like, there's so many societally important problems to solve right now.", 'start': 6058.697, 'duration': 5.663}, {'end': 6069.362, 'text': "I just, I don't find it a really interesting question to even answer.", 'start': 6064.46, 'duration': 4.902}, {'end': 6075.809, 'text': "So, in terms of societally important problems, what's the problem that is within reach?", 'start': 6070.367, 'duration': 5.442}, {'end': 6079.811, 'text': 'Well, I mean, for example, there are problems that AI creates right?', 'start': 6075.909, 'duration': 3.902}, {'end': 6091.735, 'text': 'So, more specifically, labor force displacement is going to be huge, and people keep making this frivolous econometric argument of being like oh,', 'start': 6079.831, 'duration': 11.904}], 'summary': 'Societally important problems like labor force displacement due to ai need addressing.', 'duration': 33.038, 'max_score': 6058.697, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk6058697.jpg'}, {'end': 6181.232, 'src': 'embed', 'start': 6131.85, 'weight': 0, 'content': [{'end': 6138.932, 'text': 'And you see this turning into anxiety and despair and even violence.', 'start': 6131.85, 'duration': 7.082}, {'end': 6141.032, 'text': 'So I very much worry about that.', 'start': 6139.512, 'duration': 1.52}, {'end': 6145.494, 'text': "You've written quite a bit about ethics too.", 'start': 6143.413, 'duration': 2.081}, {'end': 6158.839, 'text': "I do think that every data scientist working with Deep learning needs to recognize they have an incredibly high leverage tool that they're using that can influence society in lots of ways.", 'start': 6145.794, 'duration': 13.045}, {'end': 6163.762, 'text': "And if they're doing research, that that research is going to be used by people doing this kind of work.", 'start': 6159.039, 'duration': 4.723}, {'end': 6173.888, 'text': 'And they have a responsibility to consider the consequences and to think about things like How will humans be in the loop here?', 'start': 6164.582, 'duration': 9.306}, {'end': 6175.989, 'text': 'How do we avoid runaway feedback loops?', 'start': 6173.968, 'duration': 2.021}, {'end': 6181.232, 'text': 'How do we ensure an appeals process for humans that are impacted by my algorithm?', 'start': 6176.59, 'duration': 4.642}], 'summary': 'Data scientists using deep learning must consider ethical consequences and societal impact, including human involvement and feedback loops.', 'duration': 49.382, 'max_score': 6131.85, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk6131850.jpg'}, {'end': 6230.278, 'src': 'embed', 'start': 6206.678, 'weight': 3, 'content': [{'end': 6213.484, 'text': "Well, you're in a perfect position to educate them better, to read literature, to read history, to learn from history.", 'start': 6206.678, 'duration': 6.806}, {'end': 6221.37, 'text': 'Well, Jeremy, thank you so much for everything you do, for inspiring huge amount of people,', 'start': 6215.786, 'duration': 5.584}, {'end': 6228.616, 'text': "getting them into deep learning and having the ripple effects, the flap of a butterfly's wings that will probably change the world.", 'start': 6221.37, 'duration': 7.246}, {'end': 6229.537, 'text': 'So thank you very much.', 'start': 6228.677, 'duration': 0.86}, {'end': 6230.278, 'text': 'Cheers.', 'start': 6230.098, 'duration': 0.18}], 'summary': 'Encouraging deep learning, inspiring many, with potential to change the world.', 'duration': 23.6, 'max_score': 6206.678, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk6206678.jpg'}], 'start': 5541.443, 'title': 'Learning with spaced repetition', 'summary': "Discusses the effectiveness of spaced repetition learning, highlighting ebbinghaus's study and the simple algorithm for revising information. it emphasizes the importance of active learning and practice for continual improvement and memory retention.", 'chapters': [{'end': 6230.278, 'start': 5541.443, 'title': 'Learning with spaced repetition', 'summary': "Discusses the effectiveness of spaced repetition learning, where the psychologist ebbinghaus's study led to the development of a simple algorithm for revising information, and the importance of actively learning and practicing new things for continual improvement and memory retention.", 'duration': 688.835, 'highlights': ["Spaced repetition learning is based on a simple algorithm for revising information, leading to a dramatically smaller probability of forgetting when revised at optimal intervals. Spaced repetition learning is based on Ebbinghaus's study, which led to the development of a simple algorithm for revising information at optimal intervals to improve memory retention.", 'The speaker emphasizes the need for actively learning and practicing new things, such as using mnemonics, stories, and context to effectively retain information. The speaker stresses the importance of actively learning and practicing new things using mnemonics, stories, and context to improve memory retention.', 'The importance of continuously learning and practicing new things is highlighted, with the speaker committing at least half of every day to learning or practicing something new for the past 25 years. The speaker emphasizes the importance of continuous learning and practice, committing at least half of every day to learning or practicing something new for the past 25 years.', 'The potential societal impact of AI and the ethical responsibilities of data scientists using deep learning are discussed, emphasizing the need to consider the consequences and human issues in their work. The potential societal impact of AI and the ethical responsibilities of data scientists using deep learning are discussed, emphasizing the need to consider the consequences and human issues in their work.']}], 'duration': 688.835, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/J6XcP4JOHmk/pics/J6XcP4JOHmk5541443.jpg', 'highlights': ["Spaced repetition learning is based on Ebbinghaus's study, leading to a simple algorithm for revising information at optimal intervals.", 'The speaker emphasizes the importance of actively learning and practicing new things using mnemonics, stories, and context.', 'The speaker commits at least half of every day to learning or practicing something new for the past 25 years.', 'The potential societal impact of AI and the ethical responsibilities of data scientists using deep learning are discussed.']}], 'highlights': ['FastAI is a top resource for deep learning beginners due to its free, easy, insightful, and accessible nature.', 'The focus on practical application and hands-on exploration makes FastAI useful to experts as well.', 'Jeremy Howard and team achieved top leaderboard positions in Dawn Bench competition for CIFAR-10 and ImageNet, demonstrating efficient and cost-effective model training.', 'The potential of super convergence in deep learning allows for 10 times faster training with a 10 times higher learning rate.', 'The Fast.ai library provides an API that allows training of state-of-the-art neural networks in three lines of code.', 'The limitations of Python for TensorFlow are highlighted, with its runtime being 10 times slower than PyTorch.', 'The need for less data for impactful results is emphasized, with transfer learning being a critical technique for requiring orders of magnitude less data, challenging the assumption of needing more data.', 'Deep learning can address the 10x shortage of doctors in the world, particularly in developing countries like India and China, by providing diagnostic and treatment planning assistance with limited human expertise.', 'AI can enable healthcare workers with minimal training to provide high-quality assessments for diseases such as malaria or tuberculosis, potentially reducing the need for highly trained doctors in certain cases.', 'The potential societal impact of AI and the ethical responsibilities of data scientists using deep learning are discussed.']}