title
Lesson 3: Deep Learning 2018
description
NB: Please go to http://course.fast.ai to view this video since there is important updated information there. If you have questions, use the forums at http://forums.fast.ai
We explain convolutional networks from several different angles: the theory, a video visualization, and an Excel demo. You’ll see how to use deep learning for structured/tabular data, such as time-series sales data.
We also teach a couple of key bits of math that you really need for deep learning: exponentiation and the logarithm. You will learn how to download data to your deep learning server, how to submit to a Kaggle competition, and key differences between PyTorch and Keras/TensorFlow.
detail
{'title': 'Lesson 3: Deep Learning 2018', 'heatmap': [{'end': 824.376, 'start': 737.62, 'weight': 0.746}, {'end': 1150.566, 'start': 1065.945, 'weight': 0.761}, {'end': 2870.627, 'start': 2704.615, 'weight': 0.946}, {'end': 3609.273, 'start': 3439.057, 'weight': 0.854}, {'end': 3857.418, 'start': 3764.423, 'weight': 0.726}, {'end': 5172.397, 'start': 5080.555, 'weight': 0.775}, {'end': 5990.502, 'start': 5814.277, 'weight': 1}, {'end': 7381.853, 'start': 7291.832, 'weight': 0.754}], 'summary': 'Covers recent student contributions, deep learning applications, data science techniques, keras for image data augmentation, model recompilation, kaggle submission process, implementing cnn in excel, convolutional neural networks, neural network training, multi-label classification, and structured data in machine learning, emphasizing practical resources for sharing knowledge and achieving high accuracy in model development and competitions.', 'chapters': [{'end': 126.883, 'segs': [{'end': 78.393, 'src': 'embed', 'start': 25.56, 'weight': 0, 'content': [{'end': 27.181, 'text': 'I just wanted to highlight a few.', 'start': 25.56, 'duration': 1.621}, {'end': 36.336, 'text': "I've actually Posted to the wiki thread a few of these, but there's there's lots more.", 'start': 27.181, 'duration': 9.155}, {'end': 40.918, 'text': 'Russian has posted a whole bunch of nice introductory tutorials.', 'start': 36.336, 'duration': 4.582}, {'end': 45.36, 'text': "so, for example, if you're having any trouble getting connected with AWS,", 'start': 40.918, 'duration': 4.442}, {'end': 53.543, 'text': "She's got a whole step-by-step How to go about logging in and getting everything working, which I think is a really terrific thing.", 'start': 45.36, 'duration': 8.183}, {'end': 61.502, 'text': "and so it's a kind of thing that if you Writing some notes for yourself to remind you how to do it,", 'start': 53.543, 'duration': 7.959}, {'end': 66.865, 'text': 'You may as well post them for others to do it as well, and by using a markdown file like this.', 'start': 61.502, 'duration': 5.363}, {'end': 73.97, 'text': "And it's actually good practice if you haven't used github before, if you put it up on github, Everybody can now use it, or, of course,", 'start': 66.865, 'duration': 7.105}, {'end': 75.932, 'text': 'you can just put it in the forum.', 'start': 73.97, 'duration': 1.962}, {'end': 78.393, 'text': 'so more advanced.', 'start': 75.932, 'duration': 2.461}], 'summary': 'Russian has posted aws tutorials and encourages sharing on github or the forum.', 'duration': 52.833, 'max_score': 25.56, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc25560.jpg'}], 'start': 0.345, 'title': 'Forum activity and tutorial highlights', 'summary': 'Discusses the recent increase in helpful materials created by students, including aws tutorials and tmux usage, emphasizing the importance of sharing knowledge and providing practical resources for others.', 'chapters': [{'end': 126.883, 'start': 0.345, 'title': 'Forum activity and tutorial highlights', 'summary': 'Discusses the recent increase in helpful materials created by students, including aws tutorials and tmux usage, emphasizing the importance of sharing knowledge and providing practical resources for others.', 'duration': 126.538, 'highlights': ["Russian's AWS tutorials and Reshma's tmux tutorial are highlighted as valuable resources created by students to aid in understanding and sharing knowledge. Russian has posted step-by-step AWS tutorials to help with connection issues, while Reshma has shared a tutorial on using tmux with additional content on GitHub.", 'Encouraging students to share their notes and materials for the benefit of the entire class, emphasizing the use of GitHub for broader accessibility. The speaker encourages students to share their notes and materials, highlighting the use of GitHub for broader accessibility and the importance of contributing to the collective learning experience.']}], 'duration': 126.538, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc345.jpg', 'highlights': ["Russian's AWS tutorials and Reshma's tmux tutorial are valuable resources for understanding and sharing knowledge.", 'Encouraging students to share notes and materials for the benefit of the entire class, emphasizing the use of GitHub for broader accessibility.']}, {'end': 766.408, 'segs': [{'end': 227.377, 'src': 'embed', 'start': 175.781, 'weight': 0, 'content': [{'end': 179.259, 'text': "and so it's kind of Something which hasn't really been blogged about before.", 'start': 175.781, 'duration': 3.478}, {'end': 181.521, 'text': "so this is the first time I've seen this blogged about.", 'start': 179.259, 'duration': 2.262}, {'end': 187.466, 'text': "so when I put this on Twitter a link to Pavel's post it's been shared now hundreds of times.", 'start': 181.521, 'duration': 5.945}, {'end': 196.433, 'text': "It's been really really popular and viewed many thousands of times So that's some great content Redec has posted lots of cool stuff.", 'start': 188.047, 'duration': 8.386}, {'end': 202.697, 'text': 'I really like this practitioners guide to PyTorch, which, again This is more for more advanced students,', 'start': 196.833, 'duration': 5.864}, {'end': 210.883, 'text': "but it's like digging into people who have never used PyTorch before but know a bit about Numerical programming in general,", 'start': 202.697, 'duration': 8.186}, {'end': 213.105, 'text': "and it's a quick introduction to how PyTorch is different.", 'start': 210.883, 'duration': 2.222}, {'end': 220.035, 'text': "and And then there's been some interesting little bits of research like what's the relationship between learning rate and batch size?", 'start': 213.105, 'duration': 6.93}, {'end': 227.377, 'text': 'So one of the students actually asked me this before class and I said well, one of the other students has written an analysis of exactly that.', 'start': 220.535, 'duration': 6.842}], 'summary': "Pavel's post shared hundreds of times, viewed thousands of times. great content on pytorch and research on learning rate vs. batch size.", 'duration': 51.596, 'max_score': 175.781, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc175781.jpg'}, {'end': 286.896, 'src': 'embed', 'start': 262.7, 'weight': 5, 'content': [{'end': 276.027, 'text': "but a really interesting piece of research got some introductions to convolutional neural networks and And then something that we'll be learning about towards the end of this course.", 'start': 262.7, 'duration': 13.327}, {'end': 278.669, 'text': "but I'm sure you've noticed we're using something called ResNet,", 'start': 276.027, 'duration': 2.642}, {'end': 286.896, 'text': "and Anand Saha actually posted a pretty impressive analysis of what's a ResNet and why is it interesting,", 'start': 278.669, 'duration': 8.227}], 'summary': "Introducing convolutional neural networks and resnet, with anand saha's analysis.", 'duration': 24.196, 'max_score': 262.7, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc262700.jpg'}, {'end': 360.805, 'src': 'embed', 'start': 335.806, 'weight': 3, 'content': [{'end': 343.568, 'text': "try to answer in a way that's friendly to people that maybe have no more than a year of programming experience and haven't done any machine learning before.", 'start': 335.806, 'duration': 7.762}, {'end': 355.922, 'text': 'So you know, I hope Other people in the class Feel like you can contribute as well and just remember all of the people we just looked at,', 'start': 347.175, 'duration': 8.747}, {'end': 360.805, 'text': 'or many of them, I believe, have never Posted anything to the internet before.', 'start': 355.922, 'duration': 4.883}], 'summary': 'Encouraging inclusive participation in a beginner-friendly environment, even for those with no prior internet presence.', 'duration': 24.999, 'max_score': 335.806, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc335806.jpg'}, {'end': 469.829, 'src': 'embed', 'start': 434.404, 'weight': 2, 'content': [{'end': 438.287, 'text': "And then we're going to kind of go through this cycle where we're going to dig.", 'start': 434.404, 'duration': 3.883}, {'end': 445.517, 'text': "We're going to do a little intro into a whole bunch of application areas, using neural nets for structured data,", 'start': 439.313, 'duration': 6.204}, {'end': 452.202, 'text': 'so kind of like logistics or forecasting, or you know, financial data or that kind of thing,', 'start': 445.517, 'duration': 6.685}, {'end': 457.726, 'text': 'and then looking at language applications and LP applications using recurrent neural nets,', 'start': 452.202, 'duration': 5.524}, {'end': 469.829, 'text': "and and then collaborative filtering for Recommendation systems, and so these will all be like similar to what we've done for CNN's images.", 'start': 457.726, 'duration': 12.103}], 'summary': 'Discussion on using neural nets for structured data, language processing, and recommendation systems.', 'duration': 35.425, 'max_score': 434.404, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc434404.jpg'}, {'end': 587.133, 'src': 'embed', 'start': 555.136, 'weight': 6, 'content': [{'end': 568.32, 'text': "So to download from Kaggle We use something called Kaggle CLI, Which is here, and to install it, I think it's already in.", 'start': 555.136, 'duration': 13.184}, {'end': 569.16, 'text': "Let's just double check.", 'start': 568.32, 'duration': 0.84}, {'end': 579.029, 'text': "Yeah, so it's, or it should already be in your environment.", 'start': 574.106, 'duration': 4.923}, {'end': 584.432, 'text': 'But to make sure, one thing that happens is because this is downloading from the cattle website through like screen scraping.', 'start': 579.029, 'duration': 5.403}, {'end': 587.133, 'text': 'every time cattle changes the website it breaks.', 'start': 584.432, 'duration': 2.701}], 'summary': 'Use kaggle cli for downloading from kaggle; website changes may cause breaks.', 'duration': 31.997, 'max_score': 555.136, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc555136.jpg'}], 'start': 126.883, 'title': 'Recent student contributions and deep learning applications', 'summary': "Highlights recent student contributions such as a popular blog post on learning rate finder and research on learning rate and batch size. it also covers forum discussions, advanced students' contributions, and a plan for exploring deep learning applications across domains.", 'chapters': [{'end': 286.896, 'start': 126.883, 'title': 'Recent course highlights', 'summary': 'Highlights recent student contributions including a popular blog post on learning rate finder, a guide to pytorch for advanced students, and research on the relationship between learning rate and batch size, with hundreds of shares and thousands of views.', 'duration': 160.013, 'highlights': ["Pavel's post on learning rate finder has been shared hundreds of times and viewed thousands of times, making it popular among the audience.", "Redek's practitioners guide to PyTorch provides a quick introduction for advanced students who have never used PyTorch before, targeting those with a background in numerical programming.", "Research on the relationship between learning rate and batch size has sparked interest, with experiments conducted and shared by students, including Redek's own research on stochastic gradient descent with restarts.", "Anand Saha's analysis of ResNet provides a comprehensive understanding of the architecture and its significance for the course."]}, {'end': 766.408, 'start': 286.896, 'title': 'Forum discussions and deep learning applications', 'summary': "Covers forum discussions for beginners, advanced students' contributions, and a detailed plan for exploring deep learning applications across various domains, including practical tips for downloading data and tools usage.", 'duration': 479.512, 'highlights': ["The chapter emphasizes the importance of a beginner forum and encourages contributions from advanced students to support those with basic programming experience. The beginner forum aims to provide a less intimidating space for asking questions, and advanced students are encouraged to answer questions in a friendly manner, considering the audience's level of programming experience.", 'The chapter outlines the plan to delve into the underlying theory of Convolutional Neural Networks (CNNs) and explores various applications of neural nets for structured data, language processing, and recommendation systems. The plan includes exploring the theory of CNNs, applications of neural nets for structured data, language processing, and recommendation systems, followed by a deep dive into collaborative filtering, structured data analysis, and recurrent neural networks.', 'The chapter provides practical guidance on downloading data from Kaggle using Kaggle CLI and a Chrome extension called curl W get, along with tips for efficient data management. Practical guidance is offered for downloading data from Kaggle using Kaggle CLI, and a Chrome extension called curl W get is recommended for efficient data management, especially for large datasets.']}], 'duration': 639.525, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc126883.jpg', 'highlights': ["Pavel's post on learning rate finder is popular, shared hundreds of times and viewed thousands of times.", 'Research on the relationship between learning rate and batch size has sparked interest, with experiments conducted and shared by students.', 'The chapter outlines the plan to delve into the underlying theory of Convolutional Neural Networks (CNNs) and explores various applications of neural nets for structured data, language processing, and recommendation systems.', 'The chapter emphasizes the importance of a beginner forum and encourages contributions from advanced students to support those with basic programming experience.', "Redek's practitioners guide to PyTorch provides a quick introduction for advanced students who have never used PyTorch before, targeting those with a background in numerical programming.", "Anand Saha's analysis of ResNet provides a comprehensive understanding of the architecture and its significance for the course.", 'Practical guidance is offered for downloading data from Kaggle using Kaggle CLI, and a Chrome extension called curl W get is recommended for efficient data management, especially for large datasets.']}, {'end': 1301.194, 'segs': [{'end': 949.959, 'src': 'embed', 'start': 919.395, 'weight': 1, 'content': [{'end': 928.302, 'text': "So what I've got here is is, in a single window, is an entire End-to-end process to get a state-of-the-art result for cats versus dogs right?", 'start': 919.395, 'duration': 8.907}, {'end': 936.148, 'text': "I've, the only step I've skipped is the bit where we've downloaded it from cattle and then where we unzipped it right.", 'start': 928.982, 'duration': 7.166}, {'end': 940.471, 'text': 'so These are literally all the steps.', 'start': 936.148, 'duration': 4.323}, {'end': 949.959, 'text': 'and So we import our Libraries and actually, if you import this one conf learner that basically imports everything else.', 'start': 940.471, 'duration': 9.488}], 'summary': 'End-to-end process for cats vs dogs with skipped steps, all the key steps included', 'duration': 30.564, 'max_score': 919.395, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc919395.jpg'}, {'end': 1150.566, 'src': 'heatmap', 'start': 1065.945, 'weight': 0.761, 'content': [{'end': 1073.069, 'text': "It's just a shortcut which caches some of the intermediate steps that don't have to be recalculated each time.", 'start': 1065.945, 'duration': 7.124}, {'end': 1078.872, 'text': "And remember that when we are using pre computed activations, data augmentation doesn't work right?", 'start': 1073.069, 'duration': 5.803}, {'end': 1082.974, 'text': "So even if you ask for a data augmentation, if you've got pre compute equals, true,", 'start': 1079.152, 'duration': 3.822}, {'end': 1090.999, 'text': "It doesn't actually do any data augmentation because it's using the cached non augmented activations.", 'start': 1082.974, 'duration': 8.025}, {'end': 1093.24, 'text': 'So in this case, to keep this as simple as possible,', 'start': 1090.999, 'duration': 2.241}, {'end': 1103.885, 'text': 'I have no pre computed anything going on, and So I do three cycles of length one, and then I can then unfreeze.', 'start': 1093.26, 'duration': 10.625}, {'end': 1105.267, 'text': "so it's now going to train the whole thing.", 'start': 1103.885, 'duration': 1.382}, {'end': 1112.14, 'text': "Something we haven't seen before and we'll learn about in the second half is called BNFreeze.", 'start': 1106.574, 'duration': 5.566}, {'end': 1123.433, 'text': "For now, all you need to know is that if you're using a model like a bigger, deeper model like ResNet-50 or ResNext-101 on a dataset that's very,", 'start': 1112.541, 'duration': 10.892}, {'end': 1126.316, 'text': 'very similar to ImageNet like this.', 'start': 1123.433, 'duration': 2.883}, {'end': 1127.698, 'text': 'cats and dogs data set.', 'start': 1126.316, 'duration': 1.382}, {'end': 1137.425, 'text': "So in other words, it's like side on photos of standard objects, you know, of a similar size to ImageNet, like somewhere between 200 and 500 pixels.", 'start': 1127.718, 'duration': 9.707}, {'end': 1142.089, 'text': 'You should probably add this line when you unfreeze.', 'start': 1138.886, 'duration': 3.203}, {'end': 1150.566, 'text': "For those of you that are more advanced, what it's doing is it's causing the batch normalization moving averages to not be updated.", 'start': 1142.72, 'duration': 7.846}], 'summary': 'Pre-computed activations affect data augmentation. bnfreeze for resnet-50 on similar datasets.', 'duration': 84.621, 'max_score': 1065.945, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1065945.jpg'}, {'end': 1127.698, 'src': 'embed', 'start': 1079.152, 'weight': 0, 'content': [{'end': 1082.974, 'text': "So even if you ask for a data augmentation, if you've got pre compute equals, true,", 'start': 1079.152, 'duration': 3.822}, {'end': 1090.999, 'text': "It doesn't actually do any data augmentation because it's using the cached non augmented activations.", 'start': 1082.974, 'duration': 8.025}, {'end': 1093.24, 'text': 'So in this case, to keep this as simple as possible,', 'start': 1090.999, 'duration': 2.241}, {'end': 1103.885, 'text': 'I have no pre computed anything going on, and So I do three cycles of length one, and then I can then unfreeze.', 'start': 1093.26, 'duration': 10.625}, {'end': 1105.267, 'text': "so it's now going to train the whole thing.", 'start': 1103.885, 'duration': 1.382}, {'end': 1112.14, 'text': "Something we haven't seen before and we'll learn about in the second half is called BNFreeze.", 'start': 1106.574, 'duration': 5.566}, {'end': 1123.433, 'text': "For now, all you need to know is that if you're using a model like a bigger, deeper model like ResNet-50 or ResNext-101 on a dataset that's very,", 'start': 1112.541, 'duration': 10.892}, {'end': 1126.316, 'text': 'very similar to ImageNet like this.', 'start': 1123.433, 'duration': 2.883}, {'end': 1127.698, 'text': 'cats and dogs data set.', 'start': 1126.316, 'duration': 1.382}], 'summary': 'Avoid data augmentation when pre-compute equals true. use bnfreeze for deeper models on similar datasets like cats and dogs.', 'duration': 48.546, 'max_score': 1079.152, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1079152.jpg'}], 'start': 766.408, 'title': 'Data science and model building', 'summary': 'Discusses techniques for data manipulation and model building in data science, including an end-to-end process for classifying cats versus dogs with minimal code. it also explores precomputed activations, bnfreeze, test time augmentation, and achieving 99.45% accuracy, along with the use of keras as an alternative to fast.ai.', 'chapters': [{'end': 1034.906, 'start': 766.408, 'title': 'Data science and model building', 'summary': 'Discusses techniques for downloading and manipulating data for data science, including using symbolic links, and illustrates an end-to-end process to build a model for classifying cats versus dogs, requiring minimal code.', 'duration': 268.498, 'highlights': ['End-to-end process to build a model for classifying cats versus dogs Illustrates a complete process to achieve a state-of-the-art result for classifying cats versus dogs, requiring minimal code beyond downloading and unzipping the data.', 'Use of symbolic links for data storage and access Demonstrates the use of symbolic links to store and access data, providing flexibility in data storage locations and simplifying access to resources.', "Techniques for data manipulation suitable for specific models Explains the transformation of data to suit a specific model's requirements, such as assuming side-on photos and zooming in up to 10% each time, showcasing data manipulation techniques for model training."]}, {'end': 1301.194, 'start': 1034.946, 'title': 'Training and using precomputed activations with fast.ai', 'summary': 'Covers the use of precomputed activations, the impact on data augmentation, the introduction of bnfreeze, and the use of test time augmentation, resulting in a 99.45% accuracy. additionally, it discusses the use of keras as an alternative to fast.ai.', 'duration': 266.248, 'highlights': ['The use of precomputed activations can speed up the initial step, caching some intermediate steps and saving time, with no impact on data augmentation (2.5 minutes for precompute equals true)', 'BNFreeze is recommended for deeper models like ResNet-50 or ResNext-101 on similar datasets to ImageNet, preserving batch normalization moving averages and achieving 99.45% accuracy', "Introduction to Keras as an alternative library to Fast.ai, replicating parts of Fast.ai's lesson one to illustrate its functionality and usage"]}], 'duration': 534.786, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc766408.jpg', 'highlights': ['Achieving 99.45% accuracy using BNFreeze on deeper models like ResNet-50 or ResNext-101', 'End-to-end process for classifying cats versus dogs with minimal code', 'Use of precomputed activations to speed up initial step, caching intermediate steps']}, {'end': 1754.148, 'segs': [{'end': 1391.01, 'src': 'embed', 'start': 1335.932, 'weight': 0, 'content': [{'end': 1344.237, 'text': "so it's going to tell it where the training set and the validation set are Twice, what batch size to use.", 'start': 1335.932, 'duration': 8.305}, {'end': 1345.438, 'text': "Now you'll notice in Keras.", 'start': 1344.237, 'duration': 1.201}, {'end': 1355.336, 'text': 'We need much, much, much more code to do the same thing and, More importantly, each part of that code has many, many,', 'start': 1345.898, 'duration': 9.438}, {'end': 1360.26, 'text': 'many more things you have to set, and if you set them wrong, Everything breaks right.', 'start': 1355.336, 'duration': 4.924}, {'end': 1365.003, 'text': "so I'll give you a summary of what they are.", 'start': 1360.26, 'duration': 4.743}, {'end': 1370.907, 'text': "so you're basically, rather than creating a single Data object in Keras,", 'start': 1365.003, 'duration': 5.904}, {'end': 1376.151, 'text': 'we first of all have to define something called a data generator to say how to generate the data.', 'start': 1370.907, 'duration': 5.244}, {'end': 1382.565, 'text': 'and so a data generator, we basically have to say what kind of data augmentation is we want to do?', 'start': 1376.151, 'duration': 6.414}, {'end': 1391.01, 'text': 'And we also, we actually have to say what kind of normalization do we want to do?', 'start': 1384.366, 'duration': 6.644}], 'summary': 'Keras requires more complex and detailed code for data generation and normalization compared to other methods.', 'duration': 55.078, 'max_score': 1335.932, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1335932.jpg'}, {'end': 1502.455, 'src': 'embed', 'start': 1476.768, 'weight': 1, 'content': [{'end': 1482.43, 'text': 'So an example of like where things get a little more complex is you have to do the same thing for the validation set.', 'start': 1476.768, 'duration': 5.662}, {'end': 1490.752, 'text': "So it's up to you to create a data generator that doesn't have data augmentation, Because obviously for the validation set, unless you're using TTA,", 'start': 1482.43, 'duration': 8.322}, {'end': 1493.172, 'text': "that's going to stuff things up.", 'start': 1490.752, 'duration': 2.42}, {'end': 1502.455, 'text': "you also, when you train, You randomly reorder the images so that they're always shown in different orders to make it more random.", 'start': 1493.172, 'duration': 9.283}], 'summary': 'Data generator for validation set must exclude data augmentation and images should be randomly reordered during training.', 'duration': 25.687, 'max_score': 1476.768, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1476768.jpg'}, {'end': 1585.852, 'src': 'embed', 'start': 1557.129, 'weight': 5, 'content': [{'end': 1565.034, 'text': 'and So, having done that in Keras, you basically say okay, this is my model, and then again,', 'start': 1557.129, 'duration': 7.905}, {'end': 1570.776, 'text': "there isn't a concept of automatically freezing things or an API for that.", 'start': 1565.034, 'duration': 5.742}, {'end': 1578.859, 'text': 'so you just have to loop through the layers that you want to freeze and call trainable equals false on them.', 'start': 1570.776, 'duration': 8.083}, {'end': 1585.852, 'text': "In Keras, there's a concept we don't have in fast.ai or PyTorch of compiling a model.", 'start': 1580.908, 'duration': 4.944}], 'summary': 'In keras, freezing layers requires manual iteration and setting trainable to false.', 'duration': 28.723, 'max_score': 1557.129, 'thumbnail': ''}, {'end': 1704.486, 'src': 'embed', 'start': 1660.945, 'weight': 2, 'content': [{'end': 1667.462, 'text': 'and So, as you can see, I got to a validation accuracy of 95%.', 'start': 1660.945, 'duration': 6.517}, {'end': 1673.044, 'text': 'But as you can also see, something really weird happened where after 1 it was like 49, and then it was 69, and then 95.', 'start': 1667.462, 'duration': 5.582}, {'end': 1676.305, 'text': "I don't know why these are so low.", 'start': 1673.044, 'duration': 3.261}, {'end': 1677.706, 'text': "That's not normal.", 'start': 1676.325, 'duration': 1.381}, {'end': 1681.647, 'text': 'There may be a bug in Keras, there may be a bug in my code.', 'start': 1679.026, 'duration': 2.621}, {'end': 1685.989, 'text': "I reached out on Twitter to see if anybody could figure it out, but they couldn't.", 'start': 1683.008, 'duration': 2.981}, {'end': 1689.562, 'text': 'I guess this is one of the challenges with using Like.', 'start': 1686.409, 'duration': 3.153}, {'end': 1694.944, 'text': "this is one of the reasons I wanted to use fast AI for this course is it's much harder to screw things up.", 'start': 1689.562, 'duration': 5.382}, {'end': 1698.205, 'text': "So I don't know if I screwed something up or somebody else did.", 'start': 1694.944, 'duration': 3.261}, {'end': 1704.486, 'text': 'yes, you know, This is using the tensorflow back-end.', 'start': 1698.205, 'duration': 6.281}], 'summary': 'Validation accuracy reached 95%, but unusual low values occurred. possible bug in keras or code.', 'duration': 43.541, 'max_score': 1660.945, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1660945.jpg'}], 'start': 1301.194, 'title': 'Keras for image data augmentation and fine-tuning resnet-50', 'summary': 'Delves into the complexities of using keras for image data augmentation, emphasizing the need for defining data generators, setting normalization and augmentation parameters, and creating separate generators for training and validation sets. it also covers the process of fine-tuning resnet-50 in keras, achieving a validation accuracy of 95% but encountering unusual accuracy fluctuations, and the challenges of manual model construction and freezing layers in keras.', 'chapters': [{'end': 1511.616, 'start': 1301.194, 'title': 'Keras image data augmentation', 'summary': 'Covers the complexities of using keras for image data augmentation, including the need to define data generators, set normalization and augmentation parameters, and create separate generators for training and validation sets, highlighting the increased code complexity compared to fastai.', 'duration': 210.422, 'highlights': ['Defining data generators in Keras requires specifying data augmentation and normalization parameters, unlike FastAI which automatically handles these requirements for models like ResNet-50. specifying data augmentation and normalization parameters, increased code complexity compared to FastAI', 'Creating separate data generators for training and validation sets in Keras, with the need to ensure no data augmentation is applied to the validation set, compared to the random image reordering for training set. separate data generators for training and validation sets, no data augmentation for the validation set', 'The complexity and sensitivity of setting various parameters in Keras, with the risk of breaking the code if set incorrectly. risk of code breakage, sensitivity of setting parameters', 'The need for detailed understanding of the expected data and the necessity to carefully set multiple parameters for Keras, making it more prone to errors compared to FastAI. detailed understanding of expected data, increased error susceptibility compared to FastAI']}, {'end': 1754.148, 'start': 1512.076, 'title': 'Fine-tuning resnet-50 in keras', 'summary': 'Explains the process of fine-tuning resnet-50 in keras, achieving a validation accuracy of 95% but encountering unusual accuracy fluctuations, and the challenges of manual model construction and freezing layers in keras.', 'duration': 242.072, 'highlights': ['The validation accuracy reached 95% while fine-tuning ResNet-50 in Keras. The model achieved a validation accuracy of 95% during the fine-tuning process.', 'Encountered unusual accuracy fluctuations of 49%, 69%, and 95% during the fine-tuning process. The validation accuracy fluctuated unexpectedly, with values of 49%, 69%, and 95% at different stages.', 'Challenges of manually constructing the model and freezing layers in Keras. The process of manual model construction and freezing specific layers in Keras was highlighted as a challenge.']}], 'duration': 452.954, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1301194.jpg', 'highlights': ['Defining data generators in Keras requires specifying data augmentation and normalization parameters, increasing code complexity compared to FastAI', 'Creating separate data generators for training and validation sets in Keras, with no data augmentation for the validation set', 'The need for detailed understanding of the expected data and the necessity to carefully set multiple parameters for Keras, increasing error susceptibility compared to FastAI', 'The validation accuracy reached 95% while fine-tuning ResNet-50 in Keras', 'Encountered unusual accuracy fluctuations of 49%, 69%, and 95% during the fine-tuning process', 'Challenges of manually constructing the model and freezing layers in Keras', 'The complexity and sensitivity of setting various parameters in Keras, with the risk of breaking the code if set incorrectly']}, {'end': 2646.652, 'segs': [{'end': 1824.457, 'src': 'embed', 'start': 1754.148, 'weight': 0, 'content': [{'end': 1763.155, 'text': "you have to recompile the model and then after that I then ran another step and again I don't know what happened here the accuracy of the training set stayed about the same,", 'start': 1754.148, 'duration': 9.007}, {'end': 1767.518, 'text': 'but the validation set totally fill in the hole.', 'start': 1763.155, 'duration': 4.363}, {'end': 1773.191, 'text': "but I mean the main thing to note is, even if we put aside the validation set and We're getting.", 'start': 1767.518, 'duration': 5.673}, {'end': 1780.693, 'text': "I mean I guess the main thing is there's a hell of a lot more code here, Which is kind of annoying, but also the performance is very different.", 'start': 1773.191, 'duration': 7.502}, {'end': 1781.913, 'text': "So we're also here.", 'start': 1780.693, 'duration': 1.22}, {'end': 1789.556, 'text': 'even on the training set, we were getting like 97 percent After four epochs that took a total of about eight minutes.', 'start': 1781.913, 'duration': 7.643}, {'end': 1797.538, 'text': 'You know, over here we had 99.5 percent on the validation set and it ran a lot faster.', 'start': 1789.556, 'duration': 7.982}, {'end': 1802.556, 'text': 'so it was like four or five minutes right?', 'start': 1797.538, 'duration': 5.018}, {'end': 1812.291, 'text': 'So Depending on what you do, particularly if you end up wanting to deploy stuff to mobile devices,', 'start': 1803.299, 'duration': 8.992}, {'end': 1818.054, 'text': 'at the moment the kind of PyTorch on mobile situation is very early.', 'start': 1812.291, 'duration': 5.763}, {'end': 1824.457, 'text': "so you may find yourself wanting to use TensorFlow or you may work for a company that's kind of settled on TensorFlow.", 'start': 1818.054, 'duration': 6.403}], 'summary': 'After recompiling the model, the accuracy on the validation set improved to 99.5% and ran faster, taking about four to five minutes, while the training set accuracy was around 97% after four epochs lasting approximately eight minutes.', 'duration': 70.309, 'max_score': 1754.148, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1754148.jpg'}, {'end': 1917.63, 'src': 'embed', 'start': 1890.691, 'weight': 4, 'content': [{'end': 1896.715, 'text': "I actually spoke to Google about this a few weeks ago, and they're very interested in getting fast AI ported to TensorFlow.", 'start': 1890.691, 'duration': 6.024}, {'end': 1900.618, 'text': "So maybe by the time you're looking at this on the MOOC, maybe that'll exist.", 'start': 1897.496, 'duration': 3.122}, {'end': 1901.899, 'text': 'I certainly hope so.', 'start': 1901.198, 'duration': 0.701}, {'end': 1903.44, 'text': 'We will see.', 'start': 1903.039, 'duration': 0.401}, {'end': 1914.888, 'text': 'Anyway, so Keras and TensorFlow are certainly not that difficult to handle,', 'start': 1903.46, 'duration': 11.428}, {'end': 1917.63, 'text': "and so I don't think you should worry if you're told you have to learn them.", 'start': 1914.888, 'duration': 2.742}], 'summary': 'Google is interested in fast ai porting to tensorflow, and keras and tensorflow are not difficult to handle.', 'duration': 26.939, 'max_score': 1890.691, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1890691.jpg'}, {'end': 2180.21, 'src': 'embed', 'start': 2153.646, 'weight': 2, 'content': [{'end': 2157.768, 'text': 'so then we just have to go exp of that to get back our probabilities.', 'start': 2153.646, 'duration': 4.122}, {'end': 2165.191, 'text': 'So in this case the test set had 10,357 images in it, and there are 120 possible breeds.', 'start': 2158.848, 'duration': 6.343}, {'end': 2168.832, 'text': 'So we get back a matrix of that size.', 'start': 2166.131, 'duration': 2.701}, {'end': 2174.575, 'text': 'And so we now need to turn that into something that looks like this.', 'start': 2170.093, 'duration': 4.482}, {'end': 2178.05, 'text': 'And so the easiest way to do that is with pandas.', 'start': 2176.249, 'duration': 1.801}, {'end': 2180.21, 'text': "If you're not familiar with pandas,", 'start': 2178.71, 'duration': 1.5}], 'summary': 'The test set had 10,357 images with 120 possible breeds, and pandas can help with the conversion.', 'duration': 26.564, 'max_score': 2153.646, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc2153646.jpg'}], 'start': 1754.148, 'title': 'Model recompilation and kaggle submission process', 'summary': 'Discusses recompiling the model, comparing the performance on training and validation sets in pytorch and tensorflow, and efforts to develop a keras-compatible version of fastai. it also covers the kaggle submission process, including obtaining predictions, creating the required file format, and submitting to kaggle, with a focus on understanding the classes, generating predictions, and creating the submission file, including tips for using tta and dealing with test set labels.', 'chapters': [{'end': 1917.63, 'start': 1754.148, 'title': 'Model recompilation and performance comparison', 'summary': 'Discusses recompiling the model, comparing the performance on training and validation sets, and highlights the differences in accuracy and execution time between pytorch and tensorflow, with a mention of efforts to develop a keras-compatible version of fastai.', 'duration': 163.482, 'highlights': ['The performance on the validation set showed a significant improvement, with an accuracy of 99.5% after the recompilation, compared to about 97% on the training set, indicating the effectiveness of the changes made (quantifiable data: accuracy percentages).', 'The recompiled model achieved faster execution, taking only four to five minutes compared to the original eight minutes, showcasing a notable reduction in processing time (quantifiable data: time comparison).', 'The differences in performance between PyTorch and TensorFlow are highlighted, with the mention of the early stage of PyTorch on mobile devices and the potential need to use TensorFlow for deployment (key points: performance differences, PyTorch on mobile devices, potential use of TensorFlow for deployment).', 'The additional effort required to achieve state-of-the-art results in TensorFlow, including the need to replicate fastai algorithms and implement various techniques such as stochastic gradient descent with restarts, differential learning rates, and batch norm freezing, is emphasized, pointing out the challenges and differences in achieving comparable results (key points: additional effort in TensorFlow, challenges in replicating fastai algorithms and techniques).', 'The mention of ongoing efforts to create a Keras-compatible version of fastai, along with the interest from Google to port fastai to TensorFlow, indicates the potential for future developments in making fastai accessible in TensorFlow, highlighting the possibility of a future Keras-compatible or TensorFlow-compatible version of fastai (key points: ongoing efforts to create Keras-compatible version, interest from Google to port fastai to TensorFlow).']}, {'end': 2646.652, 'start': 1918.437, 'title': 'Kaggle submission process', 'summary': 'Covers the kaggle submission process, including obtaining predictions, creating the required file format, and submitting to kaggle, with a focus on understanding the classes, generating predictions, and creating the submission file, including tips for using tta and dealing with test set labels.', 'duration': 728.215, 'highlights': ['The chapter covers the Kaggle submission process, including obtaining predictions, creating the required file format, and submitting to Kaggle. This is the overarching theme of the transcript, emphasizing the focus on the Kaggle submission process.', 'The test set had 10,357 images and 120 possible breeds, resulting in a matrix of that size which needs to be converted into the required CSV file format. Quantifiable data about the test set size and number of possible breeds, showcasing the need for converting the matrix into the required format for submission.', 'A brief demonstration of using pandas to create the required CSV file format for submission, including handling file name adjustments and file compression. Details the practical steps involved in using pandas to create the submission file, including specific adjustments and file compression to facilitate the submission process.', 'Tips for using TTA to improve predictions, particularly when the model is less accurate, and obtaining predictions on the test set rather than the validation set. Emphasizes the importance of using TTA for improving predictions and highlights the distinction between obtaining predictions on the test set versus the validation set.', 'Explanation of transforming and preparing an image for prediction, including the need to create a mini-batch of images when dealing with single images. Provides insights into the practical considerations for transforming and preparing images for prediction, highlighting the need to handle single images as mini-batches.']}], 'duration': 892.504, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc1754148.jpg', 'highlights': ['The performance on the validation set showed a significant improvement, with an accuracy of 99.5% after the recompilation, compared to about 97% on the training set, indicating the effectiveness of the changes made (quantifiable data: accuracy percentages).', 'The recompiled model achieved faster execution, taking only four to five minutes compared to the original eight minutes, showcasing a notable reduction in processing time (quantifiable data: time comparison).', 'The test set had 10,357 images and 120 possible breeds, resulting in a matrix of that size which needs to be converted into the required CSV file format. Quantifiable data about the test set size and number of possible breeds, showcasing the need for converting the matrix into the required format for submission.', 'The differences in performance between PyTorch and TensorFlow are highlighted, with the mention of the early stage of PyTorch on mobile devices and the potential need to use TensorFlow for deployment (key points: performance differences, PyTorch on mobile devices, potential use of TensorFlow for deployment).', 'The mention of ongoing efforts to create a Keras-compatible version of fastai, along with the interest from Google to port fastai to TensorFlow, indicates the potential for future developments in making fastai accessible in TensorFlow, highlighting the possibility of a future Keras-compatible or TensorFlow-compatible version of fastai (key points: ongoing efforts to create Keras-compatible version, interest from Google to port fastai to TensorFlow).']}, {'end': 3088.775, 'segs': [{'end': 2690.894, 'src': 'embed', 'start': 2648.915, 'weight': 0, 'content': [{'end': 2653.437, 'text': "And I want to kind of step you through it because I think it explains really, really well what's going on.", 'start': 2648.915, 'duration': 4.522}, {'end': 2663.061, 'text': "And then after we look at the video, we're going to see how to implement an entire set of layers of a convolutional neural network in Microsoft Excel.", 'start': 2653.457, 'duration': 9.604}, {'end': 2668.584, 'text': "So whether you're a visual learner or a spreadsheet learner, hopefully you'll be able to understand all this.", 'start': 2664.122, 'duration': 4.462}, {'end': 2672.4, 'text': "So we're going to start with an image,", 'start': 2669.758, 'duration': 2.642}, {'end': 2677.424, 'text': "and something that we're going to do later in the course is we're going to learn to recognize digits and we'll do it like end-to-end.", 'start': 2672.4, 'duration': 5.024}, {'end': 2678.184, 'text': "we'll do the whole thing.", 'start': 2677.424, 'duration': 0.76}, {'end': 2680.006, 'text': 'So this is pretty similar.', 'start': 2678.885, 'duration': 1.121}, {'end': 2683.268, 'text': "So we're going to try and recognize, in this case, letters.", 'start': 2680.826, 'duration': 2.442}, {'end': 2688.512, 'text': "So here's an A which obviously it's actually a grid of numbers, right?", 'start': 2683.669, 'duration': 4.843}, {'end': 2690.894, 'text': "And so there's the grid of numbers.", 'start': 2689.153, 'duration': 1.741}], 'summary': 'Explanation of implementing convolutional neural network in excel to recognize letters and digits.', 'duration': 41.979, 'max_score': 2648.915, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc2648915.jpg'}, {'end': 2870.627, 'src': 'heatmap', 'start': 2704.615, 'weight': 0.946, 'content': [{'end': 2710.062, 'text': "So it's like zero, zero, zero or maybe negative one negative one, negative one, zero, zero, zero, one, one, one.", 'start': 2704.615, 'duration': 5.447}, {'end': 2716.228, 'text': "and so we're taking each three by three part of the image and Multiplying it by that three by three.", 'start': 2710.062, 'duration': 6.166}, {'end': 2720.195, 'text': 'Matrix, not as a matrix product, but an element wise product.', 'start': 2716.228, 'duration': 3.967}, {'end': 2730.272, 'text': "and so you can see what happens is everywhere where the the white edge of Matching the edge of the a and the black edge isn't, we're getting green,", 'start': 2720.195, 'duration': 10.077}, {'end': 2733.374, 'text': "We're getting a positive, and everywhere where it's the opposite.", 'start': 2730.272, 'duration': 3.102}, {'end': 2734.334, 'text': "We're getting a negative.", 'start': 2733.434, 'duration': 0.9}, {'end': 2736.275, 'text': "We're getting a red right.", 'start': 2734.374, 'duration': 1.901}, {'end': 2743.36, 'text': "and so that's the first filter creating the first, The result of the first kernel right.", 'start': 2736.275, 'duration': 7.085}, {'end': 2744.78, 'text': "and so here's a new kernel.", 'start': 2743.36, 'duration': 1.42}, {'end': 2748.182, 'text': 'this one is is got a white stripe along the top right.', 'start': 2744.78, 'duration': 3.402}, {'end': 2762.174, 'text': "so we literally scan it through every three by three part of the matrix and multiplying those three bits of the a a nine bits of the a by the nine bits of the filter to find out whether It's red or green and how red or green it is.", 'start': 2748.182, 'duration': 13.992}, {'end': 2769.26, 'text': 'Okay. and so this is assuming we had two filters one was a bottom edge, One was a left edge and you can see here the top edge.', 'start': 2762.174, 'duration': 7.086}, {'end': 2770.661, 'text': "not surprisingly It's red here.", 'start': 2769.26, 'duration': 1.401}, {'end': 2776.126, 'text': 'so a bottom edge was red here and green here, the right edge, red here and green here.', 'start': 2770.661, 'duration': 5.465}, {'end': 2784.021, 'text': 'and then in the next step we add a non-linearity and The rectified linear unit, which literally means throw away the negatives.', 'start': 2776.126, 'duration': 7.895}, {'end': 2786.302, 'text': 'so here the reds all gone.', 'start': 2784.021, 'duration': 2.281}, {'end': 2788.182, 'text': "Okay. so here's layer 1, the input.", 'start': 2786.302, 'duration': 1.88}, {'end': 2792.663, 'text': "Here's layer 2, the result of two convolutional filters.", 'start': 2788.562, 'duration': 4.101}, {'end': 2799.545, 'text': "Here's layer 3, which is which is throw away all of the red stuff, and that's called a rectified linear unit.", 'start': 2792.663, 'duration': 6.882}, {'end': 2802.286, 'text': 'and Then layer 4 is something called a max pool.', 'start': 2799.545, 'duration': 2.741}, {'end': 2810.77, 'text': 'On Layer 4, we replace every 2x2 part of this grid and we replace it with its maximum.', 'start': 2803.208, 'duration': 7.562}, {'end': 2813.151, 'text': 'So it basically makes it half the size.', 'start': 2811.41, 'duration': 1.741}, {'end': 2815.872, 'text': "It's basically the same thing, but half the size.", 'start': 2813.831, 'duration': 2.041}, {'end': 2818.893, 'text': 'And then we can go through and do exactly the same thing.', 'start': 2816.532, 'duration': 2.361}, {'end': 2826.595, 'text': 'We can have some new filter, 3x3 filter, that we put through each of the two results of the previous layer.', 'start': 2819.233, 'duration': 7.362}, {'end': 2830.576, 'text': 'And again, we can throw away the red bits.', 'start': 2828.695, 'duration': 1.881}, {'end': 2832.905, 'text': 'So get rid of all the negatives.', 'start': 2831.663, 'duration': 1.242}, {'end': 2833.926, 'text': 'so we just keep the positives.', 'start': 2832.905, 'duration': 1.021}, {'end': 2843.256, 'text': "that's called applying a rectified linear unit, And that gets us to our next layer of this convolutional neural network.", 'start': 2833.926, 'duration': 9.33}, {'end': 2849.208, 'text': 'so you can see that by you know, at this layer back here, and It was kind of very interpretable.', 'start': 2843.256, 'duration': 5.952}, {'end': 2856.955, 'text': "It's like we've either got bottom edges or left edges, But then the next layer was combining the results of convolution.", 'start': 2849.268, 'duration': 7.687}, {'end': 2860.999, 'text': "So it's starting to become a lot less clear, like intuitively, what's happening.", 'start': 2856.955, 'duration': 4.044}, {'end': 2862.48, 'text': "but it's doing the same thing.", 'start': 2860.999, 'duration': 1.481}, {'end': 2865.002, 'text': 'And then we do another max pull right.', 'start': 2862.48, 'duration': 2.522}, {'end': 2870.627, 'text': 'so we replace every two by two or three by three section with a single Digit.', 'start': 2865.002, 'duration': 5.625}], 'summary': 'Applying convolutional filters and max pooling to downsize and extract features from the input image.', 'duration': 166.012, 'max_score': 2704.615, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc2704615.jpg'}, {'end': 2843.256, 'src': 'embed', 'start': 2792.663, 'weight': 2, 'content': [{'end': 2799.545, 'text': "Here's layer 3, which is which is throw away all of the red stuff, and that's called a rectified linear unit.", 'start': 2792.663, 'duration': 6.882}, {'end': 2802.286, 'text': 'and Then layer 4 is something called a max pool.', 'start': 2799.545, 'duration': 2.741}, {'end': 2810.77, 'text': 'On Layer 4, we replace every 2x2 part of this grid and we replace it with its maximum.', 'start': 2803.208, 'duration': 7.562}, {'end': 2813.151, 'text': 'So it basically makes it half the size.', 'start': 2811.41, 'duration': 1.741}, {'end': 2815.872, 'text': "It's basically the same thing, but half the size.", 'start': 2813.831, 'duration': 2.041}, {'end': 2818.893, 'text': 'And then we can go through and do exactly the same thing.', 'start': 2816.532, 'duration': 2.361}, {'end': 2826.595, 'text': 'We can have some new filter, 3x3 filter, that we put through each of the two results of the previous layer.', 'start': 2819.233, 'duration': 7.362}, {'end': 2830.576, 'text': 'And again, we can throw away the red bits.', 'start': 2828.695, 'duration': 1.881}, {'end': 2832.905, 'text': 'So get rid of all the negatives.', 'start': 2831.663, 'duration': 1.242}, {'end': 2833.926, 'text': 'so we just keep the positives.', 'start': 2832.905, 'duration': 1.021}, {'end': 2843.256, 'text': "that's called applying a rectified linear unit, And that gets us to our next layer of this convolutional neural network.", 'start': 2833.926, 'duration': 9.33}], 'summary': 'Convolutional neural network layers involve rectified linear units and max pooling to reduce grid size.', 'duration': 50.593, 'max_score': 2792.663, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc2792663.jpg'}, {'end': 2985.847, 'src': 'embed', 'start': 2960.395, 'weight': 3, 'content': [{'end': 2964.977, 'text': 'And so then we can do it with a new letter A or letter B or whatever.', 'start': 2960.395, 'duration': 4.582}, {'end': 2968.512, 'text': 'and keep going through that process.', 'start': 2965.549, 'duration': 2.963}, {'end': 2975.658, 'text': "So, as you can see, that's a far nicer visualization than I could have created, because I'm not Octavio.", 'start': 2970.373, 'duration': 5.285}, {'end': 2981.083, 'text': "so thanks to him for sharing this with us, because it's totally awesome.", 'start': 2975.658, 'duration': 5.425}, {'end': 2982.444, 'text': 'This is not done by hand.', 'start': 2981.283, 'duration': 1.161}, {'end': 2985.847, 'text': 'He actually wrote a piece of computer software to actually do these convolutions.', 'start': 2982.484, 'duration': 3.363}], 'summary': 'Octavio created computer software for convolutions, improving visualization process.', 'duration': 25.452, 'max_score': 2960.395, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc2960395.jpg'}], 'start': 2648.915, 'title': 'Implementing cnn in excel', 'summary': 'Details the process of implementing a convolutional neural network in microsoft excel to recognize letters from an image using convolutional filters, rectified linear unit, and max pooling. it also covers visualization of convolutions in spreadsheet form without manual work.', 'chapters': [{'end': 2865.002, 'start': 2648.915, 'title': 'Implementing convolutional neural network in excel', 'summary': 'Explains the process of implementing a convolutional neural network in microsoft excel, including the steps of applying convolutional filters, rectified linear unit, and max pooling to recognize letters from an image.', 'duration': 216.087, 'highlights': ['The process involves implementing an entire set of layers of a convolutional neural network in Microsoft Excel to recognize letters from an image.', 'Explaining the application of convolutional filters, including the recognition of edges and the creation of the first kernel.', 'Describing the use of rectified linear unit to discard negative values and the application of max pooling to reduce the grid size while retaining the essential features.', 'Illustrating the combination of results from convolution and the application of additional filters to progress through the layers of the convolutional neural network.']}, {'end': 3088.775, 'start': 2865.002, 'title': 'Convolution visualization in spreadsheet form', 'summary': 'Explains the process of convolutional filters and the application of a pre-trained model on an image without any changes, illustrating the dynamic visualization in spreadsheet form and emphasizing the absence of manual work in creating the convolutions.', 'duration': 223.773, 'highlights': ['The process of convolutional filters and the application of a pre-trained model on images is detailed. The chapter discusses the use of convolutional filters to compare and match sections of an image, without conducting any training, to determine the probability of the image being a certain letter.', 'Dynamic visualization in spreadsheet form is presented, emphasizing the absence of manual work in creating the convolutions. The dynamic visualization of convolutions in spreadsheet form is showcased, indicating the absence of manual creation as Octavio developed a computer software for this purpose.', 'Explanation of the pixel values and their relevance in PyTorch is provided. The relevance of pixel values in PyTorch, specifically the conversion to a number between 0 and 1, is discussed, with a focus on the typical representation of pixel values as floats in PyTorch.']}], 'duration': 439.86, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc2648915.jpg', 'highlights': ['The process involves implementing an entire set of layers of a convolutional neural network in Microsoft Excel to recognize letters from an image.', 'The chapter discusses the use of convolutional filters to compare and match sections of an image, without conducting any training, to determine the probability of the image being a certain letter.', 'Describing the use of rectified linear unit to discard negative values and the application of max pooling to reduce the grid size while retaining the essential features.', 'Dynamic visualization in spreadsheet form is presented, emphasizing the absence of manual work in creating the convolutions.', 'Illustrating the combination of results from convolution and the application of additional filters to progress through the layers of the convolutional neural network.']}, {'end': 4096.781, 'segs': [{'end': 3144.422, 'src': 'embed', 'start': 3116.592, 'weight': 7, 'content': [{'end': 3121.134, 'text': "So here I've created a filter which is designed to detect top edges.", 'start': 3116.592, 'duration': 4.542}, {'end': 3122.995, 'text': 'So this is a 3x3 filter.', 'start': 3121.995, 'duration': 1}, {'end': 3130.016, 'text': "Okay, and I've got ones along the top, zeros in the middle, minus ones at the bottom, right.", 'start': 3124.474, 'duration': 5.542}, {'end': 3131.817, 'text': "so let's take a look at an example.", 'start': 3130.016, 'duration': 1.801}, {'end': 3136.559, 'text': "that's Here right, and so if I hit that too,", 'start': 3131.817, 'duration': 4.742}, {'end': 3144.422, 'text': 'you can see here highlighted This is the three by three part of the input that this particular thing is calculating right.', 'start': 3136.559, 'duration': 7.863}], 'summary': 'Created a 3x3 filter to detect top edges in an image.', 'duration': 27.83, 'max_score': 3116.592, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3116592.jpg'}, {'end': 3243.229, 'src': 'embed', 'start': 3204.803, 'weight': 8, 'content': [{'end': 3208.724, 'text': 'So when I say an activation, I mean a number.', 'start': 3204.803, 'duration': 3.921}, {'end': 3214.896, 'text': 'number that is calculated.', 'start': 3212.115, 'duration': 2.781}, {'end': 3227.64, 'text': 'and it is calculated by taking some numbers from the input and Applying some kind of linear operation, in this case a convolutional kernel,', 'start': 3214.896, 'duration': 12.744}, {'end': 3228.92, 'text': 'to calculate an output.', 'start': 3227.64, 'duration': 1.28}, {'end': 3240.324, 'text': "Right, you'll notice that, other than going inputs multiplied by Kernel and Summing it together right.", 'start': 3228.92, 'duration': 11.404}, {'end': 3243.229, 'text': "so here's my sum and And here's my multiply.", 'start': 3240.324, 'duration': 2.905}], 'summary': 'An activation is calculated by applying a convolutional kernel to input numbers to produce an output.', 'duration': 38.426, 'max_score': 3204.803, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3204803.jpg'}, {'end': 3312.357, 'src': 'embed', 'start': 3278.499, 'weight': 5, 'content': [{'end': 3281.521, 'text': "But if I'm not saying I'm simplifying, that's the entirety.", 'start': 3278.499, 'duration': 3.022}, {'end': 3292.025, 'text': 'So a rectified linear unit in its entirety is this, and a convolution in its entirety this Okay,', 'start': 3281.781, 'duration': 10.244}, {'end': 3302.431, 'text': 'so a single layer of a convolutional neural network is being implemented in its entirety Here in Excel.', 'start': 3292.025, 'duration': 10.406}, {'end': 3312.037, 'text': "okay, and so you can see what it's done is it's deleted Pretty much the vertical edges and highlighted the horizontal edges.", 'start': 3302.431, 'duration': 9.606}, {'end': 3312.357, 'text': 'So again,', 'start': 3312.037, 'duration': 0.32}], 'summary': 'Demonstrating the implementation of a single layer of a convolutional neural network using excel.', 'duration': 33.858, 'max_score': 3278.499, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3278499.jpg'}, {'end': 3628.475, 'src': 'heatmap', 'start': 3439.057, 'weight': 6, 'content': [{'end': 3450.047, 'text': "right, because my next filter is going to have to contain a of these three by threes, because I'm going to have to say how do I want to bring,", 'start': 3439.057, 'duration': 10.99}, {'end': 3453.308, 'text': 'how do I want to Wait these three things?', 'start': 3450.047, 'duration': 3.261}, {'end': 3457.249, 'text': 'and at the same time, how do I want to wait the corresponding three things down here?', 'start': 3453.308, 'duration': 3.941}, {'end': 3465.791, 'text': 'But because in pi torch this is going to be, this whole thing here is going to be stored as a Multi-dimensional tensor.', 'start': 3457.689, 'duration': 8.102}, {'end': 3475.034, 'text': "right, so you shouldn't really think of this now as two, three by three kernels, but one, two by three by three kernels.", 'start': 3465.791, 'duration': 9.243}, {'end': 3492.965, 'text': "Okay, so to calculate this value here, I've got the sum product of all of that, plus the sum product of Scroll down.", 'start': 3476.178, 'duration': 16.787}, {'end': 3495.446, 'text': 'all of that, okay?', 'start': 3492.965, 'duration': 2.481}, {'end': 3502.767, 'text': 'And so the top ones are being multiplied by this part of the kernel and the bottom ones are being multiplied by this part of the kernel.', 'start': 3496.846, 'duration': 5.921}, {'end': 3513.209, 'text': 'And so over time, you want to start to get very comfortable with the idea of these higher-dimensional linear combinations.', 'start': 3503.187, 'duration': 10.022}, {'end': 3523.231, 'text': "It's harder to draw it on the screen, like I had to put one above the other, but conceptually just stack it in your mind like this.", 'start': 3513.749, 'duration': 9.482}, {'end': 3524.691, 'text': "That's really how you want to think.", 'start': 3523.391, 'duration': 1.3}, {'end': 3533.397, 'text': 'And actually, Geoffrey Hinton, in his original 2012 neural nets Coursera class, has a tip,', 'start': 3526.175, 'duration': 7.222}, {'end': 3538.598, 'text': 'which is how all computer scientists deal with like very high dimensional spaces,', 'start': 3533.397, 'duration': 5.201}, {'end': 3547.301, 'text': 'Which is that they basically just visualize the two-dimensional space and then say, like 12 dimensions, really fast in their head lots of times.', 'start': 3538.598, 'duration': 8.703}, {'end': 3548.361, 'text': "So that's it right.", 'start': 3547.301, 'duration': 1.06}, {'end': 3555.082, 'text': 'we can see two dimensions on the screen and then you just got to try to trust that That you can have more dimensions.', 'start': 3548.361, 'duration': 6.721}, {'end': 3559.323, 'text': "like the concepts just you know there's, there's nothing different about them.", 'start': 3555.082, 'duration': 4.241}, {'end': 3560.943, 'text': 'And so you can see in excel.', 'start': 3559.323, 'duration': 1.62}, {'end': 3564.564, 'text': "you know excel doesn't have the ability to handle three-dimensional tensors.", 'start': 3560.943, 'duration': 3.621}, {'end': 3572.106, 'text': 'So I had to like say, okay, take this two-dimensional Dot product, add on this two-dimensional dot product.', 'start': 3564.564, 'duration': 7.542}, {'end': 3581.649, 'text': 'right, but if there was some kind of 3d excel, I could have just done that in a single formula, right, and then again apply Max zero comma,', 'start': 3572.106, 'duration': 9.543}, {'end': 3585.745, 'text': 'Otherwise known as rectified linear unit, otherwise known as value.', 'start': 3582.423, 'duration': 3.322}, {'end': 3589.628, 'text': 'okay, so here is my second layer.', 'start': 3585.745, 'duration': 3.883}, {'end': 3600.374, 'text': 'And so when people create different architectures, right, an architecture means Like how big is your kernel at layer one?', 'start': 3589.628, 'duration': 10.746}, {'end': 3602.996, 'text': 'how many filters are in your kernel at layer one?', 'start': 3600.374, 'duration': 2.622}, {'end': 3609.273, 'text': "so here? I've got a three by three, There's number one and a three by three there's number two.", 'start': 3602.996, 'duration': 6.277}, {'end': 3618.007, 'text': "so like this, Architecture I've created starts off with two, three by three convolutional kernels,", 'start': 3609.273, 'duration': 8.734}, {'end': 3626.052, 'text': 'and then my Second layer has another two kernels of size two by three by three.', 'start': 3618.007, 'duration': 8.045}, {'end': 3628.475, 'text': "So there's the first one and then down here.", 'start': 3626.052, 'duration': 2.423}], 'summary': 'Explanation of multi-dimensional tensors and higher-dimensional linear combinations, visualizing high-dimensional spaces, and creating different convolutional kernel architectures.', 'duration': 28.101, 'max_score': 3439.057, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3439057.jpg'}, {'end': 3720.12, 'src': 'embed', 'start': 3690.191, 'weight': 3, 'content': [{'end': 3695.615, 'text': "So, in this architecture we're inventing, the next step is do max pooling.", 'start': 3690.191, 'duration': 5.424}, {'end': 3701.985, 'text': "Okay, max pooling is a little hard to show in Excel, but we've got it.", 'start': 3696.34, 'duration': 5.645}, {'end': 3710.071, 'text': "So max pooling, if I do a 2x2 max pooling, it's going to halve the resolution both height and width.", 'start': 3703.146, 'duration': 6.925}, {'end': 3720.12, 'text': "So you can see here that I've replaced these four numbers with the maximum of those four numbers.", 'start': 3710.672, 'duration': 9.448}], 'summary': 'In the architecture, implementing 2x2 max pooling halves the resolution of the data.', 'duration': 29.929, 'max_score': 3690.191, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3690191.jpg'}, {'end': 3857.418, 'src': 'heatmap', 'start': 3764.423, 'weight': 0.726, 'content': [{'end': 3773.569, 'text': "And so, after our max pooling, there's a number of different things we could do next,", 'start': 3764.423, 'duration': 9.146}, {'end': 3780.035, 'text': "and I'm going to show you a kind of classic old-style Approach nowadays.", 'start': 3773.569, 'duration': 6.466}, {'end': 3786.637, 'text': 'in fact, what generally happens nowadays is we do a max pool where we kind of like max across the entire size, right?', 'start': 3780.035, 'duration': 6.602}, {'end': 3795.159, 'text': 'But on older architectures and also on all the structured data stuff we do, We actually do something called a fully connected layer.', 'start': 3787.557, 'duration': 7.602}, {'end': 3797.6, 'text': "And so here's a fully connected layer.", 'start': 3795.739, 'duration': 1.861}, {'end': 3805.382, 'text': "I'm going to take every single one of these activations and I've got to give every single one of them a weight,", 'start': 3797.6, 'duration': 7.782}, {'end': 3810.411, 'text': "right and And so then I'm going to take over here.", 'start': 3805.382, 'duration': 5.029}, {'end': 3823.643, 'text': 'here is the sum product of every one of the activations by every one of the weights for both of the two levels of my three-dimensional tensor.', 'start': 3810.411, 'duration': 13.232}, {'end': 3827.106, 'text': 'And so this is called a fully connected layer.', 'start': 3824.684, 'duration': 2.422}, {'end': 3828.888, 'text': "Notice it's different to a convolution.", 'start': 3827.206, 'duration': 1.682}, {'end': 3830.629, 'text': "I'm not going through a few at a time.", 'start': 3828.928, 'duration': 1.701}, {'end': 3835.735, 'text': "I'm creating a really big weight matrix, Right.", 'start': 3831.812, 'duration': 3.923}, {'end': 3842.581, 'text': 'so, rather than having a couple of little three by three kernels, my weight matrix is now as big as the entire input.', 'start': 3835.735, 'duration': 6.846}, {'end': 3845.944, 'text': 'And so, as you can imagine,', 'start': 3842.581, 'duration': 3.363}, {'end': 3857.418, 'text': 'Architectures that make heavy use of fully convolutional layers can have a lot of weights and Which means they can have trouble with overfitting and they can also be slow.', 'start': 3845.944, 'duration': 11.474}], 'summary': 'Fully connected layers in older architectures use large weight matrices, leading to potential overfitting and slow performance.', 'duration': 92.995, 'max_score': 3764.423, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3764423.jpg'}, {'end': 3899.876, 'src': 'embed', 'start': 3865.083, 'weight': 0, 'content': [{'end': 3877.829, 'text': 'it has up to 19 layers and VGG actually contains a fully connected layer with 4096 weights and Connected to at a hidden layer with 4,000 sorry,', 'start': 3865.083, 'duration': 12.746}, {'end': 3884.031, 'text': '4096 Activations connected to a hidden layer with 4096 activations.', 'start': 3877.829, 'duration': 6.202}, {'end': 3894.994, 'text': "so you've got like 4096 by 4096 multiplied by remember, multiplied by the number of kind of kernels that we've calculated.", 'start': 3884.031, 'duration': 10.963}, {'end': 3899.876, 'text': "so in VGG there's This.", 'start': 3894.994, 'duration': 4.882}], 'summary': 'Vgg has up to 19 layers with a fully connected layer of 4096 weights and activations.', 'duration': 34.793, 'max_score': 3865.083, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3865083.jpg'}, {'end': 3983.096, 'src': 'embed', 'start': 3954.454, 'weight': 1, 'content': [{'end': 3958.936, 'text': 'so therefore our filters are had to have two channels per filter.', 'start': 3954.454, 'duration': 4.482}, {'end': 3965.717, 'text': "And so you could imagine that this input didn't exist, and actually this was the input.", 'start': 3959.476, 'duration': 6.241}, {'end': 3970.979, 'text': 'So when you have a multi-channel input, it just means that your filters look like this.', 'start': 3966.377, 'duration': 4.602}, {'end': 3974.66, 'text': 'And so images often are full color.', 'start': 3971.479, 'duration': 3.181}, {'end': 3976.86, 'text': 'They have three, red, green, and blue.', 'start': 3974.72, 'duration': 2.14}, {'end': 3979.061, 'text': 'Sometimes they also have an alpha channel.', 'start': 3977.24, 'duration': 1.821}, {'end': 3983.096, 'text': "So however many you have that's how many inputs you need.", 'start': 3979.421, 'duration': 3.675}], 'summary': 'Filters need two channels per filter for multi-channel inputs, such as full-color images with three (rgb) or four (rgb+a) channels.', 'duration': 28.642, 'max_score': 3954.454, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3954454.jpg'}, {'end': 4050.749, 'src': 'embed', 'start': 4027.722, 'weight': 2, 'content': [{'end': 4039.305, 'text': "And so at the moment there's a Kaggle competition for iceberg detection using Some funky satellite specific data format that has two channels.", 'start': 4027.722, 'duration': 11.583}, {'end': 4041.006, 'text': "So here's how you could do that.", 'start': 4039.305, 'duration': 1.701}, {'end': 4050.749, 'text': 'you could either copy one of those two channels into the third channel, or I think what people on Kaggle are doing is to take the average of the two.', 'start': 4041.006, 'duration': 9.743}], 'summary': 'Kaggle competition for iceberg detection using satellite data with two channels', 'duration': 23.027, 'max_score': 4027.722, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4027722.jpg'}, {'end': 4096.781, 'src': 'embed', 'start': 4068.678, 'weight': 4, 'content': [{'end': 4081.993, 'text': 'I had a satellite data where the fourth channel was near infrared, and So basically, I added an extra Kind of Level to my convolutional kernels.', 'start': 4068.678, 'duration': 13.315}, {'end': 4088.157, 'text': 'that were all zeros, and so basically, like started off by ignoring the new infrared band.', 'start': 4081.993, 'duration': 6.164}, {'end': 4089.197, 'text': 'and so what happens?', 'start': 4088.157, 'duration': 1.04}, {'end': 4096.781, 'text': "it basically and you'll see this next week is that Rather than having these like carefully trained filters,", 'start': 4089.197, 'duration': 7.584}], 'summary': 'Utilized near-infrared data to modify convolutional kernels for improved results.', 'duration': 28.103, 'max_score': 4068.678, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4068678.jpg'}], 'start': 3088.775, 'title': 'Convolutional neural networks', 'summary': 'Covers implementing a single layer cnn in excel, creating cnn architecture with hidden layers, convolutional kernels, max pooling, and fully connected layers, and explores the impact of multi-channel inputs on convolutional filters.', 'chapters': [{'end': 3378.413, 'start': 3088.775, 'title': 'Convolutional neural network in excel', 'summary': 'Explains the implementation of a single layer of a convolutional neural network in excel, showcasing the use of convolutional filters to detect edges and activation calculation, with specific examples and operations highlighted.', 'duration': 289.638, 'highlights': ['Implementation of a single layer of a convolutional neural network in Excel Demonstrates the practical application of a convolutional neural network within Excel, providing a tangible example of complex neural network concepts.', 'Use of convolutional filters to detect edges Illustrates the utilization of convolutional filters to detect edges within an input, showcasing the impact of specific filter designs on edge detection.', 'Calculation of activation by applying linear operations Explains the calculation of activation through the application of linear operations, exemplifying how inputs are manipulated by convolutional kernels to generate an output.']}, {'end': 3923.759, 'start': 3379.314, 'title': 'Convolutional neural network architecture', 'summary': 'Discusses the creation of a convolutional neural network architecture, including the concept of hidden layers, the use of convolutional kernels, max pooling, and fully connected layers, with mentions of specific architectures like vgg and the challenges of using fully connected layers.', 'duration': 544.445, 'highlights': ['The architecture starts off with two, three by three convolutional kernels, and then the second layer has another two kernels of size two by three by three. The initial layer consists of two, three by three convolutional kernels, followed by a second layer with two kernels of size two by three by three.', 'Max pooling is applied to halve the resolution both in height and width using a 2x2 max pooling, replacing every 2 by 2 with its max, resulting in a decrease in resolution. Max pooling is performed using a 2x2 max pooling technique, which halves the resolution in both height and width, replacing every 2 by 2 with its max and decreasing the resolution.', 'VGG architecture contains a fully connected layer with 4096 weights and 4096 activations, resulting in a large number of weights and potential challenges with overfitting and speed. The VGG architecture includes a fully connected layer with 4096 weights and 4096 activations, leading to a large number of weights and potential issues with overfitting and speed.']}, {'end': 4096.781, 'start': 3923.759, 'title': 'Multi-channel inputs and convolutional filters', 'summary': 'Discusses the impact of multi-channel inputs on convolutional filters, including the need for matching channels, accommodating different data formats, and adapting pre-trained networks for various channel requirements.', 'duration': 173.022, 'highlights': ['Multi-channel inputs require matching channels in convolutional filters, such as using three channels for full-color images and adapting a single-channel input for a three-channel filter, as demonstrated in a medical imaging scenario with bone age calculations.', 'Adapting pre-trained networks for different channel requirements involves techniques like copying or averaging channels, as seen in the Kaggle competition for iceberg detection using a two-channel satellite-specific data format.', 'The flexibility of convolutional kernels allows for accommodating additional channels, as demonstrated with the adaptation of a three-channel ImageNet network for four-channel satellite data by adding a level of convolutional kernels with zeros for the extra infrared band.']}], 'duration': 1008.006, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc3088775.jpg', 'highlights': ['VGG architecture contains a fully connected layer with 4096 weights and 4096 activations, leading to potential issues with overfitting and speed.', 'Multi-channel inputs require matching channels in convolutional filters, such as using three channels for full-color images and adapting a single-channel input for a three-channel filter.', 'Adapting pre-trained networks for different channel requirements involves techniques like copying or averaging channels, as seen in the Kaggle competition for iceberg detection using a two-channel satellite-specific data format.', 'Max pooling is performed using a 2x2 max pooling technique, which halves the resolution in both height and width, replacing every 2 by 2 with its max and decreasing the resolution.', 'The flexibility of convolutional kernels allows for accommodating additional channels, as demonstrated with the adaptation of a three-channel ImageNet network for four-channel satellite data by adding a level of convolutional kernels with zeros for the extra infrared band.', 'Implementation of a single layer of a convolutional neural network in Excel demonstrates the practical application of a convolutional neural network within Excel, providing a tangible example of complex neural network concepts.', 'The architecture starts off with two, three by three convolutional kernels, followed by a second layer with two kernels of size two by three by three.', 'Use of convolutional filters to detect edges illustrates the utilization of convolutional filters to detect edges within an input, showcasing the impact of specific filter designs on edge detection.', 'Calculation of activation by applying linear operations explains the calculation of activation through the application of linear operations, exemplifying how inputs are manipulated by convolutional kernels to generate an output.']}, {'end': 4920.039, 'segs': [{'end': 4128.595, 'src': 'embed', 'start': 4096.781, 'weight': 0, 'content': [{'end': 4101.444, 'text': "when you're actually training something from scratch, We're actually going to start with random numbers.", 'start': 4096.781, 'duration': 4.663}, {'end': 4103.08, 'text': "and That's actually what we do.", 'start': 4101.444, 'duration': 1.636}, {'end': 4111.524, 'text': "we actually start with random numbers and then we use this thing called stochastic gradient descent Which we've kind of seen conceptually to slightly improve those random numbers,", 'start': 4103.08, 'duration': 8.444}, {'end': 4115.648, 'text': 'to make them less random, And we basically do that again and again and again.', 'start': 4111.524, 'duration': 4.124}, {'end': 4117.469, 'text': 'Okay great.', 'start': 4115.648, 'duration': 1.821}, {'end': 4128.595, 'text': "Let's take a seven minute break and we'll come back at 750, All right.", 'start': 4117.828, 'duration': 10.767}], 'summary': 'Training from scratch involves using stochastic gradient descent to improve random numbers. break at 7 minutes.', 'duration': 31.814, 'max_score': 4096.781, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4096781.jpg'}, {'end': 4293.51, 'src': 'embed', 'start': 4269.539, 'weight': 1, 'content': [{'end': 4278.796, 'text': "okay. so out of our that fully connected layer we've got in this case, we'd have five numbers and Notice at this point.", 'start': 4269.539, 'duration': 9.257}, {'end': 4280.398, 'text': "There's no relu.", 'start': 4279.537, 'duration': 0.861}, {'end': 4282.22, 'text': 'Okay, and then last layer.', 'start': 4280.418, 'duration': 1.802}, {'end': 4283.321, 'text': "There's no relu.", 'start': 4282.42, 'duration': 0.901}, {'end': 4285.042, 'text': 'Okay, so I can have negatives.', 'start': 4283.621, 'duration': 1.421}, {'end': 4293.51, 'text': 'Okay, so I want to turn these five numbers Each into a probability.', 'start': 4285.042, 'duration': 8.468}], 'summary': 'The fully connected layer has five numbers and no relu, aiming to convert them into probabilities.', 'duration': 23.971, 'max_score': 4269.539, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4269539.jpg'}, {'end': 4456.757, 'src': 'embed', 'start': 4431.745, 'weight': 3, 'content': [{'end': 4437.348, 'text': "Obviously there's lots of other nonlinearities in the world, but in deep learning this is what we mean.", 'start': 4431.745, 'duration': 5.603}, {'end': 4448.393, 'text': "So an activation function is any function that takes some activation in that's a single number and spits out some new activation like max.", 'start': 4437.888, 'duration': 10.505}, {'end': 4450.754, 'text': "So I'm now going to tell you about a different activation function.", 'start': 4448.393, 'duration': 2.361}, {'end': 4455.376, 'text': "It's slightly more complicated than ReLU, but not too much.", 'start': 4451.234, 'duration': 4.142}, {'end': 4456.757, 'text': "It's called softmax.", 'start': 4455.856, 'duration': 0.901}], 'summary': 'Deep learning uses activation functions like softmax for nonlinearity.', 'duration': 25.012, 'max_score': 4431.745, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4431745.jpg'}, {'end': 4593.009, 'src': 'embed', 'start': 4557.284, 'weight': 2, 'content': [{'end': 4567.349, 'text': 'but of all the math You just need to be super familiar with to do deep learning, The one you really need is logarithms and X right.', 'start': 4557.284, 'duration': 10.065}, {'end': 4569.551, 'text': 'all of deep learning and all of machine learning.', 'start': 4567.349, 'duration': 2.202}, {'end': 4572.273, 'text': 'they appear all the time right.', 'start': 4569.551, 'duration': 2.722}, {'end': 4593.009, 'text': 'so, For example, You absolutely need to know that log of X times y equals log of X plus log of y.', 'start': 4572.273, 'duration': 20.736}], 'summary': 'Key math for deep learning: logarithms are crucial for all ml tasks.', 'duration': 35.725, 'max_score': 4557.284, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4557284.jpg'}, {'end': 4840.998, 'src': 'embed', 'start': 4809.779, 'weight': 4, 'content': [{'end': 4818.024, 'text': "It's going to return a probability that adds up to one, and it's going to tend to want to pick one thing particularly strongly.", 'start': 4809.779, 'duration': 8.245}, {'end': 4820.926, 'text': "So that's softmax.", 'start': 4820.146, 'duration': 0.78}, {'end': 4835.493, 'text': "How would we do something that has, let's say you have an image and you want to categorize it as like cat and a dog, or like has multiple things.", 'start': 4827.887, 'duration': 7.606}, {'end': 4840.998, 'text': "What kind of function would we try to use? So happens we're going to do that right now.", 'start': 4837.135, 'duration': 3.863}], 'summary': 'Softmax returns probability adding up to 1, tends to pick one thing strongly.', 'duration': 31.219, 'max_score': 4809.779, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4809779.jpg'}], 'start': 4096.781, 'title': 'Neural network training and prediction', 'summary': 'Covers the training process of neural networks using stochastic gradient descent, transforming fully connected layer outputs into probabilities for multi-class classification, involving the calculation of ten numbers and the conversion into a probability distribution. it also explains activation functions in deep learning, emphasizing the use of relu and softmax, and their impact on model performance and multi-label classification, highlighting the importance of logarithms and their application in deep learning.', 'chapters': [{'end': 4315.777, 'start': 4096.781, 'title': 'Neural network training and prediction', 'summary': 'Discusses the training process of neural networks using stochastic gradient descent and the transformation of fully connected layer outputs into probabilities for multi-class classification, involving the calculation of ten numbers and the conversion into a probability distribution.', 'duration': 218.996, 'highlights': ['The transformation of fully connected layer outputs into probabilities for multi-class classification involves calculating ten numbers and converting them into a probability distribution. The fully connected layer calculates ten numbers for multi-class classification, ensuring that the probabilities are between zero and one and add up to one.', 'The training process of neural networks involves starting with random numbers and using stochastic gradient descent to iteratively improve them, making them less random. The training process begins with random numbers and utilizes stochastic gradient descent to iteratively improve them, gradually making them less random.']}, {'end': 4920.039, 'start': 4315.777, 'title': 'Activation functions and softmax in deep learning', 'summary': 'Explains the concept of activation functions in deep learning, emphasizing the use of relu and softmax, and their impact on model performance and multi-label classification, highlighting the importance of logarithms and their application in deep learning.', 'duration': 604.262, 'highlights': ['The chapter explains the concept of activation functions in deep learning, emphasizing the use of ReLU and softmax. It discusses the purpose of activation functions as non-linearities, and the role of ReLU and softmax as activation functions, with softmax being used in the final layer to produce probabilities between 0 and 1 that add up to 1.', 'Emphasizes the importance of logarithms in deep learning and their application in the context of activation functions and softmax. It highlights the significance of being familiar with logarithms in deep learning, particularly the properties such as turning multiplications into additions and the use of exponential functions to ensure positive values and accentuate differences.', 'Discusses the impact of softmax on model outputs, particularly in the context of multi-label classification and its properties of returning a probability that adds up to one and tending to pick one thing particularly strongly. It explains how softmax is used for multi-label classification, such as in the satellite imaging competition, to classify images based on multiple weather types and features, and how softmax tends to produce outputs with one dominant value and several smaller values, aligning with the desired outcome for classification tasks.']}], 'duration': 823.258, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4096781.jpg', 'highlights': ['The training process of neural networks involves starting with random numbers and using stochastic gradient descent to iteratively improve them, making them less random.', 'The transformation of fully connected layer outputs into probabilities for multi-class classification involves calculating ten numbers and converting them into a probability distribution.', 'Emphasizes the importance of logarithms in deep learning and their application in the context of activation functions and softmax.', 'The chapter explains the concept of activation functions in deep learning, emphasizing the use of ReLU and softmax.', 'Discusses the impact of softmax on model outputs, particularly in the context of multi-label classification and its properties of returning a probability that adds up to one and tending to pick one thing particularly strongly.']}, {'end': 7136.09, 'segs': [{'end': 5004.799, 'src': 'embed', 'start': 4976.158, 'weight': 0, 'content': [{'end': 4984.083, 'text': 'so for multi-label classification, where each sample can belong to one or more classes and We have to change a few things.', 'start': 4976.158, 'duration': 7.925}, {'end': 4985.884, 'text': "But here's the good news.", 'start': 4985.044, 'duration': 0.84}, {'end': 4988.967, 'text': "In FastAI, we don't have to change anything.", 'start': 4986.445, 'duration': 2.522}, {'end': 5004.799, 'text': 'FastAI will look at the labels in the CSV, and if there is more than one label ever for any item, it will automatically switch into multi-label mode.', 'start': 4990.708, 'duration': 14.091}], 'summary': 'Fastai makes multi-label classification easy by automatically switching into multi-label mode based on the labels in the csv.', 'duration': 28.641, 'max_score': 4976.158, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4976158.jpg'}, {'end': 5172.397, 'src': 'heatmap', 'start': 5080.555, 'weight': 0.775, 'content': [{'end': 5088.821, 'text': "you'll realize that that's a complete enumeration of everything that you can do in terms of symmetries to a square.", 'start': 5080.555, 'duration': 8.266}, {'end': 5092.864, 'text': "So it's called the dihedral group of 8.", 'start': 5089.601, 'duration': 3.263}, {'end': 5098.027, 'text': "So if you see in the code, there's actually a transform called dihedral, that's why it's called that.", 'start': 5092.864, 'duration': 5.163}, {'end': 5110.899, 'text': 'So this transforms will basically do the full set of 8 symmetric dihedral rotations and flips, plus everything which we can do to dogs and cats.', 'start': 5099.548, 'duration': 11.351}, {'end': 5117.681, 'text': 'Small 10 degree rotations, a little bit of zooming, a little bit of contrast and brightness adjustment.', 'start': 5111.879, 'duration': 5.802}, {'end': 5129.725, 'text': "So these images are a size 256x256, so I just created a little function here to let me quickly grab a data loader of any size, so here's a 256x256.", 'start': 5118.741, 'duration': 10.984}, {'end': 5141.138, 'text': "Once you've got a data object, inside it we've already seen that there's things called val ds, test ds, train ds.", 'start': 5132.33, 'duration': 8.808}, {'end': 5144.921, 'text': "They're things that you can just index into and grab a particular image.", 'start': 5141.198, 'duration': 3.723}, {'end': 5146.702, 'text': 'So you can just use square brackets, zero.', 'start': 5144.941, 'duration': 1.761}, {'end': 5149.945, 'text': "You'll also see that all of those things have a DL.", 'start': 5147.643, 'duration': 2.302}, {'end': 5151.126, 'text': "That's a data loader.", 'start': 5150.185, 'duration': 0.941}, {'end': 5153.928, 'text': 'So DS is data set DL is data loader.', 'start': 5151.226, 'duration': 2.702}, {'end': 5156.37, 'text': 'These are concepts from PyTorch.', 'start': 5154.289, 'duration': 2.081}, {'end': 5161.975, 'text': 'So if you google PyTorch data set or PyTorch data loader, you can basically see what it means.', 'start': 5156.37, 'duration': 5.605}, {'end': 5172.397, 'text': 'But the basic idea is a data set gives you a single image or a single object, that a data loader gives you back a mini-batch and, And specifically,', 'start': 5161.975, 'duration': 10.422}], 'summary': 'Discussion on dihedral group of 8, image transformations, and pytorch concepts.', 'duration': 91.842, 'max_score': 5080.555, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc5080555.jpg'}, {'end': 5341.497, 'src': 'embed', 'start': 5286.979, 'weight': 2, 'content': [{'end': 5297.523, 'text': 'So here I am using standard Python iterators and next, to grab my next mini batch From the validation sets, data loader,', 'start': 5286.979, 'duration': 10.544}, {'end': 5303.165, 'text': "and that's going to return two things It's going to return the images in the mini batch and the labels in the mini batch.", 'start': 5297.523, 'duration': 5.642}, {'end': 5306.626, 'text': 'so standard Python approach, I can pull them apart like so.', 'start': 5303.165, 'duration': 3.461}, {'end': 5312.426, 'text': 'and So here is one mini batch of labels.', 'start': 5306.626, 'duration': 5.8}, {'end': 5318.219, 'text': "And so, not surprisingly, since I said that my batch size, Let's go back and find it.", 'start': 5312.426, 'duration': 5.793}, {'end': 5326.945, 'text': "Oh, Actually it's the batch size by default is 64.", 'start': 5322.402, 'duration': 4.543}, {'end': 5329.047, 'text': "so I didn't pass in a batch size.", 'start': 5326.945, 'duration': 2.102}, {'end': 5334.111, 'text': 'So just remember shift tab to see like what are the things you can pass and what are the defaults.', 'start': 5329.047, 'duration': 5.064}, {'end': 5337.353, 'text': 'so by default? My batch size is 64.', 'start': 5334.111, 'duration': 3.242}, {'end': 5341.497, 'text': "so I've got back something of size 64 by 17.", 'start': 5337.353, 'duration': 4.144}], 'summary': 'Using standard python iterators and next to grab mini batches of 64 images and labels from the validation sets.', 'duration': 54.518, 'max_score': 5286.979, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc5286979.jpg'}, {'end': 5990.502, 'src': 'heatmap', 'start': 5814.277, 'weight': 1, 'content': [{'end': 5824.1, 'text': "so so in other words, you know Starting out by training very small images Works pretty well when you're using stuff like satellites.", 'start': 5814.277, 'duration': 9.823}, {'end': 5835.048, 'text': 'So in this case, I started right back at 64 by 64 grab some data Built my model Found out what learning rate to use.', 'start': 5824.6, 'duration': 10.448}, {'end': 5840.072, 'text': "I'm. interestingly, it turned out to be quite high.", 'start': 5835.749, 'duration': 4.323}, {'end': 5843.895, 'text': "It seems that because, like it's so, unlike ImageNet,", 'start': 5840.072, 'duration': 3.823}, {'end': 5850.5, 'text': 'I needed to do quite a bit more fitting of just that last layer before it started to flatten out.', 'start': 5843.895, 'duration': 6.605}, {'end': 5859.724, 'text': 'Then I unfreezed it, and again this is the difference to ImageNet-like datasets.', 'start': 5851.26, 'duration': 8.464}, {'end': 5860.845, 'text': 'My learning rate.', 'start': 5859.764, 'duration': 1.081}, {'end': 5866.407, 'text': 'in the initial layer I set to divided by 9, the middle layers I set to divided by 3,,', 'start': 5860.845, 'duration': 5.562}, {'end': 5871.23, 'text': 'whereas for stuff like ImageNet I had a multiple of 10 for each of those.', 'start': 5866.407, 'duration': 4.823}, {'end': 5884.3, 'text': 'You know again, the idea being that the earlier layers Probably are not as close to what they need to be compared to the image net, like data sets.', 'start': 5872.272, 'duration': 12.028}, {'end': 5888.992, 'text': 'So again, unfreeze train for a while and And you can kind of see here.', 'start': 5884.3, 'duration': 4.692}, {'end': 5891.013, 'text': "You know there's cycle one.", 'start': 5889.253, 'duration': 1.76}, {'end': 5892.173, 'text': "There's cycle two.", 'start': 5891.133, 'duration': 1.04}, {'end': 5902.296, 'text': "There's cycle three And then I kind of increase, double the size, with my images fit for a while, unfreeze fit for a while.", 'start': 5892.233, 'duration': 10.063}, {'end': 5908.404, 'text': 'double the size of the images again, fit for a while, unfreeze fit for a while and and then add TTA.', 'start': 5902.296, 'duration': 6.108}, {'end': 5910.726, 'text': 'As I mentioned last time we looked at this.', 'start': 5908.584, 'duration': 2.142}, {'end': 5918.811, 'text': 'this process ends up getting us about 30th place in this competition, which is really cool, because a lot of very,', 'start': 5910.726, 'duration': 8.085}, {'end': 5923.314, 'text': 'very smart people just a few months ago worked very, very hard on this competition.', 'start': 5918.811, 'duration': 4.503}, {'end': 5928.517, 'text': 'A couple of things people have asked about.', 'start': 5924.755, 'duration': 3.762}, {'end': 5939.018, 'text': 'One is what does this data.resize do? A couple of different pieces here.', 'start': 5928.837, 'duration': 10.181}, {'end': 5950.288, 'text': "The first is that when we say back here what transforms do we apply, and here's our transforms, we actually pass in a size.", 'start': 5939.078, 'duration': 11.21}, {'end': 5958.276, 'text': 'So one of the things our data loader does is to resize the images on-demand every time it sees them.', 'start': 5950.969, 'duration': 7.307}, {'end': 5963.992, 'text': 'This has got nothing to do with that .resize method.', 'start': 5961.411, 'duration': 2.581}, {'end': 5967.033, 'text': 'This is the thing that happens at the end.', 'start': 5964.632, 'duration': 2.401}, {'end': 5973.616, 'text': "Whatever's passed in before our data loader spits it out, it's going to resize it to this size.", 'start': 5967.513, 'duration': 6.103}, {'end': 5979.998, 'text': 'If the initial input is like 1000x1000,', 'start': 5975.836, 'duration': 4.162}, {'end': 5990.502, 'text': 'reading that JPEG and resizing it to 64x64 turns out to actually take more time than training the ConvNet dots for each batch.', 'start': 5979.998, 'duration': 10.504}], 'summary': 'Training small images, adjusting learning rates, and using tta led to 30th place in the competition.', 'duration': 176.225, 'max_score': 5814.277, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc5814277.jpg'}, {'end': 5928.517, 'src': 'embed', 'start': 5872.272, 'weight': 4, 'content': [{'end': 5884.3, 'text': 'You know again, the idea being that the earlier layers Probably are not as close to what they need to be compared to the image net, like data sets.', 'start': 5872.272, 'duration': 12.028}, {'end': 5888.992, 'text': 'So again, unfreeze train for a while and And you can kind of see here.', 'start': 5884.3, 'duration': 4.692}, {'end': 5891.013, 'text': "You know there's cycle one.", 'start': 5889.253, 'duration': 1.76}, {'end': 5892.173, 'text': "There's cycle two.", 'start': 5891.133, 'duration': 1.04}, {'end': 5902.296, 'text': "There's cycle three And then I kind of increase, double the size, with my images fit for a while, unfreeze fit for a while.", 'start': 5892.233, 'duration': 10.063}, {'end': 5908.404, 'text': 'double the size of the images again, fit for a while, unfreeze fit for a while and and then add TTA.', 'start': 5902.296, 'duration': 6.108}, {'end': 5910.726, 'text': 'As I mentioned last time we looked at this.', 'start': 5908.584, 'duration': 2.142}, {'end': 5918.811, 'text': 'this process ends up getting us about 30th place in this competition, which is really cool, because a lot of very,', 'start': 5910.726, 'duration': 8.085}, {'end': 5923.314, 'text': 'very smart people just a few months ago worked very, very hard on this competition.', 'start': 5918.811, 'duration': 4.503}, {'end': 5928.517, 'text': 'A couple of things people have asked about.', 'start': 5924.755, 'duration': 3.762}], 'summary': 'Training and fine-tuning process led to 30th place in the competition.', 'duration': 56.245, 'max_score': 5872.272, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc5872272.jpg'}, {'end': 6199.958, 'src': 'embed', 'start': 6173.985, 'weight': 5, 'content': [{'end': 6193.933, 'text': "you'll see there's a function there called f2 and And so F2 simply calls F beta score from scikit or scipy I can't remember where it came from and does a couple of little tweaks that are particularly important.", 'start': 6173.985, 'duration': 19.948}, {'end': 6199.958, 'text': 'But the important thing is you can write any metric you like.', 'start': 6196.495, 'duration': 3.463}], 'summary': 'F2 function calls f beta score from scikit or scipy and allows writing any metric', 'duration': 25.973, 'max_score': 6173.985, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc6173985.jpg'}, {'end': 6350.912, 'src': 'embed', 'start': 6298.038, 'weight': 1, 'content': [{'end': 6306.181, 'text': 'But if we were trying to predict something that was like 0, 0, 1, 1, 0, Then softmax would be a terrible choice,', 'start': 6298.038, 'duration': 8.143}, {'end': 6310.462, 'text': "because it's very hard to come up with something where both of these are high.", 'start': 6306.181, 'duration': 4.281}, {'end': 6316.405, 'text': "in fact It's impossible because they have to add up to one, so the closest they could be would be 0.5.", 'start': 6310.462, 'duration': 5.943}, {'end': 6323.287, 'text': 'so for multi-label classification, our activation function is called sigmoid.', 'start': 6316.405, 'duration': 6.882}, {'end': 6331.784, 'text': 'okay, and again the fastai library is does this automatically for you if it notices you have a multi-label problem,', 'start': 6323.287, 'duration': 8.497}, {'end': 6337.546, 'text': 'and it does that by checking your dataset to see if anything has more than one label applied to it.', 'start': 6331.784, 'duration': 5.762}, {'end': 6350.912, 'text': 'And so sigmoid is a function which is basically the same thing, except we never add up all of these exps,', 'start': 6338.327, 'duration': 12.585}], 'summary': 'For multi-label classification, use sigmoid activation instead of softmax due to its better suitability for predicting multiple high values.', 'duration': 52.874, 'max_score': 6298.038, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc6298038.jpg'}, {'end': 6960.127, 'src': 'embed', 'start': 6936.221, 'weight': 8, 'content': [{'end': 6943.572, 'text': 'When you use the differential learning rates, those three learning rates, do they just kind of spread evenly across the layers?', 'start': 6936.221, 'duration': 7.351}, {'end': 6951.401, 'text': "Yeah, we'll talk more about this later in the course, but in the fastai library there's a concept of layer groups.", 'start': 6944.537, 'duration': 6.864}, {'end': 6960.127, 'text': "So in something like a ResNet-50, you know, there's hundreds of layers, and I figured you don't want to write down hundreds of learning rates.", 'start': 6952.082, 'duration': 8.045}], 'summary': 'Fastai library uses layer groups to assign differential learning rates in complex models like resnet-50.', 'duration': 23.906, 'max_score': 6936.221, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc6936221.jpg'}], 'start': 4920.68, 'title': 'Multi-label classification and data handling in fastai', 'summary': "Covers the importance of activation functions for multi-label classification, fastai's automatic switch to multi-label mode, python iterators and next for mini-batches, differences between planet competition and imagenet networks, achieving 30th place in the competition, use of custom f2 metric, sigmoid activation function, and training differential learning rates.", 'chapters': [{'end': 5286.558, 'start': 4920.68, 'title': 'Multi-label classification and data handling in fastai', 'summary': "Explores the importance of using suitable activation functions for multi-label classification, showcases fastai's automatic switch to multi-label mode, and discusses data handling and transformations in pytorch for image classification.", 'duration': 365.878, 'highlights': ['FastAI automatically switches to multi-label mode when there are multiple labels for any item in the CSV file. FastAI simplifies multi-label classification by automatically switching to multi-label mode based on the labels in the CSV file.', 'The importance of using suitable activation functions for multi-label classification, such as not using Softmax for predicting multiple things. Emphasizes the significance of choosing appropriate activation functions for multi-label classification and advises against using Softmax for predicting multiple things.', 'Data handling and transformations in PyTorch for image classification, including the usage of data set and data loader concepts. Discusses the concepts of data set and data loader in PyTorch for image classification and highlights the usage of data handling and transformations.']}, {'end': 5687.99, 'start': 5286.979, 'title': 'Python iterators and multi-label classification', 'summary': 'Covers using standard python iterators and next to grab mini-batches, with a default batch size of 64, for multi-label classification and the process of one hot encoded labels for single-label classification, and how to manipulate and interpret image matrices.', 'duration': 401.011, 'highlights': ['Using standard Python iterators and next to grab mini-batches The speaker uses standard Python iterators and next to grab mini-batches from the validation sets, data loader, and returns images and labels in the mini-batch.', 'Default batch size of 64 for multi-label classification The default batch size for the mini-batch is 64, and the speaker shows that there are 17 possible classes, elaborating on how to extract information from the mini-batch.', 'Process of one hot encoded labels for single-label classification The speaker explains the process of converting labels into one hot encoded labels for single-label classification, and compares the actual values to the softmax activations to calculate error.', 'Manipulating and interpreting image matrices The speaker demonstrates manipulating image matrices to improve visibility and interprets the matrices to identify features in the images, highlighting the practical application of matrix manipulation in image processing.']}, {'end': 6077.39, 'start': 5690.112, 'title': 'Planet competition and imagenet networks', 'summary': 'Discusses the differences between the planet competition and imagenet networks, emphasizing the need to resize images for specialized competitions and the impact on training and learning rates, ultimately achieving a 30th place in the competition.', 'duration': 387.278, 'highlights': ['Training small images works well for specialized competitions like satellites, requiring adjustments to learning rates and resizing strategies. Achieving 30th place in the competition.', 'Resizing images to 64x64 and retraining the entire set destroys the pre-trained weights designed for larger images like those in ImageNet (224x224 or 299x299). Differentiating the impact of resizing on pre-trained ImageNet networks.', 'The resize method is a speed-up convenience function for handling large images, creating resized images for faster processing and does not affect the original data. Explaining the purpose and functionality of the resize method.']}, {'end': 6445.058, 'start': 6079.09, 'title': 'Custom metrics and activation functions', 'summary': 'Discusses the use of custom f2 metric for evaluation, the flexibility to create custom metrics, and the use of sigmoid activation function for multi-label classification, emphasizing the automatic handling by the fastai library.', 'duration': 365.968, 'highlights': ['The chapter discusses the use of custom F2 metric for evaluation, the flexibility to create custom metrics, and the use of sigmoid activation function for multi-label classification, emphasizing the automatic handling by the fastai library. The chapter explains the use of F2 metric for evaluation, allowing flexibility to create custom metrics, and highlights the automatic handling of sigmoid activation function for multi-label classification by the fastai library.', "The F2 metric is a specific way of weighting false negatives and false positives, and it is used because the competition organizers desired to use this particular F beta metric. The F2 metric is used for its specific way of weighting false negatives and false positives, based on the competition organizers' preference.", 'The transcript also emphasizes the flexibility to create custom metrics, allowing the creation of any metric as long as it takes in a set of predictions and targets, returning a number. The transcript highlights the flexibility to create custom metrics, enabling the creation of any metric that takes in predictions and targets and returns a number.', 'For multi-label classification, the fastai library automatically handles the use of sigmoid activation function, which allows for the modeling of probabilities with the property of handling multiple high values and asymptotes at the top to 1 and at the bottom to 0. The fastai library automatically handles the use of sigmoid activation function for multi-label classification, allowing the modeling of probabilities and asymptoting at the top to 1 and at the bottom to 0.']}, {'end': 7136.09, 'start': 6445.098, 'title': 'Training differential learning rates', 'summary': 'Discusses the concept of training differential learning rates, starting with training the latest playwright and then unfreezing subsets of layers with different learning rates, emphasizing the importance of training fully connected layers before unfreezing convolutional layers for better performance, and the use of layer groups in the fast.ai library to set learning rates.', 'duration': 690.992, 'highlights': ['The chapter discusses the concept of training differential learning rates The discussion revolves around training the latest playwright and then unfreezing subsets of layers with different learning rates.', 'Emphasizing the importance of training fully connected layers before unfreezing convolutional layers for better performance The fully connected layers are trained first to improve performance before unfreezing convolutional layers to avoid affecting the early layer weights when the later ones are still random.', 'Use of layer groups in the Fast.ai library to set learning rates The Fast.ai library uses the concept of layer groups to split learning rates, where the last one refers to the fully connected layers and the others are split generally about halfway through.']}], 'duration': 2215.41, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc4920680.jpg', 'highlights': ['FastAI automatically switches to multi-label mode based on labels in CSV file', 'Importance of suitable activation functions for multi-label classification', 'Using standard Python iterators and next to grab mini-batches', 'Default batch size of 64 for multi-label classification', 'Achieving 30th place in the competition for specialized satellite images', 'Use of custom F2 metric for evaluation and flexibility to create custom metrics', 'Fastai library automatically handles sigmoid activation function for multi-label classification', 'Training fully connected layers before unfreezing convolutional layers for better performance', 'Fast.ai library uses layer groups to set learning rates']}, {'end': 8192.138, 'segs': [{'end': 7381.853, 'src': 'heatmap', 'start': 7291.832, 'weight': 0.754, 'content': [{'end': 7298.418, 'text': "Funnily enough, academics in the deep learning world don't really give a shit about structured data.", 'start': 7291.832, 'duration': 6.586}, {'end': 7302.08, 'text': "Because it's pretty hard to get published in fancy conference.", 'start': 7299.339, 'duration': 2.741}, {'end': 7309.024, 'text': "proceed proceedings If you're like, if you've got a better logistics model, you know it's the thing that makes the world goes round.", 'start': 7302.08, 'duration': 6.944}, {'end': 7315.767, 'text': "It's a thing that makes everybody you know money and efficiency and make stuff work.", 'start': 7309.024, 'duration': 6.743}, {'end': 7325.163, 'text': "But it's largely ignored, sadly, and So we're not going to ignore it because we're practical deep learning, and Kaggle doesn't ignore it either,", 'start': 7315.767, 'duration': 9.396}, {'end': 7328.964, 'text': 'because people put prize money up on Kaggle to solve real world problems.', 'start': 7325.163, 'duration': 3.801}, {'end': 7332.085, 'text': 'So there are some great Kaggle competitions we can look at.', 'start': 7329.384, 'duration': 2.701}, {'end': 7333.466, 'text': "There's one running right now.", 'start': 7332.105, 'duration': 1.361}, {'end': 7339.201, 'text': "Which is the grocery sales forecasting competition for Ecuador's largest chain?", 'start': 7334.658, 'duration': 4.543}, {'end': 7344.163, 'text': "It's always a little.", 'start': 7342.342, 'duration': 1.821}, {'end': 7350.807, 'text': "I've got to be a little careful about how much I show you about currently running competitions, because I don't want to, You know, help you cheat.", 'start': 7344.163, 'duration': 6.644}, {'end': 7352.307, 'text': 'but it so happens.', 'start': 7350.807, 'duration': 1.5}, {'end': 7359.311, 'text': "there was a competition a year or two ago For one of Germany's largest grocery chains, which is almost identical.", 'start': 7352.307, 'duration': 7.004}, {'end': 7361.032, 'text': "so I'm going to show you how to do that, okay?", 'start': 7359.311, 'duration': 1.721}, {'end': 7375.331, 'text': "So that was called the Rossmann stores data, And so I would suggest you know, first of all, try practicing what we're learning on Rossmann right,", 'start': 7364.707, 'duration': 10.624}, {'end': 7381.853, 'text': 'but then see if you can get it working on grocery, because currently On the leaderboard,', 'start': 7375.331, 'duration': 6.522}], 'summary': "Deep learning overlooks structured data, but kaggle focuses on real-world problems with competitions, like the grocery sales forecasting for ecuador's largest chain.", 'duration': 90.021, 'max_score': 7291.832, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc7291832.jpg'}, {'end': 7339.201, 'src': 'embed', 'start': 7315.767, 'weight': 4, 'content': [{'end': 7325.163, 'text': "But it's largely ignored, sadly, and So we're not going to ignore it because we're practical deep learning, and Kaggle doesn't ignore it either,", 'start': 7315.767, 'duration': 9.396}, {'end': 7328.964, 'text': 'because people put prize money up on Kaggle to solve real world problems.', 'start': 7325.163, 'duration': 3.801}, {'end': 7332.085, 'text': 'So there are some great Kaggle competitions we can look at.', 'start': 7329.384, 'duration': 2.701}, {'end': 7333.466, 'text': "There's one running right now.", 'start': 7332.105, 'duration': 1.361}, {'end': 7339.201, 'text': "Which is the grocery sales forecasting competition for Ecuador's largest chain?", 'start': 7334.658, 'duration': 4.543}], 'summary': "Kaggle offers practical deep learning challenges, including a grocery sales forecasting competition for ecuador's largest chain.", 'duration': 23.434, 'max_score': 7315.767, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc7315767.jpg'}, {'end': 7526.309, 'src': 'embed', 'start': 7465.5, 'weight': 1, 'content': [{'end': 7473.905, 'text': 'I mention that because you can use that particular library without any of the other parts of fastai, so that can be handy.', 'start': 7465.5, 'duration': 8.405}, {'end': 7477.568, 'text': "And then we're also going to use fastai.columndata,", 'start': 7474.846, 'duration': 2.722}, {'end': 7487.861, 'text': 'Which is basically some stuff that allows us to do fast AI PyTorch stuff with columnar structured data.', 'start': 7478.137, 'duration': 9.724}, {'end': 7492.283, 'text': 'For structured data, we need to use pandas a lot.', 'start': 7487.861, 'duration': 4.422}, {'end': 7495.864, 'text': "Anybody who's used our data frames will be very familiar with pandas.", 'start': 7492.283, 'duration': 3.581}, {'end': 7509.174, 'text': "pandas is basically an attempt to kind of replicate data friends in Python, You know, and a bit more if you're Not entirely familiar with pandas.", 'start': 7495.864, 'duration': 13.31}, {'end': 7523.626, 'text': "There's a great book, which I think I might have mentioned before, Python for Data Analysis by Wes McKinney.", 'start': 7510.675, 'duration': 12.951}, {'end': 7526.309, 'text': "There's a new edition that just came out a couple of weeks ago.", 'start': 7523.947, 'duration': 2.362}], 'summary': 'Using fastai and pandas for structured data analysis and modeling.', 'duration': 60.809, 'max_score': 7465.5, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc7465500.jpg'}, {'end': 7581.72, 'src': 'embed', 'start': 7555.006, 'weight': 3, 'content': [{'end': 7559.629, 'text': "and there's a video of that online where we kind of have a brief mention of all of those tools.", 'start': 7555.006, 'duration': 4.623}, {'end': 7566.292, 'text': 'Structured data is generally shared as CSV files.', 'start': 7563.191, 'duration': 3.101}, {'end': 7567.893, 'text': "There's no difference in this competition.", 'start': 7566.332, 'duration': 1.561}, {'end': 7571.615, 'text': "As you'll see, there's a hyperlink to the Rossmann dataset here.", 'start': 7568.674, 'duration': 2.941}, {'end': 7577.658, 'text': "If you look at the bottom of my screen, you'll see this goes to files.fast.ai,", 'start': 7573.816, 'duration': 3.842}, {'end': 7581.72, 'text': "Because this doesn't require any login or anything to grab this data set.", 'start': 7577.658, 'duration': 4.062}], 'summary': 'Structured data shared as csv files, access via files.fast.ai', 'duration': 26.714, 'max_score': 7555.006, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc7555006.jpg'}, {'end': 7953.858, 'src': 'embed', 'start': 7928.866, 'weight': 0, 'content': [{'end': 7935.37, 'text': "It's kind of pretty much takes it as it sits in RAM and dumps it to the disk, and so it's like really, really, really fast.", 'start': 7928.866, 'duration': 6.504}, {'end': 7945.234, 'text': "and the reason that you need to know this is because the Ecuadorian grocery competition that's on now has 350 million records.", 'start': 7935.37, 'duration': 9.864}, {'end': 7947.935, 'text': 'So you will care about how long things take.', 'start': 7945.234, 'duration': 2.701}, {'end': 7953.858, 'text': 'it took, I believe, about six seconds for me to save 350 million records to feather format.', 'start': 7947.935, 'duration': 5.923}], 'summary': 'Feather format takes 6 seconds to save 350m records, ensuring speed is crucial for ecuadorian grocery competition.', 'duration': 24.992, 'max_score': 7928.866, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc7928866.jpg'}], 'start': 7137.541, 'title': 'Structured data in machine learning', 'summary': 'Discusses the importance of structured data in practical deep learning and its relevance in real-world problem-solving. it also highlights the use of specific libraries and tools such as fastai.structured and pandas, providing a comprehensive overview of working with columnar structured data.', 'chapters': [{'end': 7184.932, 'start': 7137.541, 'title': 'Learning rate finder on small data set', 'summary': 'Discusses the challenges of using a learning rate finder on a small dataset, recommending a smaller batch size to address the issue.', 'duration': 47.391, 'highlights': ['The learning rate finder struggles with small datasets due to insufficient mini-batches, necessitating a smaller batch size like four or eight to address the issue.', 'The speaker collected a small dataset from Google images and encountered difficulties with the learning rate finder, leading to an empty plot and incomprehensible numbers.']}, {'end': 7555.006, 'start': 7184.952, 'title': 'Structured data in machine learning', 'summary': 'Discusses structured data and its importance in practical deep learning, highlighting the relevance of structured data in real-world problem-solving and the use of specific libraries and tools such as fastai.structured and pandas for working with columnar structured data.', 'duration': 370.054, 'highlights': ['The importance of structured data in real-world problem-solving and practical deep learning is emphasized, with a mention of how academics in the deep learning world tend to overlook structured data. Importance of structured data, overlook by academics', "The relevance of structured data in Kaggle competitions is highlighted, with a specific reference to the grocery sales forecasting competition for Ecuador's largest chain and the potential for improvement in the competition's leaderboard through the application of machine learning techniques. Kaggle competitions, grocery sales forecasting competition, potential for improvement", 'The recommendation to practice using the learnings on the Rossmann stores data and the emphasis on the use of specific libraries such as fastai.structured and fastai.columndata for working with columnar structured data are mentioned. Practice with Rossmann stores data, use of specific libraries for structured data', "The usage of pandas for working with structured data is emphasized, along with a reference to the book 'Python for Data Analysis' by Wes McKinney for further learning. Usage of pandas, reference to 'Python for Data Analysis' book"]}, {'end': 8192.138, 'start': 7555.006, 'title': 'Rossmann data pre-processing', 'summary': 'Discusses data pre-processing for the rossmann dataset, covering structured data shared as csv files, pre-processing techniques borrowed from the third place winner of the competition, and the use of pandas and fast ai library for data processing and feature engineering.', 'duration': 637.132, 'highlights': ['The chapter discusses data pre-processing techniques borrowed from the third place winner of the competition, simplifying and porting their entire pipeline for use in the Rossmann dataset. Use of pre-processing techniques from the third place winner of the competition', 'The use of pandas and fast AI library for data processing and feature engineering, including the addition of date parts and the creation of a relational dataset by joining various tables. Utilization of pandas and fast AI library for data processing and feature engineering', 'The adoption of the feather format for saving the pre-processed dataset, resulting in significantly faster processing time, such as saving 350 million records in approximately six seconds. Utilization of the feather format for saving pre-processed dataset, achieving fast processing time']}], 'duration': 1054.597, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/9C06ZPF8Uuc/pics/9C06ZPF8Uuc7137541.jpg', 'highlights': ['The adoption of the feather format for saving the pre-processed dataset, resulting in significantly faster processing time, such as saving 350 million records in approximately six seconds.', "The usage of pandas for working with structured data is emphasized, along with a reference to the book 'Python for Data Analysis' by Wes McKinney for further learning.", 'The recommendation to practice using the learnings on the Rossmann stores data and the emphasis on the use of specific libraries such as fastai.structured and fastai.columndata for working with columnar structured data are mentioned.', 'The chapter discusses data pre-processing techniques borrowed from the third place winner of the competition, simplifying and porting their entire pipeline for use in the Rossmann dataset.', "The relevance of structured data in Kaggle competitions is highlighted, with a specific reference to the grocery sales forecasting competition for Ecuador's largest chain and the potential for improvement in the competition's leaderboard through the application of machine learning techniques."]}], 'highlights': ['Achieving 99.45% accuracy using BNFreeze on deeper models like ResNet-50 or ResNext-101', 'The performance on the validation set showed a significant improvement, with an accuracy of 99.5% after the recompilation, compared to about 97% on the training set, indicating the effectiveness of the changes made (quantifiable data: accuracy percentages)', 'The recompiled model achieved faster execution, taking only four to five minutes compared to the original eight minutes, showcasing a notable reduction in processing time (quantifiable data: time comparison)', "Pavel's post on learning rate finder is popular, shared hundreds of times and viewed thousands of times", 'The adoption of the feather format for saving the pre-processed dataset, resulting in significantly faster processing time, such as saving 350 million records in approximately six seconds', 'The chapter outlines the plan to delve into the underlying theory of Convolutional Neural Networks (CNNs) and explores various applications of neural nets for structured data, language processing, and recommendation systems', 'The mention of ongoing efforts to create a Keras-compatible version of fastai, along with the interest from Google to port fastai to TensorFlow, indicates the potential for future developments in making fastai accessible in TensorFlow, highlighting the possibility of a future Keras-compatible or TensorFlow-compatible version of fastai', 'The process involves implementing an entire set of layers of a convolutional neural network in Microsoft Excel to recognize letters from an image', 'The chapter discusses data pre-processing techniques borrowed from the third place winner of the competition, simplifying and porting their entire pipeline for use in the Rossmann dataset', 'The recommendation to practice using the learnings on the Rossmann stores data and the emphasis on the use of specific libraries such as fastai.structured and fastai.columndata for working with columnar structured data are mentioned']}