title
Lesson 9: Deep Learning Foundations to Stable Diffusion, 2022

description
(All lesson resources are available at http://course.fast.ai.) This is the first lesson of part 2 of Practical Deep Learning for Coders. It starts with a tutorial on how to use pipelines in the Diffusers library to generate images. Diffusers is (in our opinion!) the best library available at the moment for image generation. It has many features and is very flexible. We explain how to use its many features, and discuss options for accessing the GPU resources needed to use the library. We talk about some of the nifty tweaks available when using Stable Diffusion in Diffusers, and show how to use them: guidance scale (for varying the amount the prompt is used), negative prompts (for removing concepts from an image), image initialisation (for starting with an existing image), textual inversion (for adding your own concepts to generated images), Dreambooth (an alternative approach to textual inversion). The second half of the lesson covers the key concepts involved in Stable Diffusion: - CLIP embeddings - The VAE (variational autoencoder) - Predicting noise with the unet - Removing noise with schedulers. You can discuss this lesson, and access links to all notebooks and resources from it, at this forum topic: https://forums.fast.ai/t/lesson-9-part-2-preview/101336 0:00 - Introduction 6:38 - This course vs DALL-E 2 10:38 - How to take full advantage of this course 12:14 - Cloud computing options 14:58 - Getting started (Github, notebooks to play with, resources) 20:48 - Diffusion notebook from Hugging Face 26:59 - How stable diffusion works 30:06 - Diffusion notebook (guidance scale, negative prompts, init image, textual inversion, Dreambooth) 45:00 - Stable diffusion explained 53:04 - Math notation correction 1:14:37 - Creating a neural network to predict noise in an image 1:27:46 - Working with images and compressing the data with autoencoders 1:40:12 - Explaining latents that will be input into the unet 1:43:54 - Adding text as one hot encoded input to the noise and drawing (aka guidance) 1:47:06 - How to represent numbers vs text embeddings in our model with CLIP encoders 1:53:13 - CLIP encoder loss function 2:00:55 - Caveat regarding "time steps" 2:07:04 Why don’t we do this all in one step? Thanks to fmussari for the transcript, and to Raymond-Wu (on forums.fast.ai) for the timestamps.

detail
{'title': 'Lesson 9: Deep Learning Foundations to Stable Diffusion, 2022', 'heatmap': [{'end': 8111.694, 'start': 8035.588, 'weight': 1}], 'summary': "Covers deep learning foundations: part 2, stable diffusion advancements reducing steps from a thousand to just four, making it 256 times faster, and emphasizing the importance of prior knowledge. it also introduces 'strumer' and 'dreambooth' by alon, ai art prompts, image grid creation, handwritten digit generation, efficient image and data compression, and the concept of time steps, noising schedule, and optimization in diffusion models.", 'chapters': [{'end': 380.167, 'segs': [{'end': 52.254, 'src': 'embed', 'start': 0.93, 'weight': 1, 'content': [{'end': 6.632, 'text': 'Hi everybody and welcome to Deep Learning Foundations to Stable Diffusion.', 'start': 0.93, 'duration': 5.702}, {'end': 12.413, 'text': "Hopefully it's not too confusing that this is described here as Lesson 9.", 'start': 7.432, 'duration': 4.981}, {'end': 18.535, 'text': "That's because strictly speaking we treat this as Part 2 of the Practical Deep Learning for Coders series.", 'start': 12.413, 'duration': 6.122}, {'end': 26.338, 'text': 'So that part one had eight lessons, so this is lesson nine.', 'start': 21.656, 'duration': 4.682}, {'end': 28.739, 'text': "But don't worry, you didn't miss anything.", 'start': 26.658, 'duration': 2.081}, {'end': 36.923, 'text': "It's the first lesson of part two, which is called Deep Learning Foundations to Stable Diffusion.", 'start': 28.979, 'duration': 7.944}, {'end': 43.606, 'text': 'And maybe, rather than calling it Practical Deep Learning for Coders, we should call this Impractical Deep Learning for Coders,', 'start': 37.623, 'duration': 5.983}, {'end': 52.254, 'text': 'in the sense that we are certainly not going to be spending all of our time seeing exactly how to do important things with deep learning,', 'start': 43.606, 'duration': 8.648}], 'summary': 'Lesson 9 is part two of the deep learning foundations to stable diffusion series, following eight lessons in part one.', 'duration': 51.324, 'max_score': 0.93, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls930.jpg'}, {'end': 108.495, 'src': 'embed', 'start': 75.919, 'weight': 4, 'content': [{'end': 80.502, 'text': "stuff like that, then it is going to be very helpful to learn the details we'll be talking about.", 'start': 75.919, 'duration': 4.583}, {'end': 86.947, 'text': "So here in Lesson 9, there's kind of going to be two parts to it.", 'start': 82.364, 'duration': 4.583}, {'end': 93.972, 'text': 'One is just a quick run-through, quick-ish run-through of using stable diffusion.', 'start': 87.627, 'duration': 6.345}, {'end': 99.672, 'text': "um, because we're all dying to play with it right.", 'start': 95.65, 'duration': 4.022}, {'end': 108.495, 'text': "and then, um, the other thing that i'll be doing is describing in some detail what's going on.", 'start': 99.672, 'duration': 8.823}], 'summary': 'Lesson 9 covers stable diffusion with a quick run-through and detailed description.', 'duration': 32.576, 'max_score': 75.919, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls75919.jpg'}, {'end': 238.801, 'src': 'embed', 'start': 211.335, 'weight': 6, 'content': [{'end': 216.62, 'text': 'Generally speaking, you know, we expect people to be spending about 10 hours of work.', 'start': 211.335, 'duration': 5.285}, {'end': 218.147, 'text': 'on each video.', 'start': 217.446, 'duration': 0.701}, {'end': 221.651, 'text': 'Um, having said that some people spend a hell of a lot more and go very deep.', 'start': 218.828, 'duration': 2.823}, {'end': 229.739, 'text': 'Some people will spend a whole year, you know, sabbatical studying practical deep learning for coders in order to really fully understand everything.', 'start': 221.711, 'duration': 8.028}, {'end': 232.322, 'text': "So really it's up to you as to how deep you go.", 'start': 230.56, 'duration': 1.762}, {'end': 238.801, 'text': "Okay Um, so with that said, um, Let's jump into it.", 'start': 233.123, 'duration': 5.678}], 'summary': 'Expect 10 hours of work on each video, but some spend much more; some spend a full year on deep learning.', 'duration': 27.466, 'max_score': 211.335, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls211335.jpg'}, {'end': 337.628, 'src': 'embed', 'start': 309.463, 'weight': 0, 'content': [{'end': 318.426, 'text': "But then as of, yeah, Last night, papers just come out that saying it's now down to four, and it's 256 times faster.", 'start': 309.463, 'duration': 8.963}, {'end': 331.846, 'text': "And another paper has come out with a separate, I think orthogonal approach, which makes it another Let's see, 10 to 20 times faster.", 'start': 319.047, 'duration': 12.799}, {'end': 334.987, 'text': 'So things are very exciting.', 'start': 332.486, 'duration': 2.501}, {'end': 337.628, 'text': 'Things are moving very quickly.', 'start': 336.368, 'duration': 1.26}], 'summary': 'New research has made the process 256 times faster and another approach makes it 10 to 20 times faster.', 'duration': 28.165, 'max_score': 309.463, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls309463.jpg'}, {'end': 389.015, 'src': 'embed', 'start': 362.47, 'weight': 3, 'content': [{'end': 371.3, 'text': "And once you know the foundations, these kinds of details that you'll find in these papers, you'll be like, oh, I see.", 'start': 362.47, 'duration': 8.83}, {'end': 374.162, 'text': 'They did all these things the same way as usual and they made this little change.', 'start': 371.36, 'duration': 2.802}, {'end': 380.167, 'text': "So that's why we do things from the foundations, so that you can keep up with the research.", 'start': 375.544, 'duration': 4.623}, {'end': 389.015, 'text': 'do your own research by taking advantage of this foundational knowledge which all these papers are building on top of.', 'start': 380.167, 'duration': 8.848}], 'summary': 'Understanding foundational knowledge in research papers helps in conducting own research effectively.', 'duration': 26.545, 'max_score': 362.47, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls362470.jpg'}], 'start': 0.93, 'title': 'Deep learning foundations: part 2 introduction', 'summary': 'Introduces part 2 of the practical deep learning for coders series, focusing on generative model-y fun things and deep learning understanding, not necessarily essential for practical applications. it discusses stable diffusion advancements, reducing steps from a thousand to just four, making it 256 times faster, and the foundational aspects that remain constant over time. emphasizes the importance of prior knowledge and suggests an expected 10 hours of work per video.', 'chapters': [{'end': 75.919, 'start': 0.93, 'title': 'Deep learning foundations: part 2 introduction', 'summary': 'Introduces part 2 of the practical deep learning for coders series, focusing on generative model-y fun things and deep learning understanding, not necessarily essential for practical applications.', 'duration': 74.989, 'highlights': ['The chapter introduces Part 2 of the Practical Deep Learning for Coders series, focusing on generative model-y fun things and deep learning understanding, not necessarily essential for practical applications.', 'The lesson is structured as the first of part two, called Deep Learning Foundations to Stable Diffusion, which follows the initial eight lessons of part one.', 'The content is described as Impractical Deep Learning for Coders, emphasizing a focus on research and complex customization requirements over practical application.']}, {'end': 232.322, 'start': 75.919, 'title': 'Understanding practical deep learning', 'summary': 'Discusses the two parts of lesson 9, including a quick run-through of using stable diffusion and a detailed description of the working process, aiming to provide a reasonable intuitive understanding of the concepts. it emphasizes the importance of prior knowledge in deep learning and suggests an expected 10 hours of work per video, with some individuals spending significantly more time.', 'duration': 156.403, 'highlights': ['The chapter discusses the two parts of Lesson 9, including a quick run-through of using stable diffusion and a detailed description of the working process, aiming to provide a reasonable intuitive understanding of the concepts.', 'It emphasizes the importance of prior knowledge in deep learning and suggests an expected 10 hours of work per video, with some individuals spending significantly more time.']}, {'end': 380.167, 'start': 233.123, 'title': 'Stable diffusion in ai', 'summary': 'Discusses the rapid advancements in stable diffusion, with recent developments reducing the number of steps required from a thousand to just four, making it 256 times faster, as well as the foundational aspects that remain constant over time.', 'duration': 147.044, 'highlights': ['Recent developments have reduced the number of steps required for stable diffusion from a thousand to just four, making it 256 times faster.', 'The foundational aspects of stable diffusion remain constant over time, allowing individuals to keep up with the research despite rapid advancements.', 'The chapter emphasizes the importance of learning from the foundations to understand the changes and advancements in stable diffusion.']}], 'duration': 379.237, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls930.jpg', 'highlights': ['Recent developments have reduced the number of steps required for stable diffusion from a thousand to just four, making it 256 times faster.', 'The lesson is structured as the first of part two, called Deep Learning Foundations to Stable Diffusion, which follows the initial eight lessons of part one.', 'The chapter introduces Part 2 of the Practical Deep Learning for Coders series, focusing on generative model-y fun things and deep learning understanding, not necessarily essential for practical applications.', 'The foundational aspects of stable diffusion remain constant over time, allowing individuals to keep up with the research despite rapid advancements.', 'The chapter discusses the two parts of Lesson 9, including a quick run-through of using stable diffusion and a detailed description of the working process, aiming to provide a reasonable intuitive understanding of the concepts.', 'The content is described as Impractical Deep Learning for Coders, emphasizing a focus on research and complex customization requirements over practical application.', 'It emphasizes the importance of prior knowledge in deep learning and suggests an expected 10 hours of work per video, with some individuals spending significantly more time.']}, {'end': 1079.084, 'segs': [{'end': 464.991, 'src': 'embed', 'start': 412.902, 'weight': 1, 'content': [{'end': 419.925, 'text': "Um, so, you know, the cool thing is that we're now at a point we can build this stuff ourself.", 'start': 412.902, 'duration': 7.023}, {'end': 422.728, 'text': 'and run this stuff ourselves.', 'start': 421.727, 'duration': 1.001}, {'end': 424.689, 'text': "We won't actually be using this particular model.", 'start': 422.768, 'duration': 1.921}, {'end': 430.992, 'text': 'DALI 2 will be using a different model, stable diffusion, but it has very similar kind of outputs.', 'start': 424.729, 'duration': 6.263}, {'end': 433.033, 'text': 'But we can go even further now.', 'start': 431.753, 'duration': 1.28}, {'end': 443.719, 'text': "So one of our wonderful alumni, Alon, actually recently started a new company called I don't know who said it Strumer,", 'start': 435.235, 'duration': 8.484}, {'end': 453.82, 'text': "where you can use something that we'll be learning about today called Dreambooth, to put any object, person whatever into an image.", 'start': 443.719, 'duration': 10.101}, {'end': 464.991, 'text': 'And so he was kind enough to do a quick DreamBruth run for me and added these various pictures of me using his service.', 'start': 454.54, 'duration': 10.451}], 'summary': "We can now build and run our own models, with similar outputs to dali 2's model, and utilize dreambooth to add objects or people into images.", 'duration': 52.089, 'max_score': 412.902, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls412902.jpg'}, {'end': 620.672, 'src': 'embed', 'start': 592.626, 'weight': 3, 'content': [{'end': 604.057, 'text': 'And then Taneesh, everybody in the Fast.ai community probably already knows, now at Stability.ai working on stable diffusion models.', 'start': 592.626, 'duration': 11.431}, {'end': 607.601, 'text': 'His expertise particularly is in medical applications.', 'start': 604.918, 'duration': 2.683}, {'end': 620.672, 'text': 'So really folks from all the key groups pretty much around stable diffusion and stuff are working on this together.', 'start': 608.742, 'duration': 11.93}], 'summary': 'Taneesh, from fast.ai, now at stability.ai, specializes in stable diffusion models for medical applications.', 'duration': 28.046, 'max_score': 592.626, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls592626.jpg'}, {'end': 852.429, 'src': 'embed', 'start': 816.392, 'weight': 4, 'content': [{'end': 820.554, 'text': 'So check course.fast.ai to find out what our current recommendations are.', 'start': 816.392, 'duration': 4.162}, {'end': 825.918, 'text': 'Now, Lambda Labs and Jarvis Labs are also both good options.', 'start': 822.595, 'duration': 3.323}, {'end': 833.664, 'text': 'Jarvis was created by an alum of the course and has some just really fantastic options at a very reasonable price.', 'start': 825.978, 'duration': 7.686}, {'end': 839.049, 'text': 'And a lot of fast.ai students use them and love them.', 'start': 835.105, 'duration': 3.944}, {'end': 852.429, 'text': 'And also check out Lambda Labs who are the most recent provider on this page and they are rapidly adding new features.', 'start': 840.41, 'duration': 12.019}], 'summary': 'Recommended providers include lambda labs and jarvis labs for their fantastic options and reasonable prices.', 'duration': 36.037, 'max_score': 816.392, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls816392.jpg'}, {'end': 967.175, 'src': 'embed', 'start': 934.641, 'weight': 0, 'content': [{'end': 936.182, 'text': "which hopefully he'll keep up to date.", 'start': 934.641, 'duration': 1.541}, {'end': 939.443, 'text': 'So even if you come here later, this will still be up to date.', 'start': 936.262, 'duration': 3.181}, {'end': 949.445, 'text': "Because he knows so much about this area, he's been able to pull out some of the best stuff out there for just starting to play.", 'start': 940.544, 'duration': 8.901}, {'end': 959.189, 'text': "And And I think it's actually important to play because that way you can really understand what the capabilities are and what the constraints are.", 'start': 949.465, 'duration': 9.724}, {'end': 962.311, 'text': 'So then you can think about like well, what could you do with that?', 'start': 959.589, 'duration': 2.722}, {'end': 967.175, 'text': 'And also like what kind of research opportunities might there be?', 'start': 963.992, 'duration': 3.183}], 'summary': 'Access up-to-date information on playing and research opportunities in this area.', 'duration': 32.534, 'max_score': 934.641, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls934641.jpg'}], 'start': 380.167, 'title': 'Importance of foundational knowledge and fast.ai course update 2022', 'summary': "Discusses the significance of foundational knowledge and the ability to build and run models using dali 2 and stable diffusion. it also introduces 'strumer' and 'dreambooth' by alon, emphasizes input from fast.ai alumni in stable diffusion model development, and recommends compute options including colab, paper space gradient, lambda labs, and jarvis labs.", 'chapters': [{'end': 433.033, 'start': 380.167, 'title': 'Building on foundational knowledge', 'summary': 'Discusses the importance of foundational knowledge and the ability to build and run models using dali 2 and stable diffusion.', 'duration': 52.866, 'highlights': ['The importance of foundational knowledge and building on top of it for research and model development.', 'Ability to build and run models using DALI 2 and stable diffusion.', 'The capability to create similar outputs as Dali 2 illustrations of Twitter bios.']}, {'end': 1079.084, 'start': 435.235, 'title': 'Fast.ai course update 2022', 'summary': "Introduces the new company 'strumer' and its service 'dreambooth' by alon, highlights the input from fast.ai alumni in the development of stable diffusion models, and provides recommendations for compute options including colab, paper space gradient, lambda labs, and jarvis labs.", 'duration': 643.849, 'highlights': ["Alon's new company 'Strumer' and its service 'Dreambooth' is introduced, allowing users to put any object or person into an image, with examples like 'me as a dwarf' showcased.", 'The chapter emphasizes the significant input from Fast.ai alumni in the development of stable diffusion models, including contributions from Jonathan Whittaker, Waseem, Pedro, and Taneesh, with their expertise in generative models, machine learning, and medical applications.', 'Recommendations for compute options are provided, including Colab, Paper Space Gradient, Lambda Labs, and Jarvis Labs, with insights into the changing landscape of compute options and the suggestion to consider purchasing a personal machine due to decreased GPU prices.', 'The chapter directs viewers to course.fast.ai for additional materials, detailed educational material, and community resources, including forums and additional videos, to aid in understanding the content of the course.', "The importance of playing with suggested tools and resources, such as Colab notebooks and Pharma Psychotic's list, is highlighted to understand the capabilities, constraints, and research opportunities of stable diffusion models."]}], 'duration': 698.917, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls380167.jpg', 'highlights': ['The significance of foundational knowledge and its role in research and model development', 'The ability to build and run models using DALI 2 and stable diffusion', "Introduction of 'Strumer' and 'Dreambooth' by Alon, allowing users to put any object or person into an image", 'Emphasis on input from Fast.ai alumni in stable diffusion model development', 'Recommendations for compute options including Colab, Paper Space Gradient, Lambda Labs, and Jarvis Labs']}, {'end': 1805.89, 'segs': [{'end': 1128.586, 'src': 'embed', 'start': 1079.725, 'weight': 0, 'content': [{'end': 1086.612, 'text': "Um, but I just wanted to know these kinds of things are out there and they're basically like ready to go applications that you can start playing with.", 'start': 1079.725, 'duration': 6.887}, {'end': 1089.075, 'text': 'So play a lot.', 'start': 1088.154, 'duration': 0.921}, {'end': 1100.952, 'text': "Um, What you'll find is that most of them, at least at the moment, expect you to input some text to say what you want to create a picture of.", 'start': 1090.877, 'duration': 10.075}, {'end': 1108.958, 'text': "It turns out that as we'll learn, we'll learn in detail why the text you pick.", 'start': 1102.633, 'duration': 6.325}, {'end': 1114.363, 'text': "it's not very easy to know what to write, and that gives kind of interesting results.", 'start': 1108.958, 'duration': 5.405}, {'end': 1121.044, 'text': "At the moment, it's quite an artisanal thing to understand what to write.", 'start': 1116.284, 'duration': 4.76}, {'end': 1124.485, 'text': "And the best way to learn what to write, it's called the prompt.", 'start': 1121.624, 'duration': 2.861}, {'end': 1128.586, 'text': "The best way to learn about prompts is to look at other people's prompts and their outputs.", 'start': 1124.985, 'duration': 3.601}], 'summary': 'Ready-to-use applications for creating pictures from text, but finding the right prompt is challenging.', 'duration': 48.861, 'max_score': 1079.725, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls1079725.jpg'}, {'end': 1334.313, 'src': 'embed', 'start': 1299.29, 'weight': 1, 'content': [{'end': 1306.515, 'text': 'And this notebook is largely been built thanks to the wonderful folks at Hugging Face.', 'start': 1299.29, 'duration': 7.225}, {'end': 1311.338, 'text': 'And Hugging Face have a library called Diffusers.', 'start': 1307.255, 'duration': 4.083}, {'end': 1316.762, 'text': 'So any of you that have done part one of the course will be very familiar with Hugging Face.', 'start': 1312.959, 'duration': 3.803}, {'end': 1318.523, 'text': 'We used a lot of their libraries in part one.', 'start': 1316.802, 'duration': 1.721}, {'end': 1327.105, 'text': 'Diffuses is their library for doing stable diffusion, and stuff like stable diffusion.', 'start': 1320.476, 'duration': 6.629}, {'end': 1334.313, 'text': 'At the moment.', 'start': 1333.713, 'duration': 0.6}], 'summary': "Hugging face's diffusers library is used for stable diffusion.", 'duration': 35.023, 'max_score': 1299.29, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls1299290.jpg'}], 'start': 1079.725, 'title': "Ai art prompts and hugging face's diffusers", 'summary': "Covers ai art prompts, focusing on the challenge of creating effective prompts and the use of specific words to influence ai art outputs. it also discusses working with hugging face's diffusers library for stable diffusion, highlighting the process of using pipelines to generate images and the evolving nature of diffusion models.", 'chapters': [{'end': 1227.174, 'start': 1079.725, 'title': 'Understanding ai art prompts', 'summary': 'Discusses the availability of ready-to-use ai art applications, the challenge of creating effective prompts, and the use of specific words to influence ai art outputs, with lexica being a recommended platform for learning and exploring ai artworks.', 'duration': 147.449, 'highlights': ["The availability of ready-to-use AI art applications for creating AI artworks is discussed, highlighting the need to input text to generate pictures. (e.g. 'ready to go applications', 'expect you to input some text')", "The challenge of creating effective prompts for AI art is emphasized, noting the difficulty in determining what to write and the significance of learning from others' prompts. (e.g. 'not very easy to know what to write', 'the best way to learn about prompts is to look at other people's prompts and their outputs')", "The recommendation to use Lexica as a platform for learning and exploring AI artworks is provided, with a focus on understanding prompts and influencing AI outputs by adding specific words related to artists and art styles. (e.g. 'perhaps the best way to do that is Lexica', 'add a bunch of artists' names or places that they put art')"]}, {'end': 1805.89, 'start': 1227.174, 'title': "Working with hugging face's diffusers", 'summary': "Discusses working with hugging face's diffusers library for stable diffusion, highlighting the process of using pipelines to generate images and the evolving nature of diffusion models, emphasizing the importance of understanding the basic concept despite the changes in speed and steps for the process.", 'duration': 578.716, 'highlights': ["Using Hugging Face's Diffusers library for stable diffusion and pipelines", 'Evolving nature of diffusion models and changes in speed and steps for the process', 'Acknowledgment and gratitude towards Pedro Cuenca and the team at Hugging Face for building the Diffusers library']}], 'duration': 726.165, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls1079725.jpg', 'highlights': ["The challenge of creating effective prompts for AI art is emphasized, noting the difficulty in determining what to write and the significance of learning from others' prompts.", "Using Hugging Face's Diffusers library for stable diffusion and pipelines.", 'The availability of ready-to-use AI art applications for creating AI artworks is discussed, highlighting the need to input text to generate pictures.']}, {'end': 2732.285, 'segs': [{'end': 1906.72, 'src': 'embed', 'start': 1852.422, 'weight': 0, 'content': [{'end': 1866.248, 'text': 'but basically what this does is it says to what degree should we be kind of focusing on the specific caption versus just creating an image?', 'start': 1852.422, 'duration': 13.826}, {'end': 1874.496, 'text': "So we're going to try a few different guidance scales, about 1, 3, 7, 14, generally 7.5 I believe at this stage is the default.", 'start': 1866.268, 'duration': 8.228}, {'end': 1876.578, 'text': 'That might have changed by the time you watch this.', 'start': 1874.917, 'duration': 1.661}, {'end': 1880.102, 'text': 'And so each row here is a different guidance scale.', 'start': 1878.02, 'duration': 2.082}, {'end': 1885.094, 'text': "So you can see in the first row, it hasn't really listened to us very much at all.", 'start': 1881.431, 'duration': 3.663}, {'end': 1890.738, 'text': 'These are very weird looking things and none of them really look like astronauts riding a horse.', 'start': 1885.114, 'duration': 5.624}, {'end': 1901.378, 'text': 'At guided scale of three, They look more like things riding horses that they might be astronaut-ish.', 'start': 1891.959, 'duration': 9.419}, {'end': 1906.72, 'text': 'And at 7.5, they certainly on the whole look like astronauts riding a horse.', 'start': 1902.018, 'duration': 4.702}], 'summary': 'Testing different guidance scales, 7.5 being default, improves image creation.', 'duration': 54.298, 'max_score': 1852.422, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls1852422.jpg'}, {'end': 1976.643, 'src': 'embed', 'start': 1949.914, 'weight': 7, 'content': [{'end': 1961.042, 'text': 'creating two versions um one version of the image with the prompt an astronaut riding a horse and one version of the image with no prompt.', 'start': 1949.914, 'duration': 11.128}, {'end': 1962.883, 'text': "So it's just some random thing.", 'start': 1961.562, 'duration': 1.321}, {'end': 1966.546, 'text': 'And then it takes the average basically of those two things.', 'start': 1963.544, 'duration': 3.002}, {'end': 1971.22, 'text': "And that's how that's what guidance scale does,", 'start': 1968.278, 'duration': 2.942}, {'end': 1976.643, 'text': "and you can kind of think of the guidance scale as being a bit like a number that's used to weight the average.", 'start': 1971.22, 'duration': 5.423}], 'summary': 'Creating two image versions with and without prompt, then averaging them using guidance scale.', 'duration': 26.729, 'max_score': 1949.914, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls1949914.jpg'}, {'end': 2039.268, 'src': 'embed', 'start': 2012.518, 'weight': 2, 'content': [{'end': 2016.259, 'text': 'And you can pass in this thing negative prompt to diffusers.', 'start': 2012.518, 'duration': 3.741}, {'end': 2025.283, 'text': 'And what that will do is it will take the prompt, which in this case is Labrador in the style of Vermeer, and create a second image effectively,', 'start': 2016.92, 'duration': 8.363}, {'end': 2029.684, 'text': 'which is just responding to the prompt blue, and effectively subtract one from the other.', 'start': 2025.283, 'duration': 4.401}, {'end': 2032.265, 'text': "The details are slightly different to that, but that's the basic idea.", 'start': 2029.784, 'duration': 2.481}, {'end': 2039.268, 'text': 'And that way we get a non-blue Labrador in the style of Vermeer.', 'start': 2033.586, 'duration': 5.682}], 'summary': 'Using negative prompt to create non-blue labrador in vermeer style.', 'duration': 26.75, 'max_score': 2012.518, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2012518.jpg'}, {'end': 2112.518, 'src': 'embed', 'start': 2063.587, 'weight': 3, 'content': [{'end': 2069.809, 'text': "You'll need an image to image pipeline and with the image to image pipeline, you can grab.", 'start': 2063.587, 'duration': 6.222}, {'end': 2072.719, 'text': 'rather sketchy looking sketch.', 'start': 2071.36, 'duration': 1.359}, {'end': 2092.425, 'text': 'And you can then pass to this eye-to-eye, image-to-image pipeline, the initial image to start with.', 'start': 2077.001, 'duration': 15.424}, {'end': 2105.072, 'text': 'And basically, what this is going to do is, rather than starting diffusion process with random noise,', 'start': 2094.565, 'duration': 10.507}, {'end': 2112.518, 'text': "it's going to basically start it with a noisy version of this drawing.", 'start': 2105.072, 'duration': 7.446}], 'summary': 'Using image-to-image pipeline to start diffusion process with a noisy version of the drawing.', 'duration': 48.931, 'max_score': 2063.587, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2063587.jpg'}, {'end': 2329.591, 'src': 'embed', 'start': 2295.544, 'weight': 4, 'content': [{'end': 2304.009, 'text': 'And then, this is really neat, he then used an image captioning model to automatically generate captions for each of those images.', 'start': 2295.544, 'duration': 8.465}, {'end': 2310.734, 'text': 'And then he fine-tuned the stable diffusion model using those image and caption pairs.', 'start': 2304.97, 'duration': 5.764}, {'end': 2315.997, 'text': "So here's an example of one of the captions and one of the images.", 'start': 2313.656, 'duration': 2.341}, {'end': 2329.591, 'text': 'And then took that fine-tuned model and passed it prompts, like girl with a girl, girl with a pearl earring and cute Obama creature,', 'start': 2317.898, 'duration': 11.693}], 'summary': 'A stable diffusion model was fine-tuned using image captioning model to generate captions for images.', 'duration': 34.047, 'max_score': 2295.544, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2295544.jpg'}, {'end': 2399.225, 'src': 'embed', 'start': 2359.779, 'weight': 5, 'content': [{'end': 2366.204, 'text': 'Fine-tuning can take quite a bit of data and quite a bit of time, but you can actually do some special kinds of fine-tuning.', 'start': 2359.779, 'duration': 6.425}, {'end': 2375.621, 'text': "One that you can do It's called Textual Inversion, which is where we actually fine tune just a single embedding.", 'start': 2367.305, 'duration': 8.316}, {'end': 2392.897, 'text': "So for example, we can create a new embedding where we're trying to make things that look like this.", 'start': 2377.703, 'duration': 15.194}, {'end': 2399.225, 'text': 'So what we can do is we can give this concept a name.', 'start': 2395.683, 'duration': 3.542}], 'summary': 'Textual inversion allows fine-tuning of single embeddings to create new concepts.', 'duration': 39.446, 'max_score': 2359.779, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2359779.jpg'}, {'end': 2502.593, 'src': 'embed', 'start': 2476.694, 'weight': 6, 'content': [{'end': 2491.21, 'text': 'Another example, very similar to Textual Inversion, is something called Dreambooth, which, as mentioned here,', 'start': 2476.694, 'duration': 14.516}, {'end': 2496.311, 'text': "what it does is it takes an existing token, but one that isn't used much like, say, SKS.", 'start': 2491.21, 'duration': 5.101}, {'end': 2502.593, 'text': 'almost nothing has SKS and fine-tunes a model to bring that token, as it says here, close to the images we provide.', 'start': 2496.311, 'duration': 6.282}], 'summary': 'Dreambooth fine-tunes a model to bring the token sks close to provided images.', 'duration': 25.899, 'max_score': 2476.694, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2476694.jpg'}], 'start': 1806.731, 'title': 'Image grid, guidance scale, and diffusion training', 'summary': 'Covers creating image grids with varying guidance scales resulting in different resemblance levels, blending images using guidance scale and negative prompts, and stable diffusion training examples including textual inversion and dreambooth.', 'chapters': [{'end': 1927.163, 'start': 1806.731, 'title': 'Creating image grid with different guidance scales', 'summary': "Demonstrates creating an image grid using a prompt of 'astronaut riding a horse' and varying guidance scales including 1, 3, 7, 14, and 15, resulting in different levels of resemblance to the prompt.", 'duration': 120.432, 'highlights': ['Varying guidance scales, including 1, 3, 7, 14, and 15, resulted in different levels of resemblance to the prompt, with 7.5 being the default.', 'At a guidance scale of 7.5, the images generally resembled astronauts riding a horse, while at 14 or 15, they showed strong resemblance but tended to be more abstract at times.', 'At a guidance scale of 3, the images looked more like things riding horses with some astronaut-like features, whereas at a guidance scale of 1, the resemblance to the prompt was minimal.', 'The chapter also mentioned the possibility of slight problems in how the algorithm works, which may be addressed in the future.']}, {'end': 2228.967, 'start': 1930.344, 'title': 'Guidance scale and negative prompt', 'summary': 'Discusses the use of guidance scale to blend two versions of images and the concept of using negative prompts to create new images. it also explores the image-to-image pipeline for creating drawings that match a specific composition.', 'duration': 298.623, 'highlights': ['The guidance scale creates two versions of an image based on a prompt and then takes the average, providing a way to blend images with different prompts.', 'Using negative prompts, one can effectively subtract one image from another, resulting in new creations, such as a non-blue Labrador in the style of Vermeer.', 'The image-to-image pipeline allows starting the diffusion process with a noisy version of an initial drawing, leading to improved compositions while maintaining the original structure.', 'It is possible to utilize the output images as initial images for creating new compositions, such as doing an oil painting of an output image, showcasing the potential of combining Python code for creative experimentation.']}, {'end': 2732.285, 'start': 2230.655, 'title': 'Stable diffusion training examples', 'summary': 'Discusses examples of stable diffusion training, including fine-tuning models with image and caption pairs, textual inversion, and dreambooth, showcasing the process and challenges associated with it.', 'duration': 501.63, 'highlights': ['Fine-tuning models with image and caption pairs', 'Textual Inversion for training embeddings', 'Dreambooth for fine-tuning tokens']}], 'duration': 925.554, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls1806731.jpg', 'highlights': ['Varying guidance scales resulted in different resemblance levels, with 7.5 as default.', 'Guidance scale of 7.5 resulted in images resembling astronauts riding a horse.', 'Using negative prompts effectively subtracts one image from another.', 'Image-to-image pipeline allows starting diffusion process with a noisy version of an initial drawing.', 'Fine-tuning models with image and caption pairs.', 'Textual Inversion for training embeddings.', 'Dreambooth for fine-tuning tokens.', 'Guidance scale creates two versions of an image based on a prompt and then takes the average.']}, {'end': 4076.651, 'segs': [{'end': 2853.471, 'src': 'embed', 'start': 2816.753, 'weight': 0, 'content': [{'end': 2823.855, 'text': "And it's going to spit out the probability that this thing you passed in is a handwritten digit.", 'start': 2816.753, 'duration': 7.102}, {'end': 2829.357, 'text': "So for this one, so let's say this image is called x1.", 'start': 2824.696, 'duration': 4.661}, {'end': 2839.56, 'text': 'The probability that x1 is a handwritten digit, it might say is 0.98.', 'start': 2831.258, 'duration': 8.302}, {'end': 2845.022, 'text': 'And so then you pass something else into this magic API endpoint.', 'start': 2839.56, 'duration': 5.462}, {'end': 2848.67, 'text': 'which looks like this.', 'start': 2847.81, 'duration': 0.86}, {'end': 2853.471, 'text': 'You pass that in.', 'start': 2852.791, 'duration': 0.68}], 'summary': 'A system can determine the probability of an image being a handwritten digit, with an example showing a 98% likelihood.', 'duration': 36.718, 'max_score': 2816.753, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2816753.jpg'}, {'end': 3004.73, 'src': 'embed', 'start': 2955.028, 'weight': 1, 'content': [{'end': 2963.515, 'text': 'if you have this function, which can take an image and tell you the probability that that is a handwritten digit,', 'start': 2955.028, 'duration': 8.487}, {'end': 2969.204, 'text': 'how could you use it to generate new images?', 'start': 2963.515, 'duration': 5.689}, {'end': 2982.487, 'text': 'Well, imagine you wanted to turn this mess into something that did look like an image.', 'start': 2970.464, 'duration': 12.023}, {'end': 2984.287, 'text': "Here's something you could do.", 'start': 2983.447, 'duration': 0.84}, {'end': 2999.648, 'text': "Let's say it's a 28 by 28 image, which is what 786.", 'start': 2984.307, 'duration': 15.341}, {'end': 3003.249, 'text': 'Oopsy-dozy, 28 times 28, 784.', 'start': 2999.648, 'duration': 3.601}, {'end': 3004.73, 'text': 'So 794 pixels.', 'start': 3003.25, 'duration': 1.48}], 'summary': 'Function can predict handwritten digit probability. potential to generate new images from given data.', 'duration': 49.702, 'max_score': 2955.028, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2955028.jpg'}, {'end': 3433.384, 'src': 'embed', 'start': 3400.885, 'weight': 4, 'content': [{'end': 3416.588, 'text': "Now functions are not necessarily just of one variable, so here we've got x and y, but they could be of two variables, x and y and z, say.", 'start': 3400.885, 'duration': 15.703}, {'end': 3426.374, 'text': 'And so we could have functions which, for example, could be like a 3D parabola with this kind of curvature, for instance.', 'start': 3417.408, 'duration': 8.966}, {'end': 3433.384, 'text': 'And so you can still find the derivative.', 'start': 3428.315, 'duration': 5.069}], 'summary': 'Functions can have multiple variables, like x, y, and z, leading to 3d parabolas with specific curvature.', 'duration': 32.499, 'max_score': 3400.885, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls3400885.jpg'}, {'end': 3690.169, 'src': 'embed', 'start': 3655.883, 'weight': 3, 'content': [{'end': 3662.525, 'text': 'Um? the actual answer um, of like which digit should it be?', 'start': 3655.883, 'duration': 6.642}, {'end': 3677.391, 'text': "So let's call that y minus the predicted y, which is some neural network with some weights and our pixels.", 'start': 3665.044, 'duration': 12.347}, {'end': 3681.225, 'text': 'So that would be like delving in one layer deeply.', 'start': 3679.124, 'duration': 2.101}, {'end': 3688.348, 'text': "But none of these details really matter too much of like what's the loss function or the neural network or whatever.", 'start': 3681.765, 'duration': 6.583}, {'end': 3690.169, 'text': "It's just some function that's calculating loss.", 'start': 3688.388, 'duration': 1.781}], 'summary': 'The loss function in the neural network calculates the difference between the actual and predicted values.', 'duration': 34.286, 'max_score': 3655.883, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls3655883.jpg'}, {'end': 4023.475, 'src': 'embed', 'start': 3994.746, 'weight': 2, 'content': [{'end': 4001.91, 'text': 'And if you just change one thing at a time, and calculate the derivative of the loss against that one thing, then we get these things called partials.', 'start': 3994.746, 'duration': 7.164}, {'end': 4012.256, 'text': 'And if you then do it for all the different things that you could change, such as every pixel, you get this whole gradient vector,', 'start': 4002.791, 'duration': 9.465}, {'end': 4015.158, 'text': 'which we use the upside down triangle to represent.', 'start': 4012.256, 'duration': 2.902}, {'end': 4023.475, 'text': 'And then finally, the non upside down triangle simply refers to a small change.', 'start': 4016.653, 'duration': 6.822}], 'summary': 'Using derivatives to calculate partials and gradients for changes in a system.', 'duration': 28.729, 'max_score': 3994.746, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls3994746.jpg'}], 'start': 2732.866, 'title': 'Generating handwritten digits and derivatives', 'summary': 'Discusses generating handwritten digits using innovative directions and probabilities, adjusting pixel values for accurate representation, understanding derivatives in multivariable functions, and calculating loss and derivatives for improving image quality.', 'chapters': [{'end': 2955.028, 'start': 2732.866, 'title': 'Generating handwritten digits with innovative directions', 'summary': 'Discusses the innovative directions for generating handwritten digits using a web service that returns probabilities for images, demonstrating its ability to generate handwritten digits based on the returned probabilities.', 'duration': 222.162, 'highlights': ['The web service returns probabilities for images, such as 0.98 for one image and 0.4 for another, indicating the likelihood that they are handwritten digits.', 'The discussion explores the possibility of using the function behind the web API to generate handwritten digits, showcasing its magical potential.', 'The chapter delves into the concept of utilizing a function to generate handwritten digits, offering a glimpse into the innovative potential of the approach.']}, {'end': 3287.159, 'start': 2955.028, 'title': 'Generating images using probability', 'summary': 'Discusses using a function that assesses the probability of an image being a handwritten digit to generate new images by adjusting pixel values one at a time, resulting in a more accurate representation of a handwritten digit, involving 28 by 28 images and 784 pixels.', 'duration': 332.131, 'highlights': ['The process involves adjusting pixel values one at a time to increase the probability that an image is a handwritten digit, resulting in a more accurate representation of handwritten digits (784 pixels).', 'The function calculates the gradient of the probability that a specific image is a handwritten digit with respect to its pixels, involving 784 values in a 28 by 28 input.', 'The speaker acknowledges their mistake in notation and expresses the importance of understanding the notation, even though they have previously written a tutorial on matrix calculus.', 'The speaker emphasizes the significance of understanding the key concepts, despite having authored a lengthy tutorial on matrix calculus with their friend Terence.']}, {'end': 3560.236, 'start': 3289.82, 'title': 'Understanding derivatives in multivariable functions', 'summary': 'Explains the concept of derivatives in multivariable functions, including the rules for calculating derivatives and the concept of partial derivatives, with a practical example of a 28 by 28 pixel image.', 'duration': 270.416, 'highlights': ['The concept of derivatives in multivariable functions is explained, including the rules for calculating derivatives.', 'The concept of partial derivatives is introduced, with a practical example of a 28 by 28 pixel image.']}, {'end': 3903.42, 'start': 3564.637, 'title': 'Calculating loss and derivatives', 'summary': 'Explains the calculation of loss using mean squared error and the process of calculating derivatives for each pixel to improve the loss in a 7x7 low-resolution image, emphasizing the importance of understanding the direction of change for pixel values.', 'duration': 338.783, 'highlights': ['The loss is calculated using the mean squared error of the actual answer y and the predicted y by a neural network, emphasizing the importance of understanding the direction of change for pixel values.', 'The chapter explains the process of calculating derivatives for each pixel in a 7x7 low-resolution image to determine whether the pixel should be made brighter or darker to improve the loss.', "The concept of 'upside down triangle x loss' is introduced as a vector of all the derivatives of the loss with respect to each pixel, providing a convenient notation to avoid writing out all the individual derivatives."]}, {'end': 4076.651, 'start': 3903.44, 'title': 'Understanding derivatives in mathematics', 'summary': 'Explains the concept of derivatives using the example of small changes in loss and variables, emphasizing the use of derivatives in multivariable functions and the representation of gradient vectors.', 'duration': 173.211, 'highlights': ['The derivative is a small change in loss divided by a small change in a specific variable, which can be represented by the classic rise over run concept, leading to the calculation of partials and gradient vectors.', 'Changing one variable in a multivariable function, such as a pixel value in an image or a weight, and calculating the derivative of the loss against that variable results in partials and a gradient vector represented by the upside down triangle.', 'The concept of a non-upside down triangle refers to a small change in loss caused by changing a specific variable by a small bit, eventually leading to the derivative.']}], 'duration': 1343.785, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls2732866.jpg', 'highlights': ['The web service returns probabilities for images, such as 0.98 for one image and 0.4 for another, indicating the likelihood that they are handwritten digits.', 'The process involves adjusting pixel values one at a time to increase the probability that an image is a handwritten digit, resulting in a more accurate representation of handwritten digits (784 pixels).', 'The derivative is a small change in loss divided by a small change in a specific variable, which can be represented by the classic rise over run concept, leading to the calculation of partials and gradient vectors.', 'The loss is calculated using the mean squared error of the actual answer y and the predicted y by a neural network, emphasizing the importance of understanding the direction of change for pixel values.', 'The concept of derivatives in multivariable functions is explained, including the rules for calculating derivatives.']}, {'end': 5124.485, 'segs': [{'end': 4315.945, 'src': 'embed', 'start': 4204.481, 'weight': 0, 'content': [{'end': 4223.99, 'text': "And so we'll now get a new gradient here 794 values and we can use that to change every pixel to make it look a little bit more like a handwritten digit.", 'start': 4204.481, 'duration': 19.509}, {'end': 4243.883, 'text': 'So, as you can see, if we have this magic function, we can use it to turn any arbitrary noisy input into something that looks like a valid input,', 'start': 4226.431, 'duration': 17.452}, {'end': 4250.143, 'text': 'something that has a high p-value from that function, by using this derivative.', 'start': 4243.883, 'duration': 6.26}, {'end': 4261.578, 'text': 'So a key thing to remember here is this saying is saying as I change the input pixels, how does it change the probability that this is a digit?', 'start': 4250.664, 'duration': 10.914}, {'end': 4268.006, 'text': 'And that tells me which pixels to make darker and which pixels to make lighter.', 'start': 4262.198, 'duration': 5.808}, {'end': 4282.964, 'text': 'Now, those of you who remember your high school calculus may recall that when you do this by changing each pixel one at a time to calculate a derivative,', 'start': 4272.574, 'duration': 10.39}, {'end': 4289.05, 'text': 'this is called the finite differencing method of calculating derivatives.', 'start': 4282.964, 'duration': 6.086}, {'end': 4297.181, 'text': "And it's very slow because we have to call… Sorry, I can't spell.", 'start': 4289.97, 'duration': 7.211}, {'end': 4306.503, 'text': "Differencing It's very slow because we have to call this function 784 times for every single one.", 'start': 4298.241, 'duration': 8.262}, {'end': 4309.524, 'text': "We don't have to use finite differencing.", 'start': 4307.703, 'duration': 1.821}, {'end': 4315.945, 'text': 'Assuming the folks running this magic API endpoint used Python,', 'start': 4311.084, 'duration': 4.861}], 'summary': 'Using a new gradient with 794 values to transform noisy inputs into valid digits, optimizing pixel changes for high p-value, avoiding slow finite differencing method.', 'duration': 111.464, 'max_score': 4204.481, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls4204481.jpg'}, {'end': 4486.339, 'src': 'embed', 'start': 4448.548, 'weight': 3, 'content': [{'end': 4450.529, 'text': 'we create a neural net and we train it.', 'start': 4448.548, 'duration': 1.981}, {'end': 4465.63, 'text': 'So we want to train a neural net that tells us which pixels to change to make a digit look more, to make an image look more like a handwritten digit.', 'start': 4452.591, 'duration': 13.039}, {'end': 4469.312, 'text': "Okay, so here's how we can do that.", 'start': 4467.591, 'duration': 1.721}, {'end': 4482.497, 'text': 'We could create some training data and use that training data to get the information we want.', 'start': 4470.973, 'duration': 11.524}, {'end': 4486.339, 'text': 'We could pass in something that looks a lot like a handwritten digit.', 'start': 4483.117, 'duration': 3.222}], 'summary': 'Creating a neural net to train and identify handwritten digits.', 'duration': 37.791, 'max_score': 4448.548, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls4448548.jpg'}, {'end': 4712.093, 'src': 'embed', 'start': 4654.618, 'weight': 5, 'content': [{'end': 4662.564, 'text': 'but the important thing to using this stuff well is to think about neural nets as being something that has some inputs.', 'start': 4654.618, 'duration': 7.946}, {'end': 4693.493, 'text': 'some outputs and some loss function, which takes those two, and then the derivative is used to update the weights right?', 'start': 4665.856, 'duration': 27.637}, {'end': 4694.774, 'text': "That's really what we care about.", 'start': 4693.553, 'duration': 1.221}, {'end': 4696.645, 'text': 'those four things.', 'start': 4695.745, 'duration': 0.9}, {'end': 4704.569, 'text': 'Now the inputs to our model is this.', 'start': 4697.666, 'duration': 6.903}, {'end': 4712.093, 'text': "Okay, that's the inputs to our model.", 'start': 4710.452, 'duration': 1.641}], 'summary': 'Neural nets involve inputs, outputs, loss function, and derivatives for weight updates.', 'duration': 57.475, 'max_score': 4654.618, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls4654618.jpg'}, {'end': 4850.804, 'src': 'embed', 'start': 4809.942, 'weight': 6, 'content': [{'end': 4820.95, 'text': "And so we can do something we've done a thousand times, which is we can divide it by the count squared, and then we can sum all that up.", 'start': 4809.942, 'duration': 11.008}, {'end': 4828.597, 'text': 'And this here is the mean squared error, which we use all the time.', 'start': 4822.912, 'duration': 5.685}, {'end': 4840.501, 'text': "So the mean squared error means that we've now got inputs, which is noisy digits.", 'start': 4832.738, 'duration': 7.763}, {'end': 4844.402, 'text': "We've got outputs, which is noise.", 'start': 4841.601, 'duration': 2.801}, {'end': 4850.804, 'text': 'And so this neural network is trying to predict this noise.', 'start': 4845.042, 'duration': 5.762}], 'summary': 'Neural network aims to predict noise using mean squared error.', 'duration': 40.862, 'max_score': 4809.942, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls4809942.jpg'}, {'end': 5022.468, 'src': 'embed', 'start': 4984.447, 'weight': 7, 'content': [{'end': 5001.23, 'text': "How?. Because now we can take this trained neural network, so I'm going to copy it down here and we can pass it something very, very, very noisy,", 'start': 4984.447, 'duration': 16.783}, {'end': 5002.311, 'text': 'which is pure noise.', 'start': 5001.23, 'duration': 1.081}, {'end': 5015.822, 'text': "We pass it to the neural net and it's going to spit out information saying which part of that does it think is noise?", 'start': 5007.135, 'duration': 8.687}, {'end': 5022.468, 'text': "And it's going to leave behind the bits that look the most like a digit, just like we did back here.", 'start': 5016.283, 'duration': 6.185}], 'summary': 'A neural network can filter out noise to identify digits.', 'duration': 38.021, 'max_score': 4984.447, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls4984447.jpg'}, {'end': 5141.508, 'src': 'embed', 'start': 5104.289, 'weight': 8, 'content': [{'end': 5105.85, 'text': "Don't know, Michelangelo, what's happening for you.", 'start': 5104.289, 'duration': 1.561}, {'end': 5120.844, 'text': "Okay, And to answer your earlier question about how am I drawing, I'm using a graphics tablet, which I'm not very expert at,", 'start': 5107.691, 'duration': 13.153}, {'end': 5124.485, 'text': 'because on Windows you can just draw directly on the screen, which is why this is particularly messy.', 'start': 5120.844, 'duration': 3.641}, {'end': 5141.508, 'text': "All right, in practice, at the moment, this might change by the time you've watched this, we use a particular type of neural net.", 'start': 5128.726, 'duration': 12.782}], 'summary': 'Using a graphics tablet to draw, not expert, employing a specific type of neural net.', 'duration': 37.219, 'max_score': 5104.289, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls5104289.jpg'}], 'start': 4077.411, 'title': 'Enhancing handwritten digit recognition', 'summary': 'Explains how to modify pixels using gradients to improve the resemblance of a handwritten digit, resulting in a higher probability of being recognized, with an example probability of 0.2. it also discusses calculating derivatives, training neural nets, and predicting noise in neural networks for improved image recognition and noise removal.', 'chapters': [{'end': 4282.964, 'start': 4077.411, 'title': 'Modifying pixels for handwritten digit recognition', 'summary': 'Explains how to modify pixels using gradients to improve the resemblance of a handwritten digit, resulting in a higher probability of being recognized as a digit, with an example probability of 0.2, using a magic function and derivatives to guide the pixel modifications.', 'duration': 205.553, 'highlights': ['Using gradients to modify pixels can improve the resemblance of a handwritten digit, resulting in a higher probability of recognition, with an example probability of 0.2.', 'The process involves taking the new image, modifying every pixel by subtracting its gradient multiplied by a constant, and then running it through a function to create a new version with a higher probability of being a handwritten digit.', 'By changing each pixel one at a time and calculating the derivative, it is possible to determine which pixels to make darker and lighter, guided by the change in probability of being a digit.']}, {'end': 4712.093, 'start': 4282.964, 'title': 'Calculating derivatives and training neural nets', 'summary': 'Discusses the process of calculating derivatives using the finite differencing method and suggests using the analytic derivatives for efficiency. it also introduces the concept of training a neural net to determine which pixels to change to make an image more closely resemble a handwritten digit.', 'duration': 429.129, 'highlights': ['The finite differencing method of calculating derivatives is slow, requiring 784 function calls for every single one, and suggests using the analytic derivatives for efficiency.', 'Introducing the concept of training a neural net to determine which pixels to change to make an image more closely resemble a handwritten digit.', 'Creating a neural net with inputs, outputs, and a loss function, emphasizing the importance of thinking about neural nets in this way.']}, {'end': 5124.485, 'start': 4713.885, 'title': 'Predicting noise in neural networks', 'summary': 'Discusses using neural networks to predict and remove noise from images by utilizing mean squared error as the loss function, resulting in a model that can generate digit-like images and remove noise.', 'duration': 410.6, 'highlights': ['Neural network outputs measure of noise with mean squared error as loss function', 'Predicting and removing noise from images using trained neural network', 'Utilizing graphics tablet to draw directly on screen for visual aid']}], 'duration': 1047.074, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls4077411.jpg', 'highlights': ['Using gradients to modify pixels can improve the resemblance of a handwritten digit, resulting in a higher probability of recognition, with an example probability of 0.2.', 'The process involves taking the new image, modifying every pixel by subtracting its gradient multiplied by a constant, and then running it through a function to create a new version with a higher probability of being a handwritten digit.', 'By changing each pixel one at a time and calculating the derivative, it is possible to determine which pixels to make darker and lighter, guided by the change in probability of being a digit.', 'Introducing the concept of training a neural net to determine which pixels to change to make an image more closely resemble a handwritten digit.', 'The finite differencing method of calculating derivatives is slow, requiring 784 function calls for every single one, and suggests using the analytic derivatives for efficiency.', 'Creating a neural net with inputs, outputs, and a loss function, emphasizing the importance of thinking about neural nets in this way.', 'Neural network outputs measure of noise with mean squared error as loss function', 'Predicting and removing noise from images using trained neural network', 'Utilizing graphics tablet to draw directly on screen for visual aid']}, {'end': 6209.958, 'segs': [{'end': 5259.578, 'src': 'embed', 'start': 5209.091, 'weight': 0, 'content': [{'end': 5223.479, 'text': 'And the output is the noise, such that if we subtract the output from the input, we end up with the unnoisy image, or at least an approximation of it.', 'start': 5209.091, 'duration': 14.388}, {'end': 5229.803, 'text': "So that's the UNet.", 'start': 5229.022, 'duration': 0.781}, {'end': 5234.596, 'text': "Now here's the problem.", 'start': 5233.715, 'duration': 0.881}, {'end': 5235.897, 'text': "Well, here's our problem.", 'start': 5234.956, 'duration': 0.941}, {'end': 5245.386, 'text': 'We have, oh, why do I keep forgetting this? We have 28 times 28, 784.', 'start': 5238.059, 'duration': 7.327}, {'end': 5246.447, 'text': 'I should write that down.', 'start': 5245.386, 'duration': 1.061}, {'end': 5247.928, 'text': 'We have in these things, 784 pixels.', 'start': 5246.487, 'duration': 1.441}, {'end': 5259.578, 'text': "And that's quite a lot.", 'start': 5258.737, 'duration': 0.841}], 'summary': 'Unet removes noise, resulting in unnoisy image. input has 784 pixels.', 'duration': 50.487, 'max_score': 5209.091, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls5209091.jpg'}, {'end': 5443.873, 'src': 'embed', 'start': 5407.329, 'weight': 3, 'content': [{'end': 5409.631, 'text': "You know, you don't really have to do every one individually.", 'start': 5407.329, 'duration': 2.302}, {'end': 5417.277, 'text': 'There are faster, more concise ways of storing what an image is.', 'start': 5409.651, 'duration': 7.626}, {'end': 5422.001, 'text': 'We know this is true because, for example,', 'start': 5419.219, 'duration': 2.782}, {'end': 5430.087, 'text': 'a JPEG picture is far fewer bytes than the number of bytes you would get if you multiplied its height by its width, by its channels.', 'start': 5422.001, 'duration': 8.086}, {'end': 5434.191, 'text': "So we know that it's possible to compress.", 'start': 5431.769, 'duration': 2.422}, {'end': 5443.873, 'text': 'pictures So let me show you a really interesting way to compress pictures.', 'start': 5437.028, 'duration': 6.845}], 'summary': 'Jpeg pictures are far fewer bytes than the product of height, width, and channels, demonstrating compression potential.', 'duration': 36.544, 'max_score': 5407.329, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls5407329.jpg'}, {'end': 5589.237, 'src': 'embed', 'start': 5535.177, 'weight': 1, 'content': [{'end': 5554.174, 'text': "Okay so here's a neural network and so the number of pixels in this version is now 64 times 64 times 4, 16384.", 'start': 5535.177, 'duration': 18.997}, {'end': 5558.217, 'text': "So there's 16384 pixels here.", 'start': 5554.175, 'duration': 4.042}, {'end': 5564.02, 'text': "Okay, so we've compressed it from 786, 432 to 16384, which is a 48 times decrease.", 'start': 5558.237, 'duration': 5.783}, {'end': 5568.722, 'text': "Now that's no use if we've lost our image.", 'start': 5565.58, 'duration': 3.142}, {'end': 5589.237, 'text': 'So can we get the image back again? Sure.', 'start': 5583.996, 'duration': 5.241}], 'summary': 'Neural network has 16384 pixels, a 48x decrease from 786,432.', 'duration': 54.06, 'max_score': 5535.177, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls5535177.jpg'}, {'end': 5957.043, 'src': 'embed', 'start': 5919.104, 'weight': 4, 'content': [{'end': 5931.446, 'text': 'well, as long as they have a copy of the decoder on their computer, they can feed those bytes into the decoder and get back the original image.', 'start': 5919.104, 'duration': 12.342}, {'end': 5938.171, 'text': "So what we've just done is we've created a compression algorithm.", 'start': 5933.168, 'duration': 5.003}, {'end': 5951.419, 'text': "That's pretty amazing, isn't it? And in fact, these compression algorithms work extremely, extremely well.", 'start': 5942.774, 'duration': 8.645}, {'end': 5957.043, 'text': "And notice that we didn't train this on just this one image.", 'start': 5952.6, 'duration': 4.443}], 'summary': 'Compression algorithm created, works extremely well.', 'duration': 37.939, 'max_score': 5919.104, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls5919104.jpg'}], 'start': 5128.726, 'title': 'Efficient image and data compression', 'summary': 'Covers the implementation of u-net for image processing, challenges of handling large image datasets, and the need for efficient image storage. additionally, it demonstrates image compression using convolutional neural networks to reduce image size by 48 times and data compression using an autoencoder to achieve a 48 times reduction in image data size.', 'chapters': [{'end': 5407.309, 'start': 5128.726, 'title': 'Efficient image processing with u-net', 'summary': 'Explains the implementation of u-net for image processing, the challenges of handling large image datasets, and the need for efficient image storage and processing, alluding to the potential time and cost savings.', 'duration': 278.583, 'highlights': ['The U-Net model is explained and its role in denoising images is detailed, with the input being noisy images and the output being the noise, resulting in an approximation of the unnoisy image.', 'The challenges of processing large image datasets are highlighted, with the mention of 784 pixels in handwritten digits and 786,432 pixels in high-definition photos, emphasizing the potential time-consuming nature of training such models.', 'The need for efficient image storage and processing is emphasized, suggesting the exploration of alternative methods for storing pixel values to optimize efficiency and reduce computational load.']}, {'end': 5853.666, 'start': 5407.329, 'title': 'Image compression using convolutional neural networks', 'summary': 'Demonstrates a method of compressing images using a series of convolutional layers, reducing the image from 786,432 to 16,384 pixels with a 48 times decrease, and then reconstructing the original image using an inverse convolutional process, ultimately creating an autoencoder model for image reconstruction.', 'duration': 446.337, 'highlights': ['By applying a series of convolutional layers and reducing the image from 786,432 to 16,384 pixels, a 48 times decrease in size is achieved, demonstrating effective image compression.', 'The process involves reconstructing the original image using an inverse convolutional method, effectively creating an autoencoder model for image reconstruction.', 'The method involves building a neural network with convolutions that decrease in size followed by convolutions that increase in size, ultimately resulting in an autoencoder model for image reconstruction.', "The model's objective is to successfully reconstruct the input image, aiming for a mean squared error of zero, making it an interesting and effective approach for image compression and reconstruction."]}, {'end': 6209.958, 'start': 5854.346, 'title': 'Data compression using autoencoder', 'summary': 'Explains how an autoencoder can compress image data by 48 times, using only 16,384 bytes, and then decode it back to the original image, enabling the creation of a powerful compression algorithm.', 'duration': 355.612, 'highlights': ['The autoencoder compresses the original image data by 48 times, resulting in just 16,384 bytes, enabling significant data reduction and efficient storage and transmission.', 'The compressed data, if decoded with the decoder, can be reconstructed back to the original image, showcasing the effectiveness of the compression algorithm.', 'By training the autoencoder on millions of images, it can efficiently compress and decompress diverse image data, making it a powerful compression algorithm for sharing and storage.', "The process of training a UNet on millions of pictures using the autoencoder's encoder and then using the decoder to reconstruct the images results in efficient and intelligent data processing.", "The introduction of Latents as a smaller representation of images enables noise subtraction and efficient image reconstruction using the autoencoder's decoder, enhancing the compression and reconstruction process."]}], 'duration': 1081.232, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls5128726.jpg', 'highlights': ['The U-Net model denoises images, approximating unnoisy images (relevance: 5)', 'Image compression reduces size by 48 times using convolutional layers (relevance: 4)', 'Autoencoder compresses image data by 48 times, enabling efficient storage (relevance: 3)', 'Efficient image storage and processing are emphasized for optimization (relevance: 2)', "Autoencoder's decoder efficiently reconstructs images, enhancing compression (relevance: 1)"]}, {'end': 7274.916, 'segs': [{'end': 6248.445, 'src': 'embed', 'start': 6209.958, 'weight': 1, 'content': [{'end': 6214.541, 'text': "you would probably rather everybody was doing stuff on the thing that's 48 times smaller.", 'start': 6209.958, 'duration': 4.583}, {'end': 6221.577, 'text': 'So the VAE is optional, but it saves us a whole lot of time and a whole lot of money.', 'start': 6215.731, 'duration': 5.846}, {'end': 6224.06, 'text': "So that's good.", 'start': 6223.519, 'duration': 0.541}, {'end': 6230.466, 'text': "Okay, what's next??", 'start': 6227.363, 'duration': 3.103}, {'end': 6234.711, 'text': "Well, there's something else, which is….", 'start': 6232.188, 'duration': 2.523}, {'end': 6240.083, 'text': "We have not just been um in this morning's.", 'start': 6236.482, 'duration': 3.601}, {'end': 6248.445, 'text': "sorry, in the first half of today's lesson, we weren't just um saying um, produce me an image.", 'start': 6240.083, 'duration': 8.362}], 'summary': 'Using vae saves time and money, making it beneficial for all.', 'duration': 38.487, 'max_score': 6209.958, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6209958.jpg'}, {'end': 6390.112, 'src': 'embed', 'start': 6356.219, 'weight': 2, 'content': [{'end': 6362.243, 'text': "So we would expect this model to be better at predicting noise than the previous one, because we're giving it more information.", 'start': 6356.219, 'duration': 6.024}, {'end': 6366.854, 'text': 'This was a 3, this was a 6, this was a 7.', 'start': 6363.024, 'duration': 3.83}, {'end': 6376.702, 'text': 'So this neural net is going to learn to estimate noise better by taking advantage of the fact that it knows what actual the input was.', 'start': 6366.854, 'duration': 9.848}, {'end': 6380.124, 'text': 'And why is that useful?', 'start': 6379.143, 'duration': 0.981}, {'end': 6390.112, 'text': "Well, the reason that's useful is because now, when we feed in the number three, like the actual digit three, is a one hot encoded vector plus noise.", 'start': 6380.705, 'duration': 9.407}], 'summary': 'New model better predicts noise using more information. input: 3, 6, 7.', 'duration': 33.893, 'max_score': 6356.219, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6356219.jpg'}, {'end': 6522.55, 'src': 'embed', 'start': 6482.776, 'weight': 3, 'content': [{'end': 6491.678, 'text': 'So we have to do something else to turn this into an embedding, something other than grabbing a one-hot encoded version of this.', 'start': 6482.776, 'duration': 8.902}, {'end': 6493.679, 'text': 'So what do we do?', 'start': 6493.119, 'duration': 0.56}, {'end': 6522.55, 'text': "So what we're going to do is we're going to try to create a model that can take a sentence like a cute teddy and can return a vector of numbers that in some way represents what cute teddies look like.", 'start': 6496.085, 'duration': 26.465}], 'summary': 'Creating a model to generate vector representation of sentences.', 'duration': 39.774, 'max_score': 6482.776, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6482776.jpg'}, {'end': 6695.378, 'src': 'embed', 'start': 6603.449, 'weight': 4, 'content': [{'end': 6610.636, 'text': 'So what we can now do with these is we can create two models.', 'start': 6603.449, 'duration': 7.187}, {'end': 6622.472, 'text': 'one model which is a text encoder, and one model which is an image encoder.', 'start': 6612.486, 'duration': 9.986}, {'end': 6634.474, 'text': 'Okay So again, these are neural nets.', 'start': 6622.492, 'duration': 11.982}, {'end': 6637.916, 'text': "We don't care about what their architectures are or whatever.", 'start': 6634.554, 'duration': 3.362}, {'end': 6645.578, 'text': "We know that they're just black boxes, which contain weights, which means they need inputs and outputs in a loss function.", 'start': 6638.436, 'duration': 7.142}, {'end': 6647.599, 'text': "And then they'll do something.", 'start': 6646.438, 'duration': 1.161}, {'end': 6655.201, 'text': "Once we've defined inputs and outputs in a loss function, the neural nets will then do something.", 'start': 6650.039, 'duration': 5.162}, {'end': 6658.122, 'text': "So here's a really interesting idea.", 'start': 6656.601, 'duration': 1.521}, {'end': 6680.963, 'text': 'What if we take this image, and what if we then also take the text, a graceful swan.', 'start': 6664.309, 'duration': 16.654}, {'end': 6689.27, 'text': "Okay And we're going to feed these.", 'start': 6680.983, 'duration': 8.287}, {'end': 6695.378, 'text': 'into their respective models, which initially they of course have random weights.', 'start': 6691.235, 'duration': 4.143}], 'summary': 'Creating two models: text encoder and image encoder for neural nets.', 'duration': 91.929, 'max_score': 6603.449, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6603449.jpg'}, {'end': 6907.077, 'src': 'embed', 'start': 6872.229, 'weight': 7, 'content': [{'end': 6874.072, 'text': 'And this thing here is called the dot product.', 'start': 6872.229, 'duration': 1.843}, {'end': 6891.534, 'text': 'And so we could take the dot product of the features from the image model for the swan and the dot product of the features from the text model for the works graceful swan and take their dot product.', 'start': 6880.692, 'duration': 10.842}, {'end': 6895.955, 'text': 'And we want that number to be nice and big.', 'start': 6891.974, 'duration': 3.981}, {'end': 6907.077, 'text': "And the scene from Hitchcock's features should be very similar to the text scene from Hitchcock's features.", 'start': 6899.315, 'duration': 7.762}], 'summary': 'Dot product used to compare image and text features for similarity.', 'duration': 34.848, 'max_score': 6872.229, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6872229.jpg'}, {'end': 7020.133, 'src': 'embed', 'start': 6984.525, 'weight': 5, 'content': [{'end': 6991.851, 'text': 'And we need them to spit out features for things that they are not paired with, which are not similar.', 'start': 6984.525, 'duration': 7.326}, {'end': 7011.07, 'text': "And so if we can do that, then we're going to end up with a text encoder that we can feed in things like a graceful swan, some beautiful swan,", 'start': 6993.972, 'duration': 17.098}, {'end': 7012.411, 'text': 'such a lovely swan.', 'start': 7011.07, 'duration': 1.341}, {'end': 7020.133, 'text': 'And these should all give very similar embeddings because these would all represent very similar pictures.', 'start': 7013.451, 'duration': 6.682}], 'summary': 'Developing a text encoder to generate similar embeddings for non-paired, dissimilar items.', 'duration': 35.608, 'max_score': 6984.525, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6984525.jpg'}, {'end': 7162.782, 'src': 'embed', 'start': 7105.572, 'weight': 0, 'content': [{'end': 7111.055, 'text': "So the model that's used, or the pair of models that's used here, is called clip.", 'start': 7105.572, 'duration': 5.483}, {'end': 7121.388, 'text': 'This thing where we want these to be bigger and these to be smaller is called a contrastive loss.', 'start': 7115.646, 'duration': 5.742}, {'end': 7128.892, 'text': 'And now you know where the CL comes from.', 'start': 7127.211, 'duration': 1.681}, {'end': 7138.956, 'text': 'So here we have a clip text encoder.', 'start': 7133.674, 'duration': 5.282}, {'end': 7151.497, 'text': 'Its input is some text.', 'start': 7147.856, 'duration': 3.641}, {'end': 7158.74, 'text': "Its output is, we call it an embedding, it's just some features.", 'start': 7154.459, 'duration': 4.281}, {'end': 7162.782, 'text': 'Oops, embedding.', 'start': 7158.76, 'duration': 4.022}], 'summary': 'The model used is called clip, utilizing a contrastive loss to generate embeddings from input text.', 'duration': 57.21, 'max_score': 7105.572, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7105572.jpg'}, {'end': 7242.762, 'src': 'embed', 'start': 7205.175, 'weight': 8, 'content': [{'end': 7210.759, 'text': "We've got a text encoder which can allow us to train a unit which is guided by captions.", 'start': 7205.175, 'duration': 5.584}, {'end': 7225.71, 'text': 'So the last thing we need is the question about how exactly do we do this inference process here?', 'start': 7214.822, 'duration': 10.888}, {'end': 7241.462, 'text': "So how exactly, once we've got something that gives us the gradients we want, and, by the way, these gradients are often called the score function,", 'start': 7228.7, 'duration': 12.762}, {'end': 7242.762, 'text': 'just in case you come across that.', 'start': 7241.462, 'duration': 1.3}], 'summary': 'Text encoder enables guided training with captions, needing clarity on inference process and gradients for score function.', 'duration': 37.587, 'max_score': 7205.175, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7205175.jpg'}], 'start': 6209.958, 'title': 'Image and text multimodal models', 'summary': 'Discusses using variational autoencoder (vae) for image generation, and matching images and text through neural nets. it also introduces a multimodal model, clip, enabling the training of a unit guided by captions.', 'chapters': [{'end': 6522.55, 'start': 6209.958, 'title': 'Vae and image generation', 'summary': 'Discusses the use of variational autoencoder (vae) to save time and money, and the process of generating specific images using a neural net, with a focus on predicting noise and providing guidance for image creation.', 'duration': 312.592, 'highlights': ['The VAE is optional but saves a whole lot of time and money', 'The neural net learns to predict noise better by taking advantage of additional information about the original image', 'The process involves creating a model to return a vector representing the characteristics of a specific object or concept, such as a cute teddy']}, {'end': 6935.761, 'start': 6524.768, 'title': 'Image and text matching using neural nets', 'summary': 'Introduces the concept of using neural nets to match images and text by creating models for image and text encoders, and measuring similarity using dot product of their features.', 'duration': 410.993, 'highlights': ['The chapter introduces the concept of using neural nets to match images and text.', 'Creation of text and image encoders.', 'Measuring similarity using dot product of features.']}, {'end': 7274.916, 'start': 6939.136, 'title': 'Text and image multimodal model', 'summary': 'Introduces a multimodal model, called clip, which puts text and images into the same space by using contrastive loss, creating embeddings that represent similar pictures and enabling the training of a unit guided by captions.', 'duration': 335.78, 'highlights': ['The multimodal model, CLIP, creates embeddings for text and images using a contrastive loss function, ensuring that text and image embeddings are very similar for paired items and dissimilar for unpaired items.', "The text encoder allows the model to feed in text like 'graceful swan' and 'cute teddy bear' to produce very similar embeddings, enabling the model to represent similar pictures.", 'The model includes a denoiser unit, a decoder, and a text encoder, which collectively facilitate the training of a unit guided by captions and the inference process using score functions.']}], 'duration': 1064.958, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls6209958.jpg', 'highlights': ['The multimodal model, CLIP, creates embeddings for text and images using a contrastive loss function, ensuring that text and image embeddings are very similar for paired items and dissimilar for unpaired items.', 'The VAE is optional but saves a whole lot of time and money', 'The neural net learns to predict noise better by taking advantage of additional information about the original image', 'The process involves creating a model to return a vector representing the characteristics of a specific object or concept, such as a cute teddy', 'The chapter introduces the concept of using neural nets to match images and text.', "The text encoder allows the model to feed in text like 'graceful swan' and 'cute teddy bear' to produce very similar embeddings, enabling the model to represent similar pictures.", 'Creation of text and image encoders.', 'Measuring similarity using dot product of features.', 'The model includes a denoiser unit, a decoder, and a text encoder, which collectively facilitate the training of a unit guided by captions and the inference process using score functions.']}, {'end': 8113.259, 'segs': [{'end': 7337.604, 'src': 'embed', 'start': 7307.798, 'weight': 3, 'content': [{'end': 7318.423, 'text': "But to see what time steps are, even though it's got nothing to do with time in real life, consider the fact that we used varying levels of noise.", 'start': 7307.798, 'duration': 10.625}, {'end': 7325.966, 'text': "Some things were very noisy, some things were not noisy at all, some things had no noise, and some I haven't drawn here would have been pure noise.", 'start': 7318.443, 'duration': 7.523}, {'end': 7337.604, 'text': 'you could basically create a kind of a noising schedule where along here, you could put, say,', 'start': 7328.979, 'duration': 8.625}], 'summary': 'Explanation of time steps and noise levels in a hypothetical scenario.', 'duration': 29.806, 'max_score': 7307.798, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7307798.jpg'}, {'end': 7786.822, 'src': 'embed', 'start': 7758.737, 'weight': 4, 'content': [{'end': 7763.798, 'text': 'And in fact we even got better ways of doing that, where we kind of say well, what about what happens as the variance changes?', 'start': 7758.737, 'duration': 5.061}, {'end': 7765.158, 'text': 'Maybe we can look at that as well.', 'start': 7763.858, 'duration': 1.3}, {'end': 7767.599, 'text': 'And that gives us something called Adam.', 'start': 7765.178, 'duration': 2.421}, {'end': 7771.819, 'text': 'And these are types of optimizer.', 'start': 7769.559, 'duration': 2.26}, {'end': 7785.962, 'text': 'And so maybe you might be wondering, could we use these kinds of tricks? And the answer based on our very early research is yes.', 'start': 7774.2, 'duration': 11.762}, {'end': 7786.822, 'text': 'Yes, we can.', 'start': 7786.322, 'duration': 0.5}], 'summary': 'Introducing adam optimizer for variance changes, early research shows promising results.', 'duration': 28.085, 'max_score': 7758.737, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7758737.jpg'}, {'end': 7840.724, 'src': 'embed', 'start': 7812.571, 'weight': 0, 'content': [{'end': 7819.774, 'text': 'which is really all about taking these like little steps, little steps, little steps, and trying to figure out how to take bigger steps.', 'start': 7812.571, 'duration': 7.203}, {'end': 7828.198, 'text': 'And so different differential equation solvers use a lot of the same kind of ideas, if you squint, as optimizers.', 'start': 7820.515, 'duration': 7.683}, {'end': 7840.724, 'text': 'One thing that differential equations solvers do, which is kind of interesting though, is that they tend to take t as an input.', 'start': 7830.359, 'duration': 10.365}], 'summary': 'Using small steps to take bigger steps, differential equation solvers use similar ideas as optimizers and tend to take t as an input.', 'duration': 28.153, 'max_score': 7812.571, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7812571.jpg'}, {'end': 7934.052, 'src': 'embed', 'start': 7904.529, 'weight': 1, 'content': [{'end': 7912.997, 'text': "And so actually, Jono's started playing with this and experimenting a bit, and early results suggest that yeah, actually,", 'start': 7904.529, 'duration': 8.468}, {'end': 7920.143, 'text': 'when we rethink about the whole thing as being about learning rates and optimizers, maybe it actually works a bit better.', 'start': 7912.997, 'duration': 7.146}, {'end': 7922.726, 'text': "In fact, there's all kinds of things we could do.", 'start': 7921.184, 'duration': 1.542}, {'end': 7934.052, 'text': "Once we stop thinking about them as differential equations and worry about the math don't worry about the math so much about Gaussians and whatever we can really switch things around.", 'start': 7924.105, 'duration': 9.947}], 'summary': 'Early results suggest that rethinking learning rates and optimizers works better, allowing for various adjustments.', 'duration': 29.523, 'max_score': 7904.529, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7904529.jpg'}, {'end': 8113.259, 'src': 'heatmap', 'start': 8031.127, 'weight': 2, 'content': [{'end': 8034.768, 'text': 'this is some of the kind of stuff that we are starting to research at the moment.', 'start': 8031.127, 'duration': 3.641}, {'end': 8045.091, 'text': 'And the early results are extremely positive, both in terms of how quickly we can do things and what kind of outputs we seem to be getting.', 'start': 8035.588, 'duration': 9.503}, {'end': 8053.253, 'text': "Okay So I think that's probably a good place to stop it.", 'start': 8049.432, 'duration': 3.821}, {'end': 8068.808, 'text': "So what we're going to do in the next lesson is we're going to finish our journey into this notebook to see some of the code behind the scenes of what's in a pipeline when I get there.", 'start': 8053.313, 'duration': 15.495}, {'end': 8076.913, 'text': "So we'll do looking inside the pipeline and see exactly what's going on behind the scenes a bit more in terms of the code.", 'start': 8071.309, 'duration': 5.604}, {'end': 8086.727, 'text': "And then we're going to do a huge rewind the from the foundations and we're going to build up from some very tricky ground rules.", 'start': 8077.694, 'duration': 9.033}, {'end': 8096.714, 'text': "Our ground rules would be we're only allowed to use pure Python, the Python standard library and nothing else,", 'start': 8086.787, 'duration': 9.927}, {'end': 8105.96, 'text': 'and build up from there until we have recreated all of this and possibly some new research directions at the same time.', 'start': 8096.714, 'duration': 9.246}, {'end': 8108.165, 'text': "So that's our goal.", 'start': 8107.363, 'duration': 0.802}, {'end': 8111.694, 'text': 'And so strap in and see you all next time.', 'start': 8108.406, 'duration': 3.288}, {'end': 8113.259, 'text': 'See you.', 'start': 8112.978, 'duration': 0.281}], 'summary': 'Researching with positive results, exploring code and building from python standards. goal: recreate and explore new directions.', 'duration': 82.132, 'max_score': 8031.127, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls8031127.jpg'}], 'start': 7274.916, 'title': 'Time steps, noising schedule, and optimization in diffusion models', 'summary': 'Covers the concept of time steps, noising schedule, and their comparison to deep learning optimizers, as well as the transition to optimizers in diffusion models, emphasizing the use of learning rates and potential research directions, with early positive results in efficiency and outputs.', 'chapters': [{'end': 7786.822, 'start': 7274.916, 'title': 'Understanding time steps and noising schedule', 'summary': 'Explains the concept of time steps and the use of a noising schedule to determine the amount of noise to be added to images during training, and also draws comparisons to deep learning optimizers.', 'duration': 511.906, 'highlights': ['The concept of time steps and the use of a noising schedule to determine the amount of noise to be added to images during training', 'Comparisons to deep learning optimizers and the use of tricks such as momentum and Adam']}, {'end': 8113.259, 'start': 7790.694, 'title': 'Maths and optimization in diffusion models', 'summary': 'Discusses the transition from differential equations to optimizers in diffusion models, highlighting the exploration of using learning rates and optimizers for better results, and the potential for novel research directions, with early positive results in terms of efficiency and outputs.', 'duration': 322.565, 'highlights': ['The transition from differential equations to optimizers in diffusion models is being explored, showing early positive results in terms of efficiency and outputs.', 'The idea of using learning rates and optimizers instead of differential equations is being considered, with early results suggesting improved performance.', 'Novel research directions are being pursued in the realm of diffusion models, with the potential for exploring new approaches and achieving positive outcomes.']}], 'duration': 838.343, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/_7rMfsA24Ls/pics/_7rMfsA24Ls7274916.jpg', 'highlights': ['Transition from differential equations to optimizers in diffusion models showing early positive results', 'Exploration of using learning rates and optimizers instead of differential equations with improved performance', 'Pursuing novel research directions in diffusion models for exploring new approaches and achieving positive outcomes', 'Concept of time steps and use of noising schedule to determine noise amount during training', 'Comparisons to deep learning optimizers and use of tricks such as momentum and Adam']}], 'highlights': ['Recent developments have reduced the number of steps for stable diffusion from a thousand to just four, making it 256 times faster.', 'The U-Net model denoises images, approximating unnoisy images (relevance: 5)', 'The web service returns probabilities for images, such as 0.98 for one image and 0.4 for another, indicating the likelihood that they are handwritten digits.', 'The significance of foundational knowledge and its role in research and model development', 'The ability to build and run models using DALI 2 and stable diffusion', "The challenge of creating effective prompts for AI art is emphasized, noting the difficulty in determining what to write and the significance of learning from others' prompts.", 'Using gradients to modify pixels can improve the resemblance of a handwritten digit, resulting in a higher probability of recognition, with an example probability of 0.2.', 'The lesson is structured as the first of part two, called Deep Learning Foundations to Stable Diffusion, which follows the initial eight lessons of part one.', 'The foundational aspects of stable diffusion remain constant over time, allowing individuals to keep up with the research despite rapid advancements.', 'The chapter introduces Part 2 of the Practical Deep Learning for Coders series, focusing on generative model-y fun things and deep learning understanding, not necessarily essential for practical applications.']}