title
But what is a neural network? | Chapter 1, Deep learning

description
What are the neurons, why are there layers, and what is the math underlying it? Help fund future projects: https://www.patreon.com/3blue1brown Written/interactive form of this series: https://www.3blue1brown.com/topics/neural-networks Additional funding for this project provided by Amplify Partners Typo correction: At 14 minutes 45 seconds, the last index on the bias vector is n, when it's supposed to in fact be a k. Thanks for the sharp eyes that caught that! For those who want to learn more, I highly recommend the book by Michael Nielsen introducing neural networks and deep learning: https://goo.gl/Zmczdy There are two neat things about this book. First, it's available for free, so consider joining me in making a donation Nielsen's way if you get something out of it. And second, it's centered around walking through some code and data which you can download yourself, and which covers the same example that I introduce in this video. Yay for active learning! https://github.com/mnielsen/neural-networks-and-deep-learning I also highly recommend Chris Olah's blog: http://colah.github.io/ For more videos, Welch Labs also has some great series on machine learning: https://youtu.be/i8D90DkCLhI https://youtu.be/bxe2T-V8XRs For those of you looking to go *even* deeper, check out the text "Deep Learning" by Goodfellow, Bengio, and Courville. Also, the publication Distill is just utterly beautiful: https://distill.pub/ Lion photo by Kevin Pluck ----------------- Timeline: 0:00 - Introduction example 1:07 - Series preview 2:42 - What are neurons? 3:35 - Introducing layers 5:31 - Why layers? 8:38 - Edge detection example 11:34 - Counting weights and biases 12:30 - How learning relates 13:26 - Notation and linear algebra 15:17 - Recap 16:27 - Some final words 17:03 - ReLU vs Sigmoid Correction 14:45 - The final index on the bias vector should be "k" ------------------ Animations largely made using manim, a scrappy open source python library. https://github.com/3b1b/manim If you want to check it out, I feel compelled to warn you that it's not the most well-documented tool, and has many other quirks you might expect in a library someone wrote with only their own use in mind. Music by Vincent Rubinetti. Download the music on Bandcamp: https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown Stream the music on Spotify: https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u If you want to contribute translated subtitles or to help review those that have already been made by others and need approval, you can click the gear icon in the video and go to subtitles/cc, then "add subtitles/cc". I really appreciate those who do this, as it helps make the lessons accessible to more people. ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that). If you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended Various social media stuffs: Website: https://www.3blue1brown.com Twitter: https://twitter.com/3Blue1Brown Patreon: https://patreon.com/3blue1brown Facebook: https://www.facebook.com/3blue1brown Reddit: https://www.reddit.com/r/3Blue1Brown

detail
{'title': 'But what is a neural network? | Chapter 1, Deep learning', 'heatmap': [{'end': 661.877, 'start': 503.979, 'weight': 0.771}, {'value': 0.7343514405119909, 'end_time': 661.877, 'start_time': 589.897}], 'summary': 'Delves into the significance of machine learning and neural networks for present and future, discussing the use of a neural network with 784 input neurons, 10 output neurons, and two hidden layers with 16 neurons each for recognizing handwritten digits, and the role of almost 13,000 parameters, weighted sum, biases, and activation functions in pattern recognition.', 'chapters': [{'end': 148.421, 'segs': [{'end': 36.821, 'src': 'embed', 'start': 4.378, 'weight': 0, 'content': [{'end': 5.939, 'text': 'This is a 3.', 'start': 4.378, 'duration': 1.561}, {'end': 14.265, 'text': "It's sloppily written and rendered at an extremely low resolution of 28x28 pixels, but your brain has no trouble recognizing it as a 3.", 'start': 5.939, 'duration': 8.326}, {'end': 19.088, 'text': 'And I want you to take a moment to appreciate how crazy it is that brains can do this so effortlessly.', 'start': 14.265, 'duration': 4.823}, {'end': 28.215, 'text': 'I mean, this, this, and this are also recognizable as 3s, even though the specific values of each pixel is very different from one image to the next.', 'start': 19.669, 'duration': 8.546}, {'end': 36.821, 'text': 'The particular light-sensitive cells in your eye that are firing when you see these three are very different from the ones firing when you see these three.', 'start': 28.795, 'duration': 8.026}], 'summary': 'Brains recognize 3s at 28x28 pixels effortlessly.', 'duration': 32.443, 'max_score': 4.378, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk4378.jpg'}, {'end': 94.362, 'src': 'embed', 'start': 60.063, 'weight': 1, 'content': [{'end': 66.151, 'text': 'telling you what it thinks the digit is well, the task goes from comically trivial to dauntingly difficult.', 'start': 60.063, 'duration': 6.088}, {'end': 68.649, 'text': "Unless you've been living under a rock,", 'start': 67.168, 'duration': 1.481}, {'end': 74.632, 'text': 'I think I hardly need to motivate the relevance and importance of machine learning and neural networks to the present and to the future.', 'start': 68.649, 'duration': 5.983}, {'end': 82.015, 'text': "But what I want to do here is show you what a neural network actually is, assuming no background, and to help visualize what it's doing.", 'start': 75.072, 'duration': 6.943}, {'end': 84.397, 'text': 'Not as a buzzword, but as a piece of math.', 'start': 82.436, 'duration': 1.961}, {'end': 94.362, 'text': 'My hope is just that you come away feeling like the structure itself is motivated and to feel like you know what it means when you read or you hear about a neural network quote-unquote learning.', 'start': 84.977, 'duration': 9.385}], 'summary': "Neural network's importance explained with basic overview.", 'duration': 34.299, 'max_score': 60.063, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk60063.jpg'}], 'start': 4.378, 'title': 'Neural networks', 'summary': "Delves into the comparison between brain's recognition of handwritten digits and computer programming, highlighting the significance of machine learning and neural networks for the present and future.", 'chapters': [{'end': 148.421, 'start': 4.378, 'title': 'Neural networks explained', 'summary': 'Discusses the fascinating ability of the brain to effortlessly recognize handwritten digits, contrasts it with the difficulty of programming a computer to do the same, and emphasizes the relevance and importance of machine learning and neural networks to the present and future.', 'duration': 144.043, 'highlights': ['The brain effortlessly recognizes handwritten digits, despite variations in pixel values and light-sensitive cells, while programming a computer to do the same is dauntingly difficult.', 'Emphasizes the relevance and importance of machine learning and neural networks to the present and future.', 'Introducing the structure of a neural network and its visualization, aiming to help the audience understand it as a piece of math rather than just a buzzword.']}], 'duration': 144.043, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk4378.jpg', 'highlights': ['The brain effortlessly recognizes handwritten digits despite variations in pixel values and light-sensitive cells.', 'Emphasizes the relevance and importance of machine learning and neural networks to the present and future.', 'Introducing the structure of a neural network and its visualization, aiming to help the audience understand it as a piece of math rather than just a buzzword.']}, {'end': 510.421, 'segs': [{'end': 177.758, 'src': 'embed', 'start': 149.082, 'weight': 3, 'content': [{'end': 156.244, 'text': 'But even in this simplest form, it can learn to recognize handwritten digits, which is a pretty cool thing for a computer to be able to do.', 'start': 149.082, 'duration': 7.162}, {'end': 161.926, 'text': "And at the same time, you'll see how it does fall short of a couple hopes that we might have for it.", 'start': 157.525, 'duration': 4.401}, {'end': 166.808, 'text': 'As the name suggests, neural networks are inspired by the brain.', 'start': 163.607, 'duration': 3.201}, {'end': 168.429, 'text': "But let's break that down.", 'start': 167.388, 'duration': 1.041}, {'end': 171.61, 'text': 'What are the neurons and in what sense are they linked together?', 'start': 168.889, 'duration': 2.721}, {'end': 177.758, 'text': 'Right now, when I say neuron, all I want you to think about is a thing that holds a number.', 'start': 172.472, 'duration': 5.286}], 'summary': 'Neural networks can recognize handwritten digits, but fall short in some areas.', 'duration': 28.676, 'max_score': 149.082, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk149082.jpg'}, {'end': 268.196, 'src': 'embed', 'start': 194.665, 'weight': 0, 'content': [{'end': 200.95, 'text': 'Each one of these holds a number that represents the grayscale value of the corresponding pixel,', 'start': 194.665, 'duration': 6.285}, {'end': 204.313, 'text': 'ranging from 0 for black pixels up to 1 for white pixels.', 'start': 200.95, 'duration': 3.363}, {'end': 208.444, 'text': 'This number inside the neuron is called its activation.', 'start': 205.302, 'duration': 3.142}, {'end': 213.967, 'text': 'And the image you might have in mind here is that each neuron is lit up when its activation is a high number.', 'start': 209.084, 'duration': 4.883}, {'end': 221.572, 'text': 'So all of these 784 neurons make up the first layer of our network.', 'start': 216.689, 'duration': 4.883}, {'end': 231.302, 'text': 'Now jumping over to the last layer, this has 10 neurons, each representing one of the digits.', 'start': 226.397, 'duration': 4.905}, {'end': 242.093, 'text': 'The activation in these neurons represents how much the system thinks a given image corresponds with a given digit.', 'start': 232.003, 'duration': 10.09}, {'end': 248.124, 'text': "There's also a couple layers in between, called the hidden layers, which, for the time being,", 'start': 242.881, 'duration': 5.243}, {'end': 253.547, 'text': 'should just be a giant question mark for how on earth this process of recognizing digits is going to be handled.', 'start': 248.124, 'duration': 5.423}, {'end': 257.769, 'text': 'In this network, I chose two hidden layers, each one with 16 neurons.', 'start': 254.248, 'duration': 3.521}, {'end': 260.512, 'text': "And admittedly, that's kind of an arbitrary choice.", 'start': 258.23, 'duration': 2.282}, {'end': 264.774, 'text': 'To be honest, I chose two layers based on how I want to motivate the structure in just a moment.', 'start': 261.012, 'duration': 3.762}, {'end': 268.196, 'text': 'And 16, well, that was just a nice number to fit on the screen.', 'start': 265.375, 'duration': 2.821}], 'summary': 'Neural network with 784 input neurons, 2 hidden layers with 16 neurons each, and 10 output neurons for digit recognition.', 'duration': 73.531, 'max_score': 194.665, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk194665.jpg'}, {'end': 510.421, 'src': 'embed', 'start': 473.123, 'weight': 4, 'content': [{'end': 480.227, 'text': 'Moreover, you can imagine how being able to detect edges and patterns like this would be really useful for other image recognition tasks.', 'start': 473.123, 'duration': 7.104}, {'end': 487.272, 'text': 'And even beyond image recognition, there are all sorts of intelligent things you might want to do that break down into layers of abstraction.', 'start': 480.928, 'duration': 6.344}, {'end': 495.457, 'text': 'Parsing speech, for example, involves taking raw audio and picking out distinct sounds, which combine to make certain syllables,', 'start': 487.892, 'duration': 7.565}, {'end': 499.88, 'text': 'which combine to form words, which combine to make up phrases and more abstract thoughts, etc.', 'start': 495.457, 'duration': 4.423}, {'end': 503.979, 'text': 'But getting back to how any of this actually works,', 'start': 501.158, 'duration': 2.821}, {'end': 510.421, 'text': 'picture yourself designing how exactly the activations in one layer might determine the activations in the next.', 'start': 503.979, 'duration': 6.442}], 'summary': 'Detecting edges and patterns for image recognition has broader intelligent applications.', 'duration': 37.298, 'max_score': 473.123, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk473123.jpg'}], 'start': 149.082, 'title': 'Neural network for pattern recognition', 'summary': 'Discusses a neural network with 784 input neurons, 10 output neurons, and two hidden layers with 16 neurons each for recognizing handwritten digits, as well as its layered structure for pattern recognition, including processes like recognizing subcomponents, detecting edges and patterns, and potential applications for image recognition and speech parsing.', 'chapters': [{'end': 391.533, 'start': 149.082, 'title': 'Neural network for recognizing handwritten digits', 'summary': 'Discusses the structure of a neural network that recognizes handwritten digits, comprising 784 input neurons, 10 output neurons, and two hidden layers with 16 neurons each, highlighting the process of pattern recognition and the expectations from the middle layers.', 'duration': 242.451, 'highlights': ["The network consists of 784 input neurons representing the grayscale values of a 28x28 image, and 10 output neurons representing the system's recognition of each digit.", "The 2 hidden layers, each with 16 neurons, play a crucial role in recognizing specific subcomponents of digits, demonstrating the layered structure's potential for intelligent behavior.", 'The activation of neurons in the second to last layer is expected to correspond with specific subcomponents of digits, allowing for pattern recognition and digit identification.']}, {'end': 510.421, 'start': 392.406, 'title': 'Layered structure for pattern recognition', 'summary': 'Discusses the layered structure of neural networks for pattern recognition, highlighting the process of recognizing subcomponents, detecting edges and patterns, and the potential applications for image recognition and speech parsing.', 'duration': 118.015, 'highlights': ['The layered structure of neural networks is discussed, emphasizing the process of recognizing subcomponents and how they correspond to specific digits, paving the way for recognizing patterns and edges.', 'The potential applications of detecting edges and patterns in image recognition tasks and other intelligent activities are highlighted, underscoring the usefulness of layered abstraction in various tasks.', 'The process of parsing speech is mentioned, illustrating the breakdown of raw audio into distinct sounds, syllables, words, and phrases, showcasing the layered abstraction in speech recognition.']}], 'duration': 361.339, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk149082.jpg', 'highlights': ["The network consists of 784 input neurons representing the grayscale values of a 28x28 image, and 10 output neurons representing the system's recognition of each digit.", "The 2 hidden layers, each with 16 neurons, play a crucial role in recognizing specific subcomponents of digits, demonstrating the layered structure's potential for intelligent behavior.", 'The activation of neurons in the second to last layer is expected to correspond with specific subcomponents of digits, allowing for pattern recognition and digit identification.', 'The layered structure of neural networks is discussed, emphasizing the process of recognizing subcomponents and how they correspond to specific digits, paving the way for recognizing patterns and edges.', 'The potential applications of detecting edges and patterns in image recognition tasks and other intelligent activities are highlighted, underscoring the usefulness of layered abstraction in various tasks.', 'The process of parsing speech is mentioned, illustrating the breakdown of raw audio into distinct sounds, syllables, words, and phrases, showcasing the layered abstraction in speech recognition.']}, {'end': 1105.806, 'segs': [{'end': 557.674, 'src': 'embed', 'start': 511.101, 'weight': 2, 'content': [{'end': 518.943, 'text': 'The goal is to have some mechanism that could conceivably combine pixels into edges, or edges into patterns, or patterns into digits.', 'start': 511.101, 'duration': 7.842}, {'end': 522.624, 'text': "And to zoom in on one very specific example, let's say,", 'start': 519.384, 'duration': 3.24}, {'end': 530.367, 'text': 'the hope is for one particular neuron in the second layer to pick up on whether or not the image has an edge in this region here.', 'start': 522.624, 'duration': 7.743}, {'end': 535.059, 'text': 'The question at hand is what parameters should the network have?', 'start': 531.394, 'duration': 3.665}, {'end': 547.575, 'text': "What dials and knobs should you be able to tweak so that it's expressive enough to potentially capture this pattern or any other pixel pattern or the pattern that several edges can make a loop and other such things?", 'start': 535.639, 'duration': 11.936}, {'end': 555.531, 'text': "What we'll do is assign a weight to each one of the connections between our neuron and the neurons from the first layer.", 'start': 548.783, 'duration': 6.748}, {'end': 557.674, 'text': 'These weights are just numbers.', 'start': 556.272, 'duration': 1.402}], 'summary': 'Goal: develop a mechanism to combine pixels into edges, patterns, and digits using parameter tweaking and connection weights.', 'duration': 46.573, 'max_score': 511.101, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk511101.jpg'}, {'end': 759.133, 'src': 'embed', 'start': 732.5, 'weight': 0, 'content': [{'end': 737.203, 'text': 'The connections between the other layers also have a bunch of weights and biases associated with them.', 'start': 732.5, 'duration': 4.703}, {'end': 745.108, 'text': 'All said and done, this network has almost exactly 13, 000 total weights and biases.', 'start': 738.345, 'duration': 6.763}, {'end': 749.83, 'text': '13, 000 knobs and dials that can be tweaked and turned to make this network behave in different ways.', 'start': 745.128, 'duration': 4.702}, {'end': 759.133, 'text': "So, when we talk about learning, what that's referring to is getting the computer to find a valid setting for all of these many, many numbers,", 'start': 750.95, 'duration': 8.183}], 'summary': 'The network has almost exactly 13,000 total weights and biases, offering numerous knobs and dials for behavior modification.', 'duration': 26.633, 'max_score': 732.5, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk732500.jpg'}, {'end': 921.413, 'src': 'embed', 'start': 895.874, 'weight': 4, 'content': [{'end': 900.617, 'text': 'So, once you write down this weight matrix and these vectors as their own symbols,', 'start': 895.874, 'duration': 4.743}, {'end': 907.402, 'text': 'you can communicate the full transition of activations from one layer to the next in an extremely tight and neat little expression.', 'start': 900.617, 'duration': 6.785}, {'end': 915.608, 'text': 'And this makes the relevant code both a lot simpler and a lot faster, since many libraries optimize the heck out of matrix multiplication.', 'start': 908.243, 'duration': 7.365}, {'end': 921.413, 'text': 'Remember how earlier I said these neurons are simply things that hold numbers?', 'start': 917.731, 'duration': 3.682}], 'summary': 'Representing activations with weight matrix and vectors simplifies and speeds up code, leveraging matrix multiplication optimization.', 'duration': 25.539, 'max_score': 895.874, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk895874.jpg'}], 'start': 511.101, 'title': 'Neural network concepts', 'summary': 'Covers neural network pattern recognition for detecting edges in images and discusses the role of almost 13,000 parameters, weighted sum, biases, and activation functions, emphasizing the use of linear algebra in representing connections between layers.', 'chapters': [{'end': 557.674, 'start': 511.101, 'title': 'Neural network pattern recognition', 'summary': 'Discusses the goal of developing a mechanism to combine pixels into higher-level patterns, focusing on a specific example of detecting edges in an image and addressing the parameters and weights necessary for the network to capture these patterns.', 'duration': 46.573, 'highlights': ['Assigning weights to connections between neurons and the first layer is crucial for capturing specific patterns, such as detecting edges in images.', 'The goal is to develop a mechanism that can combine pixels into higher-level patterns, such as edges, patterns, and digits, by adjusting the parameters and weights of the network.', 'The focus is on enabling a particular neuron in the second layer to identify the presence of edges in specific regions of an image.']}, {'end': 1105.806, 'start': 558.575, 'title': 'Neural network weights and biases', 'summary': 'Explains the concept of weighted sum, biases, and activation functions in a neural network, which involves almost 13,000 parameters, and the role of linear algebra in representing the connections between layers.', 'duration': 547.231, 'highlights': ['The network involves almost exactly 13,000 total weights and biases, which are adjusted during learning to solve the problem at hand.', 'The weights tell what pixel pattern the neuron is picking up on, and the bias determines how high the weighted sum needs to be for the neuron to start being active.', 'Representation of connections between layers using matrix-vector multiplication leads to a tight and neat expression, making the relevant code simpler and faster.', 'The entire network consists of 13,000 parameters in the form of weights and biases, designed to recognize patterns and iterate over matrix-vector products and the sigmoid function.', 'Modern networks tend to use ReLU function instead of sigmoid due to its ease of training and effectiveness for deep neural networks.']}], 'duration': 594.705, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/aircAruvnKk/pics/aircAruvnKk511101.jpg', 'highlights': ['The network involves almost exactly 13,000 total weights and biases', 'The entire network consists of 13,000 parameters in the form of weights and biases', 'Assigning weights to connections between neurons and the first layer is crucial for capturing specific patterns', 'The focus is on enabling a particular neuron in the second layer to identify the presence of edges in specific regions of an image', 'Representation of connections between layers using matrix-vector multiplication leads to a tight and neat expression']}], 'highlights': ['The network involves almost exactly 13,000 total weights and biases', 'The entire network consists of 13,000 parameters in the form of weights and biases', "The network consists of 784 input neurons representing the grayscale values of a 28x28 image, and 10 output neurons representing the system's recognition of each digit", "The 2 hidden layers, each with 16 neurons, play a crucial role in recognizing specific subcomponents of digits, demonstrating the layered structure's potential for intelligent behavior", 'The brain effortlessly recognizes handwritten digits despite variations in pixel values and light-sensitive cells', 'Emphasizes the relevance and importance of machine learning and neural networks to the present and future', 'Introducing the structure of a neural network and its visualization, aiming to help the audience understand it as a piece of math rather than just a buzzword', 'Assigning weights to connections between neurons and the first layer is crucial for capturing specific patterns', 'The activation of neurons in the second to last layer is expected to correspond with specific subcomponents of digits, allowing for pattern recognition and digit identification', 'The layered structure of neural networks is discussed, emphasizing the process of recognizing subcomponents and how they correspond to specific digits, paving the way for recognizing patterns and edges', 'The potential applications of detecting edges and patterns in image recognition tasks and other intelligent activities are highlighted, underscoring the usefulness of layered abstraction in various tasks', 'The process of parsing speech is mentioned, illustrating the breakdown of raw audio into distinct sounds, syllables, words, and phrases, showcasing the layered abstraction in speech recognition', 'Representation of connections between layers using matrix-vector multiplication leads to a tight and neat expression']}