title
Neural Networks Part 8: Image Classification with Convolutional Neural Networks (CNNs)

description
One of the coolest things that Neural Networks can do is classify images, and this is often done with a type of Neural Network called a Convolutional Neural Network (or CNN for short). In this StatQuest, we walk through how Convolutional Neural Networks work, one step at a time, and highlight the main ideas behind filters and pooling. NOTE: This StatQuest assumes that you are already familiar with... The main ideas behind neural networks: https://youtu.be/CqOfi41LfDw The main ideas behind backpropagation: https://youtu.be/IN2XmBhILt4 Neural networks with multiple inputs and outputs: https://youtu.be/aObJUevCVDc For a complete index of all the StatQuest videos, check out: https://statquest.org/video-index/ If you'd like to support StatQuest, please consider... Buying my book, The StatQuest Illustrated Guide to Machine Learning: PDF - https://statquest.gumroad.com/l/wvtmc Paperback - https://www.amazon.com/dp/B09ZCKR4H6 Kindle eBook - https://www.amazon.com/dp/B09ZG79HXC Patreon: https://www.patreon.com/statquest ...or... YouTube Membership: https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join ...a cool StatQuest t-shirt or sweatshirt: https://shop.spreadshirt.com/statquest-with-josh-starmer/ ...buying one or two of my songs (or go large and get a whole album!) https://joshuastarmer.bandcamp.com/ ...or just donating to StatQuest! https://www.paypal.me/statquest Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter: https://twitter.com/joshuastarmer 0:00 Awesome song and introduction 1:51 Image classification with a normal Neural Network 4:28 The main ideas of Convolutional Neural Networks 4:59 Creating a Feature Map with a Filter 7:58 Pooling 9:48 Using the Pooled values as input for a Neural Network 11:29 Classifying an image of the letter "X" 13:04 Classifying a shifted image of the letter "X" #StatQuest #NeuralNetworks #Convolution

detail
{'title': 'Neural Networks Part 8: Image Classification with Convolutional Neural Networks (CNNs)', 'heatmap': [{'end': 388.473, 'start': 290.593, 'weight': 0.785}, {'value': 0.7281251027298686, 'end_time': 388.473, 'start_time': 353.462}, {'end': 466.768, 'start': 431.32, 'weight': 0.743}, {'end': 502.341, 'start': 468.092, 'weight': 0.88}, {'end': 541.629, 'start': 503.482, 'weight': 0.714}, {'end': 686.849, 'start': 660.737, 'weight': 0.718}], 'summary': 'Explores applying convolutional neural networks for image classification, reducing input nodes, tolerating shifts in pixel positions, utilizing pixel correlations, max pooling, relu activation, achieving consistent results in classifying letters o and x, and interpreting the output using softmax or argmax functions.', 'chapters': [{'end': 472.735, 'segs': [{'end': 388.473, 'src': 'heatmap', 'start': 253.251, 'weight': 0, 'content': [{'end': 258.394, 'text': 'And it might be helpful if we can take advantage of the correlation that exists among each pixel.', 'start': 253.251, 'duration': 5.143}, {'end': 267.702, 'text': 'Thus, classification of large and complicated images is usually done using something called a convolutional neural network.', 'start': 259.696, 'duration': 8.006}, {'end': 273.867, 'text': 'Convolutional neural networks do three things to make image classification practical.', 'start': 268.823, 'duration': 5.044}, {'end': 278.069, 'text': 'One, they reduce the number of input nodes.', 'start': 275.328, 'duration': 2.741}, {'end': 283.37, 'text': 'Two, they tolerate small shifts in where the pixels are in the image.', 'start': 279.309, 'duration': 4.061}, {'end': 289.352, 'text': 'And three, take advantage of the correlations that we observe in complex images.', 'start': 284.671, 'duration': 4.681}, {'end': 298.355, 'text': "So let's go back to our tic-tac-toe game and see how a convolutional neural network can recognize this letter O.", 'start': 290.593, 'duration': 7.762}, {'end': 305.718, 'text': 'The first thing a convolutional neural network does is apply a filter to the input image.', 'start': 299.755, 'duration': 5.963}, {'end': 313.963, 'text': 'In convolutional neural networks, a filter is just a smaller square that is commonly 3 pixels by 3 pixels.', 'start': 306.839, 'duration': 7.124}, {'end': 319.887, 'text': 'And the intensity of each pixel in the filter is determined by back propagation.', 'start': 315.184, 'duration': 4.703}, {'end': 327.371, 'text': 'In other words, before training a convolutional neural network, we start with random pixel values.', 'start': 321.327, 'duration': 6.044}, {'end': 333.146, 'text': 'And after training with backpropagation, we end up with something more useful.', 'start': 328.723, 'duration': 4.423}, {'end': 339.311, 'text': 'To apply the filter to the input image, we overlay the filter onto the image.', 'start': 334.547, 'duration': 4.764}, {'end': 343.314, 'text': 'And then we multiply together each overlapping pixel.', 'start': 340.392, 'duration': 2.922}, {'end': 353.462, 'text': 'And then we add each product together to get a final value, which in this case is 3.', 'start': 345.235, 'duration': 8.227}, {'end': 361.897, 'text': "Oh no! It's the dreaded terminology alert! In fancy math lingo, we call this sum of products a dot product.", 'start': 353.462, 'duration': 8.435}, {'end': 370.022, 'text': 'By computing the dot product between the input and the filter, we can say that the filter is convolved with the input,', 'start': 363.058, 'duration': 6.964}, {'end': 373.365, 'text': "and that's what gives convolutional neural networks their name.", 'start': 370.022, 'duration': 3.343}, {'end': 375.006, 'text': 'Small bam.', 'start': 374.245, 'duration': 0.761}, {'end': 382.991, 'text': 'Now we add a bias term to the output of the filter and put the final value into something called a feature map.', 'start': 376.247, 'duration': 6.744}, {'end': 388.473, 'text': 'Now, in this example, we slide the filter over one pixel.', 'start': 384.631, 'duration': 3.842}], 'summary': 'Convolutional neural networks reduce input nodes and utilize correlations in images for practical classification.', 'duration': 52.467, 'max_score': 253.251, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I253251.jpg'}, {'end': 388.473, 'src': 'embed', 'start': 363.058, 'weight': 3, 'content': [{'end': 370.022, 'text': 'By computing the dot product between the input and the filter, we can say that the filter is convolved with the input,', 'start': 363.058, 'duration': 6.964}, {'end': 373.365, 'text': "and that's what gives convolutional neural networks their name.", 'start': 370.022, 'duration': 3.343}, {'end': 375.006, 'text': 'Small bam.', 'start': 374.245, 'duration': 0.761}, {'end': 382.991, 'text': 'Now we add a bias term to the output of the filter and put the final value into something called a feature map.', 'start': 376.247, 'duration': 6.744}, {'end': 388.473, 'text': 'Now, in this example, we slide the filter over one pixel.', 'start': 384.631, 'duration': 3.842}], 'summary': 'Convolutional neural networks use dot product for filter input, resulting in feature maps.', 'duration': 25.415, 'max_score': 363.058, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I363058.jpg'}, {'end': 466.768, 'src': 'heatmap', 'start': 416.791, 'weight': 4, 'content': [{'end': 419.633, 'text': 'Bip, bip, bip, bip, bip, bip, bip.', 'start': 416.791, 'duration': 2.842}, {'end': 423.395, 'text': 'Bam! We filled up the whole feature map.', 'start': 420.614, 'duration': 2.781}, {'end': 430.82, 'text': 'To summarize, we started with an input image of the letter O and then applied a filter to it.', 'start': 424.636, 'duration': 6.184}, {'end': 440.446, 'text': 'In other words, we convolved the filter with the input and added a bias term to the values and that gave us a feature map.', 'start': 431.32, 'duration': 9.126}, {'end': 447.711, 'text': 'Because each cell in the feature map corresponds to a group of neighboring pixels,', 'start': 441.806, 'duration': 5.905}, {'end': 452.335, 'text': 'the feature map helps take advantage of any correlations there might be in the image.', 'start': 447.711, 'duration': 4.624}, {'end': 460.002, 'text': 'Bam! Now, typically we run the feature map through a relu activation function.', 'start': 453.376, 'duration': 6.626}, {'end': 466.768, 'text': 'And that means that all of the negative values are set to zero and the positive values are the same as before.', 'start': 460.883, 'duration': 5.885}], 'summary': 'Applied filter to input image, obtained feature map, utilized relu activation function.', 'duration': 30.92, 'max_score': 416.791, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I416791.jpg'}], 'start': 0.249, 'title': 'Image classification with cnn', 'summary': 'Explores the application of convolutional neural networks for image classification, focusing on reducing input nodes, tolerating small shifts in pixel positions, and utilizing pixel correlations in complex images.', 'chapters': [{'end': 472.735, 'start': 0.249, 'title': 'Image classification with cnn', 'summary': 'Discusses the application of convolutional neural networks for image classification, highlighting the reduction of input nodes, tolerance for small shifts in pixel positions, and the utilization of pixel correlations in complex images.', 'duration': 472.486, 'highlights': ['Convolutional neural networks reduce the number of input nodes, making it more practical to classify large and complicated images.', 'They tolerate small shifts in where the pixels are in the image, ensuring robust performance even with slight pixel position changes.', 'CNNs take advantage of the correlations observed in complex images, allowing for efficient recognition and classification.', 'By applying a filter to the input image, CNNs convolve the filter with the input, add a bias term, and generate a feature map, aiding in recognizing image correlations.', 'Feature maps help take advantage of any correlations in the image by corresponding to groups of neighboring pixels, further enhancing image classification.']}], 'duration': 472.486, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I249.jpg', 'highlights': ['CNNs reduce input nodes, practical for classifying large images.', 'CNNs tolerate small shifts in pixel positions for robust performance.', 'CNNs utilize correlations in complex images for efficient recognition.', 'Applying a filter to input image, CNNs generate feature maps aiding in recognizing image correlations.', 'Feature maps correspond to groups of neighboring pixels, enhancing image classification.']}, {'end': 676.843, 'segs': [{'end': 610.831, 'src': 'heatmap', 'start': 503.482, 'weight': 0, 'content': [{'end': 508.706, 'text': "To see the effect that max pooling has in this example, let's go back to the input image.", 'start': 503.482, 'duration': 5.224}, {'end': 515.412, 'text': 'Here, we see that the upper left hand corner in the input image has an exact match with the filter.', 'start': 509.607, 'duration': 5.805}, {'end': 519.996, 'text': 'And this results in the highest possible value in the feature map.', 'start': 516.653, 'duration': 3.343}, {'end': 528.338, 'text': 'And then the Max Pooling step reduced this region to the one spot where the filter did the best job matching the input image.', 'start': 521.092, 'duration': 7.246}, {'end': 535.063, 'text': 'In other words, this region in the input image corresponds to this part of the feature map.', 'start': 529.679, 'duration': 5.384}, {'end': 541.629, 'text': 'And the Max Pooling step selected the spot where the filter did the best job matching the input image.', 'start': 536.164, 'duration': 5.465}, {'end': 548.134, 'text': 'Likewise, this region in the input image corresponds to this part of the feature map.', 'start': 543.11, 'duration': 5.024}, {'end': 554.897, 'text': 'And the max pooling steps selected the spot where the filter did the best job matching the input image.', 'start': 549.335, 'duration': 5.562}, {'end': 561.778, 'text': 'So we see that max pooling selects the spots where the filter did the best job matching the input image.', 'start': 556.117, 'duration': 5.661}, {'end': 566.16, 'text': "Oh no, it's another terminology alert.", 'start': 563.259, 'duration': 2.901}, {'end': 574.402, 'text': 'Alternatively, we could calculate the average value for each region and that would be called average or mean pooling.', 'start': 567.08, 'duration': 7.322}, {'end': 581.838, 'text': "Now, going back to the max pooled layer, let's move and shrink things to give us more room.", 'start': 575.754, 'duration': 6.084}, {'end': 587.401, 'text': "Now let's convert the pooled layer into a column of input nodes.", 'start': 583.219, 'duration': 4.182}, {'end': 594.206, 'text': "Lastly, let's plug the input nodes into a normal, everyday neural network.", 'start': 589.102, 'duration': 5.104}, {'end': 606.629, 'text': 'This neural network has four input nodes, a single hidden layer with a single node using the ReLU activation function and two output nodes,', 'start': 595.346, 'duration': 11.283}, {'end': 610.831, 'text': 'one for the letter O and one for the letter X.', 'start': 606.629, 'duration': 4.202}], 'summary': 'Max pooling selects best matching spots, leading to 4 input nodes, 1 hidden node, and 2 output nodes in the neural network.', 'duration': 81.152, 'max_score': 503.482, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I503482.jpg'}, {'end': 676.843, 'src': 'embed', 'start': 640.007, 'weight': 3, 'content': [{'end': 644.588, 'text': 'and thus the x-axis coordinate for the activation function is 0.34..', 'start': 640.007, 'duration': 4.581}, {'end': 660.737, 'text': 'Now we plug 0.34 into the relu activation function and the output is 0.34 because 0.34 is greater than 0.', 'start': 644.588, 'duration': 16.149}, {'end': 669.22, 'text': 'Now this connection from the hidden layer to the output for the letter O gives us 1 for the letter O,', 'start': 660.737, 'duration': 8.483}, {'end': 676.843, 'text': 'and the connection from the hidden layer to the output for the letter X gives us 0 for the letter X.', 'start': 669.22, 'duration': 7.623}], 'summary': 'Relu activation function yields 0.34 output for x-coordinate, resulting in 1 for letter o and 0 for letter x.', 'duration': 36.836, 'max_score': 640.007, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I640007.jpg'}], 'start': 474.016, 'title': 'Cnn process and max pooling', 'summary': 'Discusses the process of cnn, including converting pooled layers into input nodes, running feature maps through relu activation function, and obtaining specific connections, and also explains the concept of max pooling in cnn, highlighting the selection of maximum value in each region and the alternative approach of average pooling.', 'chapters': [{'end': 581.838, 'start': 474.016, 'title': 'Max pooling in cnn', 'summary': 'Explains the concept of max pooling in a convolutional neural network, where the maximum value in each region is selected, reducing the region to the spot where the filter best matches the input image, and also discusses the alternative approach of average pooling.', 'duration': 107.822, 'highlights': ['Max pooling selects the spots where the filter did the best job matching the input image, resulting in a reduction of the region to the spot where the filter best matches the input image.', 'The alternative approach of average pooling involves calculating the average value for each region, providing an alternative to max pooling.']}, {'end': 676.843, 'start': 583.219, 'title': 'Convolutional neural network process', 'summary': 'Explains the process of converting a pooled layer into input nodes, running the feature map through a relu activation function, and obtaining the x-axis coordinate of 0.34, resulting in an output of 0.34 for the relu activation function, with specific connections yielding 1 for the letter o and 0 for the letter x.', 'duration': 93.624, 'highlights': ['The x-axis coordinate for the activation function is 0.34, resulting in an output of 0.34 for the ReLU activation function.', 'The connection from the hidden layer to the output for the letter O gives us 1 for the letter O, and the connection from the hidden layer to the output for the letter X gives us 0 for the letter X.', 'The process involves converting the pooled layer into a column of input nodes and then plugging the input nodes into a neural network with specific architecture.']}], 'duration': 202.827, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I474016.jpg', 'highlights': ['Max pooling selects the spots where the filter did the best job matching the input image, resulting in a reduction of the region to the spot where the filter best matches the input image.', 'The alternative approach of average pooling involves calculating the average value for each region, providing an alternative to max pooling.', 'The process involves converting the pooled layer into a column of input nodes and then plugging the input nodes into a neural network with specific architecture.', 'The x-axis coordinate for the activation function is 0.34, resulting in an output of 0.34 for the ReLU activation function.', 'The connection from the hidden layer to the output for the letter O gives us 1 for the letter O, and the connection from the hidden layer to the output for the letter X gives us 0 for the letter X.']}, {'end': 922.078, 'segs': [{'end': 727.285, 'src': 'embed', 'start': 676.843, 'weight': 4, 'content': [{'end': 686.849, 'text': 'So when the input is a picture of the letter O This convolutional neural network classifies it as a picture of the letter O.', 'start': 676.843, 'duration': 10.006}, {'end': 694.034, 'text': "Bam! Now let's see what happens when the input is a picture of the letter X.", 'start': 686.849, 'duration': 7.185}, {'end': 698.797, 'text': 'Note, even though we have changed the input, the filter is the same as before.', 'start': 694.034, 'duration': 4.763}, {'end': 702.419, 'text': "And we're doing max pooling just like before.", 'start': 699.878, 'duration': 2.541}, {'end': 708.003, 'text': 'And the neural network, with its weights and biases, is the same as before.', 'start': 703.58, 'duration': 4.423}, {'end': 714.715, 'text': 'So just like before we run the filter over the input to create the feature map.', 'start': 709.631, 'duration': 5.084}, {'end': 722.041, 'text': 'Now we run the feature map through the rail U and do max pooling.', 'start': 717.697, 'duration': 4.344}, {'end': 727.285, 'text': 'And the results of max pooling become the values for the input nodes.', 'start': 723.282, 'duration': 4.003}], 'summary': 'Convolutional neural network accurately classifies letter o as input.', 'duration': 50.442, 'max_score': 676.843, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I676843.jpg'}, {'end': 836.986, 'src': 'embed', 'start': 760.197, 'weight': 0, 'content': [{'end': 765.78, 'text': 'In this case, we started with a 6x6 image or 36 potential inputs.', 'start': 760.197, 'duration': 5.583}, {'end': 770.659, 'text': 'and compress those down to just four inputs into the neural network.', 'start': 766.775, 'duration': 3.884}, {'end': 776.524, 'text': 'We also said that convolutional neural networks take correlations into account.', 'start': 771.92, 'duration': 4.604}, {'end': 782.99, 'text': 'And this is accomplished by the filter, which looks at a region of pixels instead of just one at a time.', 'start': 777.305, 'duration': 5.685}, {'end': 790.518, 'text': 'Lastly, we said that convolutional neural networks can tolerate small shifts in where the pixels are in the image.', 'start': 783.931, 'duration': 6.587}, {'end': 796.518, 'text': "So let's see what happens when we shift the picture of the letter X one pixel to the right.", 'start': 791.676, 'duration': 4.842}, {'end': 803.4, 'text': 'In other words, will this convolutional neural network still decide if the input is a picture of the letter X?', 'start': 797.518, 'duration': 5.882}, {'end': 818.208, 'text': 'And we see that the output value for the letter X, 1.23, is much closer to 1 than the output value for the letter O, negative 0.2..', 'start': 803.6, 'duration': 14.608}, {'end': 824.674, 'text': 'So this convolutional neural network decided that the input image is of the letter X.', 'start': 818.208, 'duration': 6.466}, {'end': 836.986, 'text': 'Triple bam! Note, if we wanted to, we can make the output easier to interpret by running it through the softmax function or the argmax function.', 'start': 824.674, 'duration': 12.312}], 'summary': 'Convolutional neural network compresses 6x6 image to 4 inputs, tolerates shifts, and accurately identifies image as letter x.', 'duration': 76.789, 'max_score': 760.197, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I760197.jpg'}], 'start': 676.843, 'title': 'Convolutional neural networks', 'summary': 'Demonstrates a convolutional neural network classifying the letters o and x, achieving consistent results, reducing input nodes, taking correlations into account, and achieving a 1.23 output value for the letter x. it also explores interpreting the output using the softmax or argmax function.', 'chapters': [{'end': 727.285, 'start': 676.843, 'title': 'Convolutional neural network in action', 'summary': 'Illustrates how a convolutional neural network classifies a picture of the letter o and x using the same filter and max pooling technique, demonstrating consistent results.', 'duration': 50.442, 'highlights': ['The neural network classifies a picture of the letter O as a picture of the letter O using a convolutional neural network.', 'The input of a picture of the letter X results in consistent classification using the same filter and max pooling technique.', "The filter and max pooling technique remain unchanged for different inputs, showcasing the network's consistent behavior."]}, {'end': 922.078, 'start': 728.866, 'title': 'Convolutional neural networks', 'summary': 'Explains how a simple convolutional neural network reduces input nodes, takes correlations into account, and tolerates small shifts in image pixels, achieving a 1.23 output value for the letter x. it also mentions the possibility of interpreting the output using the softmax or argmax function.', 'duration': 193.212, 'highlights': ['The convolutional neural network achieved an output value of 1.23 for the letter X, indicating its successful classification.', 'The network reduced a 6x6 image with 36 potential inputs to just four inputs, showcasing its ability to decrease the number of inputs.', 'The filter in the network looks at a region of pixels instead of just one at a time, thus taking correlations into account.', 'The chapter also mentions the possibility of interpreting the output using the softmax or argmax function to make it easier to understand.', 'The network demonstrated tolerance to small shifts in the image pixels, maintaining its ability to classify the letter X even when shifted one pixel to the right.']}], 'duration': 245.235, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/HGwBXDKFk9I/pics/HGwBXDKFk9I676843.jpg', 'highlights': ['The network achieved an output value of 1.23 for the letter X, indicating successful classification.', 'The network reduced a 6x6 image with 36 potential inputs to just four inputs, showcasing its ability to decrease the number of inputs.', 'The filter in the network looks at a region of pixels instead of just one at a time, taking correlations into account.', 'The network demonstrated tolerance to small shifts in the image pixels, maintaining its ability to classify the letter X even when shifted one pixel to the right.', 'The neural network classifies a picture of the letter O as a picture of the letter O using a convolutional neural network.', 'The input of a picture of the letter X results in consistent classification using the same filter and max pooling technique.', "The filter and max pooling technique remain unchanged for different inputs, showcasing the network's consistent behavior.", 'The chapter also mentions the possibility of interpreting the output using the softmax or argmax function to make it easier to understand.']}], 'highlights': ['CNNs reduce input nodes, practical for classifying large images.', 'CNNs tolerate small shifts in pixel positions for robust performance.', 'CNNs utilize correlations in complex images for efficient recognition.', 'Applying a filter to input image, CNNs generate feature maps aiding in recognizing image correlations.', 'Feature maps correspond to groups of neighboring pixels, enhancing image classification.', 'The network achieved an output value of 1.23 for the letter X, indicating successful classification.', 'The network reduced a 6x6 image with 36 potential inputs to just four inputs, showcasing its ability to decrease the number of inputs.', 'The filter in the network looks at a region of pixels instead of just one at a time, taking correlations into account.', 'The network demonstrated tolerance to small shifts in the image pixels, maintaining its ability to classify the letter X even when shifted one pixel to the right.', 'The neural network classifies a picture of the letter O as a picture of the letter O using a convolutional neural network.', 'The input of a picture of the letter X results in consistent classification using the same filter and max pooling technique.', "The filter and max pooling technique remain unchanged for different inputs, showcasing the network's consistent behavior.", 'Max pooling selects the spots where the filter did the best job matching the input image, resulting in a reduction of the region to the spot where the filter best matches the input image.', 'The alternative approach of average pooling involves calculating the average value for each region, providing an alternative to max pooling.', 'The process involves converting the pooled layer into a column of input nodes and then plugging the input nodes into a neural network with specific architecture.', 'The x-axis coordinate for the activation function is 0.34, resulting in an output of 0.34 for the ReLU activation function.', 'The connection from the hidden layer to the output for the letter O gives us 1 for the letter O, and the connection from the hidden layer to the output for the letter X gives us 0 for the letter X.', 'The chapter also mentions the possibility of interpreting the output using the softmax or argmax function to make it easier to understand.']}