title
Building makemore Part 2: MLP

description
We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part2_mlp.ipynb - collab notebook (new)!!!: https://colab.research.google.com/drive/1YIfmkftLrz6MPTOO9Vwqrop2Q5llHIGK?usp=sharing - Bengio et al. 2003 MLP language model paper (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Useful links: - PyTorch internals ref http://blog.ezyang.com/2019/05/pytorch-internals/ Exercises: - E01: Tune the hyperparameters of the training to beat my best validation loss of 2.2 - E02: I was not careful with the intialization of the network in this video. (1) What is the loss you'd get if the predicted probabilities at initialization were perfectly uniform? What loss do we achieve? (2) Can you tune the initialization to get a starting loss that is much more similar to (1)? - E03: Read the Bengio et al 2003 paper (link above), implement and try any idea from the paper. Did it work? Chapters: 00:00:00 intro 00:01:48 Bengio et al. 2003 (MLP language model) paper walkthrough 00:09:03 (re-)building our training dataset 00:12:19 implementing the embedding lookup table 00:18:35 implementing the hidden layer + internals of torch.Tensor: storage, views 00:29:15 implementing the output layer 00:29:53 implementing the negative log likelihood loss 00:32:17 summary of the full network 00:32:49 introducing F.cross_entropy and why 00:37:56 implementing the training loop, overfitting one batch 00:41:25 training on the full dataset, minibatches 00:45:40 finding a good initial learning rate 00:53:20 splitting up the dataset into train/val/test splits and why 01:00:49 experiment: larger hidden layer 01:05:27 visualizing the character embeddings 01:07:16 experiment: larger embedding size 01:11:46 summary of our final code, conclusion 01:13:24 sampling from the model 01:14:55 google collab (new!!) notebook advertisement

detail
{'title': 'Building makemore Part 2: MLP', 'heatmap': [{'end': 776.475, 'start': 726.821, 'weight': 0.708}, {'end': 1135.445, 'start': 1040.855, 'weight': 0.715}, {'end': 4534.202, 'start': 4495.68, 'weight': 0.831}], 'summary': 'Covers implementing a multilayer perceptron model to overcome limitations of bigram language models, building neural network language models with 17,000-word vocabulary and 30-dimensional embedding matrices, creating a dataset for a neural network, efficient tensor operations, optimization techniques, mini-batch gradient descent, optimizing learning rates, and neural network training for text generation with a best validation loss of 2.17 after 200,000 steps.', 'chapters': [{'end': 163.043, 'segs': [{'end': 89.449, 'src': 'embed', 'start': 63.572, 'weight': 2, 'content': [{'end': 70.46, 'text': 'things quickly blow up and this table, the size of this table, grows, and in fact it grows exponentially with the length of the context.', 'start': 63.572, 'duration': 6.888}, {'end': 75.286, 'text': "Because if we only take a single character at a time, that's 27 possibilities of context.", 'start': 71.345, 'duration': 3.941}, {'end': 79.867, 'text': 'But if we take two characters in the past and try to predict the third one,', 'start': 76.106, 'duration': 3.761}, {'end': 84.988, 'text': 'suddenly the number of rows in this matrix you can look at it that way is 27 times 27..', 'start': 79.867, 'duration': 5.121}, {'end': 89.449, 'text': "So there's 729 possibilities for what could have come in the context.", 'start': 84.988, 'duration': 4.461}], 'summary': 'Context size grows exponentially, from 27 to 729 possibilities.', 'duration': 25.877, 'max_score': 63.572, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I63572.jpg'}, {'end': 146.338, 'src': 'embed', 'start': 109.177, 'weight': 0, 'content': [{'end': 111.94, 'text': "So that's why today we're going to move on to this bullet point here.", 'start': 109.177, 'duration': 2.763}, {'end': 117.704, 'text': "And we're going to implement a multilayer perceptron model to predict the next character in a sequence.", 'start': 112.3, 'duration': 5.404}, {'end': 124.505, 'text': "And this modeling approach that we're going to adopt follows this paper, Benjou et al, 2003.", 'start': 118.445, 'duration': 6.06}, {'end': 125.706, 'text': 'So I have the paper pulled up here.', 'start': 124.505, 'duration': 1.201}, {'end': 134.731, 'text': "Now, this isn't the very first paper that proposed the use of multilayer perceptrons or neural networks to predict the next character or token in a sequence,", 'start': 126.526, 'duration': 8.205}, {'end': 137.873, 'text': "but it's definitely one that was very influential around that time.", 'start': 134.731, 'duration': 3.142}, {'end': 142.395, 'text': "It is very often cited to stand in for this idea, and I think it's a very nice write-up.", 'start': 138.213, 'duration': 4.182}, {'end': 146.338, 'text': "And so this is the paper that we're going to first look at and then implement.", 'start': 142.696, 'duration': 3.642}], 'summary': 'Implementing a multilayer perceptron model to predict next character, following influential paper by benjou et al, 2003.', 'duration': 37.161, 'max_score': 109.177, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I109177.jpg'}], 'start': 0.799, 'title': 'Implementing multilayer perceptron model', 'summary': 'Discusses the limitations of the bigram language model and introduces the implementation of a multilayer perceptron model to predict the next character in a sequence, following a paper by benjou et al, 2003, due to the exponential growth in the size of the context table when using the bigram model.', 'chapters': [{'end': 163.043, 'start': 0.799, 'title': 'Implementing multilayer perceptron model for predicting sequences', 'summary': 'Discusses the limitations of the bigram language model and introduces the implementation of a multilayer perceptron model to predict the next character in a sequence, following a paper by benjou et al, 2003, due to the exponential growth in the size of the context table when using the bigram model.', 'duration': 162.244, 'highlights': ["The bigram language model's limitations are discussed due to the exponential growth of the context table, with 729 possibilities for three characters as the context, leading to the implementation of a multilayer perceptron model.", 'The implementation of a multilayer perceptron model is introduced, following the influential paper by Benjou et al, 2003, as a solution to the limitations of the bigram language model.', 'The paper by Benjou et al, 2003, is highlighted as influential in proposing the use of multilayer perceptrons or neural networks to predict the next character or token in a sequence.']}], 'duration': 162.244, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I799.jpg', 'highlights': ['The implementation of a multilayer perceptron model is introduced, following the influential paper by Benjou et al, 2003, as a solution to the limitations of the bigram language model.', 'The paper by Benjou et al, 2003, is highlighted as influential in proposing the use of multilayer perceptrons or neural networks to predict the next character or token in a sequence.', "The bigram language model's limitations are discussed due to the exponential growth of the context table, with 729 possibilities for three characters as the context, leading to the implementation of a multilayer perceptron model."]}, {'end': 549.867, 'segs': [{'end': 206.452, 'src': 'embed', 'start': 181.468, 'weight': 0, 'content': [{'end': 190.152, 'text': "Now what they do is basically, they propose to take every one of these words 17, 000 words and they're going to associate to each word a, say,", 'start': 181.468, 'duration': 8.684}, {'end': 191.733, 'text': '30-dimensional feature vector.', 'start': 190.152, 'duration': 1.581}, {'end': 197.687, 'text': 'So every word is now embedded into a 30-dimensional space.', 'start': 192.864, 'duration': 4.823}, {'end': 198.668, 'text': 'You can think of it that way.', 'start': 197.827, 'duration': 0.841}, {'end': 203.31, 'text': 'So we have 17, 000 points or vectors in a 30-dimensional space.', 'start': 199.308, 'duration': 4.002}, {'end': 206.452, 'text': "And you might imagine that's very crowded.", 'start': 204.051, 'duration': 2.401}], 'summary': '17,000 words mapped to 30-dimensional vectors for embedding.', 'duration': 24.984, 'max_score': 181.468, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I181468.jpg'}, {'end': 280.806, 'src': 'embed', 'start': 242.755, 'weight': 1, 'content': [{'end': 248.878, 'text': 'They are using a multilayer neural network to predict the next word given the previous words, and to train the neural network,', 'start': 242.755, 'duration': 6.123}, {'end': 252.159, 'text': 'they are maximizing the log likelihood of the training data, just like we did.', 'start': 248.878, 'duration': 3.281}, {'end': 255.281, 'text': 'So the modeling approach itself is identical.', 'start': 253, 'duration': 2.281}, {'end': 257.742, 'text': 'Now here they have a concrete example of this intuition.', 'start': 255.821, 'duration': 1.921}, {'end': 264.805, 'text': 'Why does it work? Basically, suppose that, for example, you are trying to predict a dog was running in a blank.', 'start': 258.982, 'duration': 5.823}, {'end': 271.302, 'text': 'Now suppose that the exact phrase a dog was running in A has never occurred in the training data.', 'start': 265.799, 'duration': 5.503}, {'end': 274.503, 'text': 'And here you are at sort of test time later,', 'start': 271.902, 'duration': 2.601}, {'end': 280.806, 'text': "when the model is deployed somewhere and it's trying to make a sentence and it's saying a dog was running in A blank.", 'start': 274.503, 'duration': 6.303}], 'summary': 'Using a neural network to predict next words with log likelihood maximization, identical to our approach.', 'duration': 38.051, 'max_score': 242.755, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I242755.jpg'}, {'end': 473.557, 'src': 'embed', 'start': 446.26, 'weight': 3, 'content': [{'end': 454.985, 'text': 'So say there were a hundred neurons here, all of them would be fully connected to the 90 words or 90 numbers that make up these three words.', 'start': 446.26, 'duration': 8.725}, {'end': 457.307, 'text': 'So this is a fully connected layer.', 'start': 456.006, 'duration': 1.301}, {'end': 461.569, 'text': "Then there's a 10-inch long linearity, and then there's this output layer.", 'start': 457.327, 'duration': 4.242}, {'end': 466.052, 'text': 'And because there are 17, 000 possible words that could come next.', 'start': 462.15, 'duration': 3.902}, {'end': 473.557, 'text': 'this layer has 17, 000 neurons and all of them are fully connected to all of these neurons in the hidden layer.', 'start': 466.052, 'duration': 7.505}], 'summary': 'Neural network has 100 neurons fully connected to 90 words or numbers, with 17,000 neurons in the output layer.', 'duration': 27.297, 'max_score': 446.26, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I446260.jpg'}], 'start': 164.079, 'title': 'Building neural network language models', 'summary': 'Discusses building character-level and word-level language models with 17,000-word vocabulary and 30-dimensional embedding matrices, using backpropagation for training and achieving improved prediction accuracy and performance.', 'chapters': [{'end': 225.341, 'start': 164.079, 'title': 'Character-level language model', 'summary': 'Discusses building a character-level language model with a vocabulary of 17,000 words, each embedded into a 30-dimensional space, which are then tuned using backpropagation during neural network training.', 'duration': 61.262, 'highlights': ['They propose to associate each of the 17,000 words with a 30-dimensional feature vector, which are initially randomly spread out but then tuned using backpropagation during neural network training.', 'The vocabulary consists of 17,000 possible words, embedded into a 30-dimensional space, and during training, these embedded points or vectors move around in this space.']}, {'end': 345.108, 'start': 226.001, 'title': 'Neural network for word prediction', 'summary': 'Discusses the use of multilayer neural networks for word prediction, explaining how the model leverages embedding spaces to generalize and transfer knowledge, ultimately improving prediction accuracy and performance.', 'duration': 119.107, 'highlights': ['The modeling approach involves using a multilayer neural network to predict the next word given the previous words, maximizing the log likelihood of the training data.', 'The approach allows for generalization by leveraging embedding spaces, where similar words or phrases are placed nearby each other, enabling knowledge transfer and improved prediction in novel scenarios.', 'The example illustrates how the model can predict words in phrases not encountered in the training data by transferring knowledge through the embedding space, enabling accurate word prediction in out-of-distribution scenarios.']}, {'end': 549.867, 'start': 345.108, 'title': 'Neural network word prediction', 'summary': 'Explains the process of using a neural network to predict the next word in a sequence, utilizing a 17,000-word vocabulary, a 30-dimensional embedding matrix, and a fully connected output layer with 17,000 neurons, optimized using backpropagation.', 'duration': 204.759, 'highlights': ['The neural network utilizes a vocabulary of 17,000 words, with a 30-dimensional embedding matrix to convert each word index into a corresponding vector.', 'The fully connected output layer consists of 17,000 neurons, with parameters optimized using backpropagation.', "The size of the hidden neural layer is a hyperparameter and can be adjusted based on design choices, potentially impacting the network's performance."]}], 'duration': 385.788, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I164079.jpg', 'highlights': ['The vocabulary consists of 17,000 possible words, embedded into a 30-dimensional space, and during training, these embedded points or vectors move around in this space.', 'The modeling approach involves using a multilayer neural network to predict the next word given the previous words, maximizing the log likelihood of the training data.', 'The neural network utilizes a vocabulary of 17,000 words, with a 30-dimensional embedding matrix to convert each word index into a corresponding vector.', 'The fully connected output layer consists of 17,000 neurons, with parameters optimized using backpropagation.', 'The example illustrates how the model can predict words in phrases not encountered in the training data by transferring knowledge through the embedding space, enabling accurate word prediction in out-of-distribution scenarios.']}, {'end': 1543.278, 'segs': [{'end': 776.475, 'src': 'heatmap', 'start': 713.596, 'weight': 0, 'content': [{'end': 721.219, 'text': 'From these five words, we have created a dataset of 32 examples, and each input to the neural net is three integers,', 'start': 713.596, 'duration': 7.623}, {'end': 724.78, 'text': 'and we have a label that is also an integer y.', 'start': 721.219, 'duration': 3.561}, {'end': 726.06, 'text': 'So x looks like this.', 'start': 724.78, 'duration': 1.28}, {'end': 730.502, 'text': 'These are the individual examples, and then y are the labels.', 'start': 726.821, 'duration': 3.681}, {'end': 739.029, 'text': "So given this, let's now write a neural network that takes these Xs and predicts the Ys.", 'start': 732.566, 'duration': 6.463}, {'end': 743.311, 'text': "First, let's build the embedding lookup table C.", 'start': 739.389, 'duration': 3.922}, {'end': 747.694, 'text': "So we have 27 possible characters and we're going to embed them in a lower dimensional space.", 'start': 743.311, 'duration': 4.383}, {'end': 754.857, 'text': 'In the paper, they have 17, 000 words and they embed them in spaces as small dimensional as 30.', 'start': 748.534, 'duration': 6.323}, {'end': 760.045, 'text': 'So they crammed 17, 000 words words into 30 dimensional space.', 'start': 754.857, 'duration': 5.188}, {'end': 763.027, 'text': 'In our case, we have only 27 possible characters.', 'start': 760.665, 'duration': 2.362}, {'end': 767.75, 'text': "So let's cram them in something as small as to start with, for example, a two dimensional space.", 'start': 763.547, 'duration': 4.203}, {'end': 776.475, 'text': "So this lookup table will be random numbers and we'll have 27 rows and we'll have two columns right?", 'start': 768.75, 'duration': 7.725}], 'summary': 'Created dataset of 32 examples with 3 integer inputs and 1 integer label, building neural network for prediction.', 'duration': 29.715, 'max_score': 713.596, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I713596.jpg'}, {'end': 797.364, 'src': 'embed', 'start': 763.547, 'weight': 2, 'content': [{'end': 767.75, 'text': "So let's cram them in something as small as to start with, for example, a two dimensional space.", 'start': 763.547, 'duration': 4.203}, {'end': 776.475, 'text': "So this lookup table will be random numbers and we'll have 27 rows and we'll have two columns right?", 'start': 768.75, 'duration': 7.725}, {'end': 780.778, 'text': 'So each 20, each one of 27 characters will have a two dimensional embedding.', 'start': 776.495, 'duration': 4.283}, {'end': 787.321, 'text': "So that's our matrix C of embeddings in the beginning, initialized randomly.", 'start': 782.219, 'duration': 5.102}, {'end': 797.364, 'text': 'Now, before we embed all of the integers inside the input X using this lookup table C, let me actually just try to embed a single individual integer,', 'start': 788.021, 'duration': 9.343}], 'summary': 'Using a 2d space with 27 rows and 2 columns to embed integers.', 'duration': 33.817, 'max_score': 763.547, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I763547.jpg'}, {'end': 1011.492, 'src': 'embed', 'start': 980.781, 'weight': 1, 'content': [{'end': 983.762, 'text': 'We can simply ask PyTorch to retrieve the fifth row of C.', 'start': 980.781, 'duration': 2.981}, {'end': 987.903, 'text': 'or the row index 5 of C.', 'start': 984.662, 'duration': 3.241}, {'end': 997.647, 'text': 'But how do we simultaneously embed all of these 32 by 3 integers stored in array X? Luckily, PyTorch indexing is fairly flexible and quite powerful.', 'start': 987.903, 'duration': 9.744}, {'end': 1003.949, 'text': "So it doesn't just work to ask for a single element 5 like this.", 'start': 998.287, 'duration': 5.662}, {'end': 1006.03, 'text': 'You can actually index using lists.', 'start': 1004.569, 'duration': 1.461}, {'end': 1011.492, 'text': 'So for example, we can get the rows 5, 6, and 7, and this will just work like this.', 'start': 1006.47, 'duration': 5.022}], 'summary': 'Pytorch indexing is powerful, allowing retrieval of multiple rows at once.', 'duration': 30.711, 'max_score': 980.781, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I980781.jpg'}, {'end': 1135.445, 'src': 'heatmap', 'start': 1040.855, 'weight': 0.715, 'content': [{'end': 1044.857, 'text': 'but it turns out that you can also index with multi-dimensional tensors of integers.', 'start': 1040.855, 'duration': 4.002}, {'end': 1048.137, 'text': 'Here we have a two-dimensional tensor of integers.', 'start': 1045.297, 'duration': 2.84}, {'end': 1053.259, 'text': 'So we can simply just do C at X, and this just works.', 'start': 1048.758, 'duration': 4.501}, {'end': 1059.681, 'text': 'And the shape of this is 32 by three, which is the original shape.', 'start': 1054.82, 'duration': 4.861}, {'end': 1064.743, 'text': "And now for every one of those 32 by three integers, we've retrieved the embedding vector here.", 'start': 1060.242, 'duration': 4.501}, {'end': 1077.754, 'text': 'So basically we have that as an example, the 13th or example index 13, the second dimension is the integer one as an example.', 'start': 1066.092, 'duration': 11.662}, {'end': 1089.936, 'text': 'And so here, if we do C of X, which gives us that array, and then we index into 13 by two of that array, then we get the embedding here.', 'start': 1078.874, 'duration': 11.062}, {'end': 1098.324, 'text': 'And you can verify that C at one, which is the integer at that location, is indeed equal to this.', 'start': 1090.656, 'duration': 7.668}, {'end': 1100.685, 'text': "You see they're equal.", 'start': 1100.045, 'duration': 0.64}, {'end': 1111.51, 'text': 'So, basically, long story short, PyTorch indexing is awesome and to embed simultaneously all of the integers in X, we can simply do C of X,', 'start': 1101.865, 'duration': 9.645}, {'end': 1114.331, 'text': 'and that is our embedding, and that just works.', 'start': 1111.51, 'duration': 2.821}, {'end': 1117.772, 'text': "Now let's construct this layer here, the hidden layer.", 'start': 1115.211, 'duration': 2.561}, {'end': 1125.3, 'text': "So we have that W1, as I'll call it, are these weights, which we will initialize randomly.", 'start': 1118.993, 'duration': 6.307}, {'end': 1133.164, 'text': 'Now the number of inputs to this layer is going to be three times two, right? Because we have two-dimensional embeddings and we have three of them.', 'start': 1126.18, 'duration': 6.984}, {'end': 1135.445, 'text': 'So the number of inputs is six.', 'start': 1133.884, 'duration': 1.561}], 'summary': 'Pytorch indexing allows embedding of integers, with 32x3 shape and 6 inputs for hidden layer.', 'duration': 94.59, 'max_score': 1040.855, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1040855.jpg'}, {'end': 1193.844, 'src': 'embed', 'start': 1162.455, 'weight': 3, 'content': [{'end': 1163.676, 'text': 'This is roughly what we want to do.', 'start': 1162.455, 'duration': 1.221}, {'end': 1169.22, 'text': 'But the problem here is that these embeddings are stacked up in the dimensions of this input tensor.', 'start': 1164.637, 'duration': 4.583}, {'end': 1177.316, 'text': "So this will not work, this matrix multiplication, because this is a shape 32 by three by two and I can't multiply that by six by 100.", 'start': 1170.069, 'duration': 7.247}, {'end': 1184.182, 'text': 'So somehow we need to concatenate these inputs here together so that we can do something along these lines, which currently does not work.', 'start': 1177.316, 'duration': 6.866}, {'end': 1193.844, 'text': 'So how do we transform this 32 by three by two into a 32 by six, so that we can actually perform this multiplication over here?', 'start': 1185.273, 'duration': 8.571}], 'summary': 'Embeddings need concatenation to perform matrix multiplication for 32x6 transformation.', 'duration': 31.389, 'max_score': 1162.455, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1162455.jpg'}, {'end': 1330.79, 'src': 'embed', 'start': 1303.77, 'weight': 4, 'content': [{'end': 1310.274, 'text': 'torch.cat takes a sequence of tensors and then we have to tell it along which dimension to concatenate.', 'start': 1303.77, 'duration': 6.504}, {'end': 1317.479, 'text': 'So in this case, all of these are 32 by two and we want to concatenate not across dimension zero but across dimension one.', 'start': 1311.334, 'duration': 6.145}, {'end': 1324.725, 'text': "So passing in one gives us a result that the shape of this is 32 by six, exactly as we'd like.", 'start': 1318.38, 'duration': 6.345}, {'end': 1330.79, 'text': 'So that basically took 32 and squashed these by concatenating them into 32 by six.', 'start': 1325.566, 'duration': 5.224}], 'summary': 'Torch.cat concatenates tensors along dimension one to get 32 by six shape.', 'duration': 27.02, 'max_score': 1303.77, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1303770.jpg'}, {'end': 1478.071, 'src': 'embed', 'start': 1452.057, 'weight': 5, 'content': [{'end': 1457.28, 'text': 'As long as the total number of elements here multiply to be the same, this will just work.', 'start': 1452.057, 'duration': 5.223}, {'end': 1463.404, 'text': 'And in PyTorch, this operation calling that view is extremely efficient.', 'start': 1458.321, 'duration': 5.083}, {'end': 1469.128, 'text': "And the reason for that is that in each tensor, there's something called the underlying storage.", 'start': 1464.225, 'duration': 4.903}, {'end': 1474.507, 'text': 'And the storage is just the numbers always as a one-dimensional vector.', 'start': 1470.663, 'duration': 3.844}, {'end': 1478.071, 'text': 'And this is how this tensor is represented in the computer memory.', 'start': 1474.927, 'duration': 3.144}], 'summary': "Pytorch's efficient view operation utilizes underlying storage for one-dimensional vector representation.", 'duration': 26.014, 'max_score': 1452.057, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1452057.jpg'}], 'start': 550.968, 'title': 'Neural network dataset creation and pytorch indexing', 'summary': 'Covers creating a dataset for a neural network with 32 examples and x input, and explains pytorch indexing with a 32x3 array and a weight matrix of 3x100.', 'chapters': [{'end': 914.885, 'start': 550.968, 'title': 'Neural network dataset creation', 'summary': 'Covers the creation of a dataset for a neural network, explaining the process of building the vocabulary, compiling the dataset, defining block size, and embedding lookup table c, with a dataset of 32 examples and x input as three integers for the neural net.', 'duration': 363.917, 'highlights': ['The dataset consists of 32 examples, with each input to the neural net being three integers and having a label that is also an integer y.', 'Building the vocabulary of characters and all the mappings from the characters as strings to integers and vice versa.', 'Explaining the process of defining block size as the context length of how many characters to predict the next one, and building out the X and Y for the neural net.', 'Creating a neural network that takes Xs and predicts the Ys, including building the embedding lookup table C with 27 possible characters and embedding them in a lower dimensional space.']}, {'end': 1162.214, 'start': 915.225, 'title': 'Pytorch indexing and embedding', 'summary': 'Explains pytorch indexing, demonstrating how to simultaneously embed multiple integers using two-dimensional tensors and construct a hidden layer with 100 neurons, based on a 32x3 array and a weight matrix of 3x100.', 'duration': 246.989, 'highlights': ['PyTorch indexing allows the simultaneous embedding of all the integers in X by using C of X, leveraging the shape of 32 by three, and demonstrating the ease of this process.', 'The construction of the hidden layer involves initializing W1 as the weights, where the number of inputs is six and the number of neurons in this layer is 100, with randomly initialized biases.', 'PyTorch indexing is flexible, allowing the use of lists and tensors of integers for indexing, as well as the ability to index with multi-dimensional tensors of integers.']}, {'end': 1345.219, 'start': 1162.455, 'title': 'Concatenating embeddings for matrix multiplication', 'summary': 'Explains the process of concatenating embeddings to transform a 32 by three by two shape into a 32 by six shape, enabling matrix multiplication and highlights the usage of torch.cat function to achieve this transformation.', 'duration': 182.764, 'highlights': ['The process of concatenating embeddings to transform a 32 by three by two shape into a 32 by six shape enables matrix multiplication, facilitating efficient data processing.', 'The usage of Torch.cat function allows for the concatenation of tensors along a specified dimension, providing a versatile solution for transforming and manipulating tensors.', 'The extensive functionality of Torch library offers numerous functions for transforming, creating, multiplying, and adding tensors, providing a wide array of possibilities for data manipulation.']}, {'end': 1543.278, 'start': 1345.999, 'title': 'Efficient tensor manipulation in pytorch', 'summary': 'Explains how to efficiently manipulate tensors in pytorch by using functions like unbind and view, enabling quick re-representation of tensors without changing the underlying storage, thus enhancing performance and flexibility.', 'duration': 197.279, 'highlights': ['Calling torch.view efficiently re-represents tensors without changing the underlying storage, allowing for quick manipulation of tensor attributes and interpretation of the one-dimensional sequence, enhancing performance and flexibility.', 'The function torch.unbind efficiently removes a tensor dimension and returns a tuple of all slices on a given dimension, providing a way to manipulate tensors effectively and create lists of tensors with various dimensions.', 'The explanation delves into the internals of Torch.Tensor, highlighting how the view of a tensor is represented and the manipulation of attributes like storage offset, strides, and shapes to interpret the one-dimensional sequence as different n-dimensional arrays, ensuring efficient memory usage and manipulation.']}], 'duration': 992.31, 'thumbnail': '', 'highlights': ['Creating a dataset with 32 examples and 3 integer inputs for a neural network.', 'Explaining PyTorch indexing with a 32x3 array and a weight matrix of 3x100.', 'Building the embedding lookup table C with 27 possible characters for a neural network.', 'Concatenating embeddings to transform a 32x3x2 shape into a 32x6 shape for matrix multiplication.', 'Using Torch.cat function for concatenating tensors along a specified dimension.', 'Calling torch.view efficiently re-represents tensors without changing the underlying storage.']}, {'end': 1955.242, 'segs': [{'end': 1573.394, 'src': 'embed', 'start': 1544.602, 'weight': 0, 'content': [{'end': 1547.424, 'text': 'For here, we just note that this is an extremely efficient operation.', 'start': 1544.602, 'duration': 2.822}, {'end': 1556.79, 'text': 'And if I delete this and come back to our EMB, we see that the shape of our EMB is 32x3x2,', 'start': 1548.284, 'duration': 8.506}, {'end': 1561.533, 'text': 'but we can simply ask for PyTorch to view this instead as a 32x6..', 'start': 1556.79, 'duration': 4.743}, {'end': 1573.394, 'text': 'And the way this gets flattened into a 32x6 array just happens that these two get stacked up in a single row.', 'start': 1563.174, 'duration': 10.22}], 'summary': 'Efficient operation: reshaping 32x3x2 emb to 32x6 array.', 'duration': 28.792, 'max_score': 1544.602, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1544602.jpg'}, {'end': 1640.528, 'src': 'embed', 'start': 1618.936, 'weight': 5, 'content': [{'end': 1630.563, 'text': "we can, for example, do something like um m dot shape at zero so that we don't hard code these numbers, and this would work for any size of this m.", 'start': 1618.936, 'duration': 11.627}, {'end': 1632.664, 'text': 'or, alternatively, we can also do negative one.', 'start': 1630.563, 'duration': 2.101}, {'end': 1640.528, 'text': "when we do negative, one pytorch will infer what this should be, because the number of elements must be the same, and we're saying that this is six.", 'start': 1632.664, 'duration': 7.864}], 'summary': 'Code can be made dynamic using m.dot.shape at zero; pytorch infers negative one as six elements.', 'duration': 21.592, 'max_score': 1618.936, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1618936.jpg'}, {'end': 1689.904, 'src': 'embed', 'start': 1662.478, 'weight': 7, 'content': [{'end': 1667.962, 'text': "So new memory is being created because there's no way to concatenate tensors just by manipulating the view attributes.", 'start': 1662.478, 'duration': 5.484}, {'end': 1671.024, 'text': 'So this is inefficient and creates all kinds of new memory.', 'start': 1668.642, 'duration': 2.382}, {'end': 1674.106, 'text': 'So let me delete this now.', 'start': 1672.725, 'duration': 1.381}, {'end': 1676.268, 'text': "We don't need this.", 'start': 1675.727, 'duration': 0.541}, {'end': 1687.24, 'text': 'And here to calculate h, we want to also dot 10h of this to get our h.', 'start': 1677.406, 'duration': 9.834}, {'end': 1689.904, 'text': 'So these are now numbers between negative one and one because of the 10h.', 'start': 1687.24, 'duration': 2.664}], 'summary': 'Inefficient memory creation due to tensor manipulation, aiming to dot 10h to calculate h.', 'duration': 27.426, 'max_score': 1662.478, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1662478.jpg'}, {'end': 1747.399, 'src': 'embed', 'start': 1715.827, 'weight': 4, 'content': [{'end': 1718.628, 'text': 'So we see that the addition here will broadcast these two.', 'start': 1715.827, 'duration': 2.801}, {'end': 1724.386, 'text': 'And in particular, we have 32 by 100 broadcasting to 100.', 'start': 1719.189, 'duration': 5.197}, {'end': 1729.228, 'text': 'So broadcasting will align on the right, create a fake dimension here,', 'start': 1724.386, 'duration': 4.842}, {'end': 1738.254, 'text': 'so this will become a 1 by 100 row vector and then it will copy vertically for every one of these rows of 32 and do an element-wise addition.', 'start': 1729.228, 'duration': 9.026}, {'end': 1747.399, 'text': 'So in this case, the correcting will be happening because the same bias vector will be added to all the rows of this matrix.', 'start': 1739.034, 'duration': 8.365}], 'summary': 'Broadcasting will align on the right, creating a 1 by 100 row vector and then copy vertically for every one of the rows of 32 to do an element-wise addition.', 'duration': 31.572, 'max_score': 1715.827, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1715827.jpg'}, {'end': 1825.428, 'src': 'embed', 'start': 1765.1, 'weight': 2, 'content': [{'end': 1771.042, 'text': 'and the output number of neurons will be for us 27, because we have 27 possible characters that come next.', 'start': 1765.1, 'duration': 5.942}, {'end': 1773.703, 'text': 'So the biases will be 27 as well.', 'start': 1771.982, 'duration': 1.721}, {'end': 1785.266, 'text': 'So therefore the logits, which are the outputs of this neural net, are going to be H multiplied by W2 plus B2.', 'start': 1775.083, 'duration': 10.183}, {'end': 1795.127, 'text': 'Logits. that shape is 32 by 27, and the logits look Now exactly as we saw in the previous video.', 'start': 1787.267, 'duration': 7.86}, {'end': 1799.469, 'text': 'we want to take these logits and we want to first exponentiate them to get our fake counts.', 'start': 1795.127, 'duration': 4.342}, {'end': 1802.05, 'text': 'And then we want to normalize them into a probability.', 'start': 1800.209, 'duration': 1.841}, {'end': 1813.194, 'text': 'So prob is counts divide and now counts.sum along the first dimension and keep them as true, exactly as in the previous video.', 'start': 1803.09, 'duration': 10.104}, {'end': 1820.237, 'text': 'And so prob.shape now is 32 by 27.', 'start': 1814.575, 'duration': 5.662}, {'end': 1825.428, 'text': "And you'll see that every row of prob, sums to one, so it's normalized.", 'start': 1820.237, 'duration': 5.191}], 'summary': 'Neural net produces 32x27 logits; prob shape is 32x27, normalized to sum to one', 'duration': 60.328, 'max_score': 1765.1, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1765100.jpg'}, {'end': 1955.242, 'src': 'embed', 'start': 1914.011, 'weight': 1, 'content': [{'end': 1917.272, 'text': 'Now, just as in the previous video, we want to take these probabilities.', 'start': 1914.011, 'duration': 3.261}, {'end': 1926.016, 'text': 'we want to look at the log probability and then we want to look at the average log probability and the negative of it to create the negative log likelihood loss.', 'start': 1917.272, 'duration': 8.744}, {'end': 1930.429, 'text': 'So the loss here is 17.', 'start': 1927.628, 'duration': 2.801}, {'end': 1936.493, 'text': "And this is the loss that we'd like to minimize to get the network to predict the correct character in the sequence.", 'start': 1930.429, 'duration': 6.064}, {'end': 1939.934, 'text': 'Okay, so I rewrote everything here and made it a bit more respectable.', 'start': 1936.833, 'duration': 3.101}, {'end': 1941.495, 'text': "So here's our dataset.", 'start': 1940.615, 'duration': 0.88}, {'end': 1943.656, 'text': "Here's all the parameters that we defined.", 'start': 1942.095, 'duration': 1.561}, {'end': 1947.018, 'text': "I'm now using a generator to make it reproducible.", 'start': 1944.717, 'duration': 2.301}, {'end': 1954.162, 'text': "I clustered all the parameters into a single list of parameters so that, for example, it's easy to count them and see that in total,", 'start': 1947.898, 'duration': 6.264}, {'end': 1955.242, 'text': 'we currently have about 3, 400 parameters.', 'start': 1954.162, 'duration': 1.08}], 'summary': 'The loss to minimize for predicting characters is 17; the total parameters are 3,400.', 'duration': 41.231, 'max_score': 1914.011, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1914011.jpg'}], 'start': 1544.602, 'title': 'Efficient tensor operations and neural network training', 'summary': 'Covers efficient tensor concatenation, manipulation using pytorch, and neural network training steps including shape alignment, broadcasting, logits generation, probability normalization, loss calculation, and parameter count totaling 3,400.', 'chapters': [{'end': 1689.904, 'start': 1544.602, 'title': 'Efficient concatenation and tensor manipulation', 'summary': 'Discusses efficient tensor concatenation, tensor manipulation using pytorch, and the implications of using negative one and m dot shape at zero for tensor operations.', 'duration': 145.302, 'highlights': ['Efficient tensor concatenation and manipulation using PyTorch to flatten a 32x3x2 tensor into a 32x6 array, resulting in the same output as the original tensor.', 'Using negative one or m dot shape at zero in PyTorch for tensor operations to infer tensor dimensions, providing flexibility for different tensor sizes.', 'Highlighting the inefficiency of creating new memory when performing tensor concatenation, and the need to avoid this for efficient memory usage.']}, {'end': 1955.242, 'start': 1690.665, 'title': 'Neural network training process', 'summary': 'Explains the steps of creating and training a neural network, with a focus on shape alignment, broadcasting, logits generation, probability normalization, loss calculation, and parameter count totaling 3,400.', 'duration': 264.577, 'highlights': ["The loss is 17, which needs to be minimized to improve the network's predictive accuracy.", "The parameter count totals about 3,400, facilitating a comprehensive understanding of the network's complexity and scale.", 'The probabilities are normalized into a 32 by 27 matrix, ensuring that each row sums to one, thus providing a clear indication of the normalized distribution.', 'The shape of the logits is 32 by 27, representing the outputs of the neural network and its alignment with the 32 examples and 27 possible characters.', 'The broadcasting aligns the 32 by 100 matrix with the 100 bias vector, enabling correct addition across all rows of the matrix.']}], 'duration': 410.64, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1544602.jpg', 'highlights': ['Efficient tensor concatenation and manipulation using PyTorch to flatten a 32x3x2 tensor into a 32x6 array, resulting in the same output as the original tensor.', "The parameter count totals about 3,400, facilitating a comprehensive understanding of the network's complexity and scale.", 'The shape of the logits is 32 by 27, representing the outputs of the neural network and its alignment with the 32 examples and 27 possible characters.', 'The probabilities are normalized into a 32 by 27 matrix, ensuring that each row sums to one, thus providing a clear indication of the normalized distribution.', 'The broadcasting aligns the 32 by 100 matrix with the 100 bias vector, enabling correct addition across all rows of the matrix.', 'Using negative one or m dot shape at zero in PyTorch for tensor operations to infer tensor dimensions, providing flexibility for different tensor sizes.', "The loss is 17, which needs to be minimized to improve the network's predictive accuracy.", 'Highlighting the inefficiency of creating new memory when performing tensor concatenation, and the need to avoid this for efficient memory usage.']}, {'end': 2628.563, 'segs': [{'end': 2009.426, 'src': 'embed', 'start': 1980.087, 'weight': 0, 'content': [{'end': 1984.411, 'text': 'This is just classification and many people use classification,', 'start': 1980.087, 'duration': 4.324}, {'end': 1990.215, 'text': "and that's why there is a functional dot cross entropy function in PyTorch to calculate this much more efficiently.", 'start': 1984.411, 'duration': 5.804}, {'end': 1999.903, 'text': 'So we could just simply call f dot cross entropy and we can pass in the logits and we can pass in the array of targets y and this calculates the exact same loss.', 'start': 1990.936, 'duration': 8.967}, {'end': 2004.447, 'text': 'So in fact, we can simply put this here.', 'start': 2002.425, 'duration': 2.022}, {'end': 2009.426, 'text': "and erase these three lines, and we're going to get the exact same result.", 'start': 2005.481, 'duration': 3.945}], 'summary': 'Pytorch offers efficient cross-entropy function for classification tasks.', 'duration': 29.339, 'max_score': 1980.087, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1980087.jpg'}, {'end': 2278.524, 'src': 'embed', 'start': 2238.365, 'weight': 1, 'content': [{'end': 2244.133, 'text': 'what PyTorch does is it internally calculates the maximum value that occurs in the logits and it subtracts it.', 'start': 2238.365, 'duration': 5.768}, {'end': 2246.075, 'text': 'So in this case, it would subtract five.', 'start': 2244.694, 'duration': 1.381}, {'end': 2252.482, 'text': 'And so therefore the greatest number in logits will become zero and all the other numbers will become some negative numbers.', 'start': 2247.219, 'duration': 5.263}, {'end': 2255.884, 'text': 'And then the result of this is always well-behaved.', 'start': 2253.282, 'duration': 2.602}, {'end': 2262.988, 'text': 'So even if we have 100 here previously, not good, but because PyTorch will subtract 100, this will work.', 'start': 2256.364, 'duration': 6.624}, {'end': 2267.721, 'text': "And so there's many good reasons to call cross-entropy.", 'start': 2264.54, 'duration': 3.181}, {'end': 2270.842, 'text': 'Number one the forward pass can be much more efficient,', 'start': 2268.381, 'duration': 2.461}, {'end': 2276.223, 'text': 'the backward pass can be much more efficient and also things can be much more numerically well-behaved.', 'start': 2270.842, 'duration': 5.381}, {'end': 2278.524, 'text': "Okay, so let's now set up the training of this neural net.", 'start': 2276.663, 'duration': 1.861}], 'summary': 'Pytorch internally subtracts max logits, making forward and backward pass more efficient and numerically well-behaved.', 'duration': 40.159, 'max_score': 2238.365, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I2238365.jpg'}, {'end': 2399.398, 'src': 'embed', 'start': 2376.571, 'weight': 6, 'content': [{'end': 2385.894, 'text': "And therefore, it's very easy to make this neural net fit only these 32 examples because we have 3, 400 parameters and only 32 examples.", 'start': 2376.571, 'duration': 9.323}, {'end': 2392.916, 'text': "So we're doing what's called overfitting a single batch of the data and getting a very low loss and good predictions.", 'start': 2386.234, 'duration': 6.682}, {'end': 2396.737, 'text': "But that's just because we have so many parameters for so few examples.", 'start': 2394.276, 'duration': 2.461}, {'end': 2399.398, 'text': "So it's easy to make this be very low.", 'start': 2396.857, 'duration': 2.541}], 'summary': 'Neural net overfits with 3,400 parameters and 32 examples, resulting in low loss and good predictions.', 'duration': 22.827, 'max_score': 2376.571, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I2376571.jpg'}, {'end': 2564.565, 'src': 'embed', 'start': 2524.599, 'weight': 4, 'content': [{'end': 2528.625, 'text': "Okay, so we started with a fairly high loss and then as we're optimizing, the loss is coming down.", 'start': 2524.599, 'duration': 4.026}, {'end': 2535.549, 'text': "But you'll notice that it takes quite a bit of time for every single iteration.", 'start': 2532.148, 'duration': 3.401}, {'end': 2541.751, 'text': "So let's actually address that because we're doing way too much work forwarding and backwarding 220, 000 examples.", 'start': 2536.009, 'duration': 5.742}, {'end': 2548.933, 'text': 'In practice, what people usually do is they perform forward and backward pass and update on many batches of the data.', 'start': 2542.711, 'duration': 6.222}, {'end': 2555.518, 'text': "So what we will want to do is we want to randomly select some portion of the dataset, and that's a mini batch,", 'start': 2549.793, 'duration': 5.725}, {'end': 2558.32, 'text': 'and then only forward backward and update on that little mini batch.', 'start': 2555.518, 'duration': 2.802}, {'end': 2561.463, 'text': 'And then we iterate on those many batches.', 'start': 2558.941, 'duration': 2.522}, {'end': 2564.565, 'text': 'So in PyTorch, we can, for example, use torch.randint.', 'start': 2562.243, 'duration': 2.322}], 'summary': 'Optimizing to reduce loss, working on mini-batches for efficiency.', 'duration': 39.966, 'max_score': 2524.599, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I2524599.jpg'}], 'start': 1957.301, 'title': 'Neural network efficiency and optimization', 'summary': "Discusses the benefits of using pytorch's functional dot cross entropy for efficiency and simplicity in neural network loss calculation, emphasizing its numerical stability, handling of overflow issues, and impact on optimization with a focus on mini-batch processing.", 'chapters': [{'end': 2100.428, 'start': 1957.301, 'title': 'Neural network efficiency', 'summary': "Discusses the inefficiency of manually implementing loss calculation in neural networks, emphasizing the benefits of using pytorch's functional dot cross entropy function for efficiency and simplicity in both forward and backward passes.", 'duration': 143.127, 'highlights': ["PyTorch's functional dot cross entropy function is more efficient than manually calculating loss, as it avoids creating intermediate tensors and utilizes fused kernels for clustered mathematical operations.", 'The backward pass in neural networks can be made more efficient with functional dot cross entropy, simplifying mathematical expressions and reducing the complexity of implementation.', 'Using functional dot cross entropy in PyTorch enhances efficiency and simplicity in both the forward and backward passes of neural network operations.']}, {'end': 2628.563, 'start': 2100.428, 'title': 'Efficient neural network optimization', 'summary': 'Explains the efficiency and numerical stability of f dot cross entropy, highlighting how pytorch addresses overflow issues and the impact of dataset size on optimization, with a focus on mini-batch processing.', 'duration': 528.135, 'highlights': ['PyTorch addresses overflow issues by internally subtracting the maximum value in logits, ensuring well-behaved results and preventing potential overflow, leading to more efficient forward and backward passes.', 'Optimizing a neural net with a larger dataset (228,000 examples) results in longer optimization time, prompting the use of mini-batch processing to iteratively forward, backward, and update on smaller subsets of the data.', 'Overfitting a neural net on a small dataset (32 examples) leads to a very low loss, but the inability to achieve a loss of exactly zero, as unique inputs for unique outputs prevent complete overfitting.', 'In practice, mini-batch processing involves randomly selecting a portion of the dataset and performing forward, backward, and update operations on that subset, leading to more efficient optimization for larger datasets.']}], 'duration': 671.262, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I1957301.jpg', 'highlights': ['Using functional dot cross entropy in PyTorch enhances efficiency and simplicity in both the forward and backward passes of neural network operations.', 'PyTorch addresses overflow issues by internally subtracting the maximum value in logits, ensuring well-behaved results and preventing potential overflow, leading to more efficient forward and backward passes.', "PyTorch's functional dot cross entropy function is more efficient than manually calculating loss, as it avoids creating intermediate tensors and utilizes fused kernels for clustered mathematical operations.", 'The backward pass in neural networks can be made more efficient with functional dot cross entropy, simplifying mathematical expressions and reducing the complexity of implementation.', 'In practice, mini-batch processing involves randomly selecting a portion of the dataset and performing forward, backward, and update operations on that subset, leading to more efficient optimization for larger datasets.', 'Optimizing a neural net with a larger dataset (228,000 examples) results in longer optimization time, prompting the use of mini-batch processing to iteratively forward, backward, and update on smaller subsets of the data.', 'Overfitting a neural net on a small dataset (32 examples) leads to a very low loss, but the inability to achieve a loss of exactly zero, as unique inputs for unique outputs prevent complete overfitting.']}, {'end': 3353.498, 'segs': [{'end': 2676.137, 'src': 'embed', 'start': 2630.788, 'weight': 0, 'content': [{'end': 2634.032, 'text': 'And now this should be many batches and this should be much, much faster.', 'start': 2630.788, 'duration': 3.244}, {'end': 2636.895, 'text': "So, okay, so it's instant almost.", 'start': 2634.292, 'duration': 2.603}, {'end': 2644.703, 'text': 'So this way we can run many, many examples nearly instantly and decrease the loss much, much faster.', 'start': 2638.116, 'duration': 6.587}, {'end': 2649.92, 'text': "Now because we're only dealing with mini-batches, the quality of our gradient is lower.", 'start': 2645.819, 'duration': 4.101}, {'end': 2652.081, 'text': 'So the direction is not as reliable.', 'start': 2650.281, 'duration': 1.8}, {'end': 2653.962, 'text': "It's not the actual gradient direction.", 'start': 2652.241, 'duration': 1.721}, {'end': 2661.164, 'text': "But the gradient direction is good enough, even when it's estimating on only 32 examples, that it is useful.", 'start': 2654.822, 'duration': 6.342}, {'end': 2670.168, 'text': "And so it's much better to have an approximate gradient and just make more steps than it is to evaluate the exact gradient and take fewer steps.", 'start': 2662.005, 'duration': 8.163}, {'end': 2673.769, 'text': "So that's why in practice this works quite well.", 'start': 2670.808, 'duration': 2.961}, {'end': 2676.137, 'text': "So let's now continue the optimization.", 'start': 2674.676, 'duration': 1.461}], 'summary': 'Using mini-batches enables faster training and approximate gradients, resulting in effective optimization.', 'duration': 45.349, 'max_score': 2630.788, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I2630788.jpg'}, {'end': 2771.949, 'src': 'embed', 'start': 2740.554, 'weight': 2, 'content': [{'end': 2742.895, 'text': "So I'll show you one way to determine a reasonable learning rate.", 'start': 2740.554, 'duration': 2.341}, {'end': 2744.476, 'text': 'It works as follows.', 'start': 2743.735, 'duration': 0.741}, {'end': 2750.238, 'text': "Let's reset our parameters to the initial settings.", 'start': 2744.616, 'duration': 5.622}, {'end': 2759.702, 'text': "And now let's print in every step, but let's only do 10 steps or so, or maybe 100 steps.", 'start': 2751.198, 'duration': 8.504}, {'end': 2765.064, 'text': 'We want to find a very reasonable search range, if you will.', 'start': 2761.141, 'duration': 3.923}, {'end': 2771.949, 'text': 'So for example, if this is very low, then we see that the loss is barely decreasing.', 'start': 2765.444, 'duration': 6.505}], 'summary': 'Demonstrate determining a reasonable learning rate by printing in every step and observing loss decrease over 10-100 steps.', 'duration': 31.395, 'max_score': 2740.554, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I2740554.jpg'}, {'end': 3244.262, 'src': 'embed', 'start': 3200.26, 'weight': 1, 'content': [{'end': 3201.48, 'text': "Now there's something we have to be careful with.", 'start': 3200.26, 'duration': 1.22}, {'end': 3210.084, 'text': 'I said that we have a better model because we are achieving a lower loss 2.3 much lower than 2.45 with the bigram model previously.', 'start': 3202.681, 'duration': 7.403}, {'end': 3219.259, 'text': "Now, that's not exactly true, and the reason that's not true is that This is actually a fairly small model,", 'start': 3210.884, 'duration': 8.375}, {'end': 3222.9, 'text': 'but these models can get larger and larger if you keep adding neurons and parameters.', 'start': 3219.259, 'duration': 3.641}, {'end': 3226.402, 'text': "So you can imagine that we don't potentially have a thousand parameters.", 'start': 3223.561, 'duration': 2.841}, {'end': 3229.323, 'text': 'We could have 10, 000 or 100, 000 or millions of parameters.', 'start': 3226.462, 'duration': 2.861}, {'end': 3236.286, 'text': 'And as the capacity of the neural network grows, it becomes more and more capable of overfitting your training set.', 'start': 3230.043, 'duration': 6.243}, {'end': 3244.262, 'text': "What that means is that the loss on the training set, on the data that you're training on, will become very, very low, as low as zero.", 'start': 3237.076, 'duration': 7.186}], 'summary': 'A small model with lower loss 2.3 than 2.45, but could grow to 100,000+ parameters leading to overfitting.', 'duration': 44.002, 'max_score': 3200.26, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I3200260.jpg'}], 'start': 2630.788, 'title': 'Mini-batch gradient descent and optimizing learning rates for neural networks', 'summary': 'Discusses the benefits of mini-batch gradient descent, enabling faster processing of examples with approximate gradient direction, and the process of optimizing learning rates, achieving a lower loss of 2.3 compared to 2.45 with 3400 parameters in neural networks.', 'chapters': [{'end': 2676.137, 'start': 2630.788, 'title': 'Mini-batch gradient descent', 'summary': 'Discusses the benefits of mini-batch gradient descent, highlighting that it allows for faster processing of many examples, even though the gradient direction is approximate and lower in quality due to dealing with mini-batches, it still proves to be useful, enabling faster decrease in loss and efficient optimization.', 'duration': 45.349, 'highlights': ['Mini-batch processing allows for much faster processing of many examples, nearly instantly, resulting in a faster decrease in loss.', 'Even though dealing with mini-batches leads to lower quality gradient, the approximate gradient direction, estimated on only 32 examples, is still useful for optimization.', 'It is much better to have an approximate gradient and make more steps than to evaluate the exact gradient and take fewer steps, making mini-batch gradient descent a practical and effective approach.']}, {'end': 3353.498, 'start': 2678.517, 'title': 'Optimizing learning rates for neural networks', 'summary': 'Discusses the process of determining a reasonable learning rate, evaluating the loss for a given learning rate, and the importance of model capacity in neural networks, achieving a lower loss of 2.3 compared to 2.45 with 3400 parameters.', 'duration': 674.981, 'highlights': ['The model achieves a lower loss of 2.3 compared to 2.45 with the bigram model, using 3400 parameters.', 'The process of determining a reasonable learning rate involves resetting parameters, printing in every step to find a reasonable search range, and evaluating the losses for different learning rates.', 'The importance of model capacity in neural networks is highlighted, as larger models can overfit the training set and fail to generalize to new data.', 'The standard practice in the field involves splitting the dataset into training, dev or validation, and test sets, using them for parameter optimization, hyperparameter tuning, and model evaluation.']}], 'duration': 722.71, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I2630788.jpg', 'highlights': ['Mini-batch processing allows for much faster processing of many examples, nearly instantly, resulting in a faster decrease in loss.', 'The model achieves a lower loss of 2.3 compared to 2.45 with the bigram model, using 3400 parameters.', 'The process of determining a reasonable learning rate involves resetting parameters, printing in every step to find a reasonable search range, and evaluating the losses for different learning rates.', 'The importance of model capacity in neural networks is highlighted, as larger models can overfit the training set and fail to generalize to new data.', 'Even though dealing with mini-batches leads to lower quality gradient, the approximate gradient direction, estimated on only 32 examples, is still useful for optimization.']}, {'end': 4065.685, 'segs': [{'end': 3383.931, 'src': 'embed', 'start': 3354.278, 'weight': 3, 'content': [{'end': 3358.98, 'text': "So we're only evaluating the loss on the test split very, very sparingly and very few times,", 'start': 3354.278, 'duration': 4.702}, {'end': 3367.524, 'text': 'because every single time you evaluate your test loss and you learn something from it, you are basically starting to also train on the test split.', 'start': 3358.98, 'duration': 8.544}, {'end': 3374.127, 'text': 'So you are only allowed to test the loss on the test set very, very few times.', 'start': 3368.164, 'duration': 5.963}, {'end': 3378.809, 'text': 'Otherwise, you risk overfitting to it as well as you experiment on your model.', 'start': 3374.447, 'duration': 4.362}, {'end': 3383.931, 'text': "So let's also split up our training data into train, dev, and test.", 'start': 3379.729, 'duration': 4.202}], 'summary': 'Evaluate test loss sparingly to avoid overfitting, split data into train, dev, test', 'duration': 29.653, 'max_score': 3354.278, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I3354278.jpg'}, {'end': 3630.24, 'src': 'embed', 'start': 3599.389, 'weight': 2, 'content': [{'end': 3601.17, 'text': "Okay, so we're getting about 2.3 on dev.", 'start': 3599.389, 'duration': 1.781}, {'end': 3605.553, 'text': 'And so the neural network when it was training did not see these dev examples.', 'start': 3601.691, 'duration': 3.862}, {'end': 3606.774, 'text': "It hasn't optimized on them.", 'start': 3605.653, 'duration': 1.121}, {'end': 3611.817, 'text': 'And yet, when we evaluate the loss on these dev, we actually get a pretty decent loss.', 'start': 3607.174, 'duration': 4.643}, {'end': 3617.641, 'text': 'And so we can also look at what the loss is on all of training set.', 'start': 3612.457, 'duration': 5.184}, {'end': 3623.797, 'text': 'Oops And so we see that the training and the dev loss are about equal.', 'start': 3619.145, 'duration': 4.652}, {'end': 3625.258, 'text': "So we're not overfitting.", 'start': 3624.217, 'duration': 1.041}, {'end': 3630.24, 'text': 'This model is not powerful enough to just be purely memorizing the data.', 'start': 3626.498, 'duration': 3.742}], 'summary': "The neural network achieved a decent loss of about 2.3 on the dev set, indicating it's not overfitting.", 'duration': 30.851, 'max_score': 3599.389, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I3599389.jpg'}, {'end': 3702.376, 'src': 'embed', 'start': 3670.725, 'weight': 0, 'content': [{'end': 3672.787, 'text': 'We now have 10, 000 parameters instead of 3, 000 parameters.', 'start': 3670.725, 'duration': 2.062}, {'end': 3677.485, 'text': "And then we're not using this.", 'start': 3676.024, 'duration': 1.461}, {'end': 3683.587, 'text': "And then here, what I'd like to do is I'd like to actually keep track of that.", 'start': 3678.405, 'duration': 5.182}, {'end': 3688.77, 'text': "Okay, let's just do this.", 'start': 3684.848, 'duration': 3.922}, {'end': 3690.15, 'text': "Let's keep stats again.", 'start': 3689.13, 'duration': 1.02}, {'end': 3702.376, 'text': "And here, when we're keeping track of the loss, let's just also keep track of the steps and let's just have an eye here and let's train on 30, 000,", 'start': 3691.071, 'duration': 11.305}], 'summary': 'Increased parameters to 10,000, tracking loss and steps, training on 30,000', 'duration': 31.651, 'max_score': 3670.725, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I3670725.jpg'}, {'end': 3992.983, 'src': 'embed', 'start': 3962.659, 'weight': 1, 'content': [{'end': 3966.983, 'text': 'the network has basically learned to separate out the characters and cluster them a little bit.', 'start': 3962.659, 'duration': 4.324}, {'end': 3972.288, 'text': 'So for example, you see how the vowels A, E, I, O, U are clustered up here.', 'start': 3967.764, 'duration': 4.524}, {'end': 3976.532, 'text': "So what that's telling us is that the neural net treats these as very similar, right?", 'start': 3973.049, 'duration': 3.483}, {'end': 3982.017, 'text': 'Because when they feed into the neural net, the embedding for all of these characters is very similar.', 'start': 3976.652, 'duration': 5.365}, {'end': 3986.581, 'text': "And so the neural net thinks that they're very similar and kind of like interchangeable, if that makes sense.", 'start': 3982.438, 'duration': 4.143}, {'end': 3992.983, 'text': 'then the points that are really far away are, for example, Q.', 'start': 3989.378, 'duration': 3.605}], 'summary': 'Neural net clusters vowels, treats them as very similar. q is far away.', 'duration': 30.324, 'max_score': 3962.659, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I3962659.jpg'}], 'start': 3354.278, 'title': 'Neural network training and optimization', 'summary': "Discusses the importance of evaluating test loss to prevent overfitting, data splitting into train, dev, and test sets, and evaluating training and dev losses. additionally, it illustrates the process of optimizing a neural net by increasing its size, adjusting the batch size and learning rate, and visualizing the embedding vectors, resulting in a total of 11,000 parameters and insights into the neural net's behavior.", 'chapters': [{'end': 3647.486, 'start': 3354.278, 'title': 'Neural network training and evaluation', 'summary': 'Discusses the importance of sparingly evaluating test loss to prevent overfitting, the process of splitting data into train, dev, and test sets, and the evaluation of training and dev losses to determine model performance.', 'duration': 293.208, 'highlights': ['The chapter emphasizes the importance of sparingly evaluating test loss to prevent overfitting, with only a few evaluations allowed to avoid training on the test split.', 'It explains the process of splitting data into train, dev, and test sets, where training is conducted on the train split and evaluation is done on the test set sparingly.', 'The transcript details the creation of X and Y tensors from a list of words, shuffling the input words, and the allocation of examples for training, validation, and test sets based on percentages.', 'It highlights the evaluation of dev loss using xdev and ydev, the comparison of training and dev losses to determine overfitting or underfitting, and the potential for performance improvements by scaling up the neural net.']}, {'end': 4065.685, 'start': 3647.806, 'title': 'Neural net optimization', 'summary': 'Illustrates the process of optimizing a neural net by increasing its size, adjusting the batch size and learning rate, and visualizing the embedding vectors, leading to a total of 11,000 parameters and revealing insights into the behavior of the neural net.', 'duration': 417.879, 'highlights': ['The neural net size is increased to 10,000 parameters from 3,000 parameters, and the training is performed on 30,000 steps with a learning rate of 0.1, but optimization is not fully achieved due to the increased size of the neural net.', 'The concern is raised that the bottleneck of the network might be the two-dimensional embeddings, leading to underfitting, prompting the decision to scale up the embedding size to 10 dimensions and reduce the number of neurons in the hidden layer to 200.', "Visualization of the two-dimensional character embeddings reveals insights into the behavior of the neural net, showing clustering of similar characters and the identification of special characters with unique embedding vectors, providing valuable information for improving the neural net's performance."]}], 'duration': 711.407, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I3354278.jpg', 'highlights': ['The neural net size is increased to 10,000 parameters from 3,000 parameters, and the training is performed on 30,000 steps with a learning rate of 0.1, but optimization is not fully achieved due to the increased size of the neural net.', "Visualization of the two-dimensional character embeddings reveals insights into the behavior of the neural net, showing clustering of similar characters and the identification of special characters with unique embedding vectors, providing valuable information for improving the neural net's performance.", 'It highlights the evaluation of dev loss using xdev and ydev, the comparison of training and dev losses to determine overfitting or underfitting, and the potential for performance improvements by scaling up the neural net.', 'The chapter emphasizes the importance of sparingly evaluating test loss to prevent overfitting, with only a few evaluations allowed to avoid training on the test split.']}, {'end': 4538.646, 'segs': [{'end': 4155.295, 'src': 'embed', 'start': 4120.477, 'weight': 4, 'content': [{'end': 4126.221, 'text': 'because when you plot the loss many times it can have this hockey stick appearance and log squashes it in.', 'start': 4120.477, 'duration': 5.744}, {'end': 4128.684, 'text': 'So it just kind of like looks nicer.', 'start': 4127.182, 'duration': 1.502}, {'end': 4133.546, 'text': 'So the X axis is step I and the Y axis will be the loss I.', 'start': 4129.184, 'duration': 4.362}, {'end': 4143.375, 'text': 'And then here, this is 30.', 'start': 4133.546, 'duration': 9.829}, {'end': 4149.912, 'text': "Ideally we wouldn't be hard coding these Okay, so let's look at the loss.", 'start': 4143.375, 'duration': 6.537}, {'end': 4155.295, 'text': "Okay, it's again very thick because the mini-batch size is very small,", 'start': 4151.633, 'duration': 3.662}], 'summary': 'Using log to flatten loss curve for better visualization and understanding.', 'duration': 34.818, 'max_score': 4120.477, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I4120477.jpg'}, {'end': 4336.185, 'src': 'embed', 'start': 4308.173, 'weight': 0, 'content': [{'end': 4312.995, 'text': "So we have here 200, 000 steps of the optimization and in the first 100, 000, we're using a learning rate of 0.1, and then in the next 100, 000,", 'start': 4308.173, 'duration': 4.822}, {'end': 4318.68, 'text': "we're using a learning rate of 0.01..", 'start': 4312.995, 'duration': 5.685}, {'end': 4319.92, 'text': 'This is the loss that I achieve.', 'start': 4318.68, 'duration': 1.24}, {'end': 4323.301, 'text': 'And these are the performance on the training and validation loss.', 'start': 4320.641, 'duration': 2.66}, {'end': 4330.723, 'text': "And in particular, the best validation loss I've been able to obtain in the last 30 minutes or so is 2.17.", 'start': 4324.142, 'duration': 6.581}, {'end': 4332.444, 'text': 'So now I invite you to beat this number.', 'start': 4330.723, 'duration': 1.721}, {'end': 4336.185, 'text': 'And you have quite a few knobs available to you to, I think, surpass this number.', 'start': 4332.984, 'duration': 3.201}], 'summary': 'Achieved a validation loss of 2.17 in 200,000 optimization steps with varying learning rates; challenge to improve performance.', 'duration': 28.012, 'max_score': 4308.173, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I4308173.jpg'}, {'end': 4426.367, 'src': 'embed', 'start': 4395.276, 'weight': 1, 'content': [{'end': 4398.537, 'text': "I'm leaving that as an exercise to the reader and that's it for now.", 'start': 4395.276, 'duration': 3.261}, {'end': 4399.637, 'text': "And I'll see you next time.", 'start': 4398.797, 'duration': 0.84}, {'end': 4407.078, 'text': 'Before we wrap up, I also wanted to show how you would sample from the model.', 'start': 4404.398, 'duration': 2.68}, {'end': 4410.339, 'text': "So we're going to generate 20 samples.", 'start': 4408.299, 'duration': 2.04}, {'end': 4413.26, 'text': 'At first we begin with all dots.', 'start': 4411.219, 'duration': 2.041}, {'end': 4414.76, 'text': "So that's the context.", 'start': 4413.76, 'duration': 1}, {'end': 4426.367, 'text': "And then until we generate the zero character again, We're going to embed the current context using the embedding table C.", 'start': 4415.52, 'duration': 10.847}], 'summary': 'Demonstrating sampling from the model with 20 samples and embedding table c.', 'duration': 31.091, 'max_score': 4395.276, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I4395276.jpg'}, {'end': 4534.202, 'src': 'heatmap', 'start': 4495.68, 'weight': 0.831, 'content': [{'end': 4497.061, 'text': "Okay, sorry, there's some bonus content.", 'start': 4495.68, 'duration': 1.381}, {'end': 4500.782, 'text': 'I wanted to mention that I want to make these notebooks more accessible.', 'start': 4497.601, 'duration': 3.181}, {'end': 4505.244, 'text': "And so I don't want you to have to like install Jupyter Notebooks and Torch and everything else.", 'start': 4501.303, 'duration': 3.941}, {'end': 4512.508, 'text': 'So I will be sharing a link to a Google Colab and the Google Colab will look like a notebook in your browser.', 'start': 4505.685, 'duration': 6.823}, {'end': 4518.731, 'text': "And you can just go to a URL and you'll be able to execute all of the code that you saw in the Google Colab.", 'start': 4512.968, 'duration': 5.763}, {'end': 4523.353, 'text': 'And so this is me executing the code in this lecture and I shortened it a little bit.', 'start': 4519.351, 'duration': 4.002}, {'end': 4529.278, 'text': "But basically you're able to train the exact same network and then plot and sample from the model,", 'start': 4524.133, 'duration': 5.145}, {'end': 4533.161, 'text': 'and everything is ready for you to tinker with the numbers right there in your browser.', 'start': 4529.278, 'duration': 3.883}, {'end': 4534.202, 'text': 'no installation necessary.', 'start': 4533.161, 'duration': 1.041}], 'summary': 'Google colab link provided for easy access, no installation necessary', 'duration': 38.522, 'max_score': 4495.68, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I4495680.jpg'}, {'end': 4529.278, 'src': 'embed', 'start': 4505.685, 'weight': 2, 'content': [{'end': 4512.508, 'text': 'So I will be sharing a link to a Google Colab and the Google Colab will look like a notebook in your browser.', 'start': 4505.685, 'duration': 6.823}, {'end': 4518.731, 'text': "And you can just go to a URL and you'll be able to execute all of the code that you saw in the Google Colab.", 'start': 4512.968, 'duration': 5.763}, {'end': 4523.353, 'text': 'And so this is me executing the code in this lecture and I shortened it a little bit.', 'start': 4519.351, 'duration': 4.002}, {'end': 4529.278, 'text': "But basically you're able to train the exact same network and then plot and sample from the model,", 'start': 4524.133, 'duration': 5.145}], 'summary': 'Google colab allows executing code in browser, training network, and sampling from the model.', 'duration': 23.593, 'max_score': 4505.685, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I4505685.jpg'}], 'start': 4065.685, 'title': 'Neural net training and text generation', 'summary': 'Involves optimizing neural network training with a best validation loss of 2.17 after 200,000 steps and discusses text generation using a model with a link to a google colab for accessing the code and model training.', 'chapters': [{'end': 4394.856, 'start': 4065.685, 'title': 'Optimizing neural net training', 'summary': 'Involves optimizing the training process of a neural network, including adjusting learning rates, plotting log loss, and experimenting with various hyperparameters to improve the validation loss, achieving a best validation loss of 2.17 after 200,000 steps.', 'duration': 329.171, 'highlights': ['The best validation loss obtained after 200,000 steps is 2.17.', 'Experimenting with various hyperparameters to improve the validation loss.', 'Plotting log loss instead of the loss to achieve a smoother visualization.']}, {'end': 4538.646, 'start': 4395.276, 'title': 'Text generation and model sampling', 'summary': 'Discusses text generation using a model, demonstrating how to sample from the model and sharing a link to a google colab for easy access to the code and model training.', 'duration': 143.37, 'highlights': ["The model generates 20 samples, resulting in word-like or name-like outputs such as 'ham,' 'joes,' and 'lila,' indicating significant progress in the model's performance.", 'A Google Colab link is shared for easy access to the code and model training, allowing users to execute the code in their browser without any installation necessary.', 'The process of sampling from the model involves embedding the context, calculating probabilities using f.softmax, sampling from the probabilities using Torch.Multinomial, and decoding the integers to strings for output.']}], 'duration': 472.961, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/TCH_1BHY58I/pics/TCH_1BHY58I4065685.jpg', 'highlights': ['The best validation loss obtained after 200,000 steps is 2.17.', 'The model generates 20 samples, showing significant progress in performance.', 'A Google Colab link is shared for easy access to the code and model training.', 'Experimenting with various hyperparameters to improve the validation loss.', 'Plotting log loss for smoother visualization of the training process.']}], 'highlights': ['Neural network language model with 17,000-word vocabulary and 30-dimensional embedding matrices', 'Efficient tensor operations and optimization techniques for neural network training', 'Creating a dataset for a neural network with 32 examples and 3 integer inputs', 'Efficient tensor concatenation and manipulation using PyTorch for neural network operations', 'Functional dot cross entropy in PyTorch enhances efficiency and simplicity in neural network operations', 'Optimizing neural net with mini-batch processing for more efficient optimization', 'Determining a reasonable learning rate for neural network training', 'Visualization of character embeddings provides insights into neural net behavior', 'Best validation loss of 2.17 achieved after 200,000 steps in neural network training']}