title
Recurrent Neural Networks (RNN) - Deep Learning w/ Python, TensorFlow & Keras p.7

description
In this part we're going to be covering recurrent neural networks. The idea of a recurrent neural network is that sequences and order matters. For many operations, this definitely does. Text tutorials and sample code: https://pythonprogramming.net/recurrent-neural-network-deep-learning-python-tensorflow-keras/ Discord: https://discord.gg/sentdex Support the content: https://pythonprogramming.net/support-donate/ Twitter: https://twitter.com/sentdex Facebook: https://www.facebook.com/pythonprogramming.net/ Twitch: https://www.twitch.tv/sentdex G+: https://plus.google.com/+sentdex

detail
{'title': 'Recurrent Neural Networks (RNN) - Deep Learning w/ Python, TensorFlow & Keras p.7', 'heatmap': [{'end': 591.396, 'start': 549.862, 'weight': 0.961}, {'end': 631.263, 'start': 601.005, 'weight': 0.774}, {'end': 897.328, 'start': 858.2, 'weight': 1}], 'summary': "Delves into recurrent neural networks' significance in processing time series data and natural language, addressing deep neural networks' limitations in capturing sentence meanings, explaining lstm and gru cells' functioning, optimizing lstm in tensorflow, building a tensorflow rnn model for image recognition with mnist dataset, and achieving 98.11% accuracy in 13 seconds, showcasing faster training speed and higher validation accuracy.", 'chapters': [{'end': 186.723, 'segs': [{'end': 51.265, 'src': 'embed', 'start': 24.787, 'weight': 0, 'content': [{'end': 34.393, 'text': 'so two areas where this tends to be the case is in like time series data, where data is organized temporally or chronologically,', 'start': 24.787, 'duration': 9.606}, {'end': 36.135, 'text': 'otherwise you might have.', 'start': 34.393, 'duration': 1.742}, {'end': 43.84, 'text': 'this is important in natural language, so the order of something in a sentence carries a lot of importance as far as the meaning of that sentence.', 'start': 36.135, 'duration': 7.705}, {'end': 51.265, 'text': 'so, for example, if you said you know some people made a neural network,', 'start': 43.84, 'duration': 7.425}], 'summary': 'Temporal and sequential data, like time series and natural language, require order preservation for accurate interpretation.', 'duration': 26.478, 'max_score': 24.787, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA24787.jpg'}, {'end': 94.799, 'src': 'embed', 'start': 70.332, 'weight': 1, 'content': [{'end': 76.874, 'text': 'but the problem with a deep neural network is this sentence would have the same, you know, meaning as a neural.', 'start': 70.332, 'duration': 6.542}, {'end': 86.697, 'text': 'a neural network made some people, And obviously these two sentences have widely varying meanings and impacts, and all that.', 'start': 76.874, 'duration': 9.823}, {'end': 91.278, 'text': "So it's important the order of things in these sentences.", 'start': 87.397, 'duration': 3.881}, {'end': 94.799, 'text': "So that's the idea of a recurrent neural network.", 'start': 91.998, 'duration': 2.801}], 'summary': 'Recurrent neural network addresses sentence order importance.', 'duration': 24.467, 'max_score': 70.332, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA70332.jpg'}, {'end': 186.723, 'src': 'embed', 'start': 118.248, 'weight': 3, 'content': [{'end': 121.209, 'text': "There's also the GRU or gated recurrent unit.", 'start': 118.248, 'duration': 2.961}, {'end': 125.03, 'text': 'But again, I just pretty much see everybody use LSTM cells in all honesty.', 'start': 121.249, 'duration': 3.781}, {'end': 127.772, 'text': 'So the way these work is.', 'start': 125.871, 'duration': 1.901}, {'end': 136.336, 'text': "you know, you've got your sequential data here and the sequential data is passed in, let's say, to one of these cells, and then the cell does two.", 'start': 127.772, 'duration': 8.564}, {'end': 138.258, 'text': 'it outputs to two locations.', 'start': 136.336, 'duration': 1.922}, {'end': 144.963, 'text': 'It outputs one to the next layer, but two, and the next layer could be the next.', 'start': 138.338, 'duration': 6.625}, {'end': 150.647, 'text': 'you know, it could be input to the next recurrent type of layer, or it could be the output layer or a dense layer, who knows?', 'start': 144.963, 'duration': 5.684}, {'end': 164.37, 'text': 'But anyways, each cell is going to output somewhere to another layer, but also down to the next node in the specific recurrent layer.', 'start': 151.127, 'duration': 13.243}, {'end': 172.555, 'text': "Now, where these output to, it doesn't necessarily have to be to the next thing down.", 'start': 164.75, 'duration': 7.805}, {'end': 176.157, 'text': "And then it also doesn't have to only go in one direction.", 'start': 172.615, 'duration': 3.542}, {'end': 179.059, 'text': 'You could have bi-directional recurrent layers, for example,', 'start': 176.197, 'duration': 2.862}, {'end': 183.962, 'text': 'and you can get really crazy with the order of things and where data gets passed and stuff.', 'start': 179.059, 'duration': 4.903}, {'end': 186.723, 'text': 'in general this is a super basic one.', 'start': 184.762, 'duration': 1.961}], 'summary': 'Lstm cells are widely used for sequential data handling in recurrent networks.', 'duration': 68.475, 'max_score': 118.248, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA118248.jpg'}], 'start': 1.816, 'title': 'Recurrent neural networks', 'summary': "Discusses the significance of recurrent neural networks in processing time series data and natural language, emphasizing the importance of data order and addressing deep neural networks' limitations in capturing sentence meanings. it also explains the functioning of lstm and gru cells in sequential data processing and output to subsequent layers and nodes.", 'chapters': [{'end': 94.799, 'start': 1.816, 'title': 'Recurrent neural network', 'summary': 'Discusses the significance of recurrent neural networks in processing time series data and natural language, highlighting the importance of the order of data and addressing the limitations of deep neural networks in capturing the meaning of sentences.', 'duration': 92.983, 'highlights': ['Recurrent Neural Network (RNN) is crucial for processing time series data and natural language, where the order of data holds significance and meaning, such as in sentences where the order of words impacts the overall meaning.', 'Deep neural networks encounter challenges in capturing the meaning of sentences, as they tokenize data by word, leading to sentences with different orders having the same meaning, highlighting the importance of considering the order of data in natural language processing.', 'The order of data holds importance in natural language, as exemplified by the varying meanings and impacts of sentences with different word orders, emphasizing the necessity of considering the order of words for accurate processing.']}, {'end': 186.723, 'start': 94.819, 'title': 'Recurrent neural network cells', 'summary': 'Explains the functioning of recurrent neural network cells, predominantly focusing on lstm and gru cells, their sequential data processing, and the output to next layer and next node in the specific recurrent layer.', 'duration': 91.904, 'highlights': ['The chapter explains the functioning of recurrent neural network cells, predominantly focusing on LSTM and GRU cells. The LSTM and GRU cells are highlighted as the main types of recurrent cells used, with LSTM being the most commonly utilized.', 'The sequential data is passed into the recurrent cells, which then output to the next layer and the next node in the specific recurrent layer. The process of passing sequential data into the cells and their output to the next layer and node is outlined.', 'The possibility of bi-directional recurrent layers and the flexibility in the direction and order of data passing is mentioned. The chapter discusses the flexibility of bi-directional recurrent layers and the potential complexity in the passing of data.']}], 'duration': 184.907, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA1816.jpg', 'highlights': ['Recurrent Neural Network (RNN) is crucial for processing time series data and natural language, emphasizing the significance of data order.', 'Deep neural networks encounter challenges in capturing the meaning of sentences, highlighting the importance of considering the order of data in natural language processing.', 'The order of data holds importance in natural language, emphasizing the necessity of considering the order of words for accurate processing.', 'The chapter explains the functioning of LSTM and GRU cells, highlighting LSTM as the most commonly utilized type of recurrent cell.', 'Sequential data is passed into the recurrent cells, which then output to the next layer and node, outlining the process of data flow.', 'The chapter discusses the flexibility of bi-directional recurrent layers and the potential complexity in the passing of data.']}, {'end': 389.544, 'segs': [{'end': 268.748, 'src': 'embed', 'start': 224.519, 'weight': 1, 'content': [{'end': 233.646, 'text': 'And then, finally, okay, based on this bundle of info, what do we want to output to both the next layer and the next node that we want to output to?', 'start': 224.519, 'duration': 9.127}, {'end': 237.128, 'text': "So that's the task of the LSTM cell.", 'start': 234.606, 'duration': 2.522}, {'end': 241.791, 'text': "Now, obviously that's pretty complicated, especially when you know all these things like.", 'start': 237.168, 'duration': 4.623}, {'end': 246.033, 'text': 'the whole point is to eventually just pass along like a scalar value.', 'start': 241.791, 'duration': 4.242}, {'end': 250.296, 'text': 'So how do we do that? Very challenging stuff.', 'start': 246.393, 'duration': 3.903}, {'end': 258.761, 'text': 'If you want to learn more about how LSTM cells work specifically, there is a great timeless write-up on the LSTM cell.', 'start': 251.036, 'duration': 7.725}, {'end': 261.363, 'text': 'I link to it in the text-based version of the tutorial.', 'start': 259.321, 'duration': 2.042}, {'end': 262.944, 'text': 'So definitely go check that out if you want to.', 'start': 261.382, 'duration': 1.562}, {'end': 268.748, 'text': "Otherwise, let's get into writing our own basic recurrent neural networks.", 'start': 263.724, 'duration': 5.024}], 'summary': 'Discussing the task of lstm cell and challenges in passing along a scalar value, with a reference to a write-up on lstm cells.', 'duration': 44.229, 'max_score': 224.519, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA224519.jpg'}, {'end': 347.101, 'src': 'embed', 'start': 291.485, 'weight': 2, 'content': [{'end': 297.634, 'text': "It's just time series, it's just informal kind of data and we don't really have you know what are we actually trying to point this to?", 'start': 291.485, 'duration': 6.149}, {'end': 303.824, 'text': "So there's usually a huge amount of pre-processing involved in when you work with recurrent neural networks.", 'start': 297.674, 'duration': 6.15}, {'end': 311.292, 'text': "So what we're gonna do here is just use a simple MNIST example and in the next tutorial we'll do a realistic example with some time series data,", 'start': 303.884, 'duration': 7.408}, {'end': 314.875, 'text': 'specifically with cryptocurrency prices.', 'start': 311.292, 'duration': 3.583}, {'end': 321.68, 'text': "so uh, yeah, anyway, um, i'm gonna minimize this and let's begin with some imports.", 'start': 314.875, 'duration': 6.805}, {'end': 329.566, 'text': 'so import tensor flow, as t jerk always does this to me, import tensorflow is tf, fix that.', 'start': 321.68, 'duration': 7.886}, {'end': 336.352, 'text': "Then we're going to go from tensorflow.keros.models.", 'start': 330.567, 'duration': 5.785}, {'end': 339.655, 'text': "We're going to import the sequential type of model.", 'start': 336.512, 'duration': 3.143}, {'end': 344.299, 'text': 'from tensorflow.keros.layers.', 'start': 340.296, 'duration': 4.003}, {'end': 347.101, 'text': "we're going to import the dense layer.", 'start': 344.299, 'duration': 2.802}], 'summary': 'Tutorial covers pre-processing for recurrent neural networks and uses mnist and cryptocurrency data for examples.', 'duration': 55.616, 'max_score': 291.485, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA291485.jpg'}, {'end': 397.635, 'src': 'embed', 'start': 366.575, 'weight': 0, 'content': [{'end': 372.858, 'text': 'if you are on the GPU version of TensorFlow, check out the cuDNN LSTM cell.', 'start': 366.575, 'duration': 6.283}, {'end': 379.84, 'text': 'So, even if you are on the GPU version of TensorFlow, this LSTM cell will indeed run on your GPU,', 'start': 373.498, 'duration': 6.342}, {'end': 389.544, 'text': "but the cuDNN LSTM cell is even more optimized and it's like five plus times faster, even though both run on your GPU.", 'start': 379.84, 'duration': 9.704}, {'end': 397.635, 'text': "If you're on the CPU version of TensorFlow, this model is going to take five hours to train or something.", 'start': 390.269, 'duration': 7.366}], 'summary': 'Cudnn lstm cell in gpu tensorflow is 5x faster than regular lstm; cpu training takes 5 hours.', 'duration': 31.06, 'max_score': 366.575, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA366575.jpg'}], 'start': 186.723, 'title': 'Understanding lstm cells, recurrent neural networks, and optimizing lstm cell in tensorflow', 'summary': 'Explains the functioning of lstm cells, the challenges of structuring data for recurrent neural networks, and covers the optimization of lstm cell in tensorflow. it also mentions the use of cudnn lstm cell which is five times faster than the standard lstm cell when running on the gpu.', 'chapters': [{'end': 268.748, 'start': 186.723, 'title': 'Understanding lstm cells', 'summary': 'Explains the functioning of lstm cells, including choosing what to forget from the previous node, adding input data to the bundle of information, and outputting to the next layer and node, which are crucial in passing along a scalar value. it also mentions a recommended resource for further learning.', 'duration': 82.025, 'highlights': ["The LSTM cell's functions include choosing what to forget from the previous node, adding input data to the bundle of information, and outputting to the next layer and node, crucial in passing along a scalar value, which is a complex task.", 'The tutorial mentions a recommended timeless write-up on LSTM cells for further learning.']}, {'end': 347.101, 'start': 268.768, 'title': 'Recurrent neural networks', 'summary': 'Introduces the challenges of structuring data for recurrent neural networks, focusing on the pre-processing involved and the use of simple mnist and realistic time series data examples.', 'duration': 78.333, 'highlights': ['Recurrent neural networks require significant pre-processing due to the informal nature of the data, typically time series, and lack of targets, making it challenging to structure the data (quantifiable: significant pre-processing involved).', 'The tutorial will use a simple MNIST example and then move on to a realistic example with time series data, specifically cryptocurrency prices (quantifiable: transition from simple MNIST to cryptocurrency prices example).', "The speaker mentions import of Tensorflow and fixing an error with 'import tensorflow as tf' (quantifiable: mention of specific imports and error fix)."]}, {'end': 389.544, 'start': 347.101, 'title': 'Optimizing lstm cell in tensorflow', 'summary': 'Covers the optimization of lstm cell in tensorflow, including the use of cudnn lstm cell which is five times faster than the standard lstm cell when running on the gpu.', 'duration': 42.443, 'highlights': ['The cuDNN LSTM cell in TensorFlow is five plus times faster than the standard LSTM cell when running on the GPU.', 'The chapter discusses the use of dense layers, dropout, and cuDNN LSTM cell for optimizing LSTM cell in TensorFlow.']}], 'duration': 202.821, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA186723.jpg', 'highlights': ['The cuDNN LSTM cell in TensorFlow is five plus times faster than the standard LSTM cell when running on the GPU.', "The LSTM cell's functions include choosing what to forget from the previous node, adding input data to the bundle of information, and outputting to the next layer and node, crucial in passing along a scalar value, which is a complex task.", 'Recurrent neural networks require significant pre-processing due to the informal nature of the data, typically time series, and lack of targets, making it challenging to structure the data (quantifiable: significant pre-processing involved).', 'The tutorial will use a simple MNIST example and then move on to a realistic example with time series data, specifically cryptocurrency prices (quantifiable: transition from simple MNIST to cryptocurrency prices example).', 'The chapter discusses the use of dense layers, dropout, and cuDNN LSTM cell for optimizing LSTM cell in TensorFlow.', 'The tutorial mentions a recommended timeless write-up on LSTM cells for further learning.', "The speaker mentions import of Tensorflow and fixing an error with 'import tensorflow as tf' (quantifiable: mention of specific imports and error fix)."]}, {'end': 829.54, 'segs': [{'end': 492.001, 'src': 'embed', 'start': 390.269, 'weight': 0, 'content': [{'end': 397.635, 'text': "If you're on the CPU version of TensorFlow, this model is going to take five hours to train or something.", 'start': 390.269, 'duration': 7.366}, {'end': 399.356, 'text': 'Either way.', 'start': 398.996, 'duration': 0.36}, {'end': 402.999, 'text': 'Okay, so those are the things that we need.', 'start': 400.277, 'duration': 2.722}, {'end': 405.08, 'text': 'And now we need the dataset.', 'start': 403.459, 'duration': 1.621}, {'end': 406.721, 'text': "So we're just going to grab mnist real quick.", 'start': 405.1, 'duration': 1.621}, {'end': 410.965, 'text': 'So mnist is equal to tf.keros.datasets.mnist.', 'start': 407.142, 'duration': 3.823}, {'end': 413.126, 'text': 'and then we just need to unpack it.', 'start': 411.565, 'duration': 1.561}, {'end': 428.034, 'text': "so it's just to these two tuples, and then it's x train, y train, then it's x test, y test, and that's mnist.load underscore data.", 'start': 413.126, 'duration': 14.908}, {'end': 432.837, 'text': "so again, my apologies for using mnist, but i i think it's better to do it.", 'start': 428.034, 'duration': 4.803}, {'end': 436.537, 'text': "make this one as basic as possible And then in the next one we'll get in.", 'start': 432.837, 'duration': 3.7}, {'end': 443.581, 'text': "Because, honestly, it would take us, like the next one's probably going to take us two, three, maybe even four parts,", 'start': 436.597, 'duration': 6.984}, {'end': 446.663, 'text': 'just to get to okay now feed it through the neural network.', 'start': 443.581, 'duration': 3.082}, {'end': 448.784, 'text': "And that's the easy part, really.", 'start': 446.843, 'duration': 1.941}, {'end': 450.966, 'text': 'Feeding it through is not too hard.', 'start': 448.904, 'duration': 2.062}, {'end': 456.789, 'text': "So for example, let's just do a print xtrain.shape.", 'start': 451.566, 'duration': 5.223}, {'end': 464.574, 'text': 'And then we should be able to deduce this from the above, but xtrain.', 'start': 457.69, 'duration': 6.884}, {'end': 470.992, 'text': "let's just get the first sample dot shape, just so we get idea what we're dealing with here.", 'start': 465.882, 'duration': 5.11}, {'end': 481.92, 'text': "so um, so we can see, we've got 60 000 examples of 28 by 28 images And of course the zeroth element is just 28 by 28..", 'start': 470.992, 'duration': 10.928}, {'end': 485.76, 'text': 'So take a minute pause and think about.', 'start': 481.92, 'duration': 3.84}, {'end': 490.721, 'text': "okay, so this is our training data and it's already in sequences for us, right?", 'start': 485.76, 'duration': 4.961}, {'end': 492.001, 'text': 'And what are the sequences??', 'start': 490.801, 'duration': 1.2}], 'summary': "Training a basic model on tensorflow's cpu version may take five hours. dataset obtained from mnist includes 60,000 examples of 28x28 images.", 'duration': 101.732, 'max_score': 390.269, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA390269.jpg'}, {'end': 593.318, 'src': 'heatmap', 'start': 542.584, 'weight': 1, 'content': [{'end': 543.906, 'text': "so we're going to say model equals.", 'start': 542.584, 'duration': 1.322}, {'end': 546.157, 'text': "i'm going to Give us some space here.", 'start': 543.906, 'duration': 2.251}, {'end': 549.722, 'text': "Model equals, and it's just going to be that sequential type of model.", 'start': 546.818, 'duration': 2.904}, {'end': 557.633, 'text': "Now we're going to do model.add, and we're going to add in an LSTM cell.", 'start': 549.862, 'duration': 7.771}, {'end': 560.877, 'text': "How many cells do we want to have? We're just going to use 128 for now.", 'start': 557.913, 'duration': 2.964}, {'end': 567.642, 'text': "We're going to specify the input shape and that is 28 by 28.", 'start': 562.519, 'duration': 5.123}, {'end': 568.943, 'text': 'We already kind of know that.', 'start': 567.642, 'duration': 1.301}, {'end': 576.728, 'text': 'But we can make this somewhat dynamic by doing X train dot shape and then one colon.', 'start': 569.824, 'duration': 6.904}, {'end': 585.749, 'text': "So then, we can pass an activation function, and we'll use our good friend rectifiedLinear.", 'start': 579.401, 'duration': 6.348}, {'end': 591.396, 'text': 'And again, do check out cuDNNLSTM.', 'start': 586.81, 'duration': 4.586}, {'end': 593.318, 'text': "I'm just not going to do that yet.", 'start': 591.956, 'duration': 1.362}], 'summary': 'Creating a model with lstm cells, 128 cells, input shape 28x28, and rectifiedlinear activation function.', 'duration': 50.734, 'max_score': 542.584, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA542584.jpg'}, {'end': 631.263, 'src': 'heatmap', 'start': 601.005, 'weight': 0.774, 'content': [{'end': 601.906, 'text': "We'll just keep it here.", 'start': 601.005, 'duration': 0.901}, {'end': 611.132, 'text': "Okay, so the next thing that we want to do is one more parameter, which is return underscore sequences, and that's going to be true.", 'start': 603.247, 'duration': 7.885}, {'end': 618.276, 'text': 'So do we want this layer to return sequences like it was input, like sequences were input?', 'start': 611.772, 'duration': 6.504}, {'end': 623.52, 'text': 'Or do we want this layer to return something flat, right?', 'start': 619.117, 'duration': 4.403}, {'end': 626.442, 'text': "So if we're going to a dense layer right?", 'start': 623.8, 'duration': 2.642}, {'end': 631.263, 'text': "We wouldn't want to return sequences, because a dense layer is not going to understand what the heck's going on.", 'start': 627.382, 'duration': 3.881}], 'summary': 'Adding new parameter to return underscore sequences as true and discussing the behavior of a dense layer.', 'duration': 30.258, 'max_score': 601.005, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA601005.jpg'}, {'end': 737.482, 'src': 'embed', 'start': 708.94, 'weight': 2, 'content': [{'end': 712.703, 'text': "that's just it's just how many classes do you have so we're just going to go with 10.", 'start': 708.94, 'duration': 3.763}, {'end': 720.39, 'text': "the activation shouldn't be rectified linear, it should be soft max at this point and okay, i think we're done there with like,", 'start': 712.703, 'duration': 7.687}, {'end': 722.252, 'text': 'at least with our you know model structure.', 'start': 720.39, 'duration': 1.862}, {'end': 726.295, 'text': 'now all we need to do is do the compile and then the fitment.', 'start': 722.252, 'duration': 4.043}, {'end': 729.116, 'text': "So to do the compile, we're going to have an optimizer.", 'start': 726.435, 'duration': 2.681}, {'end': 730.738, 'text': 'Compile Compile.', 'start': 729.417, 'duration': 1.321}, {'end': 737.482, 'text': 'Anyways, tf.keros.optimizers.atom.', 'start': 731.258, 'duration': 6.224}], 'summary': 'The model has 10 classes, with soft max activation, and requires optimization for compilation.', 'duration': 28.542, 'max_score': 708.94, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA708940.jpg'}], 'start': 390.269, 'title': 'Tensorflow and rnn model building', 'summary': 'Covers training a model in tensorflow, acquiring the mnist dataset, feeding data through the neural network, and building a recurrent neural network model for image recognition using 60,000 examples of 28x28 images, configuring lstm layers with 128 cells, rectified linear activation functions, and model compilation with an atom optimizer.', 'chapters': [{'end': 464.574, 'start': 390.269, 'title': 'Tensorflow model training and dataset handling', 'summary': 'Covers the time required to train a model in tensorflow on the cpu version, the process of acquiring the mnist dataset, and the ease of feeding data through the neural network, with a mention of the shape of the xtrain data.', 'duration': 74.305, 'highlights': ['The model in TensorFlow on the CPU version takes approximately five hours to train.', 'Acquiring the mnist dataset involves unpacking two tuples for x train, y train, x test, and y test via mnist.load_data.', 'Feeding data through the neural network is relatively easy, exemplified by the print xtrain.shape command.']}, {'end': 829.54, 'start': 465.882, 'title': 'Recurrent neural network model building', 'summary': 'Discusses building a recurrent neural network model for image recognition, using 60,000 examples of 28x28 images, and configuring lstm layers with 128 cells, rectified linear activation functions, return_sequences set to true, and a soft max output layer for 10 classes, followed by model compilation with an atom optimizer, learning rate of 1e-3, and sparse categorical cross-entropy loss function.', 'duration': 363.658, 'highlights': ['Configuring LSTM layers with 128 cells, rectified linear activation functions, and return_sequences set to true. The chapter discusses adding LSTM layers with 128 cells, specifying rectified linear activation functions, and setting return_sequences to true to maintain the sequence structure.', 'Using a soft max output layer for 10 classes and sparse categorical cross-entropy loss function for model compilation. The transcript mentions the implementation of a soft max output layer for 10 classes and the use of sparse categorical cross-entropy loss function for model compilation.', 'Configuring the model with 60,000 examples of 28x28 images for training. The chapter provides information about using 60,000 examples of 28x28 images for training the neural network model for image recognition.']}], 'duration': 439.271, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA390269.jpg', 'highlights': ['The model in TensorFlow on the CPU version takes approximately five hours to train.', 'Configuring LSTM layers with 128 cells, rectified linear activation functions, and return_sequences set to true.', 'Using a soft max output layer for 10 classes and sparse categorical cross-entropy loss function for model compilation.', 'Acquiring the mnist dataset involves unpacking two tuples for x train, y train, x test, and y test via mnist.load_data.', 'Configuring the model with 60,000 examples of 28x28 images for training.', 'Feeding data through the neural network is relatively easy, exemplified by the print xtrain.shape command.']}, {'end': 1279.429, 'segs': [{'end': 897.328, 'src': 'heatmap', 'start': 858.2, 'weight': 1, 'content': [{'end': 860.741, 'text': 'Okay. so that should be good to go.', 'start': 858.2, 'duration': 2.541}, {'end': 867.204, 'text': 'Hopefully I can run this and not crash the computer because I know Sublime likes to really get me.', 'start': 860.821, 'duration': 6.383}, {'end': 870.186, 'text': "So we'll go ahead and run that now.", 'start': 867.824, 'duration': 2.362}, {'end': 872.006, 'text': "I'll probably just go through maybe one epoch.", 'start': 870.366, 'duration': 1.64}, {'end': 873.527, 'text': "Oh, let's see.", 'start': 872.106, 'duration': 1.421}, {'end': 875.728, 'text': 'Invalid syntax at the dropout.', 'start': 873.987, 'duration': 1.741}, {'end': 881.191, 'text': "What did we, and maybe not, what line is that? 14, did we not fully close this off? We didn't.", 'start': 875.788, 'duration': 5.403}, {'end': 884.092, 'text': 'Try again.', 'start': 883.752, 'duration': 0.34}, {'end': 888.924, 'text': 'Wait for it.', 'start': 888.424, 'duration': 0.5}, {'end': 891.405, 'text': 'Okay, things are looking good so far.', 'start': 889.745, 'duration': 1.66}, {'end': 895.227, 'text': 'All right, and we are off, we are learning.', 'start': 893.146, 'duration': 2.081}, {'end': 897.328, 'text': 'The accuracy does not look very good.', 'start': 895.347, 'duration': 1.981}], 'summary': 'Running code with sublime, encountering syntax error, achieving low accuracy.', 'duration': 39.128, 'max_score': 858.2, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA858200.jpg'}, {'end': 1065.324, 'src': 'embed', 'start': 1035.17, 'weight': 1, 'content': [{'end': 1037.512, 'text': "so let's see the example.", 'start': 1035.17, 'duration': 2.342}, {'end': 1040.635, 'text': "uh, yeah, i mean, look how much faster it's learning.", 'start': 1037.512, 'duration': 3.123}, {'end': 1043.396, 'text': 'now all we did was scale it between zero and one.', 'start': 1040.635, 'duration': 2.761}, {'end': 1046.56, 'text': "that's how huge of an impact that makes.", 'start': 1043.396, 'duration': 3.164}, {'end': 1047.641, 'text': "so yeah, it's super important.", 'start': 1046.56, 'duration': 1.081}, {'end': 1052.927, 'text': "Okay, so I'm going to go ahead and break.", 'start': 1051.365, 'duration': 1.562}, {'end': 1055.33, 'text': "well, I guess I don't have to break it, but I'm going to show you guys now.", 'start': 1052.927, 'duration': 2.403}, {'end': 1057.093, 'text': "CUDI and NLSTM, because it's so fast.", 'start': 1055.33, 'duration': 1.763}, {'end': 1062.74, 'text': "Anyways, paste, paste, and then I'm going to delete this and delete this.", 'start': 1057.734, 'duration': 5.006}, {'end': 1065.324, 'text': 'Whoops, we do want to have that though.', 'start': 1063.982, 'duration': 1.342}], 'summary': 'Scaling between 0 and 1 speeds up learning significantly. cudi and nlstm are fast.', 'duration': 30.154, 'max_score': 1035.17, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA1035170.jpg'}, {'end': 1112.177, 'src': 'embed', 'start': 1082.858, 'weight': 0, 'content': [{'end': 1085.039, 'text': 'I probably should be running this in a console instead.', 'start': 1082.858, 'duration': 2.181}, {'end': 1088.582, 'text': 'but whatever, look at this, look how fast this is.', 'start': 1085.039, 'duration': 3.543}, {'end': 1092.425, 'text': 'this is the best, the best thing ever.', 'start': 1088.582, 'duration': 3.843}, {'end': 1097.331, 'text': 'What a great little addition.', 'start': 1096.291, 'duration': 1.04}, {'end': 1098.992, 'text': 'I mean, recurrent nets are so slow.', 'start': 1097.531, 'duration': 1.461}, {'end': 1101.473, 'text': 'This QDNNLSTM cell is just glorious.', 'start': 1099.032, 'duration': 2.441}, {'end': 1106.495, 'text': "So we're already at an accuracy of 96%.", 'start': 1102.233, 'duration': 4.262}, {'end': 1112.177, 'text': "And before epochs, we're taking, I don't know, we were pretty deep into that one, and it was like two minutes and 30 seconds to go.", 'start': 1106.495, 'duration': 5.682}], 'summary': 'Qdnnlstm cell achieves 96% accuracy in 2.5 minutes.', 'duration': 29.319, 'max_score': 1082.858, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA1082858.jpg'}, {'end': 1161.876, 'src': 'embed', 'start': 1132.811, 'weight': 2, 'content': [{'end': 1133.852, 'text': 'validation actually did better.', 'start': 1132.811, 'duration': 1.041}, {'end': 1142.1, 'text': "98.11. I don't know if I can get back to the other layers or not.", 'start': 1133.852, 'duration': 8.248}, {'end': 1144.362, 'text': 'I did want to show like on this.', 'start': 1142.22, 'duration': 2.142}, {'end': 1153.11, 'text': 'So like on something like this, I think the reason why validation accuracy is higher than accuracy is because this is an average for the entire epoch.', 'start': 1144.802, 'duration': 8.308}, {'end': 1156.992, 'text': 'Whereas this is like at the end of the epoch, we run it.', 'start': 1153.83, 'duration': 3.162}, {'end': 1161.876, 'text': "So, and that's also why like this is at the, this is an average for the epoch.", 'start': 1157.473, 'duration': 4.403}], 'summary': 'Validation accuracy is higher at 98.11% than accuracy due to epoch average.', 'duration': 29.065, 'max_score': 1132.811, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA1132811.jpg'}], 'start': 829.54, 'title': 'Optimizing model training and improving lstm model performance', 'summary': 'Discusses model training with 3 epochs, the impact of data normalization on training speed, and the use of qdnnlstm for faster learning. it also highlights the achievement of 98.11% accuracy in just 13 seconds, showcasing faster training speed and higher validation accuracy.', 'chapters': [{'end': 1082.858, 'start': 829.54, 'title': 'Optimizing model training and normalization', 'summary': 'Discusses model training with 3 epochs, the impact of data normalization on training speed, and the use of qdnnlstm for faster learning, highlighting the importance of data scaling and its impact on training speed.', 'duration': 253.318, 'highlights': ['The importance of data normalization is highlighted, as scaling the data between 0 and 1 significantly speeds up the learning process, as seen in the example where learning is much faster after scaling.', 'The use of QDNNLSTM to improve learning speed is addressed, with a focus on the tanh activation function and the potential for improved accuracy, providing insights into alternative approaches for faster training.', 'The discussion of model training reveals the slow learning process and the need for data normalization, emphasizing the impact of scaling on training speed and the significant difference it makes in accelerating learning.', 'The significance of data normalization is emphasized, as the lack of normalization leads to slow learning and poor accuracy, demonstrating the crucial role of data preprocessing in enhancing training efficiency.']}, {'end': 1279.429, 'start': 1082.858, 'title': 'Improving lstm model performance', 'summary': 'Discusses the significant improvement in model performance, with the qdnnlstm cell achieving an accuracy of 98.11% in just 13 seconds, highlighting the faster training speed and higher validation accuracy.', 'duration': 196.571, 'highlights': ['The QDNNLSTM cell achieved an accuracy of 98.11% in just 13 seconds, showcasing the significant improvement in training speed and model accuracy.', 'The validation accuracy of 97.73% highlights the consistent high performance of the model, with potential for further training to enhance accuracy.', 'The explanation regarding the discrepancy between validation accuracy and accuracy provides insight into the evaluation process and potential for continuous model improvement.', 'Acknowledgment of the diverse audience and gratitude towards supporters adds a personal touch and highlights the global reach of the content.']}], 'duration': 449.889, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/BSpXCRTOLJA/pics/BSpXCRTOLJA829540.jpg', 'highlights': ['QDNNLSTM cell achieved 98.11% accuracy in 13 seconds, showcasing significant improvement in training speed and model accuracy.', 'Importance of data normalization is highlighted, as scaling data between 0 and 1 significantly speeds up the learning process.', 'Validation accuracy of 97.73% highlights consistent high performance of the model, with potential for further training to enhance accuracy.', 'Discussion of model training reveals slow learning process and need for data normalization, emphasizing the impact of scaling on training speed.']}], 'highlights': ['RNN is crucial for processing time series data and natural language, emphasizing the significance of data order.', 'Deep neural networks encounter challenges in capturing the meaning of sentences, highlighting the importance of considering the order of data in natural language processing.', 'The order of data holds importance in natural language, emphasizing the necessity of considering the order of words for accurate processing.', 'The chapter explains the functioning of LSTM and GRU cells, highlighting LSTM as the most commonly utilized type of recurrent cell.', 'The cuDNN LSTM cell in TensorFlow is five plus times faster than the standard LSTM cell when running on the GPU.', "The LSTM cell's functions include choosing what to forget from the previous node, adding input data to the bundle of information, and outputting to the next layer and node, crucial in passing along a scalar value, which is a complex task.", 'Recurrent neural networks require significant pre-processing due to the informal nature of the data, typically time series, and lack of targets, making it challenging to structure the data (quantifiable: significant pre-processing involved).', 'The tutorial will use a simple MNIST example and then move on to a realistic example with time series data, specifically cryptocurrency prices (quantifiable: transition from simple MNIST to cryptocurrency prices example).', 'The model in TensorFlow on the CPU version takes approximately five hours to train.', 'QDNNLSTM cell achieved 98.11% accuracy in 13 seconds, showcasing significant improvement in training speed and model accuracy.', 'Importance of data normalization is highlighted, as scaling data between 0 and 1 significantly speeds up the learning process.', 'Validation accuracy of 97.73% highlights consistent high performance of the model, with potential for further training to enhance accuracy.']}