title
Learn TensorFlow.js - Deep Learning and Neural Networks with JavaScript

description
This full course introduces the concept of client-side artificial neural networks. We will learn how to deploy and run models along with full deep learning applications in the browser! To implement this cool capability, we’ll be using TensorFlow.js (TFJS), TensorFlow’s JavaScript library. By the end of this video tutorial, you will have built and deployed a web application that runs a neural network in the browser to classify images! To get there, we'll learn about client-server deep learning architectures, converting Keras models to TFJS models, serving models with Node.js, tensor operations, and more! This course was created by deeplizard. Check out their YouTube channel and website for more tutorials! 🦎 http://youtube.com/deeplizard http://deeplizard.com ⭐️Course Sections⭐️ ⌨️ 0:00 - Intro to deep learning with client-side neural networks ⌨️ 6:06 - Convert Keras model to Layers API format ⌨️ 11:16 - Serve deep learning models with Node.js and Express ⌨️ 19:22 - Building UI for neural network web app ⌨️ 27:08 - Loading model into a neural network web app ⌨️ 36:55 - Explore tensor operations with VGG16 preprocessing ⌨️ 45:16 - Examining tensors with the debugger ⌨️ 1:00:37 - Broadcasting with tensors ⌨️ 1:11:30 - Running MobileNet in the browser Keras model H5 files: VGG16 - https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5 MobileNet - https://github.com/fchollet/deep-learning-models/releases/download/v0.6/mobilenet_1_0_224_tf.h5 Code: https://www.patreon.com/posts/19580029 Broadcasting notebook: https://www.patreon.com/posts/20051386 Download access to code files and notebooks are available as a perk for the deeplizard hivemind. Check out the details regarding deeplizard perks and rewards at: http://deeplizard.com/hivemind Music: Jarvic 8 by Kevin MacLeod Investigations by Kevin MacLeod Crypto by Kevin MacLeod Too Cool by Kevin MacLeod Brittle Rille by Kevin MacLeod Chillin Hard by Kevin MacLeod Dreamy Flashback by Kevin MacLeod Thinking Music by Kevin MacLeod YouTube: https://www.youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ Website: http://incompetech.com/ Licensed under Creative Commons: By Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/ -- Learn to code for free and get a developer job: https://www.freecodecamp.org Read hundreds of articles on programming: https://medium.freecodecamp.org

detail
{'title': 'Learn TensorFlow.js - Deep Learning and Neural Networks with JavaScript', 'heatmap': [{'end': 862.448, 'start': 806.829, 'weight': 0.775}, {'end': 1011.777, 'start': 954.335, 'weight': 0.937}, {'end': 1147.513, 'start': 1093.732, 'weight': 1}, {'end': 1247.98, 'start': 1192.447, 'weight': 0.72}, {'end': 1915.151, 'start': 1856.764, 'weight': 0.71}, {'end': 4737.676, 'start': 4680.336, 'weight': 0.722}], 'summary': 'Course covers client-side neural networks using tensorflow.js for deployment, setting up an express server, loading and processing vgg16 model, image data preprocessing, debugging and creating tensors, tensor broadcasting, and optimizing web app with model selection, emphasizing the shift in deployment perspective and the practicality of mobilenet over vgg16 in terms of loading speed.', 'chapters': [{'end': 45.942, 'segs': [{'end': 45.942, 'src': 'embed', 'start': 1.124, 'weight': 0, 'content': [{'end': 1.804, 'text': "What's up, guys?", 'start': 1.124, 'duration': 0.68}, {'end': 7.385, 'text': "In this video, we're going to introduce the concept of client-side artificial neural networks,", 'start': 2.004, 'duration': 5.381}, {'end': 13.107, 'text': 'which will lead us to deploying and running models along with our full deep learning applications in the browser.', 'start': 7.385, 'duration': 5.722}, {'end': 19.508, 'text': "To implement this cool capability, we'll be using TensorFlow.js, TensorFlow's JavaScript library,", 'start': 13.707, 'duration': 5.801}, {'end': 23.389, 'text': 'which will allow us to well build and access models in JavaScript.', 'start': 19.508, 'duration': 3.881}, {'end': 26.77, 'text': "I'm super excited to cover this, so let's get to it.", 'start': 24.129, 'duration': 2.641}, {'end': 39.639, 'text': "Whether you're a complete beginner to neural networks and deep learning or you just want to up your skills in this area.", 'start': 33.996, 'duration': 5.643}, {'end': 45.942, 'text': "you'll definitely want to check out all the resources available on the Deep Blizzard YouTube channel as well as deepblizzard.com.", 'start': 39.639, 'duration': 6.303}], 'summary': 'Introducing client-side artificial neural networks using tensorflow.js for deploying and running models in the browser.', 'duration': 44.818, 'max_score': 1.124, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41124.jpg'}], 'start': 1.124, 'title': 'Client-side neural networks with tensorflow.js', 'summary': 'Introduces client-side artificial neural networks using tensorflow.js to deploy and run models in the browser, highlighting accessibility and potential for beginners and skilled individuals.', 'chapters': [{'end': 45.942, 'start': 1.124, 'title': 'Client-side neural networks with tensorflow.js', 'summary': 'Introduces client-side artificial neural networks using tensorflow.js for deploying and running models in the browser, emphasizing the accessibility and potential of this approach for both beginners and skilled individuals.', 'duration': 44.818, 'highlights': ['Introduction of client-side artificial neural networks using TensorFlow.js', 'Deployment and running of models in the browser', 'Emphasis on accessibility for beginners and skilled individuals']}], 'duration': 44.818, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41124.jpg', 'highlights': ['Introduction of client-side artificial neural networks using TensorFlow.js', 'Deployment and running of models in the browser', 'Emphasis on accessibility for beginners and skilled individuals']}, {'end': 1085.906, 'segs': [{'end': 346.05, 'src': 'embed', 'start': 312.645, 'weight': 10, 'content': [{'end': 319.23, 'text': "But would we want to do everything with a model running in the browser? That's another question, and the answer is probably not.", 'start': 312.645, 'duration': 6.585}, {'end': 326.516, 'text': 'Having our models run in the browser is best suited for tasks like fine-tuning pre-trained models or, most popularly, inference.', 'start': 320.091, 'duration': 6.425}, {'end': 329.619, 'text': 'So, using a model to get predictions on new data.', 'start': 327.137, 'duration': 2.482}, {'end': 335.704, 'text': 'And this task is exactly what we saw with the project we worked on in the Keras Model Deployment series using Flask.', 'start': 330.479, 'duration': 5.225}, {'end': 340.608, 'text': 'While building new models and training models from scratch can also be done in the browser.', 'start': 336.587, 'duration': 4.021}, {'end': 346.05, 'text': 'these tasks are usually better addressed using other APIs like Keras or standard TensorFlow in Python.', 'start': 340.608, 'duration': 5.442}], 'summary': 'Running models in the browser is suited for fine-tuning pre-trained models or inference, best seen in the keras model deployment project using flask.', 'duration': 33.405, 'max_score': 312.645, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4312645.jpg'}, {'end': 509.551, 'src': 'embed', 'start': 483.854, 'weight': 1, 'content': [{'end': 490.717, 'text': 'To convert a Keras model into a TensorFlow.js model, though we need to have saved the entire model with the weights, the architecture,', 'start': 483.854, 'duration': 6.863}, {'end': 492.018, 'text': 'everything in an H5 file.', 'start': 490.717, 'duration': 1.301}, {'end': 496.023, 'text': "Currently, that's done using charismodel.save function.", 'start': 493.001, 'duration': 3.022}, {'end': 501.246, 'text': "So, given this, I already have a sample model I've created with Keras and saved to disk.", 'start': 496.463, 'duration': 4.783}, {'end': 505.569, 'text': "If you don't already have access to these Keras model files, don't worry.", 'start': 501.946, 'duration': 3.623}, {'end': 509.551, 'text': "I've included links to Keras GitHub where you can just download these files.", 'start': 505.829, 'duration': 3.722}], 'summary': 'To convert a keras model to tensorflow.js, save the model with weights and architecture in an h5 file using charismodel.save function.', 'duration': 25.697, 'max_score': 483.854, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4483854.jpg'}, {'end': 602.072, 'src': 'embed', 'start': 561.542, 'weight': 0, 'content': [{'end': 573.368, 'text': "This is going to be done directly using Python and this method is for when we're working with a Keras model and we want to go ahead and convert it on the spot to a TensorFlow.js model without necessarily needing to save it to an H5 file first.", 'start': 561.542, 'duration': 11.826}, {'end': 579.392, 'text': "So we're in a Jupyter Notebook where we're importing Keras in the TensorFlow.js library.", 'start': 574.747, 'duration': 4.645}, {'end': 585.658, 'text': "And I'm going to demo this with the VGG16 model because we'll be making use of this one in a future section anyway.", 'start': 579.692, 'duration': 5.966}, {'end': 589.021, 'text': 'But this conversion will work for any model you build with Keras.', 'start': 586.138, 'duration': 2.883}, {'end': 594.586, 'text': "So we have this VGG16 model that's created by calling keras.applications.vgg16.vgg16.", 'start': 589.381, 'duration': 5.205}, {'end': 602.072, 'text': 'And then we call tensorflowjs.converters.saveKerasModel.', 'start': 597.469, 'duration': 4.603}], 'summary': 'Demonstrating direct conversion of keras model to tensorflow.js without saving to h5 file, using vgg16 model.', 'duration': 40.53, 'max_score': 561.542, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4561542.jpg'}, {'end': 686.925, 'src': 'embed', 'start': 653.108, 'weight': 3, 'content': [{'end': 658.011, 'text': 'But the VGG16 model we converted, on the other hand, with over 140 million learnable parameters, has 144 corresponding weight files.', 'start': 653.108, 'duration': 4.903}, {'end': 666.419, 'text': "Alright, so that's how we can convert our existing Keras models into TensorFlow.js models.", 'start': 661.977, 'duration': 4.442}, {'end': 671.88, 'text': "We'll see how these models and their corresponding weights are loaded in the browser in a future section,", 'start': 666.919, 'duration': 4.961}, {'end': 675.081, 'text': 'when we start building our browser application to run these models.', 'start': 671.88, 'duration': 3.201}, {'end': 676.042, 'text': "I'll see you there.", 'start': 675.442, 'duration': 0.6}, {'end': 686.925, 'text': "In this section we'll go through the process of getting a web server set up to host deep learning web applications and serve deep learning models with Express for Node.js.", 'start': 677.702, 'duration': 9.223}], 'summary': 'Vgg16 model has over 140m parameters and 144 weight files. converting keras models to tensorflow.js. setting up web server for deep learning web applications with express for node.js.', 'duration': 33.817, 'max_score': 653.108, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4653108.jpg'}, {'end': 756.238, 'src': 'embed', 'start': 722.999, 'weight': 4, 'content': [{'end': 727.305, 'text': 'it makes sense that we might want to use a JavaScript-based technology to host our apps,', 'start': 722.999, 'duration': 4.306}, {'end': 731.791, 'text': "since we're kind of breaking away from Python and embracing JavaScript in this series.", 'start': 727.305, 'duration': 4.486}, {'end': 735.155, 'text': 'So, enter Express for Node.js.', 'start': 732.672, 'duration': 2.483}, {'end': 741.961, 'text': 'Express is a minimalist web framework very similar to Flask, but is for Node.js, not Python.', 'start': 735.935, 'duration': 6.026}, {'end': 747.266, 'text': "And if you're not already familiar with Node.js, then you're probably wondering what it is as well.", 'start': 742.522, 'duration': 4.744}, {'end': 756.238, 'text': "Node.js, which we'll refer to most of the time as just Node, is an open-source runtime environment that executes JavaScript on the server-side.", 'start': 748.254, 'duration': 7.984}], 'summary': 'Using express for node.js to host apps, embracing javascript over python.', 'duration': 33.239, 'max_score': 722.999, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4722999.jpg'}, {'end': 814.313, 'src': 'embed', 'start': 785.721, 'weight': 5, 'content': [{'end': 788.902, 'text': 'choose the installation for your operating system and get it installed.', 'start': 785.721, 'duration': 3.181}, {'end': 793.304, 'text': "I've installed Node on a Windows machine, but you'll still be able to follow the demos.", 'start': 789.682, 'duration': 3.622}, {'end': 796.465, 'text': "we'll see in a few moments, even if you're running another operating system.", 'start': 793.304, 'duration': 3.161}, {'end': 802.408, 'text': "Alright, after we've got Node installed, we need to create a directory that will hold all of our project files.", 'start': 797.426, 'duration': 4.982}, {'end': 805.95, 'text': "So, we have this directory here I've called TensorFlow.js.", 'start': 803.088, 'duration': 2.862}, {'end': 814.313, 'text': "Within this directory, we'll create a subdirectory called local server, which is where the express code that will run our web server will reside.", 'start': 806.829, 'duration': 7.484}], 'summary': "Install node.js on windows, create project directory 'tensorflow.js', and subdirectory 'local server' for express code.", 'duration': 28.592, 'max_score': 785.721, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4785721.jpg'}, {'end': 862.448, 'src': 'heatmap', 'start': 806.829, 'weight': 0.775, 'content': [{'end': 814.313, 'text': "Within this directory, we'll create a subdirectory called local server, which is where the express code that will run our web server will reside.", 'start': 806.829, 'duration': 7.484}, {'end': 820.637, 'text': "And we'll also create a static directory, which is where our web pages and eventually our models will reside.", 'start': 814.894, 'duration': 5.743}, {'end': 829.703, 'text': 'Within this local server, we create a package.json file, which is going to allow us to specify the packages that our project depends on.', 'start': 821.598, 'duration': 8.105}, {'end': 831.804, 'text': "Let's go ahead and open this file.", 'start': 830.383, 'duration': 1.421}, {'end': 840.692, 'text': "I've opened this with Visual Studio Code, which is a free open source code editor developed by Microsoft that can run on Windows, Linux, and Mac OS.", 'start': 832.645, 'duration': 8.047}, {'end': 847.578, 'text': "This is what we'll be using to write our code, so you can download it and use it yourself as well, or you can use any other editor that you'd like.", 'start': 841.372, 'duration': 6.206}, {'end': 850.46, 'text': 'Alright, back to the package.json file.', 'start': 848.318, 'duration': 2.142}, {'end': 852.642, 'text': 'Within package.json.', 'start': 851.241, 'duration': 1.401}, {'end': 857.186, 'text': "we're going to specify a name for our project, which we're calling TensorFlow.js.", 'start': 852.642, 'duration': 4.544}, {'end': 859.388, 'text': 'all lowercase, per the requirements of this file.', 'start': 857.186, 'duration': 2.202}, {'end': 862.448, 'text': "We'll also specify the version of our project.", 'start': 860.287, 'duration': 2.161}], 'summary': 'Creating subdirectories for express code and static files, specifying package dependencies in package.json', 'duration': 55.619, 'max_score': 806.829, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4806829.jpg'}, {'end': 888.012, 'src': 'embed', 'start': 863.129, 'weight': 6, 'content': [{'end': 869.692, 'text': 'There are some specs that the format of this version has to meet, but most simplistically, it has to be in an x.x.x format.', 'start': 863.129, 'duration': 6.563}, {'end': 872.659, 'text': "so we're just going to go with the default of 1.0..", 'start': 869.692, 'duration': 2.967}, {'end': 881.129, 'text': '0. Alright, name and version are the only two requirements for this file, but there are several other optional items we can add, like a description,', 'start': 872.659, 'duration': 8.47}, {'end': 882.53, 'text': 'the author and a few others.', 'start': 881.129, 'duration': 1.401}, {'end': 888.012, 'text': "We're not going to worry about this stuff, but we are going to add one more thing, the dependencies.", 'start': 883.05, 'duration': 4.962}], 'summary': 'The file format requires a version in x.x.x format with name and version as mandatory fields, and optional items like description and author can be included. additionally, dependencies need to be added.', 'duration': 24.883, 'max_score': 863.129, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4863129.jpg'}, {'end': 1033.358, 'src': 'heatmap', 'start': 954.335, 'weight': 8, 'content': [{'end': 955.576, 'text': "file that we didn't have before.", 'start': 954.335, 'duration': 1.241}, {'end': 958.919, 'text': 'It contains information about the downloaded dependencies.', 'start': 956.457, 'duration': 2.462}, {'end': 960.48, 'text': "Don't delete these things.", 'start': 959.479, 'duration': 1.001}, {'end': 963.982, 'text': 'Alright, so at this point we have node.', 'start': 961.5, 'duration': 2.482}, {'end': 965.463, 'text': 'we have express.', 'start': 963.982, 'duration': 1.481}, {'end': 970.787, 'text': 'now we need to write a node program that will start the express server and will host the files that we specify.', 'start': 965.463, 'duration': 5.324}, {'end': 973.829, 'text': 'Oh, see that makes sense.', 'start': 970.807, 'duration': 3.022}, {'end': 978.665, 'text': "To do this, we'll create this file called server.js.", 'start': 975.543, 'duration': 3.122}, {'end': 985.229, 'text': 'Inside of server.js, we first import express using require express.', 'start': 981.027, 'duration': 4.202}, {'end': 990.713, 'text': 'Using require like this will import the express module and give our program access to it.', 'start': 986.23, 'duration': 4.483}, {'end': 1000.079, 'text': 'You can think of a module and node as being analogous to a library in JavaScript or Python just a group of functions that we want to have access to from within our program.', 'start': 991.353, 'duration': 8.726}, {'end': 1006.035, 'text': 'And then we create an express application using the express module, which we assign to app.', 'start': 1000.993, 'duration': 5.042}, {'end': 1011.777, 'text': 'An express app is essentially a series of calls to functions that we call middleware functions.', 'start': 1007.115, 'duration': 4.662}, {'end': 1021.44, 'text': "Middleware functions have access to the HTTP request and response objects, as well as the next function in the application's request response cycle,", 'start': 1012.577, 'duration': 8.863}, {'end': 1023.541, 'text': 'which just passes control to the next handler.', 'start': 1021.44, 'duration': 2.101}, {'end': 1028.114, 'text': "So within this app, when a request comes in, we're doing two things.", 'start': 1024.631, 'duration': 3.483}, {'end': 1033.358, 'text': "We're first logging information about the request to the terminal where the express server is running.", 'start': 1028.734, 'duration': 4.624}], 'summary': 'Creating a node program to start an express server and host specified files.', 'duration': 79.023, 'max_score': 954.335, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG4954335.jpg'}, {'end': 1101.979, 'src': 'embed', 'start': 1071.133, 'weight': 9, 'content': [{'end': 1075.294, 'text': "I've specified port 81 here, but you can specify whichever unused port you'd like.", 'start': 1071.133, 'duration': 4.161}, {'end': 1081.601, 'text': 'When the server starts up and starts listening on this port, this function will be called, which will log this message,', 'start': 1076.294, 'duration': 5.307}, {'end': 1083.523, 'text': 'letting us know that the server is up and running.', 'start': 1081.601, 'duration': 1.922}, {'end': 1085.906, 'text': "Alright, we're all set up.", 'start': 1084.785, 'duration': 1.121}, {'end': 1092.655, 'text': "Let's drop a sample HTML file into our static directory, then start up the express server and see if we can browse to the page.", 'start': 1086.146, 'duration': 6.509}, {'end': 1101.979, 'text': "We're going to actually just place the web application called predict.html that we created in the Keras deployment series into this directory as a proof of concept.", 'start': 1093.732, 'duration': 8.247}], 'summary': 'Configured server to listen on port 81, logged message when server started, added predict.html to the static directory.', 'duration': 30.846, 'max_score': 1071.133, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41071133.jpg'}], 'start': 46.382, 'title': 'Deploying client-side neural networks with tensorflow.js', 'summary': 'Discusses the deployment of client-side neural networks, highlighting the shift in deployment perspective, advantages of running models in the browser, considerations for client-side deployment, the need for smaller models, suitability of tasks like fine-tuning and inference, and the process of converting a keras model to a tensorflow.js model through the terminal and directly using python.', 'chapters': [{'end': 249.503, 'start': 46.382, 'title': 'Client-side neural networks', 'summary': 'Discusses the deployment of client-side neural networks, highlighting the shift in deployment perspective, advantages of running models in the browser, and considerations for client-side deployment, emphasizing the ease of user interaction and the ability to keep data local to user devices.', 'duration': 203.121, 'highlights': ['The model is embedded inside the frontend application, running in the browser, allowing users to keep their data local, which enhances user privacy and eliminates the need for data to travel across the internet.', 'Deployment of client-side neural networks enables highly interactive and easy-to-use applications accessible to anyone without prerequisites or installs, increasing user engagement and simplicity.', 'The chapter contrasts the traditional deployment perspective with client-side neural network deployment, emphasizing the shift from backend model hosting to embedding the model within the frontend application, leading to a more user-centric and accessible approach.']}, {'end': 431.812, 'start': 249.963, 'title': 'Deploying models in browser with tensorflow.js', 'summary': "Discusses the considerations and advantages of deploying models in the browser using tensorflow.js, highlighting the need for smaller models, the suitability of tasks like fine-tuning and inference, and the use of tensorflow's model converter tool to convert keras models into tensorflow.js models.", 'duration': 181.849, 'highlights': ['The need for smaller models to be loaded and running in the browser, with TensorFlow.js suggesting models of 30 MB or less, compared to the VGG16 model which is over 500 MB.', 'The suitability of tasks like fine-tuning pre-trained models and inference for models running in the browser, as well as the recommendation to use other APIs like Keras or standard TensorFlow in Python for building and training models from scratch.', "The use of TensorFlow's Model Converter tool to convert Keras models into TensorFlow.js models, enabling the utilization of models built and trained with Keras in the browser with TensorFlow.js."]}, {'end': 635.717, 'start': 432.312, 'title': 'Convert keras model to tensorflow.js', 'summary': 'Explains how to convert a keras model to a tensorflow.js model through the terminal and directly using python, demonstrating the process and mentioning the vgg16 model as an example, using tensorflow.js converter tool and savekerasmodel function.', 'duration': 203.405, 'highlights': ['The chapter demonstrates two ways to convert a Keras model to a TensorFlow.js model: through the terminal or command line, and directly using Python, with the latter method allowing conversion without saving to an H5 file first.', 'The process involves using the TensorFlow.js converter tool from a Python environment where Keras is installed, and specifying the input format as charis, the path to the saved H5 file, and the output directory for the converted model.', 'A specific example of the conversion is shown using the VGG16 model with tensorflowjs.converters.saveKerasModel function, highlighting the versatility of the conversion process.']}, {'end': 1085.906, 'start': 636.437, 'title': 'Converting keras models to tensorflow.js', 'summary': 'Outlines the process of converting keras models to tensorflow.js, highlighting the differences in the number of weight files between small and large models and the setup of a web server using express for node.js.', 'duration': 449.469, 'highlights': ['The VGG16 model has over 140 million learnable parameters, resulting in 144 corresponding weight files, compared to a small model with about 640 learnable parameters.', 'Express for Node.js is introduced as a web framework similar to Flask for hosting deep learning applications, allowing server-side JavaScript execution and static file serving.', 'The process of setting up a web server for hosting deep learning web applications and serving models using Express for Node.js is detailed, including the installation of Node.js and the creation of project directories and files.', 'The use of npm to install dependencies specified in the package.json file, such as Express, for the project is explained, emphasizing the role of npm as the node package manager.', 'The creation of a server.js file to start the Express server and host static files is demonstrated, indicating the use of middleware functions and specifying the port for the server to listen on.']}], 'duration': 1039.524, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG446382.jpg', 'highlights': ['The model is embedded inside the frontend application, enhancing user privacy and eliminating data travel (Paragraph 1)', 'Deployment of client-side neural networks enables highly interactive and easy-to-use applications (Paragraph 1)', 'The shift from backend model hosting to embedding the model within the frontend application (Paragraph 1)', 'The need for smaller models to be loaded and running in the browser, with TensorFlow.js suggesting models of 30 MB or less (Paragraph 2)', 'The suitability of tasks like fine-tuning pre-trained models and inference for models running in the browser (Paragraph 2)', "The use of TensorFlow's Model Converter tool to convert Keras models into TensorFlow.js models (Paragraph 2)", 'The chapter demonstrates two ways to convert a Keras model to a TensorFlow.js model: through the terminal or command line, and directly using Python (Paragraph 3)', 'The process involves using the TensorFlow.js converter tool from a Python environment where Keras is installed (Paragraph 3)', 'A specific example of the conversion is shown using the VGG16 model with tensorflowjs.converters.saveKerasModel function (Paragraph 3)', 'The VGG16 model has over 140 million learnable parameters, resulting in 144 corresponding weight files, compared to a small model with about 640 learnable parameters (Paragraph 4)', 'Express for Node.js is introduced as a web framework similar to Flask for hosting deep learning applications (Paragraph 4)', 'The process of setting up a web server for hosting deep learning web applications and serving models using Express for Node.js is detailed (Paragraph 4)']}, {'end': 1776.711, 'segs': [{'end': 1147.513, 'src': 'heatmap', 'start': 1093.732, 'weight': 1, 'content': [{'end': 1101.979, 'text': "We're going to actually just place the web application called predict.html that we created in the Keras deployment series into this directory as a proof of concept.", 'start': 1093.732, 'duration': 8.247}, {'end': 1103.66, 'text': 'So we place that here.', 'start': 1102.479, 'duration': 1.181}, {'end': 1107.163, 'text': "You can use any HTML file you'd like, though, to test this.", 'start': 1104.3, 'duration': 2.863}, {'end': 1110.365, 'text': 'Now, to start Express, we use PowerShell.', 'start': 1108.203, 'duration': 2.162}, {'end': 1116.29, 'text': "Let's make sure we're inside of the local server directory, and we run node server js.", 'start': 1110.946, 'duration': 5.344}, {'end': 1124.198, 'text': 'We get our output message letting us know that Express is serving files from our static directory on port 81.', 'start': 1117.351, 'duration': 6.847}, {'end': 1127.06, 'text': "So now let's browse to localhost or whatever.", 'start': 1124.198, 'duration': 2.862}, {'end': 1136.366, 'text': "the IP address is that you're running Express on port 81, slash predict.html, which is the name of the file we put into the static directory.", 'start': 1127.06, 'duration': 9.306}, {'end': 1138.387, 'text': 'And here we go.', 'start': 1137.687, 'duration': 0.7}, {'end': 1141.269, 'text': 'This is indeed the web page we wanted to be served.', 'start': 1138.607, 'duration': 2.662}, {'end': 1147.513, 'text': 'We can also check out the output from this request in PowerShell to view the logging that we specified.', 'start': 1142.85, 'duration': 4.663}], 'summary': 'Placed predict.html in directory, served by express on port 81.', 'duration': 53.781, 'max_score': 1093.732, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41093732.jpg'}, {'end': 1136.366, 'src': 'embed', 'start': 1108.203, 'weight': 1, 'content': [{'end': 1110.365, 'text': 'Now, to start Express, we use PowerShell.', 'start': 1108.203, 'duration': 2.162}, {'end': 1116.29, 'text': "Let's make sure we're inside of the local server directory, and we run node server js.", 'start': 1110.946, 'duration': 5.344}, {'end': 1124.198, 'text': 'We get our output message letting us know that Express is serving files from our static directory on port 81.', 'start': 1117.351, 'duration': 6.847}, {'end': 1127.06, 'text': "So now let's browse to localhost or whatever.", 'start': 1124.198, 'duration': 2.862}, {'end': 1136.366, 'text': "the IP address is that you're running Express on port 81, slash predict.html, which is the name of the file we put into the static directory.", 'start': 1127.06, 'duration': 9.306}], 'summary': 'Using powershell, start express on port 81 to serve files from the static directory.', 'duration': 28.163, 'max_score': 1108.203, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41108203.jpg'}, {'end': 1247.98, 'src': 'heatmap', 'start': 1192.447, 'weight': 0.72, 'content': [{'end': 1195.509, 'text': 'Recall, this was the app we built in that previous series.', 'start': 1192.447, 'duration': 3.062}, {'end': 1204.314, 'text': 'We had a fine-tuned VGG16 model running in the backend as a web service and as a user we would select an image of a cat or dog,', 'start': 1196.389, 'duration': 7.925}, {'end': 1206.816, 'text': 'submit the image to the model and receive a prediction.', 'start': 1204.314, 'duration': 2.502}, {'end': 1213.873, 'text': "Now, the idea of the app we'll develop with TensorFlow.js will be similar, but let's discuss the differences.", 'start': 1208.129, 'duration': 5.744}, {'end': 1216.774, 'text': "Can you see the source code that's generating the responses?", 'start': 1214.373, 'duration': 2.401}, {'end': 1225.24, 'text': 'Um yeah, we can, and we will, but first know that our model will be running entirely in the browser.', 'start': 1219.216, 'duration': 6.024}, {'end': 1231.239, 'text': 'Our app will therefore consist only of a front-end application developed with HTML and JavaScript.', 'start': 1226.112, 'duration': 5.127}, {'end': 1233.823, 'text': "So, here's what the new app will do.", 'start': 1232.04, 'duration': 1.783}, {'end': 1241.373, 'text': 'The general layout will be similar to the one we just went over where a user will select an image, submit it to the model, and get a prediction.', 'start': 1234.684, 'duration': 6.689}, {'end': 1247.98, 'text': "We won't be restricted to choosing only cat and dog images, though, because we won't be using fine-tuned models this time.", 'start': 1242.074, 'duration': 5.906}], 'summary': 'Developing a web app with tensorflow.js for browser-based image predictions.', 'duration': 55.533, 'max_score': 1192.447, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41192447.jpg'}, {'end': 1303.603, 'src': 'embed', 'start': 1277.869, 'weight': 4, 'content': [{'end': 1284.131, 'text': "Well, we're first going to go against this recommendation and use VGG16 as our model, which is over 500MB in size.", 'start': 1277.869, 'duration': 6.262}, {'end': 1287.664, 'text': 'Nice priorities.', 'start': 1286.943, 'duration': 0.721}, {'end': 1293.351, 'text': "We'll see how that works out for us, but you can imagine that it may be problematic.", 'start': 1288.665, 'duration': 4.686}, {'end': 1295.033, 'text': 'No worries, though.', 'start': 1294.152, 'duration': 0.881}, {'end': 1299.398, 'text': "we'll have MobileNet to the rescue coming in at only about 16 megabytes,", 'start': 1295.033, 'duration': 4.365}, {'end': 1303.603, 'text': "so we'll get to see how these two models compare to each other performance-wise in the browser.", 'start': 1299.398, 'duration': 4.205}], 'summary': 'Using vgg16 (over 500mb) and mobilenet (16mb) to compare performance in the browser.', 'duration': 25.734, 'max_score': 1277.869, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41277869.jpg'}, {'end': 1369.436, 'src': 'embed', 'start': 1343.05, 'weight': 9, 'content': [{'end': 1348.013, 'text': 'Each of these directories will contain the model.json and the corresponding weight files for each model.', 'start': 1343.05, 'duration': 4.963}, {'end': 1351.636, 'text': 'Navigating inside of vgg16, we can see that.', 'start': 1348.894, 'duration': 2.742}, {'end': 1353.911, 'text': 'To get these files here.', 'start': 1352.711, 'duration': 1.2}, {'end': 1364.254, 'text': 'I simply went through the conversion process in Python of loading BGG16 in MobileNet with Keras and then converting the models with the TensorFlow.js converter we previously discussed.', 'start': 1353.911, 'duration': 10.343}, {'end': 1369.436, 'text': 'So, follow that earlier section to get this same output to place in your model directories.', 'start': 1364.735, 'duration': 4.701}], 'summary': 'Directories contain model.json and weight files for vgg16 and mobilenet models.', 'duration': 26.386, 'max_score': 1343.05, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41343050.jpg'}, {'end': 1417.679, 'src': 'embed', 'start': 1387.788, 'weight': 10, 'content': [{'end': 1389.649, 'text': "Let's open it up and take a look at the structure.", 'start': 1387.788, 'duration': 1.861}, {'end': 1398.397, 'text': 'So we just have this JavaScript object called ImageNetClasses that contains the key value pairs of the ImageNet classes with associated IDs.', 'start': 1390.37, 'duration': 8.027}, {'end': 1404.129, 'text': "Alright, now let's open the Predict with TFJS HTML file and jump into the code.", 'start': 1399.485, 'duration': 4.644}, {'end': 1411.194, 'text': "We're starting off in the head by specifying the title of our webpage and importing the styling from this CSS file.", 'start': 1405.089, 'duration': 6.105}, {'end': 1413.556, 'text': 'For all the styling on the page.', 'start': 1412.095, 'duration': 1.461}, {'end': 1417.679, 'text': "we'll be using Bootstrap, which is an open source library for developing HTML,", 'start': 1413.556, 'duration': 4.123}], 'summary': 'Examining javascript object imagenetclasses with associated ids, using bootstrap for styling.', 'duration': 29.891, 'max_score': 1387.788, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41387788.jpg'}, {'end': 1452.251, 'src': 'embed', 'start': 1429.025, 'weight': 6, 'content': [{'end': 1436.247, 'text': 'Now Bootstrap uses a grid layout where you can think of the web page having containers that could be interpreted as grids,', 'start': 1429.025, 'duration': 7.222}, {'end': 1441.968, 'text': 'and then UI elements on the page are organized into the rows and the columns that make up those grids.', 'start': 1436.247, 'duration': 5.721}, {'end': 1448.19, 'text': "By setting the elements class attributes, that's how Bootstrap knows what type of styling to apply to them.", 'start': 1442.788, 'duration': 5.402}, {'end': 1452.251, 'text': "So given this, that's how we're organizing our UI elements.", 'start': 1449.17, 'duration': 3.081}], 'summary': 'Bootstrap uses a grid layout to organize ui elements.', 'duration': 23.226, 'max_score': 1429.025, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41429025.jpg'}, {'end': 1553.206, 'src': 'embed', 'start': 1495.986, 'weight': 0, 'content': [{'end': 1499.707, 'text': 'So this single line is all it takes to get TensorFlow.js into our app.', 'start': 1495.986, 'duration': 3.721}, {'end': 1503.949, 'text': 'Then we import the ImageNet class.js file we checked out earlier.', 'start': 1500.548, 'duration': 3.401}, {'end': 1508.61, 'text': 'And lastly, we import our predict.js file which, as mentioned earlier,', 'start': 1504.709, 'duration': 3.901}, {'end': 1512.772, 'text': 'contains all the logic for what our app does when a user supplies an image to it.', 'start': 1508.61, 'duration': 4.162}, {'end': 1515.337, 'text': "Alright, so that's it for the HTML.", 'start': 1513.595, 'duration': 1.742}, {'end': 1518.181, 'text': "Let's check out the page and explore the grid layout.", 'start': 1515.678, 'duration': 2.503}, {'end': 1522.787, 'text': 'First, we start up our express server, which we learned how to do in the last section.', 'start': 1518.701, 'duration': 4.086}, {'end': 1531.458, 'text': "Then, in our browser, we'll navigate to the IP address where our express server is running, port 81, predict with tfjs.html.", 'start': 1523.848, 'duration': 7.61}, {'end': 1534.272, 'text': "And here's our page.", 'start': 1533.371, 'duration': 0.901}, {'end': 1540.276, 'text': "It's pretty empty right now because we haven't selected an image, but once we write the JavaScript logic to handle what to do?", 'start': 1534.632, 'duration': 5.644}, {'end': 1544.88, 'text': 'when we select an image, then the name of the selected image file will be displayed here.', 'start': 1540.276, 'duration': 4.604}, {'end': 1549.223, 'text': 'the image will be displayed in the image section and upon clicking the predict button,', 'start': 1544.88, 'duration': 4.343}, {'end': 1553.206, 'text': 'the predictions for the image from the model will be displayed in this prediction section.', 'start': 1549.223, 'duration': 3.983}], 'summary': 'Integrating tensorflow.js into app, exploring grid layout, handling image predictions.', 'duration': 57.22, 'max_score': 1495.986, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41495986.jpg'}, {'end': 1675.302, 'src': 'embed', 'start': 1648.595, 'weight': 5, 'content': [{'end': 1652.299, 'text': "Now, we'll focus on the JavaScript that handles all the logic for this app.", 'start': 1648.595, 'duration': 3.704}, {'end': 1656.243, 'text': "We'll also start getting acquainted with the TensorFlow.js API.", 'start': 1652.96, 'duration': 3.283}, {'end': 1659.587, 'text': "Without further ado, let's get right into the code.", 'start': 1657.104, 'duration': 2.483}, {'end': 1668.732, 'text': 'Recall in the last section we created this predict.js file within the static directory but left it empty.', 'start': 1662.944, 'duration': 5.788}, {'end': 1675.302, 'text': 'This file now contains the JavaScript logic that handles what will happen when a user submits an image to the application.', 'start': 1669.193, 'duration': 6.109}], 'summary': 'Learning javascript logic and tensorflow.js for app development.', 'duration': 26.707, 'max_score': 1648.595, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41648595.jpg'}, {'end': 1783.938, 'src': 'embed', 'start': 1756.695, 'weight': 8, 'content': [{'end': 1762.458, 'text': "Here we have what's called an IIFE, or Immediately Invoked Function Expression.", 'start': 1756.695, 'duration': 5.763}, {'end': 1766.521, 'text': "An IIFE is a function that runs as soon as it's defined.", 'start': 1763.139, 'duration': 3.382}, {'end': 1774.288, 'text': 'We can see this is structured by placing the function within parentheses and then specifying the call to the function,', 'start': 1767.441, 'duration': 6.847}, {'end': 1776.711, 'text': 'with these parentheses that immediately follow.', 'start': 1774.288, 'duration': 2.423}, {'end': 1783.938, 'text': 'Within this function, we load the model by calling the TensorFlow.js function tf.loadmodel,', 'start': 1777.672, 'duration': 6.266}], 'summary': 'Iife runs as soon as defined, loads model using tf.loadmodel', 'duration': 27.243, 'max_score': 1756.695, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41756695.jpg'}], 'start': 1086.146, 'title': 'Setting up express server and client-side deep learning app', 'summary': 'Demonstrates setting up an express server for hosting a web application and configuring it to serve models and tensorflow.js apps. it also covers creating a client-side deep learning application utilizing vgg16 and mobilenet models, importing required libraries, exploring grid layout using bootstrap, and delving into javascript logic for user interactions with the tensorflow.js api.', 'chapters': [{'end': 1277.249, 'start': 1086.146, 'title': 'Setting up express server for web app', 'summary': 'Demonstrates setting up an express server to host a web application, enabling browsing to the page and configuring it to serve models and host tensorflow.js apps.', 'duration': 191.103, 'highlights': ['Express server is started and serving files from the static directory on port 81, enabling browsing to the web page.', 'Node and Express are set up to serve models and host TensorFlow.js apps, preparing for building the UI for the first client-side neural network application.', 'The new app developed with HTML and JavaScript will allow users to select an image, submit it to the model, and receive the top 5 predictions for that image from the ImageNet classes.']}, {'end': 1776.711, 'start': 1277.869, 'title': 'Client-side deep learning app', 'summary': 'Covers setting up the client-side deep learning application, utilizing vgg16 and mobilenet models, creating necessary files and directories, importing required libraries, and exploring the grid layout using bootstrap. it then delves into the javascript logic for handling user interactions and getting acquainted with the tensorflow.js api.', 'duration': 498.842, 'highlights': ['The chapter covers setting up the client-side deep learning application, utilizing VGG16 and MobileNet models, creating necessary files and directories, importing required libraries, and exploring the grid layout using Bootstrap.', 'VGG16 model, over 500MB in size, is being used along with MobileNet, which is about 16 megabytes, to compare their performance in the browser.', 'Directories for TensorFlow.js models, including subdirectories for mobilenet and vgg16, are created, each containing the model.json and corresponding weight files.', "The HTML file 'predictWithTFJS.html' and JavaScript file 'predictJS' are created to serve as the web app and hold all the JavaScript logic, respectively.", "The 'ImageNetClass.js' file containing all the ImageNet classes is included, and the structure of the JavaScript object 'ImageNetClasses' is explained.", "The UI elements of the web app are organized using Bootstrap's grid layout, with containers, rows, and columns.", "The JavaScript logic in the 'predict.js' file is detailed, including handling image file selection, reading the contents of the selected file, and displaying predictions for the image from the model.", 'An Immediately Invoked Function Expression (IIFE) is utilized to define and run a function that handles the image file selection and loading the image.']}], 'duration': 690.565, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41086146.jpg', 'highlights': ['Express server is started and serving files from the static directory on port 81, enabling browsing to the web page.', 'Node and Express are set up to serve models and host TensorFlow.js apps, preparing for building the UI for the first client-side neural network application.', 'The new app developed with HTML and JavaScript will allow users to select an image, submit it to the model, and receive the top 5 predictions for that image from the ImageNet classes.', 'The chapter covers setting up the client-side deep learning application, utilizing VGG16 and MobileNet models, creating necessary files and directories, importing required libraries, and exploring the grid layout using Bootstrap.', 'VGG16 model, over 500MB in size, is being used along with MobileNet, which is about 16 megabytes, to compare their performance in the browser.', "The HTML file 'predictWithTFJS.html' and JavaScript file 'predictJS' are created to serve as the web app and hold all the JavaScript logic, respectively.", "The UI elements of the web app are organized using Bootstrap's grid layout, with containers, rows, and columns.", "The JavaScript logic in the 'predict.js' file is detailed, including handling image file selection, reading the contents of the selected file, and displaying predictions for the image from the model.", 'An Immediately Invoked Function Expression (IIFE) is utilized to define and run a function that handles the image file selection and loading the image.', 'Directories for TensorFlow.js models, including subdirectories for mobilenet and vgg16, are created, each containing the model.json and corresponding weight files.', "The 'ImageNetClass.js' file containing all the ImageNet classes is included, and the structure of the JavaScript object 'ImageNetClasses' is explained."]}, {'end': 2318.367, 'segs': [{'end': 1823.831, 'src': 'embed', 'start': 1798.427, 'weight': 3, 'content': [{'end': 1806.511, 'text': "We're first going to be working with vgg16 as our model, so I've specified the URL to where the model.json file for vgg16 resides.", 'start': 1798.427, 'duration': 8.084}, {'end': 1815.135, 'text': 'Now, tf.loadModel returns a promise, meaning that this function promises to return the model at some point in the future.', 'start': 1807.471, 'duration': 7.664}, {'end': 1823.831, 'text': 'This await keyword pauses the execution of this wrapping function until the promise is resolved and the model is loaded.', 'start': 1816.428, 'duration': 7.403}], 'summary': 'Using vgg16 model, tf.loadmodel promises future return, await pauses execution.', 'duration': 25.404, 'max_score': 1798.427, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41798427.jpg'}, {'end': 1919.297, 'src': 'heatmap', 'start': 1856.764, 'weight': 4, 'content': [{'end': 1863.589, 'text': 'So here we are in predictwithtfjs.html, and you can see that right within this first div, the container.', 'start': 1856.764, 'duration': 6.825}, {'end': 1866.431, 'text': "I've inserted this row where the progress bar is embedded.", 'start': 1863.589, 'duration': 2.842}, {'end': 1869.894, 'text': "We'll see it in action within the UI at the end of this video.", 'start': 1867.192, 'duration': 2.702}, {'end': 1877.43, 'text': 'Alright, jumping back over to the JavaScript, we now need to write the logic for what happens when the predict button is clicked.', 'start': 1870.888, 'duration': 6.542}, {'end': 1882.552, 'text': 'When a user clicks the predict button, we first get the image from the selected image element.', 'start': 1878.23, 'duration': 4.322}, {'end': 1890.274, 'text': 'Then we need to transform the image into a rank 4 tensor object of floats with height and width dimensions of 224x224,,', 'start': 1883.312, 'duration': 6.962}, {'end': 1891.374, 'text': "since that's what the model expects.", 'start': 1890.274, 'duration': 1.1}, {'end': 1903.499, 'text': 'To do this, we create a tensor object from the image by calling the TensorFlow.js function tf.fromPixels and passing our image to it.', 'start': 1894.929, 'duration': 8.57}, {'end': 1915.151, 'text': "We then resize the image to 224 by 224, cast the tensor's type to float32, and expand the tensor's dimensions to be of rank 4.", 'start': 1904.38, 'duration': 10.771}, {'end': 1919.297, 'text': "We're doing all of this because the model expects the image data to be organized in this way.", 'start': 1915.151, 'duration': 4.146}], 'summary': 'In predictwithtfjs.html, logic for predict button is implemented, transforming image into a rank 4 tensor object.', 'duration': 24.368, 'max_score': 1856.764, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41856764.jpg'}, {'end': 2035.835, 'src': 'embed', 'start': 2006.67, 'weight': 2, 'content': [{'end': 2010.552, 'text': 'Predict returns a tensor of the output predictions for the given input.', 'start': 2006.67, 'duration': 3.882}, {'end': 2013.714, 'text': "We then call data on the prediction's tensor,", 'start': 2011.553, 'duration': 2.161}, {'end': 2020.858, 'text': 'which asynchronously loads the values from the tensor and returns a promise of a typed array after the computation completes.', 'start': 2013.714, 'duration': 7.144}, {'end': 2025.749, 'text': 'Notice the await and async keywords here that we discussed earlier.', 'start': 2021.747, 'duration': 4.002}, {'end': 2030.992, 'text': 'So this predictions array is going to be made up of 1000 elements,', 'start': 2026.79, 'duration': 4.202}, {'end': 2035.835, 'text': 'each of which corresponds to the prediction probability for an individual ImageNet class.', 'start': 2030.992, 'duration': 4.843}], 'summary': 'Predict function generates 1000 prediction probabilities for imagenet classes.', 'duration': 29.165, 'max_score': 2006.67, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42006670.jpg'}, {'end': 2177.598, 'src': 'embed', 'start': 2116.9, 'weight': 0, 'content': [{'end': 2123.021, 'text': 'So I paused the video while this model was continuing to load and it ended up taking about 40 seconds to complete.', 'start': 2116.9, 'duration': 6.121}, {'end': 2125.061, 'text': 'Not great.', 'start': 2123.941, 'duration': 1.12}, {'end': 2129.542, 'text': 'It may even take longer for you depending on your specific computing resources.', 'start': 2125.682, 'duration': 3.86}, {'end': 2137.244, 'text': "Remember though, I said we'd run into some less than ideal situations with running such a large model like VGG16 in the browser.", 'start': 2130.402, 'duration': 6.842}, {'end': 2144.585, 'text': 'So the time it takes to load the model is the first issue.', 'start': 2141.944, 'duration': 2.641}, {'end': 2151.083, 'text': "We've got over 500 MB of files to load into the browser for this model, hence the long loading time.", 'start': 2145.6, 'duration': 5.483}, {'end': 2155.785, 'text': "Alright, well our model is loaded, so let's choose an image and predict on it.", 'start': 2152.163, 'duration': 3.622}, {'end': 2172.853, 'text': 'Hmm, about a 5 second wait time to get a prediction on a single image.', 'start': 2168.371, 'duration': 4.482}, {'end': 2177.598, 'text': 'Again, not great.', 'start': 2176.357, 'duration': 1.241}], 'summary': 'Loading vgg16 model took 40 seconds, predicting image took 5 seconds.', 'duration': 60.698, 'max_score': 2116.9, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42116900.jpg'}], 'start': 1777.672, 'title': 'Loading and processing vgg16 model', 'summary': 'Covers loading vgg16 model using tf.loadmodel, transforming image data into a rank 4 tensor, emphasizing the need for preprocessing functions, and promising 1000 elements in the predictions array. it also addresses processing image predictions, model loading time of 40 seconds, and 5 seconds to predict on a single image.', 'chapters': [{'end': 2030.992, 'start': 1777.672, 'title': 'Loading and preprocessing vgg16 model', 'summary': 'Covers loading the vgg16 model using tf.loadmodel, transforming image data into a rank 4 tensor, and highlighting the need for preprocessing functions for vgg16 model in tensorflow.js, with a promise of 1000 elements in the predictions array.', 'duration': 253.32, 'highlights': ['The model is loaded using tf.loadModel, which returns a promise, and the await keyword is used to pause the execution until the model is loaded.', 'The image is transformed into a rank 4 tensor of floats with dimensions of 224x224, using tf.fromPixels and other TensorFlow.js functions.', 'VGG16 model needs specific preprocessing of image data, and the need to build preprocessing functions for it is highlighted, as TensorFlow.js does not include these functions.', "The predict function is called on the model, returning a tensor of output predictions, and calling data on the prediction's tensor asynchronously loads the values and returns a promise of a typed array with 1000 elements."]}, {'end': 2318.367, 'start': 2030.992, 'title': 'Tensor operations and preprocessing for vgg16', 'summary': 'Focuses on processing image predictions using tensor operations and preprocessing for vgg16, addressing issues such as model loading time and prediction latency, with the model taking about 40 seconds to load and 5 seconds to predict on a single image.', 'duration': 287.375, 'highlights': ['The model took about 40 seconds to load, indicating a significant loading time for the large VGG16 model.', 'There is approximately a 5-second wait time to get a prediction on a single image, highlighting the latency issues in prediction.', 'The preprocessing for VGG16 is essential for accurate predictions, and further exposure to TensorFlow.js API and tensor operations will be covered to address these issues.']}], 'duration': 540.695, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG41777672.jpg', 'highlights': ['The model took about 40 seconds to load, indicating a significant loading time for the large VGG16 model.', 'There is approximately a 5-second wait time to get a prediction on a single image, highlighting the latency issues in prediction.', "The predict function is called on the model, returning a tensor of output predictions, and calling data on the prediction's tensor asynchronously loads the values and returns a promise of a typed array with 1000 elements.", 'The model is loaded using tf.loadModel, which returns a promise, and the await keyword is used to pause the execution until the model is loaded.', 'The image is transformed into a rank 4 tensor of floats with dimensions of 224x224, using tf.fromPixels and other TensorFlow.js functions.']}, {'end': 2947.825, 'segs': [{'end': 2359.815, 'src': 'embed', 'start': 2318.367, 'weight': 0, 'content': [{'end': 2326.676, 'text': 'the authors state quote the only preprocessing we do is subtracting the mean RGB value computed on the training set from each pixel.', 'start': 2318.367, 'duration': 8.309}, {'end': 2328.278, 'text': "Let's break this down a bit.", 'start': 2327.197, 'duration': 1.081}, {'end': 2336.628, 'text': 'We know that ImageNet was the training set for VGG16, so ImageNet is the dataset for which the mean RGB values are calculated.', 'start': 2328.939, 'duration': 7.689}, {'end': 2346.031, 'text': 'To do this calculation for a single color channel, say red, we compute the average red value of all the pixels across every ImageNet image.', 'start': 2337.148, 'duration': 8.883}, {'end': 2349.352, 'text': 'The same goes for the other two color channels, green and blue.', 'start': 2346.731, 'duration': 2.621}, {'end': 2357.214, 'text': 'Then, to preprocess each image, we subtract the mean red value from the original red value in each pixel.', 'start': 2350.132, 'duration': 7.082}, {'end': 2359.815, 'text': 'We do the same for the green and blue values as well.', 'start': 2357.734, 'duration': 2.081}], 'summary': 'Authors preprocess images by subtracting mean rgb values from each pixel based on imagenet training set.', 'duration': 41.448, 'max_score': 2318.367, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42318367.jpg'}, {'end': 2491.713, 'src': 'embed', 'start': 2469.326, 'weight': 1, 'content': [{'end': 2478.109, 'text': 'Each element along the second axis of our 224x224x3 tensor represents a pixel containing a red, green, and blue value, in that order.', 'start': 2469.326, 'duration': 8.783}, {'end': 2482.971, 'text': 'So, the zeroth index in each of these pixels is the red value of the pixel.', 'start': 2478.609, 'duration': 4.362}, {'end': 2491.713, 'text': 'After gathering all the red values, we need to center them by subtracting the meanImageNet red value from each red value in our tensor.', 'start': 2484.17, 'duration': 7.543}], 'summary': 'Processing 224x224x3 tensor pixel values by centering around meanimagenet red value.', 'duration': 22.387, 'max_score': 2469.326, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42469326.jpg'}, {'end': 2563.667, 'src': 'embed', 'start': 2539.568, 'weight': 2, 'content': [{'end': 2545.691, 'text': 'Great, now we have a one-dimensional tensor containing all the centered red values from every pixel in our original tensor.', 'start': 2539.568, 'duration': 6.123}, {'end': 2550.653, 'text': 'We need to go through this same process now again to get the centered green and blue values.', 'start': 2546.491, 'duration': 4.162}, {'end': 2556.143, 'text': 'At a brief glance, you can see the code is almost exactly the same as what we went through for the red values.', 'start': 2551.601, 'duration': 4.542}, {'end': 2563.667, 'text': "The only exceptions are the indices we're passing to tf.gather and the mean ImageNet values we're passing to tf.scaler.", 'start': 2556.824, 'duration': 6.843}], 'summary': 'Processing red, green, and blue values from a tensor for imagenet', 'duration': 24.099, 'max_score': 2539.568, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42539568.jpg'}, {'end': 2629.224, 'src': 'embed', 'start': 2603.32, 'weight': 4, 'content': [{'end': 2609.242, 'text': 'This tensor represents 50176 pixels, each containing a red, green, and blue value.', 'start': 2603.32, 'duration': 5.922}, {'end': 2614.004, 'text': 'We need to reshape this tensor to be in the form that the model expects, which is 224x224x3.', 'start': 2610.143, 'duration': 3.861}, {'end': 2622.917, 'text': "Now remember at the start we said that we'd need to reverse the order of the color channels of our image from RGB to BGR.", 'start': 2616.61, 'duration': 6.307}, {'end': 2629.224, 'text': 'So we do that using the TensorFlow.js function reverse to reverse our tensor along the specified axis.', 'start': 2623.397, 'duration': 5.827}], 'summary': 'Reshape 50176 pixels to 224x224x3, reverse rgb to bgr using tensorflow.js.', 'duration': 25.904, 'max_score': 2603.32, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42603320.jpg'}], 'start': 2318.367, 'title': 'Preprocessing image data', 'summary': 'Discusses vgg16 preprocessing techniques involving zero centering rgb values and reversing pixel order, as well as tensorflow.js image data preprocessing, resulting in a pre-processed tensor with 224x224x3 dimensions for model prediction.', 'chapters': [{'end': 2398.449, 'start': 2318.367, 'title': 'Vgg16 preprocessing techniques', 'summary': 'Discusses the preprocessing techniques used for vgg16, including zero centering the mean rgb values of the imagenet dataset and reversing the order of each pixel from rgb to bgr.', 'duration': 80.082, 'highlights': ['The preprocessing for VGG16 involves subtracting the mean RGB value computed on the training set from each pixel, zero centering each color channel with respect to the ImageNet dataset.', 'To calculate the mean RGB values for the ImageNet dataset, the authors compute the average value of each color channel across all the pixels in the dataset.', 'Another preprocessing step involves reversing the order of each pixel from RGB to BGR, as the CAFE library used for training VGG16 uses a BGR color scheme for reading images.']}, {'end': 2947.825, 'start': 2399.574, 'title': 'Preprocessing image data with tensorflow.js', 'summary': 'Explains the process of preprocessing image data using tensorflow.js, involving operations such as centering red, green, and blue values, reshaping tensors, reversing color channels, and expanding dimensions, resulting in a pre-processed tensor object containing 224x224x3 dimensions, which is then used for model prediction.', 'duration': 548.251, 'highlights': ['The process involves centering red, green, and blue values by subtracting mean ImageNet values, reshaping tensors, and reversing color channels from RGB to BGR, resulting in a pre-processed tensor object containing 224x224x3 dimensions, which is then used for model prediction.', 'The TensorFlow.js function tf.gather is used to gather red, green, and blue values from the tensor, and the tf.sub function is used to subtract the mean red value from each red value in the tensor, resulting in a new tensor with centered red values.', 'The tensor containing centered red, green, and blue values is then reshaped into a one-dimensional tensor, and the process is repeated to obtain centered green and blue values, resulting in a centered RGB object containing one-dimensional tensors of centered red, green, and blue values.', "The centered red, green, and blue tensors are stacked along axis 1 to create a preprocessed tensor representing 50176 pixels, then reshaped into 224x224x3 to match the model's expectation, followed by reversing the color channels from RGB to BGR and expanding the dimensions to transform the tensor from rank 3 to rank 4 for model prediction.", 'The chapter also mentions the possibility of achieving the same pre-processed tensor using broadcasting, and highlights the upcoming coverage of broadcasting in a future section to explore alternative methods of tensor operations.']}], 'duration': 629.458, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42318367.jpg', 'highlights': ['The preprocessing for VGG16 involves subtracting the mean RGB value computed on the training set from each pixel, zero centering each color channel with respect to the ImageNet dataset.', 'The process involves centering red, green, and blue values by subtracting mean ImageNet values, reshaping tensors, and reversing color channels from RGB to BGR, resulting in a pre-processed tensor object containing 224x224x3 dimensions, which is then used for model prediction.', 'The tensor containing centered red, green, and blue values is then reshaped into a one-dimensional tensor, and the process is repeated to obtain centered green and blue values, resulting in a centered RGB object containing one-dimensional tensors of centered red, green, and blue values.', 'To calculate the mean RGB values for the ImageNet dataset, the authors compute the average value of each color channel across all the pixels in the dataset.', "The centered red, green, and blue tensors are stacked along axis 1 to create a preprocessed tensor representing 50176 pixels, then reshaped into 224x224x3 to match the model's expectation, followed by reversing the color channels from RGB to BGR and expanding the dimensions to transform the tensor from rank 3 to rank 4 for model prediction."]}, {'end': 3526.903, 'segs': [{'end': 3015.122, 'src': 'embed', 'start': 2990.034, 'weight': 3, 'content': [{'end': 3000.417, 'text': 'If we just print out this list using console dot log indices, we get back that this is an array with three things in it.', 'start': 2990.034, 'duration': 10.383}, {'end': 3005.179, 'text': "We know that each element in this array is a tensor, so let's access one of them.", 'start': 3001.017, 'duration': 4.162}, {'end': 3006.959, 'text': "Let's get the first tensor.", 'start': 3005.659, 'duration': 1.3}, {'end': 3015.122, 'text': "And it might help if we spell indices correctly, so let's try that again.", 'start': 3012.161, 'duration': 2.961}], 'summary': 'Access array elements and tensors, with 3 items in array.', 'duration': 25.088, 'max_score': 2990.034, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42990034.jpg'}, {'end': 3182.245, 'src': 'embed', 'start': 3155.412, 'weight': 4, 'content': [{'end': 3161.493, 'text': "This is where we're centering each red value by subtracting the mean red value from ImageNet using the sub function.", 'start': 3155.412, 'duration': 6.081}, {'end': 3165.894, 'text': "Let's make a new variable called centered red and mimic this operation.", 'start': 3162.233, 'duration': 3.661}, {'end': 3174.016, 'text': "So we'll define centered red equal to red and then call the sub function.", 'start': 3167.134, 'duration': 6.882}, {'end': 3177.457, 'text': "Now let's print centered red.", 'start': 3176.037, 'duration': 1.42}, {'end': 3182.245, 'text': 'and scroll up to the top.', 'start': 3181.144, 'duration': 1.101}], 'summary': 'Centered red values are obtained by subtracting the mean red value from imagenet using the sub function.', 'duration': 26.833, 'max_score': 3155.412, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43155412.jpg'}, {'end': 3254.384, 'src': 'embed', 'start': 3230.828, 'weight': 2, 'content': [{'end': 3237.593, 'text': 'The next step is to reshape this tensor to be a rank 1 tensor of size 50, 176.', 'start': 3230.828, 'duration': 6.765}, {'end': 3243.697, 'text': 'So we just want to bring all the centered red values together, which are currently each residing in their own individual tensors.', 'start': 3237.593, 'duration': 6.104}, {'end': 3248.26, 'text': "So to mimic this reshape call, we'll make a new variable called reshaped red.", 'start': 3244.637, 'duration': 3.623}, {'end': 3254.384, 'text': "So we'll scroll back down in our debugger console, and we'll copy this reshape call.", 'start': 3249.16, 'duration': 5.224}], 'summary': 'Reshape tensor to 1d with size 50, 176', 'duration': 23.556, 'max_score': 3230.828, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43230828.jpg'}, {'end': 3329.965, 'src': 'embed', 'start': 3301.781, 'weight': 1, 'content': [{'end': 3307.465, 'text': "This is where we're bringing our centered red, green, and blue values all together into a new processed tensor.", 'start': 3301.781, 'duration': 5.684}, {'end': 3313.689, 'text': "So from the console, let's run this first stack operation by creating a variable called stacked tensor.", 'start': 3308.165, 'duration': 5.524}, {'end': 3320.193, 'text': "So I'll create stack tensor, set that equal to this stack call.", 'start': 3314.669, 'duration': 5.524}, {'end': 3329.965, 'text': 'Remember, we just saw that reshaped red ended up being a rank 1 tensor of shape 50, 176.', 'start': 3323.223, 'duration': 6.742}], 'summary': 'Creating a new processed tensor using red, green, and blue values, resulting in a rank 1 tensor of shape 50,176.', 'duration': 28.184, 'max_score': 3301.781, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43301781.jpg'}, {'end': 3526.903, 'src': 'embed', 'start': 3496.632, 'weight': 0, 'content': [{'end': 3511.323, 'text': "So we'll create a new tensor called expanded tensor and set that equal to reverse tensor and we'll copy the expand dim call from over here and call that on our reverse tensor.", 'start': 3496.632, 'duration': 14.691}, {'end': 3515.466, 'text': "All right now let's check the shape of this guy to make sure it's what we expect.", 'start': 3512.744, 'duration': 2.722}, {'end': 3526.903, 'text': 'So we have this inserted dimension at the start, now making our tensor rank four with shape one by 224 by 224 by three,', 'start': 3518.959, 'duration': 7.944}], 'summary': 'Created expanded tensor from reverse tensor, resulting in a rank four tensor with shape 1x224x224x3', 'duration': 30.271, 'max_score': 3496.632, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43496632.jpg'}], 'start': 2948.565, 'title': 'Debugging and creating tensors', 'summary': 'Covers debugging tensor operations with 3 one-dimensional tensors, and creating a 4d processed tensor with specific dimensions and values, such as resizing to a rank 1 tensor of size 50, 176.', 'chapters': [{'end': 3060.069, 'start': 2948.565, 'title': 'Debugging tensor operations', 'summary': 'Focuses on debugging tensor operations, including inspecting and mimicking individual transformations, with a total of three elements in the array and each element being a one-dimensional tensor with a single value zero.', 'duration': 111.504, 'highlights': ['The chapter emphasizes the importance of having a good grip on the idea of tensor transformations to follow the debugging process.', 'It mentions being paused on defining the MeanImageNetRGB object and executing the list of Rank 1 tensors called Indices, which shows up in the Local Variable panel.', 'The transcript highlights the process of inspecting individual elements in the array, including accessing the first, second, and third tensors, all being one-dimensional tensors with a single value zero.']}, {'end': 3526.903, 'start': 3061.112, 'title': 'Creating processed tensor', 'summary': 'Explains the process of gathering, centering, reshaping, stacking, reversing, and expanding a tensor to create a 4d processed tensor with specific dimensions and values, such as reshaping a tensor to a rank 1 tensor of size 50, 176 and reversing the values from rgb to bgr.', 'duration': 465.791, 'highlights': ['Creating a tensor containing only the red pixel values and checking its shape and values, which are 224 by 224 by 1 and 56, 58, and 59 respectively.', 'Centering the red values by subtracting the mean red value from ImageNet and confirming the new values to be about minus 67, minus 65, and minus 64 for the first three values along the second axis.', 'Reshaping the centered red tensor to be a rank 1 tensor of size 50, 176 and displaying all the red values in this one-dimensional tensor.', 'Stacking the centered red, green, and blue values into a new processed tensor of shape 50, 176 by 3, and reshaping it to be of shape 224 by 224 by three before reversing the values from RGB to BGR.', 'Expanding the dimensions of the tensor to make it a rank 4 tensor with shape one by 224 by 224 by three, which is what the model requires.']}], 'duration': 578.338, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG42948565.jpg', 'highlights': ['Expanding tensor dimensions to meet model requirements', 'Creating a 4d processed tensor with specific dimensions and values', 'Reshaping red tensor to a rank 1 tensor of size 50, 176', 'Inspecting individual elements in the array and accessing one-dimensional tensors', 'Centering red values by subtracting mean red value from ImageNet']}, {'end': 4288.52, 'segs': [{'end': 3662.045, 'src': 'embed', 'start': 3629.175, 'weight': 3, 'content': [{'end': 3634.737, 'text': 'Let me know what you thought of going through this practice in the debugger to see how the tensors changed over time with each operation.', 'start': 3629.175, 'duration': 5.562}, {'end': 3636.458, 'text': "And I'll see you in the next section.", 'start': 3635.097, 'duration': 1.361}, {'end': 3645.241, 'text': "In this section, we'll learn about broadcasting and illustrate its importance and major convenience when it comes to tensor operations.", 'start': 3638.399, 'duration': 6.842}, {'end': 3646.621, 'text': "So, let's get to it.", 'start': 3645.481, 'duration': 1.14}, {'end': 3662.045, 'text': "Over the last couple of sections we've immersed ourselves in tensors and hopefully now we have a good understanding of how to work with,", 'start': 3655.023, 'duration': 7.022}], 'summary': 'Understanding tensor operations and broadcasting for efficient computation.', 'duration': 32.87, 'max_score': 3629.175, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43629175.jpg'}, {'end': 3746.383, 'src': 'embed', 'start': 3719.116, 'weight': 0, 'content': [{'end': 3723.739, 'text': "So look, all of this code in red has now been replaced with what's shown in green.", 'start': 3719.116, 'duration': 4.623}, {'end': 3726.38, 'text': "That's a pretty massive reduction of code.", 'start': 3724.379, 'duration': 2.001}, {'end': 3730.662, 'text': 'Before we show how this happened, we need to understand what broadcasting is.', 'start': 3727.22, 'duration': 3.442}, {'end': 3737.065, 'text': 'Broadcasting describes how tensors with different shapes are treated during arithmetic operations.', 'start': 3731.622, 'duration': 5.443}, {'end': 3744.402, 'text': 'For example, it might be relatively easy to look at these two rank two tensors and figure out what the sum of them would be.', 'start': 3738.021, 'duration': 6.381}, {'end': 3746.383, 'text': 'They have the same shape.', 'start': 3745.262, 'duration': 1.121}], 'summary': 'Code in red replaced with green, leading to massive reduction of code.', 'duration': 27.267, 'max_score': 3719.116, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43719116.jpg'}, {'end': 3824.651, 'src': 'embed', 'start': 3793.333, 'weight': 1, 'content': [{'end': 3797.275, 'text': "Alright, we're first going to look at the result, and then we'll go over how we arrived there.", 'start': 3793.333, 'duration': 3.942}, {'end': 3801.956, 'text': 'Our result from summing these two tensors is a 3x3 tensor.', 'start': 3798.995, 'duration': 2.961}, {'end': 3804.457, 'text': "So here's how broadcasting works.", 'start': 3802.996, 'duration': 1.461}, {'end': 3807.063, 'text': 'We have two tensors with different shapes.', 'start': 3805.222, 'duration': 1.841}, {'end': 3813.126, 'text': 'The goal of broadcasting is to make the tensors have the same shape so we can perform element-wise operations on them.', 'start': 3807.603, 'duration': 5.523}, {'end': 3818.668, 'text': "First, we have to see if the operation we're trying to do is even possible between the given tensors.", 'start': 3813.906, 'duration': 4.762}, {'end': 3824.651, 'text': "Based on the tensors' original shapes, there may not be a way to reshape them to force them to be compatible.", 'start': 3819.608, 'duration': 5.043}], 'summary': 'The result of summing two tensors is a 3x3 tensor, and broadcasting aligns shapes for element-wise operations.', 'duration': 31.318, 'max_score': 3793.333, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43793333.jpg'}, {'end': 4016.918, 'src': 'embed', 'start': 3990.306, 'weight': 2, 'content': [{'end': 3996.111, 'text': 'We can now easily take the element-wise sum of these two to get this resulting 3x3 tensor.', 'start': 3990.306, 'duration': 5.805}, {'end': 3998.253, 'text': "Let's do another example.", 'start': 3997.132, 'duration': 1.121}, {'end': 4006.531, 'text': 'What if we wanted to multiply this rank two tensor of shape one by three with this rank zero tensor, better known as a scalar?', 'start': 3999.266, 'duration': 7.265}, {'end': 4012.535, 'text': "We can do this, since there's nothing in the broadcasting rules preventing us from operating on two tensors of different ranks.", 'start': 4007.151, 'duration': 5.384}, {'end': 4013.596, 'text': "Let's see.", 'start': 4013.215, 'duration': 0.381}, {'end': 4016.918, 'text': 'We first compare the last dimensions of the two shapes.', 'start': 4014.476, 'duration': 2.442}], 'summary': 'Demonstrating tensor operations and broadcasting rules in rank 2 and rank 0 tensors.', 'duration': 26.612, 'max_score': 3990.306, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43990306.jpg'}], 'start': 3526.903, 'title': 'Tensor broadcasting', 'summary': 'Covers tensor operations, debugging in chrome, and broadcasting in tensor arithmetic, with examples demonstrating the significant reduction in code through broadcasting and explanation of how broadcasting works, resulting in a 3x3 tensor. it also explains the concept of tensor broadcasting, including examples of element-wise operations and shape comparison, illustrating how broadcasting simplifies code and reduces the need for explicit looping and handling of dimensions.', 'chapters': [{'end': 3898.997, 'start': 3526.903, 'title': 'Tensor operations and broadcasting', 'summary': 'Covers tensor operations, debugging in chrome, and broadcasting in tensor arithmetic, with examples demonstrating the significant reduction in code through broadcasting and explanation of how broadcasting works, resulting in a 3x3 tensor.', 'duration': 372.094, 'highlights': ['The chapter covers tensor operations, debugging in Chrome, and broadcasting in tensor arithmetic, with examples demonstrating the significant reduction in code through broadcasting and explanation of how broadcasting works, resulting in a 3x3 tensor.', 'The code transformation through broadcasting resulted in a significant reduction in code, replacing all explicit 1x1 tensor operations with broadcasting, showcasing improved efficiency and clarity in the predict.js file.', 'Broadcasting in tensor arithmetic is explained using examples of adding tensors with different shapes, demonstrating the goal of broadcasting to make tensors have the same shape to perform element-wise operations and resulting in a 3x3 tensor from the sum of two tensors with shapes 1x3 and 3x1.']}, {'end': 4288.52, 'start': 3898.997, 'title': 'Tensor broadcasting explained', 'summary': 'Explains the concept of tensor broadcasting, including examples of element-wise operations and shape comparison, illustrating how broadcasting simplifies code and reduces the need for explicit looping and handling of dimensions.', 'duration': 389.523, 'highlights': ['The resulting tensor will have shape 3x3, expanded from the original tensors of shape 1x3 and 3x1 to perform element-wise operations, showcasing the impact of broadcasting on tensor arithmetic.', 'Illustrating the multiplication of a rank two tensor of shape 1x3 with a scalar, resulting in a tensor of shape 1x3 through broadcasting, emphasizing the flexibility of broadcasting across tensors of different ranks.', 'The explanation of broadcasting in the context of VGG16 preprocessing code, demonstrating how broadcasting simplifies the process of subtracting MeanImageNetRGB tensor from the original tensor, leading to concise code and improved readability.']}], 'duration': 761.617, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG43526903.jpg', 'highlights': ['The code transformation through broadcasting resulted in a significant reduction in code, showcasing improved efficiency and clarity.', 'Broadcasting in tensor arithmetic is explained using examples of adding tensors with different shapes, resulting in a 3x3 tensor.', 'Illustrating the multiplication of a rank two tensor of shape 1x3 with a scalar, resulting in a tensor of shape 1x3 through broadcasting.', 'The chapter covers tensor operations, debugging in Chrome, and broadcasting in tensor arithmetic, with examples demonstrating the significant reduction in code through broadcasting.']}, {'end': 4752.471, 'segs': [{'end': 4376.534, 'src': 'embed', 'start': 4339.143, 'weight': 0, 'content': [{'end': 4343.226, 'text': "Well, we're in luck, because we'll now make use of a much smaller model MobileNet,", 'start': 4339.143, 'duration': 4.083}, {'end': 4346.929, 'text': 'which is pretty ideal size-wise for running in the browser coming in at around 16 MB.', 'start': 4343.226, 'duration': 3.703}, {'end': 4353.543, 'text': "With MobileNet, we'll see a vast decrease in time for both loading the model and obtaining predictions.", 'start': 4348.55, 'duration': 4.993}, {'end': 4357.494, 'text': "Let's go ahead and get into the code to see what modifications we need to make.", 'start': 4354.205, 'duration': 3.289}, {'end': 4367.188, 'text': "Alright, we're here in our Predict with TFJS HTML file and we're going to make a model selector where the user has the ability to choose which model to use.", 'start': 4358.723, 'duration': 8.465}, {'end': 4371.571, 'text': "For now, we'll have VGG16 and MobileNet as available options.", 'start': 4367.749, 'duration': 3.822}, {'end': 4376.534, 'text': 'Currently, the call to load the model occurs immediately when the webpage is requested,', 'start': 4372.152, 'duration': 4.382}], 'summary': 'Using mobilenet, a 16 mb model, for faster predictions and load times in the browser.', 'duration': 37.391, 'max_score': 4339.143, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44339143.jpg'}, {'end': 4433.323, 'src': 'embed', 'start': 4408.639, 'weight': 1, 'content': [{'end': 4414.963, 'text': 'Now also recall how we mentioned that until now, the model was being loaded immediately when a user arrived at the page,', 'start': 4408.639, 'duration': 6.324}, {'end': 4418.185, 'text': 'and during that time the progress bar would show to indicate the loading.', 'start': 4414.963, 'duration': 3.222}, {'end': 4421.868, 'text': "Since we'll be changing the functionality so that the model isn't loaded.", 'start': 4418.946, 'duration': 2.922}, {'end': 4427.472, 'text': "until a user chooses which model they want to use, we won't need the progress bar to show until that model is selected.", 'start': 4421.868, 'duration': 5.604}, {'end': 4433.323, 'text': "So, navigating to the progress bar element, we're going to set the display style attribute to none,", 'start': 4428.399, 'duration': 4.924}], 'summary': 'The model will no longer load immediately, and the progress bar will be hidden until a model is selected.', 'duration': 24.684, 'max_score': 4408.639, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44408639.jpg'}, {'end': 4517.547, 'src': 'embed', 'start': 4487.63, 'weight': 3, 'content': [{'end': 4489.872, 'text': 'LoadModel accepts the name of the model to be loaded.', 'start': 4487.63, 'duration': 2.242}, {'end': 4493.655, 'text': 'Once called, the progress bar will be shown to indicate the model is loading.', 'start': 4490.632, 'duration': 3.023}, {'end': 4500.319, 'text': "We initially set the model to Undefined so that in case we're in a situation where we're switching from one model to another,", 'start': 4494.495, 'duration': 5.824}, {'end': 4502.181, 'text': 'the previous model can be cleared from memory.', 'start': 4500.319, 'duration': 1.862}, {'end': 4508.077, 'text': 'Afterwards, we set model to the result of calling the TensorFlow.js function tf.loadmodel.', 'start': 4503.132, 'duration': 4.945}, {'end': 4512.802, 'text': "Remember, this function accepts the URL to the given model's model.json file.", 'start': 4508.818, 'duration': 3.984}, {'end': 4517.547, 'text': 'The models reside in folders that were given the names of the actual models themselves.', 'start': 4513.282, 'duration': 4.265}], 'summary': 'Loadmodel function loads model using tf.loadmodel and shows progress bar.', 'duration': 29.917, 'max_score': 4487.63, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44487630.jpg'}, {'end': 4648.73, 'src': 'embed', 'start': 4626.736, 'weight': 4, 'content': [{'end': 4634.927, 'text': 'So we have our Mean ImageNet RGB tensor that we defined last time here and we subtract the Mean ImageNet RGB tensor from the original tensor,', 'start': 4626.736, 'duration': 8.191}, {'end': 4638.372, 'text': 'reverse the RGB values and expand the dimensions of the tensor.', 'start': 4634.927, 'duration': 3.445}, {'end': 4642.007, 'text': 'We then return this final tensor as the result of this function.', 'start': 4639.246, 'duration': 2.761}, {'end': 4646.749, 'text': 'If MobileNet is selected on the other hand, then our preprocessing will be a bit different.', 'start': 4643.008, 'duration': 3.741}, {'end': 4648.73, 'text': 'Unlike VGG16,', 'start': 4647.329, 'duration': 1.401}], 'summary': 'Preprocessing involves subtracting mean imagenet rgb tensor, reversing rgb values, and expanding tensor dimensions for some models.', 'duration': 21.994, 'max_score': 4626.736, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44626736.jpg'}, {'end': 4737.676, 'src': 'heatmap', 'start': 4680.336, 'weight': 0.722, 'content': [{'end': 4684.12, 'text': 'Lastly, we again expand the dimensions and then return this resulting tensor.', 'start': 4680.336, 'duration': 3.784}, {'end': 4690.961, 'text': "Also, in this last case, if a model name is passed to this function that isn't one of the available ones already here,", 'start': 4685.478, 'duration': 5.483}, {'end': 4692.302, 'text': "then we'll throw this exception.", 'start': 4690.961, 'duration': 1.341}, {'end': 4695.363, 'text': "Alright, we've made all the necessary code changes.", 'start': 4693.122, 'duration': 2.241}, {'end': 4697.765, 'text': "Let's now browse to our app and see the results.", 'start': 4695.583, 'duration': 2.182}, {'end': 4703.128, 'text': "We've arrived at our application, and we now have the new model selector we added in.", 'start': 4699.205, 'duration': 3.923}, {'end': 4707.29, 'text': 'Clicking on this selector, we can select either MobileNet or VGG16.', 'start': 4703.848, 'duration': 3.442}, {'end': 4709.331, 'text': "Let's go ahead and select MobileNet.", 'start': 4707.91, 'duration': 1.421}, {'end': 4713.331, 'text': 'And you can see that loaded pretty fast.', 'start': 4711.55, 'duration': 1.781}, {'end': 4719.752, 'text': 'Remember when we loaded VGG16 in previous sections, I had to pause and resume the video since it took so long to load.', 'start': 4713.871, 'duration': 5.881}, {'end': 4721.493, 'text': 'But MobileNet was speedy.', 'start': 4720.133, 'duration': 1.36}, {'end': 4722.973, 'text': 'Alright, cool.', 'start': 4722.373, 'duration': 0.6}, {'end': 4731.356, 'text': "Now we'll select an image, click predict, and again, MobileNet was super fast relative to VGG16 in returning a prediction to us.", 'start': 4723.274, 'duration': 8.082}, {'end': 4737.676, 'text': 'So hopefully this exercise has illustrated the practicality of using mobile nets in situations like these.', 'start': 4732.731, 'duration': 4.945}], 'summary': 'Mobilenet outperformed vgg16 in speed and efficiency during model testing.', 'duration': 57.34, 'max_score': 4680.336, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44680336.jpg'}, {'end': 4739.198, 'src': 'embed', 'start': 4713.871, 'weight': 5, 'content': [{'end': 4719.752, 'text': 'Remember when we loaded VGG16 in previous sections, I had to pause and resume the video since it took so long to load.', 'start': 4713.871, 'duration': 5.881}, {'end': 4721.493, 'text': 'But MobileNet was speedy.', 'start': 4720.133, 'duration': 1.36}, {'end': 4722.973, 'text': 'Alright, cool.', 'start': 4722.373, 'duration': 0.6}, {'end': 4731.356, 'text': "Now we'll select an image, click predict, and again, MobileNet was super fast relative to VGG16 in returning a prediction to us.", 'start': 4723.274, 'duration': 8.082}, {'end': 4737.676, 'text': 'So hopefully this exercise has illustrated the practicality of using mobile nets in situations like these.', 'start': 4732.731, 'duration': 4.945}, {'end': 4739.198, 'text': "And that's it.", 'start': 4738.557, 'duration': 0.641}], 'summary': 'Mobilenet is faster than vgg16 in loading and predicting images, demonstrating its practicality.', 'duration': 25.327, 'max_score': 4713.871, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44713871.jpg'}], 'start': 4290.945, 'title': 'Optimizing deep learning web app and model selection', 'summary': 'Demonstrates the reduction of model size from 500 mb to 16 mb by replacing vgg16 with mobilenet, resulting in significant improvement in loading time and prediction speed. it also discusses the implementation of model selection using a model selector and the practicality of mobilenet over vgg16 in terms of loading speed.', 'chapters': [{'end': 4445.973, 'start': 4290.945, 'title': 'Optimizing deep learning web app', 'summary': 'Demonstrates how replacing vgg16 with mobilenet reduces model size from 500 mb to 16 mb, resulting in significant improvement in loading time and prediction speed, and modifies the web app to load the model only when selected by the user.', 'duration': 155.028, 'highlights': ['Replacing VGG16 with MobileNet reduces model size from over 500 MB to around 16 MB, resulting in a vast decrease in time for both loading the model and obtaining predictions.', "Modifying the web app to load the model once a user selects which model they'd like to use improves user experience and eliminates the need for the progress bar to show until the model is selected.", 'Adding a model selector in the HTML file allows users to choose between VGG16 and MobileNet, enhancing user interaction and customization of the web app.']}, {'end': 4752.471, 'start': 4446.554, 'title': 'Model selection and preprocessing', 'summary': 'Discusses the implementation of model selection using a model selector, the loadmodel function to load the selected model, and the preprocessimage function for image preprocessing. mobilenet loaded faster than vgg16, illustrating its practicality.', 'duration': 305.917, 'highlights': ['The loadModel function is used to load the selected model, MobileNet or VGG16, and show a progress bar while loading. The model is set to Undefined initially to clear the previous model from memory. The model is then loaded using tf.loadmodel, and the progress bar is hidden. (Relevance: 5)', 'The preprocessImage function is created to preprocess images differently based on the selected model. For VGG16, preprocessing involves subtracting the Mean ImageNet RGB tensor, reversing RGB values, and expanding dimensions. For MobileNet, preprocessing involves scaling RGB values from 0-255 to -1 to 1. (Relevance: 4)', 'MobileNet loaded faster than VGG16, demonstrating its practicality in use cases where speed is crucial. (Relevance: 3)']}], 'duration': 461.526, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/EoYfa6mYOG4/pics/EoYfa6mYOG44290945.jpg', 'highlights': ['Replacing VGG16 with MobileNet reduces model size from over 500 MB to around 16 MB, resulting in a vast decrease in time for both loading the model and obtaining predictions.', "Modifying the web app to load the model once a user selects which model they'd like to use improves user experience and eliminates the need for the progress bar to show until the model is selected.", 'Adding a model selector in the HTML file allows users to choose between VGG16 and MobileNet, enhancing user interaction and customization of the web app.', 'The loadModel function is used to load the selected model, MobileNet or VGG16, and show a progress bar while loading. The model is set to Undefined initially to clear the previous model from memory. The model is then loaded using tf.loadmodel, and the progress bar is hidden.', 'The preprocessImage function is created to preprocess images differently based on the selected model. For VGG16, preprocessing involves subtracting the Mean ImageNet RGB tensor, reversing RGB values, and expanding dimensions. For MobileNet, preprocessing involves scaling RGB values from 0-255 to -1 to 1.', 'MobileNet loaded faster than VGG16, demonstrating its practicality in use cases where speed is crucial.']}], 'highlights': ['Deployment of client-side neural networks enables highly interactive and easy-to-use applications', 'Replacing VGG16 with MobileNet reduces model size from over 500 MB to around 16 MB', 'The model took about 40 seconds to load, indicating a significant loading time for the large VGG16 model', 'The shift from backend model hosting to embedding the model within the frontend application', 'The preprocessing for VGG16 involves subtracting the mean RGB value computed on the training set from each pixel, zero centering each color channel with respect to the ImageNet dataset', 'The need for smaller models to be loaded and running in the browser, with TensorFlow.js suggesting models of 30 MB or less', 'The process involves centering red, green, and blue values by subtracting mean ImageNet values, reshaping tensors, and reversing color channels from RGB to BGR, resulting in a pre-processed tensor object containing 224x224x3 dimensions, which is then used for model prediction', 'MobileNet loaded faster than VGG16, demonstrating its practicality in use cases where speed is crucial', 'The code transformation through broadcasting resulted in a significant reduction in code, showcasing improved efficiency and clarity', "The UI elements of the web app are organized using Bootstrap's grid layout, with containers, rows, and columns"]}