title
How to Write Python Scripts to Analyze JSON APIs and Sort Results
description
In this Python Programming Tutorial, we will be learning how to grab data from a JSON API, parse out the information we want, and then sort the data using a custom key. The API we will be using is a JSON API for Homebrew Packages and we will be sorting the packages by their popularity. We cover a lot of topics in this tutorial. We will be using the Requests Library, converting to/from JSON, reading and writing to files, writing our own sorting function, and more. Let's get started...
Homebrew Packages - https://formulae.brew.sh/formula/
Homebrew Analytics - https://formulae.brew.sh/analytics/
Requests Tutorial - https://youtu.be/tb8gHvYlCFs
JSON Tutorial - https://youtu.be/9N6a-VLBa2I
Sorting Tutorial - https://youtu.be/D3JvDWO-BY4
List Comprehensions - https://youtu.be/3dt4OGnU5sM
✅ Support My Channel Through Patreon:
https://www.patreon.com/coreyms
✅ Become a Channel Member:
https://www.youtube.com/channel/UCCezIgC97PvUuR4_gbFUs5g/join
✅ One-Time Contribution Through PayPal:
https://goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
http://a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
https://www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - http://coreyms.com/
My Second Channel - https://www.youtube.com/c/coreymschafer
Facebook - https://www.facebook.com/CoreyMSchafer
Twitter - https://twitter.com/CoreyMSchafer
Instagram - https://www.instagram.com/coreymschafer/
#Python #JSON #API
detail
{'title': 'How to Write Python Scripts to Analyze JSON APIs and Sort Results', 'heatmap': [{'end': 1318.942, 'start': 1280.589, 'weight': 1}], 'summary': 'Learn to analyze homebrew package popularity and package information by accessing json apis, sorting packages, and analyzing installation data using python, while addressing challenges and emphasizing efficiency in managing json requests and creating json files for analytics.', 'chapters': [{'end': 237.839, 'segs': [{'end': 96.168, 'src': 'embed', 'start': 58.048, 'weight': 0, 'content': [{'end': 60.369, 'text': "It did exist but I'll get to that in just a second.", 'start': 58.048, 'duration': 2.321}, {'end': 64.735, 'text': "So, since I didn't think that this information existed,", 'start': 61.27, 'duration': 3.465}, {'end': 70.623, 'text': 'I figured that it would be a good idea for a real world project where we could build a solution to this on our own.', 'start': 64.735, 'duration': 5.888}, {'end': 76.992, 'text': 'So I actually went out and created a script that sorts these packages by their popularity manually.', 'start': 71.083, 'duration': 5.909}, {'end': 83.057, 'text': 'But after I did that, I discovered that they actually have this information available if you look hard enough for it.', 'start': 77.592, 'duration': 5.465}, {'end': 85.739, 'text': 'So that is the page that I have open here.', 'start': 83.417, 'duration': 2.322}, {'end': 90.583, 'text': 'So on their analytics page here, they have pages for their most installed packages.', 'start': 86.12, 'duration': 4.463}, {'end': 92.745, 'text': 'And again, I found this pretty hard to find.', 'start': 90.924, 'duration': 1.821}, {'end': 96.168, 'text': "That's why I didn't know it existed before writing my own script to this.", 'start': 93.005, 'duration': 3.163}], 'summary': 'Discovered existing package popularity data after creating manual sorting script.', 'duration': 38.12, 'max_score': 58.048, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g58048.jpg'}, {'end': 178.022, 'src': 'embed', 'start': 117.605, 'weight': 2, 'content': [{'end': 125.152, 'text': "I believe that the install on request means that the package was explicitly installed and wasn't installed as a dependency or anything like that.", 'start': 117.605, 'duration': 7.547}, {'end': 127.113, 'text': "So that's what I'm most interested in.", 'start': 125.732, 'duration': 1.381}, {'end': 132.197, 'text': 'So we can see the most installed packages for the last 30 days on this page here.', 'start': 127.994, 'duration': 4.203}, {'end': 137.261, 'text': 'So like I said, I wrote a script to do this manually without knowing that this existed.', 'start': 132.718, 'duration': 4.543}, {'end': 141.685, 'text': 'And I figured that I could still make a video showing how I got this information manually.', 'start': 137.762, 'duration': 3.923}, {'end': 145.588, 'text': "And then we can compare it to their version here to make sure that it's working properly.", 'start': 142.085, 'duration': 3.503}, {'end': 147.929, 'text': "So if you're trying to learn something,", 'start': 146.028, 'duration': 1.901}, {'end': 154.512, 'text': "then I definitely think it's a good idea to try to come up with your own solutions to problems like these anytime you run into them,", 'start': 147.929, 'duration': 6.583}, {'end': 156.292, 'text': 'even if a solution already exists.', 'start': 154.512, 'duration': 1.78}, {'end': 163.115, 'text': "Not only is it good practice, but you're also solving a problem that you're actually interested in,", 'start': 156.893, 'duration': 6.222}, {'end': 167.337, 'text': "and you're going to feel a bit more sense of pride after you've done this as well.", 'start': 163.115, 'duration': 4.222}, {'end': 173.06, 'text': 'So let me show you how I came up with my own solution to these most popular packages,', 'start': 167.797, 'duration': 5.263}, {'end': 178.022, 'text': 'based solely on what they have in their package information listed over here.', 'start': 173.06, 'duration': 4.962}], 'summary': 'Interested in analyzing most installed packages for the last 30 days and comparing own solution to the existing one.', 'duration': 60.417, 'max_score': 117.605, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g117605.jpg'}, {'end': 230.095, 'src': 'embed', 'start': 197.894, 'weight': 3, 'content': [{'end': 204.4, 'text': "So I'm going to open this up in a new tab and the URL for this is forward slash API.", 'start': 197.894, 'duration': 6.506}, {'end': 206.082, 'text': 'forward slash formula.', 'start': 204.4, 'duration': 1.682}, {'end': 207.003, 'text': 'forward slash.', 'start': 206.082, 'duration': 0.921}, {'end': 208.905, 'text': 'the name of the package dot JSON.', 'start': 207.003, 'duration': 1.902}, {'end': 210.807, 'text': 'So let me make this a little larger here.', 'start': 209.225, 'duration': 1.582}, {'end': 220.851, 'text': "Now this isn't formatted, so it's a bit jumbled together and difficult to read, but this JSON has the analytics available for this specific package.", 'start': 211.247, 'duration': 9.604}, {'end': 230.095, 'text': 'So if I hit control F to do a search and then search for 30 days, then we can see that we have the 30 days for the installs,', 'start': 221.131, 'duration': 8.964}], 'summary': 'The json provides analytics for the package, including 30-day install data.', 'duration': 32.201, 'max_score': 197.894, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g197894.jpg'}], 'start': 0.209, 'title': 'Analyzing homebrew package popularity and package information', 'summary': 'Discusses discovering analytics on the homebrew website, sorting packages by popularity, finding the most installed packages, manually extracting package information, and using json apis for analytics, highlighting problem-solving and the benefits of personal solutions.', 'chapters': [{'end': 137.261, 'start': 0.209, 'title': 'Analyzing homebrew package popularity', 'summary': 'Discusses discovering the analytics page on the homebrew website, manually creating a script to sort packages by popularity, and later finding the existing analytics page that lists the most installed packages for the last 30 days, despite its obscure location on the website.', 'duration': 137.052, 'highlights': ['Discovering the analytics page on the Homebrew website The speaker came across the analytics page on the Homebrew website, which includes information about the total number of installations for certain timeframes, such as the last 30 days, 90 days, and 365 days.', 'Manually creating a script to sort packages by popularity The speaker created a script to sort the packages by popularity since they initially thought that the information about the most popular packages was not available on the website.', 'Finding the existing analytics page listing the most installed packages for the last 30 days After writing a script to sort the packages by popularity, the speaker discovered the existing analytics page that lists the most installed packages for the last 30 days, despite its obscure location on the website.']}, {'end': 237.839, 'start': 137.762, 'title': 'Analyzing package information', 'summary': 'Demonstrates the process of manually extracting information from package details and utilizing json apis to retrieve analytics, emphasizing the importance of problem-solving and the benefits of creating personal solutions.', 'duration': 100.077, 'highlights': ['By manually extracting information and utilizing JSON APIs, the speaker demonstrates the process of analyzing package details to retrieve analytics, promoting problem-solving and independent learning.', 'Encourages viewers to seek their own solutions to problems, highlighting the benefits of hands-on practice and the sense of accomplishment gained from personal problem-solving.', "Emphasizes the importance of learning through practical application and personal problem-solving in order to gain a deeper understanding of the topic and promote a sense of pride in one's work."]}], 'duration': 237.63, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g209.jpg', 'highlights': ['Discovering the analytics page on the Homebrew website, including information about total installations for different timeframes.', 'Manually creating a script to sort packages by popularity when the information was initially thought to be unavailable.', 'Finding the existing analytics page listing the most installed packages for the last 30 days after writing a sorting script.', 'Demonstrating the process of analyzing package details to retrieve analytics using manual extraction and JSON APIs.', 'Encouraging viewers to seek their own solutions, highlighting the benefits of hands-on practice and personal problem-solving.', 'Emphasizing the importance of learning through practical application and personal problem-solving for a deeper understanding and sense of pride.']}, {'end': 863.569, 'segs': [{'end': 262.963, 'src': 'embed', 'start': 237.839, 'weight': 5, 'content': [{'end': 249.149, 'text': "I think the difference between install and install on request is that install on request is a package that is installed explicitly and doesn't count being installed as a dependency.", 'start': 237.839, 'duration': 11.31}, {'end': 251.412, 'text': "So that's what I'm going to be after in this video.", 'start': 249.51, 'duration': 1.902}, {'end': 255.736, 'text': 'Okay, so this is the JSON for this particular package.', 'start': 252.032, 'duration': 3.704}, {'end': 258.819, 'text': 'But we want to compare the analytics of all the packages.', 'start': 256.076, 'duration': 2.743}, {'end': 262.963, 'text': "So let's go back to the main page here.", 'start': 259.178, 'duration': 3.785}], 'summary': 'Exploring the distinction between install and install on request for packages, aiming to compare analytics of all packages.', 'duration': 25.124, 'max_score': 237.839, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g237839.jpg'}, {'end': 312.436, 'src': 'embed', 'start': 285.658, 'weight': 0, 'content': [{'end': 289.24, 'text': 'And this takes a little longer to load since there are more packages.', 'start': 285.658, 'duration': 3.582}, {'end': 297.766, 'text': "okay, so this is a large json file here that contains information on all of the packages, but it doesn't contain the analytics.", 'start': 289.66, 'duration': 8.106}, {'end': 307.012, 'text': "now let's create a new python script so that we can begin to analyze some of the data that they provide here and so that we can read this a little bit better.", 'start': 297.766, 'duration': 9.246}, {'end': 312.436, 'text': "so what i'm going to do is i'm going to copy the url to this json here for all of these packages.", 'start': 307.012, 'duration': 5.424}], 'summary': 'Analyzing a large json file containing package information to create a new python script for data analysis.', 'duration': 26.778, 'max_score': 285.658, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g285658.jpg'}, {'end': 378.924, 'src': 'embed', 'start': 350.202, 'weight': 1, 'content': [{'end': 357.367, 'text': 'And I want to get that URL for all of the packages for that JSON.', 'start': 350.202, 'duration': 7.165}, {'end': 365.774, 'text': "And once we get a response from that URL, we can parse out that JSON simply by saying, let's say I'll call this packages underscore JSON.", 'start': 357.707, 'duration': 8.067}, {'end': 370.758, 'text': "I'll set that equal to R, which is the request, R dot JSON.", 'start': 366.174, 'duration': 4.584}, {'end': 378.924, 'text': "Now, if you're not familiar with how to use the request library, then I do have a detailed video on how to use these requests and responses,", 'start': 371.298, 'duration': 7.626}], 'summary': 'Retrieve json data from url using request library for parsing.', 'duration': 28.722, 'max_score': 350.202, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g350202.jpg'}, {'end': 451.82, 'src': 'embed', 'start': 429.993, 'weight': 3, 'content': [{'end': 438.898, 'text': 'Now, when working with JSON data, we can use the JSON module to dump this to a string, And we can tell the string how we want it to be formatted.', 'start': 429.993, 'duration': 8.905}, {'end': 442.698, 'text': "So to do this, let's import the JSON module.", 'start': 439.578, 'duration': 3.12}, {'end': 447.039, 'text': 'So I will import JSON here at the top.', 'start': 443.159, 'duration': 3.88}, {'end': 451.82, 'text': "And now I'm just going to overwrite my print statement here.", 'start': 447.7, 'duration': 4.12}], 'summary': 'Using the json module to dump data to a string and customize formatting.', 'duration': 21.827, 'max_score': 429.993, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g429993.jpg'}, {'end': 533.156, 'src': 'embed', 'start': 506.689, 'weight': 4, 'content': [{'end': 519.623, 'text': 'So I can tell from the ending square bracket at the end here in our output that this JSON is a list of values and it looks like this is a list of all of the packages.', 'start': 506.689, 'duration': 12.934}, {'end': 525.531, 'text': 'Let me make this a bit larger here so that we can see this a bit better.', 'start': 519.683, 'duration': 5.848}, {'end': 533.156, 'text': 'So if we look at the information for one package here, then we can look through the information that it provides.', 'start': 526.012, 'duration': 7.144}], 'summary': 'The json contains a list of packages with detailed information.', 'duration': 26.467, 'max_score': 506.689, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g506689.jpg'}, {'end': 808.646, 'src': 'embed', 'start': 780.612, 'weight': 2, 'content': [{'end': 788.977, 'text': "and I'm going to instead replace that with our package name that we got from the JSON data here.", 'start': 780.612, 'duration': 8.365}, {'end': 797.942, 'text': "So now, let's make a request to that URL that we just generated and see if we get the analytics data for this single package.", 'start': 789.417, 'duration': 8.525}, {'end': 808.646, 'text': "So to do this, I'm just going to say r is equal to request.get and I want to get this package URL.", 'start': 798.322, 'duration': 10.324}], 'summary': 'Replacing with package name from json data, making request to package url to get analytics data.', 'duration': 28.034, 'max_score': 780.612, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g780612.jpg'}], 'start': 237.839, 'title': 'Package analytics and working with json data', 'summary': 'Covers the difference between install and install on request, accessing a json file containing package information, analyzing data using python, working with a large json object from an api, formatting json data for readability, and generating a specific url for analytics data.', 'chapters': [{'end': 388.512, 'start': 237.839, 'title': 'Package analytics: install on request', 'summary': 'Discusses the difference between install and install on request, accessing a json file containing information on all packages, and the process of analyzing the data using a python script and the request library.', 'duration': 150.673, 'highlights': ['Accessing a JSON file containing information on all packages and analyzing the data using a Python script and the request library. The chapter discusses the process of analyzing the data using a Python script and the request library, focusing on the difference between install and install on request, and accessing a JSON file containing information on all packages.', "Difference between install and install on request. The difference between install and install on request is explained as a package that is installed explicitly and doesn't count as being installed as a dependency.", 'Using the request library to retrieve contents from a JSON URL and parsing the JSON response. The process involves using the request library to retrieve contents from a JSON URL and parsing the JSON response to access the required data for analysis.']}, {'end': 863.569, 'start': 389.472, 'title': 'Working with json data', 'summary': 'Discusses working with a large json object retrieved from an api, formatting the json data using the json module to improve readability, and generating a specific url to obtain analytics data for a single package using the package name.', 'duration': 474.097, 'highlights': ['The JSON data is formatted using the JSON module to improve readability by dumping it into a string with an indent of 2, resulting in a more formatted response. The JSON data is formatted using the JSON module to improve readability by dumping it into a string with an indent of 2, resulting in a more formatted response.', 'Analyzing the JSON data reveals that it is a list of values representing different packages, and by accessing the first index, the output changes from a list to a single object, corresponding to the first package from the JSON. Analyzing the JSON data reveals that it is a list of values representing different packages, and by accessing the first index, the output changes from a list to a single object, corresponding to the first package from the JSON.', 'Generating a specific URL for a package by using the naming convention from the API and the package name obtained from the JSON data, and then making a request to that URL to retrieve the analytics data for that single package. Generating a specific URL for a package by using the naming convention from the API and the package name obtained from the JSON data, and then making a request to that URL to retrieve the analytics data for that single package.']}], 'duration': 625.73, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g237839.jpg', 'highlights': ['Accessing a JSON file containing information on all packages and analyzing the data using a Python script and the request library.', 'Using the request library to retrieve contents from a JSON URL and parsing the JSON response.', 'Generating a specific URL for a package by using the naming convention from the API and the package name obtained from the JSON data, and then making a request to that URL to retrieve the analytics data for that single package.', 'The JSON data is formatted using the JSON module to improve readability by dumping it into a string with an indent of 2, resulting in a more formatted response.', 'Analyzing the JSON data reveals that it is a list of values representing different packages, and by accessing the first index, the output changes from a list to a single object, corresponding to the first package from the JSON.', "Difference between install and install on request. The difference between install and install on request is explained as a package that is installed explicitly and doesn't count as being installed as a dependency."]}, {'end': 1234.796, 'segs': [{'end': 943.859, 'src': 'embed', 'start': 863.729, 'weight': 0, 'content': [{'end': 868.612, 'text': 'If I scroll up to the top here, the name of the package that we got is that A2PS.', 'start': 863.729, 'duration': 4.883}, {'end': 870.813, 'text': 'So we did get the right information.', 'start': 868.812, 'duration': 2.001}, {'end': 876.737, 'text': 'And if I scroll down here towards the bottom, then we can see that we have the analytics here.', 'start': 871.153, 'duration': 5.584}, {'end': 882.721, 'text': 'So we have the analytics for the installations for the past 30, 90, and 365 days.', 'start': 877.037, 'duration': 5.684}, {'end': 885.963, 'text': 'And we also have the analytics for the install on requests.', 'start': 883.041, 'duration': 2.922}, {'end': 889.526, 'text': 'So it looks like we did get the information that we want for that one package.', 'start': 886.263, 'duration': 3.263}, {'end': 893.588, 'text': "Okay, so now let's look at how we would access those analytics specifically.", 'start': 889.926, 'duration': 3.662}, {'end': 898.211, 'text': "So I'm going to make this a little larger here and just scroll up a little bit.", 'start': 893.608, 'duration': 4.603}, {'end': 903.575, 'text': "So if we look at this dictionary, then it's within a key named analytics.", 'start': 898.572, 'duration': 5.003}, {'end': 908.839, 'text': 'And then we want the analytics for the number of installs on request.', 'start': 904.175, 'duration': 4.664}, {'end': 916.764, 'text': 'So within analytics here, we have a key called install on requests.', 'start': 909.279, 'duration': 7.485}, {'end': 924.95, 'text': 'And then with install on requests, we have keys for 30D, which is 30 days, 90D, which is 90 days, and 365D, which is 365 days.', 'start': 917.124, 'duration': 7.826}, {'end': 937.596, 'text': "and within each key for those days there's another key with the package name, so in this case it's A2PS,", 'start': 928.972, 'duration': 8.624}, {'end': 943.859, 'text': 'and then the value of that key contains the number of installations on request.', 'start': 937.596, 'duration': 6.263}], 'summary': 'Received analytics for a2ps package, including installs for 30, 90, and 365 days, and install on request data.', 'duration': 80.13, 'max_score': 863.729, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g863729.jpg'}, {'end': 1068.157, 'src': 'embed', 'start': 1041.684, 'weight': 3, 'content': [{'end': 1049.688, 'text': "So I'll call this variable installs underscore 90 and here instead of the 30D key, I'm going to access the 90D key.", 'start': 1041.684, 'duration': 8.004}, {'end': 1055.891, 'text': "And here, this last one I'll call installs underscore 365.", 'start': 1051.048, 'duration': 4.843}, {'end': 1060.353, 'text': 'And I will access the 365d key there.', 'start': 1055.891, 'duration': 4.462}, {'end': 1068.157, 'text': 'So, so far we have the package name and the analytics data for three different date ranges.', 'start': 1060.713, 'duration': 7.444}], 'summary': 'Accessed analytics data for 30d, 90d, and 365d date ranges', 'duration': 26.473, 'max_score': 1041.684, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1041684.jpg'}, {'end': 1216.804, 'src': 'embed', 'start': 1187.131, 'weight': 4, 'content': [{'end': 1190.194, 'text': "And let me print how many packages we're going to need to analyze.", 'start': 1187.131, 'duration': 3.063}, {'end': 1194.658, 'text': 'And to do that, I will just print the LEN, which will give me the length.', 'start': 1190.494, 'duration': 4.164}, {'end': 1196.778, 'text': 'of this packages list here.', 'start': 1195.098, 'duration': 1.68}, {'end': 1203.78, 'text': 'So if I save that and run it, then we can see that there are 4718 packages.', 'start': 1197.159, 'duration': 6.621}, {'end': 1213.443, 'text': "Now the way that we're doing this right now, that means that we're going to have to make requests to the individual JSONs for 4718 packages.", 'start': 1204.2, 'duration': 9.243}, {'end': 1216.804, 'text': 'Now, if anyone has ever watched my video on web scraping,', 'start': 1213.803, 'duration': 3.001}], 'summary': 'The analysis will require processing 4718 packages, needing individual json requests.', 'duration': 29.673, 'max_score': 1187.131, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1187131.jpg'}], 'start': 863.729, 'title': 'Accessing and analyzing package installation data', 'summary': 'Details the process of accessing analytics data for a specific package, demonstrating how to retrieve the number of installations for the past 30, 90, and 365 days, and explains the process of accessing and printing package information, including challenges of analyzing 4718 packages due to potential server limitations.', 'chapters': [{'end': 1068.157, 'start': 863.729, 'title': 'Accessing analytics data for package installations', 'summary': 'Details the process of accessing analytics data for a specific package, demonstrating how to retrieve the number of installations on request for the past 30, 90, and 365 days, while emphasizing the step-by-step approach of accessing and retrieving the relevant data from a nested dictionary.', 'duration': 204.428, 'highlights': ['The chapter explains the process of accessing analytics data for a specific package, demonstrating how to retrieve the number of installations on request for the past 30, 90, and 365 days, emphasizing the step-by-step approach of accessing and retrieving the relevant data from a nested dictionary.', "The package name 'A2PS' is correctly identified, and the analytics for the installations for the past 30, 90, and 365 days, as well as the analytics for the install on requests, are obtained, indicating the successful retrieval of the desired information.", "The process involves accessing the nested dictionary within the 'analytics' key and further accessing the 'install on requests' key to retrieve the number of installations on request for the specified date ranges, demonstrating a systematic approach to accessing the required data.", "The method involves accessing the keys for 30 days, 90 days, and 365 days within the 'install on requests' key, and then accessing the package name and its corresponding value containing the number of installations on request, demonstrating the detailed steps involved in obtaining the analytics data for different date ranges."]}, {'end': 1234.796, 'start': 1068.457, 'title': 'Package data analysis', 'summary': 'Explains the process of accessing and printing package information, including package name, description, and installation data for 30, 90, and 365 days, for a single package, and discusses the challenges of analyzing 4718 packages due to potential server limitations.', 'duration': 166.339, 'highlights': ['The chapter explains the process of accessing and printing package information, including package name, description, and installation data for 30, 90, and 365 days, for a single package. Accessing and printing package information, including package name, description, and installation data for 30, 90, and 365 days', 'Discusses the challenges of analyzing 4718 packages due to potential server limitations. Challenges of analyzing 4718 packages']}], 'duration': 371.067, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g863729.jpg', 'highlights': ['Demonstrates step-by-step approach to accessing and retrieving relevant data from a nested dictionary.', "Successfully retrieves analytics for the installations of package 'A2PS' for the past 30, 90, and 365 days.", 'Systematic approach to accessing the nested dictionary and retrieving the number of installations on request for specified date ranges.', 'Detailed steps involved in obtaining analytics data for different date ranges.', 'Explains challenges of analyzing 4718 packages due to potential server limitations.']}, {'end': 1708.528, 'segs': [{'end': 1261.333, 'src': 'embed', 'start': 1235.356, 'weight': 0, 'content': [{'end': 1241.621, 'text': "Now these are pretty lightweight JSON responses that we'll be requesting, so it shouldn't put too much strain on their site.", 'start': 1235.356, 'duration': 6.265}, {'end': 1246.646, 'text': 'But I still think it would be a good idea to just put in a very slight delay before each request.', 'start': 1242.102, 'duration': 4.544}, {'end': 1249.787, 'text': "so that it's not hammering these requests one after another.", 'start': 1247.006, 'duration': 2.781}, {'end': 1253.769, 'text': "So to do this, I'm going to sleep between each request.", 'start': 1250.288, 'duration': 3.481}, {'end': 1261.333, 'text': "And for the amount of time that I'm going to sleep for, I'm just going to use the amount of time that it took me to get a response from the site.", 'start': 1254.17, 'duration': 7.163}], 'summary': 'Lightweight json responses with slight delay to prevent hammering requests.', 'duration': 25.977, 'max_score': 1235.356, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1235356.jpg'}, {'end': 1318.942, 'src': 'heatmap', 'start': 1280.589, 'weight': 1, 'content': [{'end': 1293.342, 'text': "i'm just going to say for package in packages json, since that is a list of all of the packages and now i'm going to reuse all of this logic here.", 'start': 1280.589, 'duration': 12.753}, {'end': 1297.265, 'text': 'let me get rid of the output for now so that we can see all of this.', 'start': 1293.342, 'duration': 3.923}, {'end': 1307.354, 'text': "i'm going to uncomment out all of this and i'm going to just reuse all of this logic and indent this in our for loop.", 'start': 1297.265, 'duration': 10.089}, {'end': 1309.755, 'text': 'and now i need to make a couple of changes here.', 'start': 1307.354, 'duration': 2.401}, {'end': 1318.942, 'text': "so now that i'm within this for loop, we are accessing the information for this package variable instead of packages underscore JSON.", 'start': 1309.755, 'duration': 9.187}], 'summary': "Reusing logic from 'packages.json' for package iteration within a for loop.", 'duration': 38.353, 'max_score': 1280.589, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1280589.jpg'}, {'end': 1480.043, 'src': 'embed', 'start': 1448.116, 'weight': 1, 'content': [{'end': 1453.339, 'text': "maybe you'll want to revisit the data or analyze it in a different way after we've downloaded it once,", 'start': 1448.116, 'duration': 5.223}, {'end': 1464.366, 'text': 'and it will be much easier and faster to analyze that single file on our own computer than it would be to go out and request those 4700 JSON files each time we want to look at something in a different way.', 'start': 1453.339, 'duration': 11.027}, {'end': 1471.313, 'text': 'So for now, Our main goal is to capture this information and save it to our own custom JSON file.', 'start': 1464.826, 'duration': 6.487}, {'end': 1480.043, 'text': "So to do this, above our for loop, I'm going to create an empty list, and I'm going to call this empty list results.", 'start': 1471.814, 'duration': 8.229}], 'summary': 'Create a custom json file to store and analyze 4700 json files efficiently.', 'duration': 31.927, 'max_score': 1448.116, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1448116.jpg'}, {'end': 1518.9, 'src': 'embed', 'start': 1489.049, 'weight': 2, 'content': [{'end': 1495.171, 'text': "Within our for loop, after we've grabbed the installations for the last 30, 90, and 365 days, we'll put that into a dictionary.", 'start': 1489.049, 'duration': 6.122}, {'end': 1502.074, 'text': "and let's make this a nested dictionary.", 'start': 1499.213, 'duration': 2.861}, {'end': 1506.256, 'text': "so first we'll have keys for the package name and description.", 'start': 1502.074, 'duration': 4.182}, {'end': 1510.457, 'text': "so I'll say data is equal to, and we want this to be a dictionary.", 'start': 1506.256, 'duration': 4.201}, {'end': 1518.9, 'text': "so first I'll have a key for the name and the value I will just have as package name and I'll put in a comma.", 'start': 1510.457, 'duration': 8.443}], 'summary': 'Within a for loop, installations for last 30, 90, and 365 days are put into a nested dictionary.', 'duration': 29.851, 'max_score': 1489.049, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1489049.jpg'}], 'start': 1235.356, 'title': 'Managing json requests and capturing api data', 'summary': 'Discusses implementing a slight delay between json requests to avoid overwhelming the api, using response time for sleep duration, and capturing installation data for 30, 90, and 365 days, emphasizing efficiency and web request management.', 'chapters': [{'end': 1408.671, 'start': 1235.356, 'title': 'Implementing delay for lightweight json requests', 'summary': 'Discusses implementing a slight delay between lightweight json requests to avoid overwhelming the api, using the response time to determine the sleep duration, and iterating through packages to access and print their install analytics for further testing.', 'duration': 173.315, 'highlights': ['Implementing a slight delay between requests to avoid overwhelming the API, using the response time to determine the sleep duration, and iterating through packages to access and print their install analytics for further testing.', 'Utilizing the response time to dynamically adjust the sleep duration between requests, ensuring efficient utilization of resources and preventing API overload.', 'Iterating through packages to access and print their install analytics, demonstrating a systematic approach to testing and analyzing data for multiple packages.']}, {'end': 1708.528, 'start': 1408.691, 'title': 'Capturing and saving api data', 'summary': 'Focuses on capturing information about installations for the last 30, 90, and 365 days, and saving it as a list of dictionaries to a custom json file, with an emphasis on the efficiency of analysis and the use of sleep delays to manage web requests.', 'duration': 299.837, 'highlights': ['The chapter emphasizes the importance of saving the data as a list of dictionaries to a custom JSON file for efficient and faster analysis on our own computer, rather than making 4700 JSON requests each time, which aligns with the goal of capturing and saving API data to our own machine.', 'The chapter describes the process of creating a nested dictionary to capture the package name, description, and analytics data for the last 30, 90, and 365 days, providing a customizable approach to analyze various data points, showcasing the flexibility of the script.', "The chapter outlines the use of sleep delays after each web request, leveraging the time module to manage the program's sleep duration based on the response time, aiming to provide a buffer and optimize the server's load, ensuring efficient and considerate web request management."]}], 'duration': 473.172, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1235356.jpg', 'highlights': ['Utilizing the response time to dynamically adjust the sleep duration between requests, ensuring efficient utilization of resources and preventing API overload.', 'The chapter emphasizes the importance of saving the data as a list of dictionaries to a custom JSON file for efficient and faster analysis on our own computer, rather than making 4700 JSON requests each time, which aligns with the goal of capturing and saving API data to our own machine.', 'The chapter describes the process of creating a nested dictionary to capture the package name, description, and analytics data for the last 30, 90, and 365 days, providing a customizable approach to analyze various data points, showcasing the flexibility of the script.']}, {'end': 2183.151, 'segs': [{'end': 1771.306, 'src': 'embed', 'start': 1749.812, 'weight': 2, 'content': [{'end': 1761.4, 'text': 'and then we are parsing out the information that we want from that specific JSON file And then we are creating a dictionary and putting in our own custom information here.', 'start': 1749.812, 'duration': 11.588}, {'end': 1766.783, 'text': "So we're setting the name, the description, and the analytics for that one specific package.", 'start': 1761.48, 'duration': 5.303}, {'end': 1771.306, 'text': 'And then we are appending that one specific package to our results list.', 'start': 1767.223, 'duration': 4.083}], 'summary': 'Parsing and customizing json data to create and append a package to a results list.', 'duration': 21.494, 'max_score': 1749.812, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1749812.jpg'}, {'end': 1880.954, 'src': 'embed', 'start': 1854.862, 'weight': 1, 'content': [{'end': 1863.266, 'text': "Let's go ahead and add the logic for saving that list of dictionaries to a JSON file so that we can make sure that it works with what we have now.", 'start': 1854.862, 'duration': 8.404}, {'end': 1871.089, 'text': "So notice, as I'm developing this script, I'm building this up to the final result a little bit at a time,", 'start': 1864.126, 'duration': 6.963}, {'end': 1874.131, 'text': "and I'm always doing it with small chunks of test data.", 'start': 1871.089, 'duration': 3.042}, {'end': 1880.954, 'text': 'So if I were to just take out the break statement and write up the code for saving the JSON file without testing it first,', 'start': 1874.491, 'duration': 6.463}], 'summary': 'Adding logic to save list of dictionaries to json file with incremental development and testing.', 'duration': 26.092, 'max_score': 1854.862, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1854862.jpg'}, {'end': 1955.104, 'src': 'embed', 'start': 1925.517, 'weight': 5, 'content': [{'end': 1927.659, 'text': "And we use dump when we're dumping to a file.", 'start': 1925.517, 'duration': 2.142}, {'end': 1930.042, 'text': "We use dump s when we're dumping to a string.", 'start': 1927.78, 'duration': 2.262}, {'end': 1932.103, 'text': 'So we want to do json.dump.', 'start': 1930.362, 'duration': 1.741}, {'end': 1934.606, 'text': 'We want to dump that results list.', 'start': 1932.604, 'duration': 2.002}, {'end': 1939.19, 'text': 'We want to dump that to f, which is our open file.', 'start': 1935.547, 'duration': 3.643}, {'end': 1943.014, 'text': 'And we also want to put in an indent equal to 2 here.', 'start': 1939.871, 'duration': 3.143}, {'end': 1946.357, 'text': 'And that will just make our file nice and formatted.', 'start': 1943.394, 'duration': 2.963}, {'end': 1955.104, 'text': 'So if I save this and run it, then that should create a package info JSON file in our current directory.', 'start': 1946.837, 'duration': 8.267}], 'summary': 'Using json.dump to write results list to file, with indent=2.', 'duration': 29.587, 'max_score': 1925.517, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1925517.jpg'}, {'end': 2005.587, 'src': 'embed', 'start': 1971.898, 'weight': 3, 'content': [{'end': 1975.942, 'text': 'so if I open that up, this is what got written to that file.', 'start': 1971.898, 'duration': 4.044}, {'end': 1977.923, 'text': 'okay, so we can see that this file looks good.', 'start': 1975.942, 'duration': 1.981}, {'end': 1979.625, 'text': 'this is what we expected.', 'start': 1977.923, 'duration': 1.702}, {'end': 1987.352, 'text': 'so now we can try to remove that break statement from our for loop, And it should go through all of those packages and add them to our JSON file.', 'start': 1979.625, 'duration': 7.727}, {'end': 1992.977, 'text': "Now, before I run this, I'm also going to put in some print statements so that we can keep track of where we currently are.", 'start': 1987.732, 'duration': 5.245}, {'end': 1997.84, 'text': "And I'm also going to time how long it takes to go through all of these URLs.", 'start': 1993.417, 'duration': 4.423}, {'end': 2005.587, 'text': "So first, I'm going to put in a print statement that gives us some feedback that things are still going well.", 'start': 1998.341, 'duration': 7.246}], 'summary': 'Modifying code to remove break statement, tracking progress with print statements and timing the process.', 'duration': 33.689, 'max_score': 1971.898, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1971898.jpg'}, {'end': 2171.561, 'src': 'embed', 'start': 2143.519, 'weight': 0, 'content': [{'end': 2150.725, 'text': 'Okay, so I paused the video and I let that script finish and it looks like it finished in about 1400 seconds.', 'start': 2143.519, 'duration': 7.206}, {'end': 2158.451, 'text': "So let's see if I open up my calculator that is about 1400 divided by 60.", 'start': 2151.225, 'duration': 7.226}, {'end': 2166.137, 'text': "It's about 24 and a half minutes that it took to go out and pull down the JSON of all of those package files.", 'start': 2158.451, 'duration': 7.686}, {'end': 2171.561, 'text': "Now, if we hadn't put in the sleep statement, then it probably would have done it in about half that time,", 'start': 2166.857, 'duration': 4.704}], 'summary': 'It took approximately 24.5 minutes to pull down the json of all package files, which would have been done in about half the time without the sleep statement.', 'duration': 28.042, 'max_score': 2143.519, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2143519.jpg'}], 'start': 1708.528, 'title': 'Json file creation and testing', 'summary': 'Covers building a script to create package information and analytics, testing with small data chunks, and saving to a json file. the process includes timing the loop resulting in the script finishing in approximately 1400 seconds.', 'chapters': [{'end': 1946.357, 'start': 1708.528, 'title': 'Building and testing json file creation', 'summary': 'Covers building a script step by step to create a list of packages with their information and analytics, testing the script with small chunks of data, and then saving the list of dictionaries to a json file to ensure it works, emphasizing the importance of testing to avoid wasting time.', 'duration': 237.829, 'highlights': ['The process involves building a script incrementally, testing it with small chunks of data, and saving the list of dictionaries to a JSON file to ensure it works properly, highlighting the importance of testing to avoid time wastage.', "The speaker explains the process of using 'json.dump' to write the results list to a JSON file, including specifying the file name, opening it in write mode, and formatting the file with an indent of 2.", "The script's development involves creating an empty results list, looping through packages to extract information and analytics, and creating a dictionary for each package before appending it to the results list."]}, {'end': 2183.151, 'start': 1946.837, 'title': 'Creating package info json file', 'summary': 'Covers the process of creating a package info json file in the current directory, including adding print statements and timing the loop, resulting in the script finishing in approximately 1400 seconds.', 'duration': 236.314, 'highlights': ['The script finished in about 1400 seconds, which is approximately 24 and a half minutes, to pull down the JSON of all the package files, with a sleep statement potentially doubling the time taken.', 'The process involved removing a break statement from the for loop to go through all the packages and add them to the JSON file.', 'Print statements and timing were added to track progress and measure the time it took to go through the URLs, with the script finishing in approximately 1400 seconds.']}], 'duration': 474.623, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g1708528.jpg', 'highlights': ['The script finished in about 1400 seconds, which is approximately 24 and a half minutes, to pull down the JSON of all the package files, with a sleep statement potentially doubling the time taken.', 'The process involves building a script incrementally, testing it with small chunks of data, and saving the list of dictionaries to a JSON file to ensure it works properly, highlighting the importance of testing to avoid time wastage.', "The script's development involves creating an empty results list, looping through packages to extract information and analytics, and creating a dictionary for each package before appending it to the results list.", 'The process involved removing a break statement from the for loop to go through all the packages and add them to the JSON file.', 'Print statements and timing were added to track progress and measure the time it took to go through the URLs, with the script finishing in approximately 1400 seconds.', "The speaker explains the process of using 'json.dump' to write the results list to a JSON file, including specifying the file name, opening it in write mode, and formatting the file with an indent of 2."]}, {'end': 2909.927, 'segs': [{'end': 2224.914, 'src': 'embed', 'start': 2183.852, 'weight': 6, 'content': [{'end': 2190.236, 'text': 'So now we should have all of that data saved into our JSON file in the same directory as our current script.', 'start': 2183.852, 'duration': 6.384}, {'end': 2191.656, 'text': 'So let me check that out.', 'start': 2190.576, 'duration': 1.08}, {'end': 2194.737, 'text': "So I'll reopen this packageinfo.json.", 'start': 2191.676, 'duration': 3.061}, {'end': 2201.52, 'text': 'And if we look at this file, we can see that it looks like we have all of this information for all of these packages.', 'start': 2195.338, 'duration': 6.182}, {'end': 2203.401, 'text': 'So this is looking pretty good.', 'start': 2201.8, 'duration': 1.601}, {'end': 2209.643, 'text': 'So we have each package name, we have the descriptions, and we have the analytics for each one.', 'start': 2203.841, 'duration': 5.802}, {'end': 2218.049, 'text': 'so now we can actually do what we first set out to do and use this data to determine what the most popular homebrew packages are.', 'start': 2210.143, 'duration': 7.906}, {'end': 2224.914, 'text': "so to do this, I'm going to create a new script and I'll just create this in the same directory that we are currently in.", 'start': 2218.049, 'duration': 6.865}], 'summary': 'Data saved in json file with package names, descriptions, and analytics for determining most popular homebrew packages.', 'duration': 41.062, 'max_score': 2183.852, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2183852.jpg'}, {'end': 2281.945, 'src': 'embed', 'start': 2251.894, 'weight': 5, 'content': [{'end': 2255.077, 'text': 'We can put that in explicitly if we want, but read is the default.', 'start': 2251.894, 'duration': 3.183}, {'end': 2256.498, 'text': 'So either way you want to do that.', 'start': 2255.177, 'duration': 1.321}, {'end': 2258.841, 'text': "So I'll say as F.", 'start': 2257.199, 'duration': 1.642}, {'end': 2260.784, 'text': 'So F is our open file there.', 'start': 2258.841, 'duration': 1.943}, {'end': 2265.59, 'text': 'Now say data is equal to JSON dot load.', 'start': 2261.324, 'duration': 4.266}, {'end': 2269.456, 'text': 'When we want to load JSON data from a file, we use load.', 'start': 2266.011, 'duration': 3.445}, {'end': 2273.602, 'text': 'And I want to load from F, which is our JSON file.', 'start': 2269.997, 'duration': 3.605}, {'end': 2281.945, 'text': 'Okay, so if I print this data, then it should now be a Python list of all of our dictionaries.', 'start': 2274.202, 'duration': 7.743}], 'summary': 'Using json dot load to load data from a file into a python list of dictionaries.', 'duration': 30.051, 'max_score': 2251.894, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2251894.jpg'}, {'end': 2329.463, 'src': 'embed', 'start': 2305.574, 'weight': 4, 'content': [{'end': 2312.636, 'text': "because python doesn't know what this data is or how we want it sorted, so we need to tell it how we want this sorted.", 'start': 2305.574, 'duration': 7.062}, {'end': 2319.598, 'text': "now i have a separate, more detailed video on how to do custom sorting, so i'll leave a link to that video in the description section below,", 'start': 2312.636, 'duration': 6.962}, {'end': 2323.299, 'text': "but for this example, i'll just show you how we can sort this specific list.", 'start': 2319.598, 'duration': 3.701}, {'end': 2326.701, 'text': "So first I'm going to create a function.", 'start': 2324.119, 'duration': 2.582}, {'end': 2329.463, 'text': 'that will be what we want to use to sort the list.', 'start': 2326.701, 'duration': 2.762}], 'summary': 'Demonstrating custom sorting in python through a function.', 'duration': 23.889, 'max_score': 2305.574, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2305574.jpg'}, {'end': 2444.401, 'src': 'embed', 'start': 2424.09, 'weight': 2, 'content': [{'end': 2435.036, 'text': 'We can either sort it in place using the sort method or we can keep the original data unsorted and capture the sorted data in a new variable using the sorted function.', 'start': 2424.09, 'duration': 10.946}, {'end': 2441.279, 'text': "Now, I'm just going to sort the data in place in this example, so that means we don't have to create a new variable.", 'start': 2435.516, 'duration': 5.763}, {'end': 2444.401, 'text': "So, in order to do this, I'm just going to do this here at the bottom.", 'start': 2441.68, 'duration': 2.721}], 'summary': 'Sorting data in place using sort method without creating a new variable.', 'duration': 20.311, 'max_score': 2424.09, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2424090.jpg'}, {'end': 2552.201, 'src': 'embed', 'start': 2523.022, 'weight': 3, 'content': [{'end': 2526.424, 'text': 'Now I will just print out that data string.', 'start': 2523.022, 'duration': 3.402}, {'end': 2531.747, 'text': 'So if I save that and run it, whoops, I accidentally pasted something in there.', 'start': 2526.884, 'duration': 4.863}, {'end': 2538.672, 'text': 'If I save that and run it, now let me scroll all the way up to the top here.', 'start': 2532.088, 'duration': 6.584}, {'end': 2540.573, 'text': "I'm going to have to use this over here.", 'start': 2538.692, 'duration': 1.881}, {'end': 2545.256, 'text': 'Then we should see the most installed packages of the last 30 days.', 'start': 2540.994, 'duration': 4.262}, {'end': 2548.719, 'text': 'So we have Node, we have Python.', 'start': 2546.057, 'duration': 2.662}, {'end': 2552.201, 'text': 'We have get W get yarn.', 'start': 2549.339, 'duration': 2.862}], 'summary': 'Printing data string, accidentally pasting, displaying top installed packages: node, python, w get yarn', 'duration': 29.179, 'max_score': 2523.022, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2523022.jpg'}, {'end': 2621.43, 'src': 'embed', 'start': 2596.508, 'weight': 1, 'content': [{'end': 2605.376, 'text': "Now, I do think that the way that we build up our custom script does allow us to have some additional functionality that I don't think that they have in their online API for this analytics data.", 'start': 2596.508, 'duration': 8.868}, {'end': 2611.081, 'text': 'So for example, if I go back to their website here, let me open up the browser again.', 'start': 2605.957, 'duration': 5.124}, {'end': 2617.547, 'text': 'So here is their JSON API for their analytics data for the last 30 days.', 'start': 2611.782, 'duration': 5.765}, {'end': 2621.43, 'text': 'So if I look at this, Let me make this a little bit larger here.', 'start': 2617.907, 'duration': 3.523}], 'summary': 'Custom script provides additional functionality compared to online api for analytics data.', 'duration': 24.922, 'max_score': 2596.508, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2596508.jpg'}, {'end': 2672.337, 'src': 'embed', 'start': 2641.448, 'weight': 0, 'content': [{'end': 2643.07, 'text': 'and the percentage of downloads.', 'start': 2641.448, 'duration': 1.622}, {'end': 2648.655, 'text': "So if we wanted to know the description of these packages, then we'd have to go to them one by one.", 'start': 2643.41, 'duration': 5.245}, {'end': 2652.519, 'text': 'But with our custom script that we built, we do have these descriptions.', 'start': 2649.176, 'duration': 3.343}, {'end': 2658.085, 'text': 'So that also allows us to filter by descriptions as well.', 'start': 2653.02, 'duration': 5.065}, {'end': 2663.711, 'text': 'So if I go back to my script, let me make this page a little smaller here.', 'start': 2658.485, 'duration': 5.226}, {'end': 2672.337, 'text': 'If I go back to my script, then I could simply write a list comprehension here to filter out the data based on a certain description.', 'start': 2664.091, 'duration': 8.246}], 'summary': 'Custom script enables filtering by descriptions, streamlining package analysis.', 'duration': 30.889, 'max_score': 2641.448, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2641448.jpg'}], 'start': 2183.852, 'title': 'Analyzing homebrew packages data', 'summary': 'Covers the process of saving data into a json file, analyzing package information, creating a new script to determine the most popular homebrew packages, sorting a python list based on specific criteria, comparing custom script results with the official api, and obtaining the top packages for the last 365 days.', 'chapters': [{'end': 2224.914, 'start': 2183.852, 'title': 'Using json data for package analytics', 'summary': 'Covers the process of saving data into a json file, analyzing package information, and creating a new script to determine the most popular homebrew packages.', 'duration': 41.062, 'highlights': ['Saving data into a JSON file and checking it. The data is saved into a JSON file in the same directory as the current script, and the file is checked for the saved information.', 'Analyzing package information in the JSON file. The JSON file contains package names, descriptions, and analytics for each package.', 'Creating a new script to determine the most popular homebrew packages. The new script is created in the same directory to analyze the data and determine the most popular homebrew packages.']}, {'end': 2522.501, 'start': 2224.914, 'title': 'Sorting python list with custom function', 'summary': 'Demonstrates how to load a json file, create a custom sorting function, and sort a python list based on specific criteria, resulting in a sorted list with the most installed packages of the last 30 days at the beginning of the list.', 'duration': 297.587, 'highlights': ['Demonstrates loading a JSON file using the JSON module and printing the loaded data as a Python list. The chapter demonstrates loading a JSON file using the JSON module and printing the loaded data as a Python list.', 'Explains the need for a custom sorting function and demonstrates creating a function to sort the list based on specific criteria. The chapter explains the need for a custom sorting function and demonstrates creating a function to sort the list based on specific criteria.', 'Shows how to use the sorting function to sort the list in descending order, capturing the sorted data in the original variable. The chapter shows how to use the sorting function to sort the list in descending order, capturing the sorted data in the original variable.']}, {'end': 2909.927, 'start': 2523.022, 'title': 'Analyzing homebrew packages data', 'summary': 'Discusses analyzing data on the most installed packages of the last 30 days, comparing custom script results with the official api, and demonstrating additional functionality of the custom script by filtering packages based on descriptions and obtaining the top packages for the last 365 days.', 'duration': 386.905, 'highlights': ['The custom script successfully retrieves and displays the most installed packages of the last 30 days, with Node and Python being the top two packages. The custom script retrieves the most installed packages of the last 30 days, with Node and Python being the top two packages.', "The custom script's results match the official API's data, indicating the accuracy and reliability of the custom script. The custom script's results match the official API's data, demonstrating the accuracy and reliability of the custom script.", 'The custom script provides additional functionality by including package descriptions, allowing filtering based on descriptions, and obtaining top packages for the last 365 days. The custom script includes package descriptions, enables filtering based on descriptions, and obtains top packages for the last 365 days, showcasing its additional functionality.']}], 'duration': 726.075, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1lxrb_ezP-g/pics/1lxrb_ezP-g2183852.jpg', 'highlights': ['The custom script provides additional functionality by including package descriptions, allowing filtering based on descriptions, and obtaining top packages for the last 365 days.', "The custom script's results match the official API's data, indicating the accuracy and reliability of the custom script.", 'Shows how to use the sorting function to sort the list in descending order, capturing the sorted data in the original variable.', 'The custom script successfully retrieves and displays the most installed packages of the last 30 days, with Node and Python being the top two packages.', 'Explains the need for a custom sorting function and demonstrates creating a function to sort the list based on specific criteria.', 'Demonstrates loading a JSON file using the JSON module and printing the loaded data as a Python list.', 'Creating a new script to determine the most popular homebrew packages. The new script is created in the same directory to analyze the data and determine the most popular homebrew packages.', 'Analyzing package information in the JSON file. The JSON file contains package names, descriptions, and analytics for each package.', 'Saving data into a JSON file and checking it. The data is saved into a JSON file in the same directory as the current script, and the file is checked for the saved information.']}], 'highlights': ['Accessing a JSON file containing information on all packages and analyzing the data using a Python script and the request library.', 'Utilizing the response time to dynamically adjust the sleep duration between requests, ensuring efficient utilization of resources and preventing API overload.', 'The custom script provides additional functionality by including package descriptions, allowing filtering based on descriptions, and obtaining top packages for the last 365 days.', "Successfully retrieves analytics for the installations of package 'A2PS' for the past 30, 90, and 365 days.", 'The chapter emphasizes the importance of saving the data as a list of dictionaries to a custom JSON file for efficient and faster analysis on our own computer, rather than making 4700 JSON requests each time, which aligns with the goal of capturing and saving API data to our own machine.']}