title
Web Scraping With Python | Python Tutorial | Web Scraping Tutorial | Python Beautifulsoup | Edureka

description
This Edureka live session on "WebScraping using Python" will help you understand the fundamentals of scraping along with a demo to scrape some details from Flipkart. Below are the topics covered in this tutorial: 1. Need of Web Scraping 2. What is Web Scraping? 3. Different Libraries used 4. Demo - Scrape a website

detail
{'title': 'Web Scraping With Python | Python Tutorial | Web Scraping Tutorial | Python Beautifulsoup | Edureka', 'heatmap': [{'end': 805.753, 'start': 725.71, 'weight': 1}], 'summary': 'Tutorial covers web scraping with python, including its need, process, and tools, with a live demo using flipkart for data extraction. it discusses use cases, legal considerations, fundamentals, html parsing, and product details scraping, demonstrating extraction and storage in a csv file.', 'chapters': [{'end': 84.401, 'segs': [{'end': 84.401, 'src': 'embed', 'start': 37.996, 'weight': 0, 'content': [{'end': 40.938, 'text': "So in this demo, I'll be fetching some details from Flipkart.", 'start': 37.996, 'duration': 2.942}, {'end': 43.96, 'text': 'So I hope you guys are clear with the agenda.', 'start': 41.859, 'duration': 2.101}, {'end': 51.506, 'text': 'Now let me begin with the very first topic that is the need of scraping or why do people actually scrape a website.', 'start': 46.262, 'duration': 5.244}, {'end': 57.453, 'text': 'So, scraping is basically extracting data from various websites.', 'start': 54.232, 'duration': 3.221}, {'end': 60.354, 'text': 'Now, there are many ways to collect data online.', 'start': 57.973, 'duration': 2.381}, {'end': 63.494, 'text': "But let's say I just want to collect data for comparison.", 'start': 60.834, 'duration': 2.66}, {'end': 68.436, 'text': 'So, here let me take an example of MySmartPrice which is known for best price comparison.', 'start': 63.914, 'duration': 4.522}, {'end': 71.896, 'text': 'So, what they do, they collect data directly from merchants.', 'start': 68.996, 'duration': 2.9}, {'end': 75.697, 'text': 'Now, the larger stores generally provide the feeds which are listed on their stores.', 'start': 72.277, 'duration': 3.42}, {'end': 77.438, 'text': 'But what about the smaller stores?', 'start': 76.078, 'duration': 1.36}, {'end': 84.401, 'text': 'now, one way of doing it is to either copy the prices from different websites and then paste it to your local system,', 'start': 78.118, 'duration': 6.283}], 'summary': 'Demonstration on scraping data from flipkart and other websites for comparison and collection.', 'duration': 46.405, 'max_score': 37.996, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c37996.jpg'}], 'start': 1.991, 'title': 'Web scraping with python', 'summary': 'Covers the need and importance of web scraping, the process and tools involved, and concludes with a live demo of scraping a website, using flipkart as an example, to fetch data for comparison and analysis.', 'chapters': [{'end': 84.401, 'start': 1.991, 'title': 'Web scraping with python', 'summary': 'Covers the need and importance of web scraping, the process and tools involved, and concludes with a live demo of scraping a website, using flipkart as an example, to fetch data for comparison and analysis.', 'duration': 82.41, 'highlights': ['The session covers the need and importance of web scraping, the process and tools involved, and concludes with a live demo of scraping a website, using Flipkart as an example, to fetch data for comparison and analysis.', "Web scraping involves extracting data from various websites, and it's essential for tasks like price comparison on platforms like MySmartPrice.", 'The larger stores generally provide feeds for price comparison, but scraping is necessary for collecting data from smaller stores and websites.', 'The demo will involve scraping details from Flipkart to demonstrate the process of data extraction using Python.']}], 'duration': 82.41, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1991.jpg', 'highlights': ["Web scraping involves extracting data from various websites, and it's essential for tasks like price comparison on platforms like MySmartPrice.", 'The larger stores generally provide feeds for price comparison, but scraping is necessary for collecting data from smaller stores and websites.', 'The session covers the need and importance of web scraping, the process and tools involved, and concludes with a live demo of scraping a website, using Flipkart as an example, to fetch data for comparison and analysis.', 'The demo will involve scraping details from Flipkart to demonstrate the process of data extraction using Python.']}, {'end': 232.327, 'segs': [{'end': 232.327, 'src': 'embed', 'start': 149.368, 'weight': 0, 'content': [{'end': 157.093, 'text': 'so people generally use web scraping to build the best marketing strategies, monitor them and hence upgrade their business.', 'start': 149.368, 'duration': 7.725}, {'end': 166.42, 'text': 'so in a way, web scraping provides them a way to get information, whether it can be a website, prices, it can be images, it can be data for any,', 'start': 157.093, 'duration': 9.327}, {'end': 168.361, 'text': 'for any analyzing purpose as well.', 'start': 166.42, 'duration': 1.941}, {'end': 174.671, 'text': 'So also you do scraping because many websites do not provide an API, unlike Facebook, Twitter and all.', 'start': 169.11, 'duration': 5.561}, {'end': 179.673, 'text': 'But in many cases you will find that web scraping is the only alternative to get data out of them.', 'start': 175.032, 'duration': 4.641}, {'end': 185.514, 'text': 'So we have discussed the various real life use cases which uses web scraping.', 'start': 181.793, 'duration': 3.721}, {'end': 189.731, 'text': 'But here the question is, is it actually legal? Well, maybe.', 'start': 185.914, 'duration': 3.817}, {'end': 196.634, 'text': 'Scraping is fine till you are not causing any considerable damage to the target website and you are doing it responsibly.', 'start': 190.331, 'duration': 6.303}, {'end': 202.655, 'text': 'So website is legal as long as there is no attempt to change the article or blog or anything.', 'start': 197.054, 'duration': 5.601}, {'end': 208.017, 'text': 'You should not try to override the restricted access of the website or play around with the security mechanism.', 'start': 203.036, 'duration': 4.981}, {'end': 213.319, 'text': 'Now let me tell you the companies who have faced problems while web scraping.', 'start': 209.438, 'duration': 3.881}, {'end': 217.657, 'text': 'So you all must have heard about LinkedIn.', 'start': 215.856, 'duration': 1.801}, {'end': 223.001, 'text': 'So in 2014, there was a case where they sued a group of dependents in California.', 'start': 218.138, 'duration': 4.863}, {'end': 226.843, 'text': 'Now. this case was based on Digital Millennium Copyright Act,', 'start': 223.541, 'duration': 3.302}, {'end': 232.327, 'text': 'where they used a series of bots and fake profiles to scrape the data from other member profiles.', 'start': 226.843, 'duration': 5.484}], 'summary': 'Web scraping aids in marketing, data analysis; legal if done responsibly. linkedin sued for scraping data in 2014.', 'duration': 82.959, 'max_score': 149.368, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c149368.jpg'}], 'start': 84.401, 'title': 'Web scraping use cases and legal considerations', 'summary': 'Discusses various web scraping use cases such as e-commerce, market research, and social media, and covers legal considerations, including the linkedin lawsuit in 2014.', 'chapters': [{'end': 232.327, 'start': 84.401, 'title': 'Web scraping: use cases and legal considerations', 'summary': 'Discusses the various use cases of web scraping, including e-commerce portals, market research, and social media, and also addresses the legal considerations and the case of linkedin suing a group for scraping data in 2014.', 'duration': 147.926, 'highlights': ['Web scraping has various real-life use cases such as e-commerce portals, market research, and social media. Web scraping can be used in e-commerce to scrape products, prices, ratings, and images, in market research to analyze market needs and competition, and in social media to track business profiles and online presence.', 'Web scraping can be the only alternative to obtain data from websites that do not provide an API. In cases where websites do not offer an API, web scraping becomes the sole method to extract data from them, making it a crucial tool for data acquisition.', 'Legal considerations of web scraping include not causing damage to the target website and not attempting to override restricted access or security mechanisms. Web scraping is legal as long as it does not cause significant harm to the target website and is done responsibly without attempting to override access restrictions or security mechanisms.', 'LinkedIn sued a group in 2014 for web scraping under the Digital Millennium Copyright Act. LinkedIn sued a group for using bots and fake profiles to scrape data from member profiles, highlighting the legal consequences of unauthorized web scraping.']}], 'duration': 147.926, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c84401.jpg', 'highlights': ['Web scraping can be the only alternative to obtain data from websites that do not provide an API.', 'Web scraping has various real-life use cases such as e-commerce portals, market research, and social media.', 'Legal considerations of web scraping include not causing damage to the target website and not attempting to override restricted access or security mechanisms.', 'LinkedIn sued a group in 2014 for web scraping under the Digital Millennium Copyright Act.']}, {'end': 599.281, 'segs': [{'end': 300.283, 'src': 'embed', 'start': 254.017, 'weight': 0, 'content': [{'end': 258.399, 'text': 'extract the useful information and then store the same data to your local storage.', 'start': 254.017, 'duration': 4.382}, {'end': 260.701, 'text': 'Now, this local storage can be in any form.', 'start': 258.86, 'duration': 1.841}, {'end': 264.943, 'text': 'It can be in XML form, it can be CSV or it can be any database as well.', 'start': 260.781, 'duration': 4.162}, {'end': 273.929, 'text': 'So, web scraping is a technique where you automate the procedure within a fraction of seconds instead of manually copying the data from websites.', 'start': 266.364, 'duration': 7.565}, {'end': 278.732, 'text': 'Now we know scraping is when you extract any information.', 'start': 275.751, 'duration': 2.981}, {'end': 285.156, 'text': 'Now, if you want to extract through a browser or a website or a web page, so that process is known as a web scraping.', 'start': 279.133, 'duration': 6.023}, {'end': 289.258, 'text': 'So let us see the basic steps which are involved while scraping.', 'start': 285.876, 'duration': 3.382}, {'end': 293.36, 'text': 'So the very first step is document load.', 'start': 291.178, 'duration': 2.182}, {'end': 296.681, 'text': 'Now in our case, we are doing web scraping.', 'start': 293.74, 'duration': 2.941}, {'end': 300.283, 'text': 'So we have to load the entire document, which is your HTML page.', 'start': 296.701, 'duration': 3.582}], 'summary': 'Web scraping automates data extraction from websites, enabling quick document loading in html format.', 'duration': 46.266, 'max_score': 254.017, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c254017.jpg'}, {'end': 378.503, 'src': 'embed', 'start': 346.123, 'weight': 2, 'content': [{'end': 349.885, 'text': 'Although I would recommend Python because a lot of people use it for web scraping.', 'start': 346.123, 'duration': 3.762}, {'end': 354.826, 'text': 'Well, the good part about Python is that it has libraries for everything.', 'start': 350.565, 'duration': 4.261}, {'end': 358.468, 'text': 'Now, this is why it is the go-to language for many developers.', 'start': 355.447, 'duration': 3.021}, {'end': 366.47, 'text': 'Now, as you can see here, there are a lot of libraries which are used for web scraping, such as Pattern, Scrapey, Mechanize,', 'start': 359.548, 'duration': 6.922}, {'end': 367.931, 'text': 'Beautiful Soup and Request.', 'start': 366.47, 'duration': 1.461}, {'end': 370.621, 'text': 'So I have only used scraping.', 'start': 368.74, 'duration': 1.881}, {'end': 378.503, 'text': "I have also, I've only used Python for scraping because what I like about Python is that it makes a string manipulation very easy.", 'start': 370.941, 'duration': 7.562}], 'summary': 'Python is recommended for web scraping due to its extensive libraries and ease of string manipulation.', 'duration': 32.38, 'max_score': 346.123, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c346123.jpg'}, {'end': 417.621, 'src': 'embed', 'start': 391.446, 'weight': 4, 'content': [{'end': 395.287, 'text': 'and MongoDB is what we mostly prefer to use for data storage in web scraping.', 'start': 391.446, 'duration': 3.841}, {'end': 399.053, 'text': 'Now, as you can see, there are different Python packages.', 'start': 396.192, 'duration': 2.861}, {'end': 401.174, 'text': 'Now, these are not enough.', 'start': 399.874, 'duration': 1.3}, {'end': 408.057, 'text': 'We also have some other Python libraries, such as XPath, you have PDF, you have Miner, and many more.', 'start': 401.234, 'duration': 6.823}, {'end': 414.62, 'text': 'But here, if you see, Beautiful Soup and Request are the most popular libraries in this lot.', 'start': 408.317, 'duration': 6.303}, {'end': 417.621, 'text': "And later in the demo, we'll be implementing them as well.", 'start': 415.06, 'duration': 2.561}], 'summary': 'Mongodb is preferred for web scraping data storage. popular python libraries include beautiful soup and request.', 'duration': 26.175, 'max_score': 391.446, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c391446.jpg'}], 'start': 232.867, 'title': 'Web scraping fundamentals', 'summary': 'Covers web scraping basics, explaining the purpose, methods, and steps involved, emphasizing automation and efficiency. it also introduces web scraping with python, highlighting its popularity, compatibility with nosql databases, and usage of popular libraries for data extraction.', 'chapters': [{'end': 322.699, 'start': 232.867, 'title': 'Web scraping basics', 'summary': 'Explains the concept of web scraping, highlighting its purpose, methods, and steps involved, emphasizing the automation and efficiency it offers in extracting and storing data from web pages.', 'duration': 89.832, 'highlights': ['Web scraping involves automating the extraction of data from web pages and storing it locally in formats such as XML or CSV, offering efficiency and automation. Web scraping involves automating the extraction of data from web pages and storing it locally in formats such as XML or CSV, offering efficiency and automation.', 'The basic steps of web scraping include document load, parsing, extraction, and transformation, streamlining the process of extracting specific information from HTML pages. The basic steps of web scraping include document load, parsing, extraction, and transformation, streamlining the process of extracting specific information from HTML pages.', 'Web scraping is a method to automate data extraction from web pages, providing a faster alternative to manual data copying. Web scraping is a method to automate data extraction from web pages, providing a faster alternative to manual data copying.']}, {'end': 599.281, 'start': 323.219, 'title': 'Web scraping with python', 'summary': 'Introduces web scraping with python, highlighting its popularity due to libraries for data manipulation, compatibility with nosql databases like mongodb, and the usage of popular libraries like beautiful soup and request for data extraction from html and xml files.', 'duration': 276.062, 'highlights': ['Python is popular for web scraping due to its libraries for data manipulation, with a recommendation by the speaker for its usage. Python is recommended for web scraping due to its popularity and availability of libraries for data manipulation.', "Python has libraries for everything, making it the go-to language for many developers. Python's extensive libraries make it the preferred language for developers, contributing to its popularity.", 'Beautiful Soup and Request are highlighted as the most popular libraries for web scraping in Python. Beautiful Soup and Request are emphasized as the most popular libraries for web scraping in Python.', 'Python is compatible with various databases, such as NoSQL databases like MongoDB, for data storage in web scraping. Python is compatible with NoSQL databases like MongoDB, which are preferred for data storage in web scraping.', 'The speaker demonstrates scraping a website, specifically Flipkart, to extract details like product names, configurations, ratings, and prices. The speaker demonstrates scraping Flipkart to extract details like product names, configurations, ratings, and prices.']}], 'duration': 366.414, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c232867.jpg', 'highlights': ['Web scraping involves automating the extraction of data from web pages and storing it locally in formats such as XML or CSV, offering efficiency and automation.', 'The basic steps of web scraping include document load, parsing, extraction, and transformation, streamlining the process of extracting specific information from HTML pages.', 'Python is popular for web scraping due to its libraries for data manipulation, with a recommendation by the speaker for its usage.', "Python's extensive libraries make it the preferred language for developers, contributing to its popularity.", 'Beautiful Soup and Request are emphasized as the most popular libraries for web scraping in Python.', 'Python is compatible with NoSQL databases like MongoDB, which are preferred for data storage in web scraping.']}, {'end': 1014.488, 'segs': [{'end': 672.074, 'src': 'embed', 'start': 629.011, 'weight': 0, 'content': [{'end': 631.894, 'text': 'So, as you can see, there is a list of iPhones are displayed.', 'start': 629.011, 'duration': 2.883}, {'end': 634.976, 'text': 'So here we have all the necessary details of it.', 'start': 632.394, 'duration': 2.582}, {'end': 640.541, 'text': 'We have the title, we have the ratings, we have the reviews, we have the price as well.', 'start': 635.056, 'duration': 5.485}, {'end': 644.044, 'text': 'So this is the Flipkart page.', 'start': 640.901, 'duration': 3.143}, {'end': 646.887, 'text': "Now let me go back to PyCharm and let's understand the code.", 'start': 644.425, 'duration': 2.462}, {'end': 652.844, 'text': 'So here what I have done.', 'start': 651.843, 'duration': 1.001}, {'end': 654.825, 'text': 'I have my uRequest,', 'start': 652.844, 'duration': 1.981}, {'end': 662.749, 'text': 'which will basically opens up the connection and this will grab the web page or the Flipkart page in our case and then it will directly load it.', 'start': 654.825, 'duration': 7.924}, {'end': 666.892, 'text': 'And then what I have done, I have simply saved it to a variable which is uClient.', 'start': 663.15, 'duration': 3.742}, {'end': 668.873, 'text': 'after that, what I have done.', 'start': 667.612, 'duration': 1.261}, {'end': 672.074, 'text': 'I have passed the same variable to the read function.', 'start': 668.873, 'duration': 3.201}], 'summary': 'Transcript covers details of iphones on flipkart and code for web page retrieval.', 'duration': 43.063, 'max_score': 629.011, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c629011.jpg'}, {'end': 807.134, 'src': 'heatmap', 'start': 719.665, 'weight': 1, 'content': [{'end': 724.909, 'text': 'you have the content, or you can say the description, and after that you have the price.', 'start': 719.665, 'duration': 5.244}, {'end': 725.71, 'text': 'so what i have to do?', 'start': 724.909, 'duration': 0.801}, {'end': 730.694, 'text': 'i have to first search for the main class in which all these are, all these features are present.', 'start': 725.71, 'duration': 4.984}, {'end': 736.921, 'text': 'so if i go up, so this is my main class, or you can say the main div class which covers all the element.', 'start': 730.694, 'duration': 6.227}, {'end': 739.324, 'text': 'it has title also it has price.', 'start': 736.921, 'duration': 2.403}, {'end': 741.707, 'text': 'also it has ratings, reviews and everything.', 'start': 739.324, 'duration': 2.383}, {'end': 745.773, 'text': "so i'll just copy this class and go back to my pie chart.", 'start': 741.707, 'duration': 4.066}, {'end': 752.421, 'text': 'So here I have written this.', 'start': 750.56, 'duration': 1.861}, {'end': 757.343, 'text': 'I have this pasted this, the same div class which is over here', 'start': 752.421, 'duration': 4.922}, {'end': 765.568, 'text': "So here I have page two which will help me to pass my HTML and then I've used the function which is find all.", 'start': 757.884, 'duration': 7.684}, {'end': 769.89, 'text': 'So here it will find all the div tag which is a class.', 'start': 765.928, 'duration': 3.962}, {'end': 772.251, 'text': 'Now in some cases it may be ID also.', 'start': 770.23, 'duration': 2.021}, {'end': 776.693, 'text': 'So here we have div tag which is a class and the name of the div tag.', 'start': 772.711, 'duration': 3.982}, {'end': 779.215, 'text': 'And then I have stored it into containers.', 'start': 777.314, 'duration': 1.901}, {'end': 781.556, 'text': "So next what I'll be doing?", 'start': 780.495, 'duration': 1.061}, {'end': 788.841, 'text': "I'll be printing the length of these containers, which will basically tell me how many products that I have on this webpage.", 'start': 781.556, 'duration': 7.285}, {'end': 790.962, 'text': "So I'll come in the whole code for now.", 'start': 789.341, 'duration': 1.621}, {'end': 805.753, 'text': 'So here as we see there are 24 products on the very first page of Flipkart.', 'start': 801.41, 'duration': 4.343}, {'end': 807.134, 'text': 'So let me show you again.', 'start': 806.133, 'duration': 1.001}], 'summary': 'The speaker demonstrates web scraping by finding and printing the number of products on a webpage, which is 24.', 'duration': 87.469, 'max_score': 719.665, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c719665.jpg'}, {'end': 837.215, 'src': 'embed', 'start': 806.133, 'weight': 2, 'content': [{'end': 807.134, 'text': 'So let me show you again.', 'start': 806.133, 'duration': 1.001}, {'end': 811.223, 'text': 'so here, as you can see, there are many products.', 'start': 808.84, 'duration': 2.383}, {'end': 817.192, 'text': 'now using this length function, i have calculated that there are 24 products which are displayed on the very first page of flipkart site.', 'start': 811.223, 'duration': 5.969}, {'end': 827.605, 'text': 'so this was for the one div class.', 'start': 825.583, 'duration': 2.022}, {'end': 829.467, 'text': 'or you say just one container.', 'start': 827.605, 'duration': 1.862}, {'end': 833.852, 'text': "but let's say i have to fetch details of each product which is present on the web page.", 'start': 829.467, 'duration': 4.385}, {'end': 834.713, 'text': 'so what i have to do?', 'start': 833.852, 'duration': 0.861}, {'end': 837.215, 'text': 'i have to traverse the dom elements.', 'start': 834.713, 'duration': 2.502}], 'summary': 'Using length function, 24 products found on first page of flipkart site.', 'duration': 31.082, 'max_score': 806.133, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c806133.jpg'}, {'end': 902.472, 'src': 'embed', 'start': 862.961, 'weight': 4, 'content': [{'end': 876.004, 'text': 'so let me just run this whole HTML.', 'start': 862.961, 'duration': 13.043}, {'end': 879.084, 'text': 'so here my HTML.', 'start': 876.004, 'duration': 3.08}, {'end': 880.805, 'text': 'so this is my basically HTML.', 'start': 879.084, 'duration': 1.721}, {'end': 881.725, 'text': "now what I'll do?", 'start': 880.805, 'duration': 0.92}, {'end': 885.709, 'text': "I'll copy the whole html and i'll place it somewhere else.", 'start': 881.725, 'duration': 3.984}, {'end': 893.95, 'text': "let's say, i'll place it in my notepad so that we can see it simultaneously.", 'start': 885.709, 'duration': 8.241}, {'end': 898.451, 'text': 'now let us try to read this html a bit now.', 'start': 893.95, 'duration': 4.501}, {'end': 902.472, 'text': "here first we'll figure out what things we need to scrape from here.", 'start': 898.451, 'duration': 4.021}], 'summary': 'Demonstrating the process of running and copying html for further analysis.', 'duration': 39.511, 'max_score': 862.961, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c862961.jpg'}], 'start': 599.601, 'title': 'Web scraping, html parsing and data extraction', 'summary': "Demonstrates web scraping using python's beautifulsoup library to extract details from a flipkart webpage, such as title, ratings, reviews, and price, and explains parsing a large html file and extracting data, with a specific example of finding 24 products on the first page of flipkart.", 'chapters': [{'end': 672.074, 'start': 599.601, 'title': 'Web scraping with python', 'summary': 'Demonstrates web scraping with python using the beautifulsoup library to extract details such as title, ratings, reviews, and price from a flipkart webpage, and then explains the code for opening the connection and reading the web page.', 'duration': 72.473, 'highlights': ['The chapter demonstrates web scraping with Python using the BeautifulSoup library to extract details such as title, ratings, reviews, and price from a Flipkart webpage.', 'The code for opening the connection and reading the web page using uRequest and the read function is explained.']}, {'end': 1014.488, 'start': 672.074, 'title': 'Html parsing and data extraction', 'summary': 'Covers the process of parsing a large html file, extracting data from elements using beautifulsoup, and printing the number of products on a webpage, with a specific example of finding 24 products on the first page of flipkart.', 'duration': 342.414, 'highlights': ['The process of parsing a large HTML file and extracting data from elements using BeautifulSoup is explained. The process of parsing a large HTML file and extracting data from elements using BeautifulSoup is explained.', 'Printing the number of products on a webpage, with a specific example of finding 24 products on the first page of Flipkart. Printing the number of products on a webpage, with a specific example of finding 24 products on the first page of Flipkart.', 'Demonstration of extracting specific data elements like title, rating, and price from the HTML file using BeautifulSoup. Demonstration of extracting specific data elements like title, rating, and price from the HTML file using BeautifulSoup.']}], 'duration': 414.887, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c599601.jpg', 'highlights': ['Demonstrates web scraping with Python using the BeautifulSoup library to extract details such as title, ratings, reviews, and price from a Flipkart webpage.', 'Demonstration of extracting specific data elements like title, rating, and price from the HTML file using BeautifulSoup.', 'Printing the number of products on a webpage, with a specific example of finding 24 products on the first page of Flipkart.', 'The code for opening the connection and reading the web page using uRequest and the read function is explained.', 'The process of parsing a large HTML file and extracting data from elements using BeautifulSoup is explained.']}, {'end': 1354.617, 'segs': [{'end': 1244.196, 'src': 'embed', 'start': 1214.485, 'weight': 0, 'content': [{'end': 1218.429, 'text': 'so, as you can see, i have the title, i have the price and i have the ratings.', 'start': 1214.485, 'duration': 3.944}, {'end': 1220.672, 'text': 'so now what i have to do?', 'start': 1218.429, 'duration': 2.243}, {'end': 1224.676, 'text': 'this is showing in my terminal, but what if i want to show this in my file?', 'start': 1220.672, 'duration': 4.004}, {'end': 1226.999, 'text': "so for so, now i'll be creating a file.", 'start': 1224.676, 'duration': 2.323}, {'end': 1230.523, 'text': "so i'll just again comment this.", 'start': 1226.999, 'duration': 3.524}, {'end': 1232.325, 'text': "and here i'll be creating a file.", 'start': 1230.523, 'duration': 1.802}, {'end': 1244.196, 'text': 'So here I have created a file which is named as products.csv, and then, using the open function,', 'start': 1238.993, 'duration': 5.203}], 'summary': 'Creating a file named products.csv to store title, price, and ratings.', 'duration': 29.711, 'max_score': 1214.485, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1214485.jpg'}, {'end': 1295.127, 'src': 'embed', 'start': 1267.35, 'weight': 2, 'content': [{'end': 1270.072, 'text': 'So I have these three things for one containers.', 'start': 1267.35, 'duration': 2.722}, {'end': 1275.957, 'text': 'Now I have to do the same for all the containers or you can say all the div classes which is present on the web page.', 'start': 1270.453, 'duration': 5.504}, {'end': 1278.079, 'text': 'So for that I need a for loop.', 'start': 1276.478, 'duration': 1.601}, {'end': 1281.542, 'text': "So what I'll do next, I'll create a loop.", 'start': 1278.68, 'duration': 2.862}, {'end': 1285.825, 'text': 'Now here it will say for container in containers.', 'start': 1282.082, 'duration': 3.743}, {'end': 1292.251, 'text': 'So containers is the main class where all your element is present or your features are present.', 'start': 1286.166, 'duration': 6.085}, {'end': 1295.127, 'text': 'so here i just.', 'start': 1293.365, 'duration': 1.762}], 'summary': 'Creating a for loop to iterate through all div classes on the web page to perform a task for each container.', 'duration': 27.777, 'max_score': 1267.35, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1267350.jpg'}, {'end': 1332.125, 'src': 'embed', 'start': 1308.88, 'weight': 1, 'content': [{'end': 1317.488, 'text': 'so i have copied it here and then i saved it into a variable which is product name, similarly for product, similarly for price.', 'start': 1308.88, 'duration': 8.608}, {'end': 1328.784, 'text': "similarly for price i have, i'll be copying this thing and this and i'll be saving it to a variable which is price and similarly for rating.", 'start': 1319.321, 'duration': 9.463}, {'end': 1332.125, 'text': 'so here, if you see, i have added one more function, that is strip.', 'start': 1328.784, 'duration': 3.341}], 'summary': 'Variable data for product, price, and rating extracted using functions like copy and save.', 'duration': 23.245, 'max_score': 1308.88, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1308880.jpg'}], 'start': 1015.468, 'title': 'Scraping product details from web page', 'summary': 'Details the process of scraping product details from a web page using python. it includes extracting product title, price, and ratings and storing them in a csv file. it demonstrates scraping specific html elements, attributes, and the use of for loops to extract multiple product details.', 'chapters': [{'end': 1354.617, 'start': 1015.468, 'title': 'Scraping product details from web page', 'summary': 'Details the process of scraping product details from a web page, including extracting the product title, price, and ratings, and storing them in a csv file using python. it demonstrates the scraping of specific html elements and their attributes, such as class names, and the use of for loops to extract multiple product details.', 'duration': 339.149, 'highlights': ['The chapter demonstrates the process of scraping product details, including the product title, price, and ratings, and storing them in a CSV file using Python. The chapter provides an overview of scraping product details, such as the product title, price, and ratings, and storing them in a CSV file using Python. It showcases the use of specific HTML elements and attributes, and the implementation of for loops to extract multiple product details.', 'The process involves scraping specific HTML elements and their attributes, such as class names, to extract the product details. The process involves scraping specific HTML elements, such as class names, to extract product details, including the product title, price, and ratings.', 'Demonstrates the use of for loops to extract multiple product details from the web page. The chapter demonstrates the use of for loops to extract multiple product details from the web page, including the product title, price, and ratings.']}], 'duration': 339.149, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1015468.jpg', 'highlights': ['The chapter demonstrates the process of scraping product details, including the product title, price, and ratings, and storing them in a CSV file using Python.', 'The process involves scraping specific HTML elements and their attributes, such as class names, to extract the product details.', 'Demonstrates the use of for loops to extract multiple product details from the web page.']}, {'end': 1813.762, 'segs': [{'end': 1522.97, 'src': 'embed', 'start': 1492.365, 'weight': 1, 'content': [{'end': 1499.447, 'text': 'So again, I have split the string and then I have stored the final rating in the variable, which is final underscore rating.', 'start': 1492.365, 'duration': 7.082}, {'end': 1506.249, 'text': 'Now what I have to do, I have to concatenate all these three features.', 'start': 1502.728, 'duration': 3.521}, {'end': 1511.211, 'text': "So I'll be concatenating the product name, my price and the rating.", 'start': 1506.73, 'duration': 4.481}, {'end': 1514.752, 'text': "So I'll be concatenating it with the comma operator.", 'start': 1511.591, 'duration': 3.161}, {'end': 1522.97, 'text': "Now, if you see here, Now, this I'll be storing in the CSV as well.", 'start': 1516.633, 'duration': 6.337}], 'summary': 'Split string, store rating, concatenate features, store in csv.', 'duration': 30.605, 'max_score': 1492.365, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1492365.jpg'}, {'end': 1692.616, 'src': 'embed', 'start': 1661.554, 'weight': 0, 'content': [{'end': 1670.502, 'text': 'we have fetched all the details from the flipkart website and these details are automatically fetched and then stored in a csv file which is named as products.csv.', 'start': 1661.554, 'duration': 8.948}, {'end': 1674.007, 'text': 'now one more thing.', 'start': 1672.406, 'duration': 1.601}, {'end': 1678.489, 'text': 'now there are few things which i want to show you.', 'start': 1674.007, 'duration': 4.482}, {'end': 1689.975, 'text': 'so so, apart from these, we have a complete playlist on youtube now which covers all your topics from the basics.', 'start': 1678.489, 'duration': 11.486}, {'end': 1692.616, 'text': 'so you can go ahead and look at these videos.', 'start': 1689.975, 'duration': 2.641}], 'summary': 'Details fetched from flipkart stored in products.csv. new youtube playlist available.', 'duration': 31.062, 'max_score': 1661.554, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1661554.jpg'}, {'end': 1742.34, 'src': 'embed', 'start': 1716.818, 'weight': 3, 'content': [{'end': 1721.981, 'text': 'Apart from these, we also have the interview questions which will further help you to crack any interview.', 'start': 1716.818, 'duration': 5.163}, {'end': 1729.259, 'text': 'Now let me quickly tell you about the Python core certification which has been provided by Edureka.', 'start': 1723.072, 'duration': 6.187}, {'end': 1737.376, 'text': 'So these are the weekends and weekday batches.', 'start': 1734.434, 'duration': 2.942}, {'end': 1739.458, 'text': 'So you can choose according to your timings.', 'start': 1737.676, 'duration': 1.782}, {'end': 1742.34, 'text': 'And in this course, all the concepts are covered from the scratch.', 'start': 1739.538, 'duration': 2.802}], 'summary': 'Edureka offers python core certification with weekend and weekday batches, covering all concepts from scratch.', 'duration': 25.522, 'max_score': 1716.818, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1716818.jpg'}], 'start': 1370.256, 'title': 'Scraping flipkart data to csv', 'summary': 'Showcases the process of scraping product details from flipkart, extracting prices and ratings, then concatenating and storing the data in a csv file. it also highlights the availability of python core certification courses by edureka and a discount offer.', 'chapters': [{'end': 1813.762, 'start': 1370.256, 'title': 'Scraping flipkart data to csv', 'summary': 'Showcases the process of scraping product details from flipkart, extracting prices and ratings, then concatenating and storing the data in a csv file, with mention of potential issues and additional resources. it also highlights the availability of python core certification courses by edureka and a discount offer.', 'duration': 443.506, 'highlights': ['The chapter demonstrates scraping product details from Flipkart, including 24 products, their prices, and ratings, showcasing the practical application of web scraping. (24 products, prices, and ratings)', 'The process involves splitting the price and rating strings, using functions like split and replace to manipulate the data, ultimately concatenating the product name, price, and rating using the comma operator. (Data manipulation using split, replace, and concatenation)', "The extracted data is stored in a CSV file, ensuring careful handling to avoid creating extra columns due to commas in product names and utilizing the 'right' function, demonstrating practical file handling. (Storing data in a CSV file and file handling)", 'The transcript concludes with a mention of additional resources, including a YouTube playlist covering various topics, Python core certification courses by Edureka, and a discount offer on all courses provided by Edureka, offering a comprehensive learning experience. (Additional resources and discount offer)']}], 'duration': 443.506, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/mKxFfjNyj3c/pics/mKxFfjNyj3c1370256.jpg', 'highlights': ['The chapter demonstrates scraping product details from Flipkart, including 24 products, their prices, and ratings, showcasing the practical application of web scraping. (24 products, prices, and ratings)', 'The process involves splitting the price and rating strings, using functions like split and replace to manipulate the data, ultimately concatenating the product name, price, and rating using the comma operator. (Data manipulation using split, replace, and concatenation)', "The extracted data is stored in a CSV file, ensuring careful handling to avoid creating extra columns due to commas in product names and utilizing the 'right' function, demonstrating practical file handling. (Storing data in a CSV file and file handling)", 'The transcript concludes with a mention of additional resources, including a YouTube playlist covering various topics, Python core certification courses by Edureka, and a discount offer on all courses provided by Edureka, offering a comprehensive learning experience. (Additional resources and discount offer)']}], 'highlights': ['Web scraping involves extracting data from various websites, essential for tasks like price comparison on platforms like MySmartPrice.', 'Web scraping can be the only alternative to obtain data from websites that do not provide an API.', 'Web scraping involves automating the extraction of data from web pages and storing it locally in formats such as XML or CSV, offering efficiency and automation.', 'Demonstrates web scraping with Python using the BeautifulSoup library to extract details such as title, ratings, reviews, and price from a Flipkart webpage.', 'The chapter demonstrates the process of scraping product details, including the product title, price, and ratings, and storing them in a CSV file using Python.', 'The chapter demonstrates scraping product details from Flipkart, including 24 products, their prices, and ratings, showcasing the practical application of web scraping. (24 products, prices, and ratings)']}