title
What Is Hadoop? | Introduction To Hadoop | Hadoop Tutorial For Beginners | Simplilearn

description
🔥Post Graduate Program In Data Engineering: https://www.simplilearn.com/pgp-data-engineering-certification-training-course?utm_campaign=Hadoop-iANBytZ26MI&utm_medium=Descriptionff&utm_source=youtube 🔥Big Data Engineer Masters Program (Discount Code - YTBE15): https://www.simplilearn.com/big-data-engineer-masters-program?utm_campaign=Hadoop-iANBytZ26MI&utm_medium=Descriptionff&utm_source=youtube This Hadoop tutorial will help you understand what is Big Data, what is Hadoop, how Hadoop came into existence, what are the various components of Hadoop and an explanation on Hadoop use case. Below topics are explained in this Hadoop tutorial: Below topics are explained in this Hadoop tutorial: (02:30)The rise of Big Data (06:31)What is Big Data? (09:40)Big Data and its challenges (11:17)Hadoop as a solution (11:31)What is Hadoop? (11:51)Components of Hadoop (25:16)Use case of Hadoop To learn more about Hadoop, subscribe to our YouTube channel: https://www.youtube.com/user/Simplilearn?sub_confirmation=1 To access the slides, click here: https://www.slideshare.net/Simplilearn/what-is-hadoop-what-is-big-data-hadoop-introduction-to-hadoop-hadoop-tutorial-simplilearn/Simplilearn/what-is-hadoop-what-is-big-data-hadoop-introduction-to-hadoop-hadoop-tutorial-simplilearn Watch more videos on HadoopTraining: https://www.youtube.com/watch?v=CKLzDWMsQGM&list=PLEiEAq2VkUUJqp1k-g5W1mo37urJQOdCZ #Hadoop #WhatIsHadoop #BigData #Hadooptutorial #HadoopTutorialForBeginners #LearnHadoop #HadoopTraining #HadoopCertification #SimplilearnHadoop #Simplilearn 🔥 Enroll for FREE Big Data Hadoop Spark Course & Get your Completion Certificate: https://www.simplilearn.com/learn-hadoop-spark-basics-skillup?utm_campaign=Hadoop-iANBytZ26MI&utm_medium=Description&utm_source=youtube ➡️ About Post Graduate Program In Data Engineering This Data Engineering course is ideal for professionals, covering critical topics like the Hadoop framework, Data Processing using Spark, Data Pipelines with Kafka, Big Data on AWS, and Azure cloud infrastructures. This program is delivered via live sessions, industry projects, IBM hackathons, and Ask Me Anything sessions. ✅ Key Features Post Graduate Program Certificate and Alumni Association membership - Exclusive Master Classes and Ask me Anything sessions by IBM - 8X higher live interaction in live Data Engineering online classes by industry experts - Capstone from 3 domains and 14+ Projects with Industry datasets from YouTube, Glassdoor, Facebook etc. - Simplilearn's JobAssist helps you get noticed by top hiring companies ✅ Skills Covered - Real-Time Data Processing - Data Pipelining - Big Data Analytics - Data Visualization - Provisioning data storage services - Apache Hadoop - Ingesting Streaming and Batch Data - Transforming Data - Implementing Security Requirements - Data Protection - Encryption Techniques - Data Governance and Compliance Controls 👉 Learn More At: https://www.simplilearn.com/pgp-data-engineering-certification-training-course?utm_campaign=Hadoop-iANBytZ26MI&utm_medium=Descriptionff&utm_source=youtube 🔥🔥 Interested in Attending Live Classes? Call Us: IN - 18002127688 / US - +18445327688

detail
{'title': 'What Is Hadoop? | Introduction To Hadoop | Hadoop Tutorial For Beginners | Simplilearn', 'heatmap': [{'end': 782.764, 'start': 701.207, 'weight': 0.898}, {'end': 1449.329, 'start': 1405.078, 'weight': 0.747}, {'end': 1519.613, 'start': 1495.006, 'weight': 0.8}], 'summary': 'Provides an introduction to hadoop, showcasing its concepts through farming analogy, data evolution, big data challenges, hdfs distribution, and mapreduce applications, including a case study on fraud detection at zions bank.', 'chapters': [{'end': 142.052, 'segs': [{'end': 94.455, 'src': 'embed', 'start': 66.216, 'weight': 0, 'content': [{'end': 69.159, 'text': 'With this, harvesting is done simultaneously.', 'start': 66.216, 'duration': 2.943}, {'end': 71.98, 'text': 'So, instead of him trying to harvest all this different fruit,', 'start': 69.339, 'duration': 2.641}, {'end': 76.624, 'text': 'he now has two more people in there who are putting their fruit away and harvesting it for him.', 'start': 71.98, 'duration': 4.644}, {'end': 81.847, 'text': 'Now the storage room becomes a bottleneck to store and access all the fruits in a single storage area.', 'start': 76.784, 'duration': 5.063}, {'end': 83.929, 'text': "So they can't fit all the fruit in one place.", 'start': 82.007, 'duration': 1.922}, {'end': 89.832, 'text': 'So Jack decides to distribute the storage area and give each one of them a separate storage.', 'start': 84.309, 'duration': 5.523}, {'end': 92.014, 'text': 'And you can look at this computer terms.', 'start': 90.113, 'duration': 1.901}, {'end': 94.455, 'text': 'We have our people that are the processors.', 'start': 92.094, 'duration': 2.361}], 'summary': 'Harvesting efficiency increased with 2 more people. storage distribution alleviated bottleneck.', 'duration': 28.239, 'max_score': 66.216, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI66216.jpg'}, {'end': 155.702, 'src': 'embed', 'start': 124.319, 'weight': 2, 'content': [{'end': 127.923, 'text': 'she pulls out two apples and then another storage room he pulls out three oranges.', 'start': 124.319, 'duration': 3.604}, {'end': 134.247, 'text': 'And we complete a nice fruit basket, and this solution helps them to complete the order on time without any hassle.', 'start': 128.243, 'duration': 6.004}, {'end': 135.267, 'text': 'All of them are happy.', 'start': 134.387, 'duration': 0.88}, {'end': 138.569, 'text': "They're prepared for an increase in demand in the future.", 'start': 135.528, 'duration': 3.041}, {'end': 142.052, 'text': 'So they now have this growth system where you can just keep hiring on new people.', 'start': 138.75, 'duration': 3.302}, {'end': 145.334, 'text': 'They can continue to grow and develop a very large farm.', 'start': 142.192, 'duration': 3.142}, {'end': 150.457, 'text': 'So how does this story relate to big data? And I hinted at that a little bit.', 'start': 145.454, 'duration': 5.003}, {'end': 155.702, 'text': 'Earlier, the limited data, only one processor, one storage unit, was needed.', 'start': 150.777, 'duration': 4.925}], 'summary': 'Fulfilled order with 2 apples, 3 oranges, leading to future growth; related to big data and scalability.', 'duration': 31.383, 'max_score': 124.319, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI124319.jpg'}], 'start': 4.181, 'title': 'Understanding hadoop through farming', 'summary': 'Explains the concept of hadoop through a farming scenario, demonstrating parallel processing and distributed storage to meet increasing demand, using the example of jack hiring more people to harvest and store fruits.', 'chapters': [{'end': 142.052, 'start': 4.181, 'title': 'Understanding hadoop through farming', 'summary': 'Explains the concept of hadoop by using a farming scenario, where jack hires more people to harvest and store fruits, highlighting the parallel processing and distributed storage aspects of hadoop to meet increasing demand.', 'duration': 137.871, 'highlights': ['Jack hires two more people for harvesting, enabling simultaneous harvesting and storage. The hiring of two additional people allows for simultaneous harvesting and storage, addressing the issue of limited capacity and enabling increased productivity.', 'Distributed storage is implemented to accommodate the growing amount of fruit, ensuring timely order completion. The decision to distribute the storage area and provide each person with a separate storage space illustrates the concept of distributed storage, enabling parallel processing and timely order fulfillment.', 'The scenario demonstrates the ability to handle increasing demand by continuously hiring new people. The scenario depicts a growth system where new hires can be made to accommodate increasing demand, showcasing the scalability of the solution.']}], 'duration': 137.871, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI4181.jpg', 'highlights': ['Jack hires two more people for simultaneous harvesting and storage, addressing limited capacity and increasing productivity.', 'Distributed storage accommodates growing fruit amount, ensuring timely order completion through separate storage spaces.', 'The scenario showcases the scalability of the solution by continuously hiring new people to handle increasing demand.']}, {'end': 391.139, 'segs': [{'end': 185.621, 'src': 'embed', 'start': 159.745, 'weight': 1, 'content': [{'end': 165.29, 'text': 'Instead of having a small computer, you would then spend money for a huge mainframe with all the flashing lights on it.', 'start': 159.745, 'duration': 5.545}, {'end': 167.973, 'text': 'Then the Cray computers were really massive.', 'start': 165.41, 'duration': 2.563}, {'end': 173.535, 'text': 'Nowadays, a lot of the computers that sit on our desktop are powerful as the mainframes they had back then.', 'start': 168.473, 'duration': 5.062}, {'end': 175.836, 'text': "So it's pretty amazing how time has changed.", 'start': 173.635, 'duration': 2.201}, {'end': 177.777, 'text': 'But you used to be able to do everything on one computer.', 'start': 176.016, 'duration': 1.761}, {'end': 181.619, 'text': 'And you had structured data and a database you stored your structured data in.', 'start': 177.957, 'duration': 3.662}, {'end': 185.621, 'text': 'So most of the time you were querying databases, SQL queries.', 'start': 181.759, 'duration': 3.862}], 'summary': 'Transition from massive mainframes to powerful desktop computers, now capable of mainframe-level processing.', 'duration': 25.876, 'max_score': 159.745, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI159745.jpg'}, {'end': 240.111, 'src': 'embed', 'start': 197.726, 'weight': 0, 'content': [{'end': 205.31, 'text': 'you would get yourself a nice big Sun computer or mainframe if you had a lot of data and a lot of stuff going on, and it was very easy to do.', 'start': 197.726, 'duration': 7.584}, {'end': 212.732, 'text': 'Soon though, The data generation increased, leading to high volume of data along with different data formats.', 'start': 205.551, 'duration': 7.181}, {'end': 221.518, 'text': "And so you can imagine, in today's world, this year we will generate more data than all the previous years summed together.", 'start': 212.952, 'duration': 8.566}, {'end': 226.182, 'text': 'We will generate more data just this year than all the previous years summed together.', 'start': 221.638, 'duration': 4.544}, {'end': 228.003, 'text': "And that's the way it's been going for some time.", 'start': 226.322, 'duration': 1.681}, {'end': 229.664, 'text': 'And you can see we have a variety of data.', 'start': 228.123, 'duration': 1.541}, {'end': 236.408, 'text': "We have our structured data, which is what you'd think about a database with rows and columns and easy to look at, nice spreadsheet.", 'start': 229.704, 'duration': 6.704}, {'end': 238.049, 'text': 'We have our semi-structured data.', 'start': 236.508, 'duration': 1.541}, {'end': 240.111, 'text': 'They have emails as an example here.', 'start': 238.149, 'duration': 1.962}], 'summary': 'Data generation increased, this year will generate more data than all previous years combined.', 'duration': 42.385, 'max_score': 197.726, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI197726.jpg'}, {'end': 308.534, 'src': 'embed', 'start': 280.015, 'weight': 3, 'content': [{'end': 286.058, 'text': "There's just no way that's going to happen unless people don't mind waiting a year to get the history of their tweets or look something up.", 'start': 280.015, 'duration': 6.043}, {'end': 287.979, 'text': 'Hence, we start doing multiple processors.', 'start': 286.178, 'duration': 1.801}, {'end': 292.442, 'text': "So they're used to process high volume of data, and this saved time.", 'start': 288.26, 'duration': 4.182}, {'end': 293.223, 'text': "So we're moving forward.", 'start': 292.502, 'duration': 0.721}, {'end': 294.424, 'text': 'We got multiple processors.', 'start': 293.243, 'duration': 1.181}, {'end': 299.728, 'text': 'The single storage unit became the bottleneck due to which network overhead was generated.', 'start': 294.704, 'duration': 5.024}, {'end': 308.534, 'text': 'So now you have your network coming in, and each one of these servers has to wait before it can grab the data from the single stored unit.', 'start': 300.128, 'duration': 8.406}], 'summary': 'Implementing multiple processors reduced data processing time and resolved network bottleneck.', 'duration': 28.519, 'max_score': 280.015, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI280015.jpg'}, {'end': 375.293, 'src': 'embed', 'start': 336.776, 'weight': 4, 'content': [{'end': 342.258, 'text': "You're not getting a bottleneck somewhere where people are just waiting for data being pulled or being processed.", 'start': 336.776, 'duration': 5.482}, {'end': 347.2, 'text': 'This is known as parallel processing with distributed storage.', 'start': 342.538, 'duration': 4.662}, {'end': 350.302, 'text': 'So parallel processing, distributed storage.', 'start': 347.501, 'duration': 2.801}, {'end': 355.704, 'text': 'And you can see here the parallel processing is your different computers running the processes and distributed storage.', 'start': 350.522, 'duration': 5.182}, {'end': 360.566, 'text': "So, what's in it for you? Well, today we're going to cover big data and its challenges.", 'start': 356.144, 'duration': 4.422}, {'end': 365.008, 'text': "We're going to look at Hadoop as a solution, because this is all about what is Hadoop.", 'start': 360.746, 'duration': 4.262}, {'end': 372.712, 'text': 'Hadoop is probably one of the biggest growing file systems out there that people have now gone to, and there are a lot of varieties of Hadoop.', 'start': 365.308, 'duration': 7.404}, {'end': 375.293, 'text': 'I was surprised when I started digging into it.', 'start': 372.832, 'duration': 2.461}], 'summary': 'Parallel processing and distributed storage address big data challenges, with hadoop emerging as a widely adopted solution.', 'duration': 38.517, 'max_score': 336.776, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI336776.jpg'}], 'start': 142.192, 'title': 'Data evolution and processing', 'summary': 'Delves into the evolution of data and computing, emphasizing the surge in data volumes and the shift to structured, semi-structured, and unstructured data. it also explores the adoption of parallel processing and distributed storage to tackle high volume data, with a mention of hadoop as a big data solution.', 'chapters': [{'end': 259.103, 'start': 142.192, 'title': 'Evolution of data and computing', 'summary': "Highlights the evolution of data and computing, from single processors in the 90s to the current generation's astonishing data volumes, with a specific emphasis on structured, semi-structured, and unstructured data. this year alone will generate more data than all previous years combined.", 'duration': 116.911, 'highlights': ["The data generation has increased, leading to this year's generation of more data than all previous years combined.", "The evolution of computing power from small computers in the 90s to today's desktop computers being as powerful as the mainframes back then.", 'The types of data include structured data (resembling a database), semi-structured data (emails, XML, HTML), and unstructured data (diverse photo formats).']}, {'end': 391.139, 'start': 259.142, 'title': 'Parallel processing and distributed storage', 'summary': 'Discusses the challenges of processing high volume data, the use of multiple processors to save time, and the adoption of parallel processing with distributed storage to eliminate network overhead, leading to the introduction of hadoop as a solution for big data.', 'duration': 131.997, 'highlights': ['The adoption of multiple processors to process high volume data saved time and eliminated network overhead. Using multiple processors to process high volume data saved time and eliminated network overhead, enabling easy access to storage and data.', 'The introduction of parallel processing with distributed storage eliminated network overhead and bottleneck issues, improving data processing efficiency. The adoption of parallel processing with distributed storage eliminated network overhead and bottleneck issues, improving data processing efficiency and eliminating wait times for data retrieval and processing.', 'Hadoop was introduced as a solution for big data challenges, offering a widely used file system with various specialized offshoots. Hadoop was introduced as a solution for big data challenges, offering a widely used file system with various specialized offshoots to address different data processing needs.']}], 'duration': 248.947, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI142192.jpg', 'highlights': ["This year's generation of more data than all previous years combined.", "Today's desktop computers being as powerful as the mainframes back then.", 'The types of data include structured, semi-structured, and unstructured data.', 'The adoption of multiple processors to process high volume data saved time and eliminated network overhead.', 'The introduction of parallel processing with distributed storage improved data processing efficiency.', 'Hadoop was introduced as a solution for big data challenges.']}, {'end': 915.506, 'segs': [{'end': 442.249, 'src': 'embed', 'start': 412.854, 'weight': 4, 'content': [{'end': 414.516, 'text': 'Volume is the most common one that we see.', 'start': 412.854, 'duration': 1.662}, {'end': 422.445, 'text': "Velocity, how fast is that data being generated? So you have your volume, but you might have a huge amount of data that's just streaming in.", 'start': 414.696, 'duration': 7.749}, {'end': 424.645, 'text': "You can't just process it on one computer.", 'start': 422.605, 'duration': 2.04}, {'end': 429.546, 'text': 'You need to process it on multiple computers and then send that to the appropriate storage spaces.', 'start': 424.665, 'duration': 4.881}, {'end': 433.147, 'text': 'Variety When you look at variety every step you do to go through.', 'start': 429.786, 'duration': 3.361}, {'end': 439.468, 'text': "it might be very simple to do like a spreadsheet, but now you have to look at the email and how it's stored, so it needs a different process.", 'start': 433.147, 'duration': 6.321}, {'end': 442.249, 'text': "You have to look at HTML pages and how they're stored.", 'start': 439.608, 'duration': 2.641}], 'summary': 'Big data involves managing large volumes, high velocity, and diverse data types.', 'duration': 29.395, 'max_score': 412.854, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI412854.jpg'}, {'end': 611.647, 'src': 'embed', 'start': 580.549, 'weight': 1, 'content': [{'end': 583.31, 'text': 'So big data challenges and solutions.', 'start': 580.549, 'duration': 2.761}, {'end': 588.772, 'text': 'So we looked at a lot of the things that define big data, which also define the challenges in big data.', 'start': 583.45, 'duration': 5.322}, {'end': 589.752, 'text': 'So those are all important.', 'start': 588.812, 'duration': 0.94}, {'end': 593.653, 'text': "So having a single central storage, that's a big challenge.", 'start': 589.812, 'duration': 3.841}, {'end': 595.454, 'text': 'So we do a distributed storages.', 'start': 593.913, 'duration': 1.541}, {'end': 599.897, 'text': "Now, when we talk about distributed storages, we're not talking about around the world.", 'start': 595.734, 'duration': 4.163}, {'end': 605.022, 'text': "We're not talking about you have one computer in the U.S., one in India, one in Australia.", 'start': 599.917, 'duration': 5.105}, {'end': 611.647, 'text': "We're talking about you have a rack of computers or two racks or three racks or a hundred racks of computers.", 'start': 605.102, 'duration': 6.545}], 'summary': 'Big data challenges include single central storage and require distributed storage solutions with multiple racks of computers.', 'duration': 31.098, 'max_score': 580.549, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI580549.jpg'}, {'end': 671.03, 'src': 'embed', 'start': 639.612, 'weight': 2, 'content': [{'end': 642.053, 'text': 'So one of the challenges, we had a serial processor.', 'start': 639.612, 'duration': 2.441}, {'end': 644.914, 'text': 'You have one processor, one output, one input.', 'start': 642.073, 'duration': 2.841}, {'end': 651.157, 'text': 'Now you have parallel processing, so you have processes A and B, and then they have your output going out.', 'start': 645.134, 'duration': 6.023}, {'end': 655.819, 'text': 'And then the lack of ability to process unstructured data.', 'start': 651.676, 'duration': 4.143}, {'end': 663.865, 'text': 'And this was like really where Hadoop started becoming big, was all this weird data were accumulated, all this unstructured, structured,', 'start': 655.939, 'duration': 7.926}, {'end': 664.946, 'text': 'semi-structured data.', 'start': 663.865, 'duration': 1.081}, {'end': 670.03, 'text': 'Now they have a way of actually storing it and processing it that makes a lot more sense in the Hadoop system.', 'start': 665.106, 'duration': 4.924}, {'end': 671.03, 'text': 'That was a big challenge.', 'start': 670.07, 'duration': 0.96}], 'summary': 'Challenges with serial processing, solved by parallel processing in hadoop for unstructured data.', 'duration': 31.418, 'max_score': 639.612, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI639612.jpg'}, {'end': 784.786, 'src': 'heatmap', 'start': 692.048, 'weight': 0, 'content': [{'end': 700.767, 'text': 'So what exactly is Hadoop? Hadoop is a framework that manages big data storage in a distributed way and processes it parallelly.', 'start': 692.048, 'duration': 8.719}, {'end': 704.832, 'text': "We've been talking about this now for a little bit here, so this shouldn't be a big surprise.", 'start': 701.207, 'duration': 3.625}, {'end': 711.159, 'text': 'We have our big data, we need to store it, and we have our processing and analyzing of it, so we need to do something with that data.', 'start': 704.872, 'duration': 6.287}, {'end': 712.761, 'text': 'Components of Hadoop.', 'start': 711.359, 'duration': 1.402}, {'end': 718.102, 'text': "So how does the Hadoop system do what it does? So let's look at the components of Hadoop.", 'start': 713.081, 'duration': 5.021}, {'end': 727.084, 'text': 'We have our Hadoop setup, and then you have your Hadoop HDFS, which is the storage unit of Hadoop, or Hadoop file system, which the HDFS stands for.', 'start': 718.702, 'duration': 8.382}, {'end': 730.444, 'text': 'And then you have your processing in Hadoop, which is your map reduce.', 'start': 727.264, 'duration': 3.18}, {'end': 739.186, 'text': 'And you heard me earlier talk about reduce, not just for where it fit up there, but also understanding its functionality in processing data.', 'start': 730.764, 'duration': 8.422}, {'end': 741.487, 'text': "So let's start with the Hadoop file system.", 'start': 739.446, 'duration': 2.041}, {'end': 750.774, 'text': 'Hadoop distributed file system, HDFS, is specially designed for storing huge data sets in commodity hardware.', 'start': 741.628, 'duration': 9.146}, {'end': 754.916, 'text': "So we have name node, data node, and I'll come back in just a minute, commodity there.", 'start': 750.974, 'duration': 3.942}, {'end': 756.777, 'text': 'We have name node, data node.', 'start': 755.016, 'duration': 1.761}, {'end': 759.799, 'text': 'Our name node, there is only one name node.', 'start': 756.958, 'duration': 2.841}, {'end': 764.863, 'text': "Now in the newer versions of Hadoop, there's a backup name node, but only one is active.", 'start': 760.08, 'duration': 4.783}, {'end': 766.604, 'text': 'Only one is processing everything.', 'start': 764.943, 'duration': 1.661}, {'end': 768.606, 'text': 'And there can be multiple data nodes.', 'start': 766.684, 'duration': 1.922}, {'end': 770.949, 'text': 'So you have all your racks of computers.', 'start': 768.927, 'duration': 2.022}, {'end': 772.691, 'text': 'In this case, we have three data nodes.', 'start': 770.989, 'duration': 1.702}, {'end': 775.034, 'text': "But usually there's at least five to ten.", 'start': 772.832, 'duration': 2.202}, {'end': 782.764, 'text': "That's very rare to see a company spend a lot of money on a rack that's less than five data nodes for storing for their Hadoop file system.", 'start': 775.114, 'duration': 7.65}, {'end': 784.786, 'text': "That's not to say that there isn't reason to do that.", 'start': 782.844, 'duration': 1.942}], 'summary': 'Hadoop manages big data storage and processing in a distributed, parallel manner with components including hdfs and mapreduce.', 'duration': 92.738, 'max_score': 692.048, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI692048.jpg'}], 'start': 391.519, 'title': 'Understanding big data and its challenges', 'summary': 'Explains the concept of big data, emphasizing the 5vs - volume, velocity, variety, value, and veracity, and discusses the challenges, emphasizing the need for distributed storage, parallel processing, and the role of hadoop as a solution.', 'chapters': [{'end': 580.249, 'start': 391.519, 'title': 'Understanding big data: the 5vs', 'summary': 'Explains the concept of big data, focusing on the 5vs - volume, velocity, variety, value, and veracity, and how they impact data processing and resource utilization.', 'duration': 188.73, 'highlights': ['Veracity is an important aspect of big data as uncertain data requires significant processing, adding a huge amount of processing and resource usage. Veracity is crucial in big data as uncertain data, such as Twitter content with misspellings and abbreviations, requires extensive processing and storage, significantly impacting resource usage.', "Value is the end result of processing big data, as it needs to be reduced to a certain value that provides meaningful insights and meets the stakeholders' expectations. The end goal of processing big data is to derive meaningful value that meets stakeholders' expectations, requiring the reduction of processed data to a certain value to provide valuable insights.", 'Volume is not the sole measurement of big data, as velocity and variety also play key roles, impacting data processing and resource utilization. Volume alone does not define big data, as velocity and variety also significantly impact data processing and resource utilization, requiring processing on multiple computers and different storage spaces.']}, {'end': 915.506, 'start': 580.549, 'title': 'Big data challenges & hadoop solutions', 'summary': 'Discusses the challenges in big data, emphasizing the need for distributed storage, parallel processing, and the ability to handle unstructured data, and highlights the role of hadoop as a solution and its key components.', 'duration': 334.957, 'highlights': ["Hadoop's role as a solution for big data challenges Hadoop is highlighted as a significant solution for big data challenges, as it manages big data storage in a distributed manner and processes it parallelly, being the most commonly used and highly developed framework.", 'Importance of distributed storage for big data The importance of distributed storage in big data is emphasized, where data is stored across multiple computers to overcome the challenge of having a single central storage, enabling scalability and parallel processing.', 'Challenges of serial processing and transition to parallel processing The transition from serial processing to parallel processing is explained, highlighting the challenges of serial processing and the benefits of parallel processing for efficient data processing.', 'Significance of handling unstructured data in Hadoop The significance of Hadoop in handling unstructured data is emphasized, as it provides a way to store and process unstructured, structured, and semi-structured data effectively, contributing to overcoming a big challenge in big data.', 'Components of Hadoop file system and its setup The components of the Hadoop file system, including Hadoop HDFS for storage and MapReduce for processing, are detailed, along with the setup involving name nodes and data nodes, highlighting their roles and functionalities in managing and processing data.']}], 'duration': 523.987, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI391519.jpg', 'highlights': ['Hadoop is highlighted as a significant solution for big data challenges, managing big data storage in a distributed manner and processing it parallelly.', 'The importance of distributed storage in big data is emphasized, enabling scalability and parallel processing.', 'The transition from serial processing to parallel processing is explained, highlighting the challenges of serial processing and the benefits of parallel processing for efficient data processing.', 'The significance of Hadoop in handling unstructured data is emphasized, providing a way to store and process unstructured, structured, and semi-structured data effectively.', 'Volume alone does not define big data, as velocity and variety also significantly impact data processing and resource utilization, requiring processing on multiple computers and different storage spaces.']}, {'end': 1160.478, 'segs': [{'end': 956.684, 'src': 'embed', 'start': 915.766, 'weight': 0, 'content': [{'end': 919.108, 'text': 'in HDFS data stored in a distributed manner.', 'start': 915.766, 'duration': 3.342}, {'end': 921.65, 'text': "so we're looking at 30 terabyte file.", 'start': 919.108, 'duration': 2.542}, {'end': 923.03, 'text': "that's a pretty big file.", 'start': 921.65, 'duration': 1.38}, {'end': 928.614, 'text': "even on my high-end computer it's rare for me to have more than five to ten terabytes on a desktop.", 'start': 923.03, 'duration': 5.584}, {'end': 930.695, 'text': "you know computer sitting on my desk that I'm working on.", 'start': 928.614, 'duration': 2.081}, {'end': 935.278, 'text': 'most computers probably have a one terabyte hard drive, so 30 terabytes is pretty big.', 'start': 930.695, 'duration': 4.583}, {'end': 942.585, 'text': 'the 30 terabytes of data is loaded and it goes into the name node and then the name node distributes that around.', 'start': 935.278, 'duration': 7.307}, {'end': 946.831, 'text': 'And so in this case, we divided it into 128 megabytes each.', 'start': 943.025, 'duration': 3.806}, {'end': 952.158, 'text': "And we'll designate the blocks with blue, red, and kind of a tan color of gray.", 'start': 947.171, 'duration': 4.987}, {'end': 953.339, 'text': 'So we have our data nodes.', 'start': 952.218, 'duration': 1.121}, {'end': 956.684, 'text': 'And so we go ahead and distribute that data across those data nodes.', 'start': 953.52, 'duration': 3.164}], 'summary': 'Data stored in hdfs in a distributed manner, with 30 terabytes divided into 128 mb blocks across data nodes.', 'duration': 40.918, 'max_score': 915.766, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI915766.jpg'}, {'end': 1011.911, 'src': 'embed', 'start': 977.092, 'weight': 3, 'content': [{'end': 983.713, 'text': 'so that if one of these commodity machines remember commodity versus enterprise if one of these commodity machines goes down,', 'start': 977.092, 'duration': 6.621}, {'end': 985.654, 'text': "you can swap a new one in, and it's cheap.", 'start': 983.713, 'duration': 1.941}, {'end': 987.596, 'text': 'Modeling machines are more likely to fail.', 'start': 985.894, 'duration': 1.702}, {'end': 988.596, 'text': "That's why they're cheaper.", 'start': 987.636, 'duration': 0.96}, {'end': 991.219, 'text': "Best description I heard was they're a cheap knockoff.", 'start': 988.757, 'duration': 2.462}, {'end': 995.743, 'text': "The Chinese knockoff is the way they actually rephrased it, although they're made all over the world now.", 'start': 991.599, 'duration': 4.144}, {'end': 999.926, 'text': 'So features of the Hadoop file system or the HDFS.', 'start': 995.943, 'duration': 3.983}, {'end': 1001.888, 'text': 'It provides a distributed storage.', 'start': 1000.246, 'duration': 1.642}, {'end': 1006.81, 'text': 'implemented on commodity hardware and provides data security.', 'start': 1002.248, 'duration': 4.562}, {'end': 1011.911, 'text': "So this is that replication part where you don't have a RAID backup or a high-end backup.", 'start': 1006.97, 'duration': 4.941}], 'summary': 'Hadoop file system uses commodity hardware for distributed storage, emphasizing cost-effectiveness and replication for data security.', 'duration': 34.819, 'max_score': 977.092, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI977092.jpg'}, {'end': 1055.353, 'src': 'embed', 'start': 1030.122, 'weight': 4, 'content': [{'end': 1035.667, 'text': "Let's go ahead and look at the Hadoop Map Reduce processing unit of Hadoop.", 'start': 1030.122, 'duration': 5.545}, {'end': 1042.625, 'text': 'Hadoop MapReduce is a programming technique where huge data is processed in parallel and distributed fashion.', 'start': 1036.021, 'duration': 6.604}, {'end': 1049.349, 'text': 'And when we look at this, because in Hadoop, the MapReduce is how it queries the data and how it functions, which is important to know.', 'start': 1042.825, 'duration': 6.524}, {'end': 1055.353, 'text': "Also think about the process, because thinking MapReduce is also very important if you're in the programming side.", 'start': 1049.589, 'duration': 5.764}], 'summary': 'Hadoop mapreduce processes data in parallel and distributed fashion, crucial for querying and programming.', 'duration': 25.231, 'max_score': 1030.122, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1030122.jpg'}], 'start': 915.766, 'title': 'Hdfs data distribution and hadoop benefits', 'summary': 'Explains the distribution of a 30 terabyte file in hdfs, with data divided into 128 megabyte blocks and replicated three times across different data nodes. it also discusses the benefits of hadoop, such as fault tolerance, distributed storage, and parallel processing in mapreduce, enabling efficient data querying and valuable output.', 'chapters': [{'end': 977.092, 'start': 915.766, 'title': 'Hdfs data distribution', 'summary': 'Explains the distribution of a 30 terabyte file in hdfs, with data divided into 128 megabyte blocks and replicated three times across different data nodes.', 'duration': 61.326, 'highlights': ['The 30 terabytes of data is loaded and divided into 128 megabyte blocks, which are replicated three times by default across different data nodes.', 'Data in HDFS is distributed across data nodes, allowing for the management of large files such as a 30 terabyte file, which surpasses the storage capacity of most high-end computers.', 'The distribution of data in HDFS enables the storage and replication of large files, such as the 30 terabyte file mentioned, which is significantly larger than the storage capacity of most desktop computers.']}, {'end': 1160.478, 'start': 977.092, 'title': 'Hadoop: distributed storage and processing', 'summary': 'Discusses the benefits of hadoop, such as fault tolerance, distributed storage, and parallel processing in mapreduce, which enables efficient data querying and valuable output.', 'duration': 183.386, 'highlights': ['The Hadoop file system (HDFS) provides distributed storage on commodity hardware, ensuring data security and fault tolerance through data replication in two different places.', 'Hadoop MapReduce allows parallel processing of big data, reducing the processing load on the master node and efficiently producing valuable output.', "Commodity machines are cheaper and more likely to fail, making them suitable for Hadoop's fault-tolerant design, which enables easy machine replacement and high fault tolerance."]}], 'duration': 244.712, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI915766.jpg', 'highlights': ['Data in HDFS is distributed across data nodes, allowing for the management of large files such as a 30 terabyte file, which surpasses the storage capacity of most high-end computers.', 'The 30 terabytes of data is loaded and divided into 128 megabyte blocks, which are replicated three times by default across different data nodes.', 'The distribution of data in HDFS enables the storage and replication of large files, such as the 30 terabyte file mentioned, which is significantly larger than the storage capacity of most desktop computers.', 'The Hadoop file system (HDFS) provides distributed storage on commodity hardware, ensuring data security and fault tolerance through data replication in two different places.', 'Hadoop MapReduce allows parallel processing of big data, reducing the processing load on the master node and efficiently producing valuable output.', "Commodity machines are cheaper and more likely to fail, making them suitable for Hadoop's fault-tolerant design, which enables easy machine replacement and high fault tolerance."]}, {'end': 1802.264, 'segs': [{'end': 1397.873, 'src': 'embed', 'start': 1354.933, 'weight': 0, 'content': [{'end': 1357.915, 'text': 'the shuffle and sort is all handled by the map.', 'start': 1354.933, 'duration': 2.982}, {'end': 1358.615, 'text': 'reduce, setup.', 'start': 1357.915, 'duration': 0.7}, {'end': 1362.516, 'text': 'Then we have our shuffle and sort, which handles getting all those keys together.', 'start': 1358.775, 'duration': 3.741}, {'end': 1367.818, 'text': 'And finally, under the shuffle and sort, then goes into the reduce phase, where we get our final answer.', 'start': 1362.877, 'duration': 4.941}, {'end': 1370.299, 'text': 'So components of Hadoop version 2.', 'start': 1368.019, 'duration': 2.28}, {'end': 1373.401, 'text': "Remember we said there's a version 3 out, which is in beta test at this time.", 'start': 1370.299, 'duration': 3.102}, {'end': 1377.422, 'text': 'Version 2, we have the storage unit of Hadoop, the HDFS.', 'start': 1373.461, 'duration': 3.961}, {'end': 1382.184, 'text': 'We have Hadoop YARN, which is our resource management unit of Hadoop.', 'start': 1377.722, 'duration': 4.462}, {'end': 1384.945, 'text': 'And we have our Hadoop MapReduce.', 'start': 1382.424, 'duration': 2.521}, {'end': 1386.886, 'text': "Let's touch upon the middle one.", 'start': 1385.265, 'duration': 1.621}, {'end': 1389.248, 'text': "We haven't talked about that yet, the Hadoop YARN.", 'start': 1386.906, 'duration': 2.342}, {'end': 1392.57, 'text': 'YARN, yet another resource negotiator.', 'start': 1389.408, 'duration': 3.162}, {'end': 1396.472, 'text': 'So the YARN is basically your Hadoop operating system.', 'start': 1392.71, 'duration': 3.762}, {'end': 1397.873, 'text': 'It acts like an operating system.', 'start': 1396.512, 'duration': 1.361}], 'summary': 'Hadoop version 2 components include hdfs, yarn, mapreduce. version 3 is in beta test.', 'duration': 42.94, 'max_score': 1354.933, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1354933.jpg'}, {'end': 1449.329, 'src': 'heatmap', 'start': 1405.078, 'weight': 0.747, 'content': [{'end': 1410.161, 'text': 'But it acts like just like your PC, Windows OS or Apple OS or Linux.', 'start': 1405.078, 'duration': 5.083}, {'end': 1411.241, 'text': 'So it acts like that.', 'start': 1410.401, 'duration': 0.84}, {'end': 1413.663, 'text': "It's a file system that sits on top of everything.", 'start': 1411.361, 'duration': 2.302}, {'end': 1416.466, 'text': "And it's responsible for managing cluster resources,", 'start': 1413.863, 'duration': 2.603}, {'end': 1420.992, 'text': "knowing where everything's going and making sure you don't overload one machine when one doesn't get used.", 'start': 1416.466, 'duration': 4.526}, {'end': 1425.157, 'text': "And it also does the job scheduling, making sure they're scheduled in the right place.", 'start': 1421.192, 'duration': 3.965}, {'end': 1427.64, 'text': "So what is YARN? So let's go ahead and take a look at YARN.", 'start': 1425.457, 'duration': 2.183}, {'end': 1429.302, 'text': 'We have a client, client, client.', 'start': 1427.66, 'duration': 1.642}, {'end': 1436.064, 'text': "Now, the client machine is you on your laptop, whether you're doing a query, you're putting together some code to do some data analysis.", 'start': 1429.602, 'duration': 6.462}, {'end': 1440.746, 'text': 'So we have our client machine, and the client machine submits the job request.', 'start': 1436.225, 'duration': 4.521}, {'end': 1442.407, 'text': 'So it sends it out there and says hey,', 'start': 1440.846, 'duration': 1.561}, {'end': 1449.329, 'text': 'I want you to query all the word count for all these Twitters between the current political candidates that are running.', 'start': 1442.407, 'duration': 6.922}], 'summary': 'Yarn is a file system managing cluster resources and job scheduling, handling job requests from client machines.', 'duration': 44.251, 'max_score': 1405.078, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1405078.jpg'}, {'end': 1525.497, 'src': 'heatmap', 'start': 1495.006, 'weight': 0.8, 'content': [{'end': 1499.789, 'text': "So where are we going to put the physical resources in when we're working with them? And you have an app master.", 'start': 1495.006, 'duration': 4.783}, {'end': 1503.554, 'text': 'The app master requests container from the node manager.', 'start': 1499.889, 'duration': 3.665}, {'end': 1506.898, 'text': 'So your app master says, hey, node manager, I have a process going on.', 'start': 1503.634, 'duration': 3.264}, {'end': 1509.742, 'text': 'I need so much RAM and so much CPU.', 'start': 1507.119, 'duration': 2.623}, {'end': 1516.111, 'text': 'Can you do that for me? And these all go back to the resource manager, which is in charge of the overall big picture.', 'start': 1509.822, 'duration': 6.289}, {'end': 1519.613, 'text': "So let's go ahead and take a look at the use of Hadoop.", 'start': 1516.411, 'duration': 3.202}, {'end': 1525.497, 'text': 'Why do we want Hadoop? What is the end result? What can you use it for? And there are so many use cases.', 'start': 1519.673, 'duration': 5.824}], 'summary': 'Hadoop manages physical resources for app masters, ensuring ram and cpu allocation, with resource manager overseeing the big picture and serving various use cases.', 'duration': 30.491, 'max_score': 1495.006, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1495006.jpg'}, {'end': 1567.63, 'src': 'embed', 'start': 1529.559, 'weight': 2, 'content': [{'end': 1532.801, 'text': "But we're going to look at fraudulent activities in one case study.", 'start': 1529.559, 'duration': 3.242}, {'end': 1536.263, 'text': "So we're going to do Hadoop use case combating fraudulent activities.", 'start': 1532.941, 'duration': 3.322}, {'end': 1542.927, 'text': 'And they start by detecting fraudulent transactions as one among many of the various problems any bank faces.', 'start': 1536.563, 'duration': 6.364}, {'end': 1547.53, 'text': 'And this is also true of almost any online business, but big for banks.', 'start': 1543.127, 'duration': 4.403}, {'end': 1553.854, 'text': "You have your money that's being stored there and you have a lot of fraud activities that come in from all kinds of different directions,", 'start': 1547.63, 'duration': 6.224}, {'end': 1555.535, 'text': 'where people are trying to rip the bank off.', 'start': 1553.854, 'duration': 1.681}, {'end': 1563.806, 'text': 'In this case, Zions, the Zions Bank Corporation, their main challenge was to combat the fraudulent activities which were taking place.', 'start': 1555.879, 'duration': 7.927}, {'end': 1567.63, 'text': 'Approaches used by Zions security team to combat fraudulent activities.', 'start': 1564.106, 'duration': 3.524}], 'summary': 'Zions bank used hadoop to combat fraudulent activities, detecting transactions and addressing main challenges.', 'duration': 38.071, 'max_score': 1529.559, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1529559.jpg'}, {'end': 1667.865, 'src': 'embed', 'start': 1636.686, 'weight': 3, 'content': [{'end': 1638.548, 'text': 'But all that can be dumped into Hadoop system.', 'start': 1636.686, 'duration': 1.862}, {'end': 1642.891, 'text': "And it's very affordable to do that compared to doing it across enterprise machines.", 'start': 1638.608, 'duration': 4.283}, {'end': 1645.492, 'text': 'So this is how the Hadoop solved the problems.', 'start': 1643.251, 'duration': 2.241}, {'end': 1649.515, 'text': 'Zions could now store massive amount of data using Hadoop.', 'start': 1645.692, 'duration': 3.823}, {'end': 1653.157, 'text': 'So they could now pull all that data together and store it in one place to look at it.', 'start': 1649.575, 'duration': 3.582}, {'end': 1656.839, 'text': 'Processing Processing of unstructured data like server logs.', 'start': 1653.417, 'duration': 3.422}, {'end': 1658.22, 'text': "Remember we're talking about the different data.", 'start': 1656.859, 'duration': 1.361}, {'end': 1659.2, 'text': 'Server logs are big.', 'start': 1658.26, 'duration': 0.94}, {'end': 1660.181, 'text': 'Customer data.', 'start': 1659.36, 'duration': 0.821}, {'end': 1661.922, 'text': 'All those emails going back and forth.', 'start': 1660.361, 'duration': 1.561}, {'end': 1664.443, 'text': 'Customer transactions, which is more structured.', 'start': 1662.002, 'duration': 2.441}, {'end': 1667.865, 'text': 'Usually you have like they took a loan out for X amount of dollars, X interest.', 'start': 1664.623, 'duration': 3.242}], 'summary': 'Hadoop enabled zions to store and process massive amounts of data, including unstructured data like server logs and customer emails, making it more affordable and efficient compared to enterprise machines.', 'duration': 31.179, 'max_score': 1636.686, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1636686.jpg'}], 'start': 1160.498, 'title': 'Mapreduce in hadoop and its applications', 'summary': "Explains the process of mapreduce in hadoop, including its phases and use case in combating fraudulent activities. it also discusses zions bank's use of hadoop, resulting in improved security, fraud detection, and cost savings.", 'chapters': [{'end': 1567.63, 'start': 1160.498, 'title': 'Understanding mapreduce in hadoop', 'summary': 'Explains the process of mapreduce in hadoop, detailing the splitting, mapping, shuffling, sorting, and reducing phases, and touches upon the hadoop yarn resource management unit, with a use case of combating fraudulent activities in the banking sector.', 'duration': 407.132, 'highlights': ['The process of MapReduce in Hadoop, detailing the splitting, mapping, shuffling, sorting, and reducing phases. The chapter thoroughly explains the entire process of MapReduce in Hadoop, including the splitting of input data, mapping individual lines, shuffling and sorting the keys, and finally reducing the data to obtain the final output.', 'The Hadoop YARN resource management unit, acting like an operating system, responsible for managing cluster resources and job scheduling. The Hadoop YARN is described as the resource management unit of Hadoop, acting like an operating system by managing cluster resources, ensuring balanced resource allocation, and handling job scheduling across the entire cluster.', 'Use case of combating fraudulent activities in the banking sector, focusing on detecting fraudulent transactions and the challenges faced by Zions Bank Corporation. A use case of combating fraudulent activities in the banking sector is discussed, particularly focusing on the challenges faced by Zions Bank Corporation in detecting and combating fraudulent transactions, a common problem for banks and online businesses.']}, {'end': 1802.264, 'start': 1567.91, 'title': 'Hadoop solving big data problems', 'summary': 'Discusses how zions bank used hadoop to store and analyze massive amounts of unstructured and structured data, leading to improved security and fraud detection, and efficient data processing, resulting in cost savings and better business performance.', 'duration': 234.354, 'highlights': ['Zions Bank faced challenges in storing and analyzing massive amounts of unstructured and structured data, which led to the adoption of Hadoop for data storage and processing. Massive amounts of unstructured and structured data', 'Hadoop enabled Zions Bank to store and process large volumes of unstructured data like server logs, customer data, emails, and transactions, leading to improved data analysis and security detection. Improved data analysis and security detection', 'The use of Hadoop resulted in significant time efficiency, enabling Zions Bank to process data in a matter of weeks or a month, as opposed to potentially a million years using traditional methods. Significant time efficiency in data processing', 'The adoption of Hadoop allowed Zions Bank to detect fraudulent activities like malware, spear phishing, and account takeovers, leading to cost savings and improved business performance. Cost savings and improved business performance']}], 'duration': 641.766, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iANBytZ26MI/pics/iANBytZ26MI1160498.jpg', 'highlights': ['The process of MapReduce in Hadoop, detailing the splitting, mapping, shuffling, sorting, and reducing phases.', 'The Hadoop YARN resource management unit, acting like an operating system, responsible for managing cluster resources and job scheduling.', 'Use case of combating fraudulent activities in the banking sector, focusing on detecting fraudulent transactions and the challenges faced by Zions Bank Corporation.', 'Hadoop enabled Zions Bank to store and process large volumes of unstructured data like server logs, customer data, emails, and transactions, leading to improved data analysis and security detection.', 'The adoption of Hadoop allowed Zions Bank to detect fraudulent activities like malware, spear phishing, and account takeovers, leading to cost savings and improved business performance.', 'The use of Hadoop resulted in significant time efficiency, enabling Zions Bank to process data in a matter of weeks or a month, as opposed to potentially a million years using traditional methods.', 'Zions Bank faced challenges in storing and analyzing massive amounts of unstructured and structured data, which led to the adoption of Hadoop for data storage and processing.']}], 'highlights': ['The use of Hadoop resulted in significant time efficiency, enabling Zions Bank to process data in a matter of weeks or a month, as opposed to potentially a million years using traditional methods.', 'The adoption of Hadoop allowed Zions Bank to detect fraudulent activities like malware, spear phishing, and account takeovers, leading to cost savings and improved business performance.', 'Hadoop enabled Zions Bank to store and process large volumes of unstructured data like server logs, customer data, emails, and transactions, leading to improved data analysis and security detection.', 'The process of MapReduce in Hadoop, detailing the splitting, mapping, shuffling, sorting, and reducing phases.', 'The Hadoop YARN resource management unit, acting like an operating system, responsible for managing cluster resources and job scheduling.', 'The scenario showcases the scalability of the solution by continuously hiring new people to handle increasing demand.', 'The importance of distributed storage in big data is emphasized, enabling scalability and parallel processing.', 'The transition from serial processing to parallel processing is explained, highlighting the challenges of serial processing and the benefits of parallel processing for efficient data processing.', 'The significance of Hadoop in handling unstructured data is emphasized, providing a way to store and process unstructured, structured, and semi-structured data effectively.', 'Volume alone does not define big data, as velocity and variety also significantly impact data processing and resource utilization, requiring processing on multiple computers and different storage spaces.']}