title
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure Training | Edureka
description
** Microsoft Azure Certification Training : https://www.edureka.co/microsoft-certified-azure-solution-architect-certification-training **
This Edureka "Azure Data Factory” video will give you a thorough and insightful overview of Microsoft Azure Data Factory and help you understand other related terms like Data Lakes and Data Warehousing.
Following are the offering of this video:
1. Why Azure Data Factory?
2. What Is Azure Data Factory?
3. Data Factory Concepts
4. What is Azure Data Lake?
5. Data Lake Concepts
6. Data Lake Vs Data Warehouse
7. Demo- Moving On-Premise Data To Cloud
Check out our Playlists: https://goo.gl/A1CJjM
Subscribe to our channel to get video updates. Hit the subscribe button above.
#MicrosoftAzureCertification #CloudComputing #AzureDataFactory #AzurePlatform #AzureDataLake #AzureTraining #AzureTutorial #Azure #Edureka #AzureBlobSTorage
-----------------------------------------------------------------------------------------------
How it Works?
There will be 30 hours of instructor-led interactive online classes, 40 hours of assignments and 20 hours of project. At the end of the training you will be working on a real time project for which we will provide you a Grade and a Verifiable Certificate!
--------------------------------------------------------------------
About The Course
Microsoft Azure Certification training from edureka will introduce you to the fundamental concepts of Microsoft Azure platform and services offered. This course offers you the opportunity to take an existing ASP.NET MVC application and expand its functionality by moving it to Azure Cloud and also focuses on the considerations necessary when building a highly available solution in the cloud. This Microsoft Cloud training will prepare you for the 70-532: Developing Microsoft Azure Solutions certification exam.
----------------------------------------------------------------------
Why learn Azure?
Edureka's Azure Certification training helps you conceptualize the Microsoft Azure fundamentals and services provided on its platform. You will learn to configure and deploy Web applications and also master the creation and deployment of Azure Web apps. You will learn to create and configure Azure Virtual machines and will able to create and manage storage account, while handling blobs and containers present in it. This Azure Cloud training helps you master the SQL Database instance and the implications of importing a SQL standalone database. You will learn about the integrals of Azure Ad (Active Directory) instance and also learn to create a virtual network and implement a point-to-site network.
What are the objectives of this Azure certification
· Compare the services available in the Azure platform
· Configure and deploy Web Applications
· Design and implement Azure PaaS compute and Web and Mobile Services
· Create and Configure Azure Virtual Machines
· Create and Manage a Storage account
· Manage blobs and containers in a Storage account
· Create, Configure and Connect to a SQL Database instance
· Identify the implications of importing a SQL standalone database
· Manage users, groups and subscriptions in an Azure Active Directory instance
· Create a Virtual Network
· Implement a point-to-site network
---------------------------------------------------------------------------------------------------------------------------------------
Who should go for this Azure Certification?
This Microsoft Azure training is designed for the IT professionals who want to pursue a career in Cloud Computing and become Microsoft Azure Developer Specialist. This Azure course is a best fit for:
· IT Professionals
· Application Developers
· .Net Developers
· Solutions Architects
· DevOps Engineers
· Professionals who want Project Experience in migrating and deploying Azure solutions
· Professionals who wants to learn designing, programming, implementing, automating and monitoring Microsoft Azure solutions
· Professionals who wants to become proficient with development tools, techniques and approaches used to build scalable and resilient Azure solutions.
---------------------------------------------------------------------------------------------------------
Cognizant, Dell, KPMG, Hitachi, Wipro, Avanade, Annik Inc., Brillio and over 45,000 MNCs across 185 countries use Power BI
For more information, Please write back to us at sales@edureka.in or call us at IND: 9606058406 / US: 18338555775 (toll free).
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
detail
{'title': 'Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure Training | Edureka', 'heatmap': [{'end': 1932.91, 'start': 1905.159, 'weight': 1}], 'summary': 'Provides an overview of azure data factory, discussing its significance in data integration, management, and movement to the cloud, including concepts, differences between data lake and data warehouse, and a demo on creating and managing azure databases.', 'chapters': [{'end': 57.002, 'segs': [{'end': 57.002, 'src': 'embed', 'start': 7.332, 'weight': 0, 'content': [{'end': 11.774, 'text': 'Hey guys, I am Vishal and I welcome you all to this session by Dureka today.', 'start': 7.332, 'duration': 4.442}, {'end': 18.298, 'text': "I'm going to give you an overview of yet another Microsoft Azure concept, and today's topic of discussion is Azure Data Factory,", 'start': 11.794, 'duration': 6.504}, {'end': 22.02, 'text': 'which is very important from data integration and data management perspective.', 'start': 18.298, 'duration': 3.722}, {'end': 28.924, 'text': "But before we do go ahead and dive into the concepts of this particular term, let's quickly start by taking a look at today's agenda first.", 'start': 22.52, 'duration': 6.404}, {'end': 34.587, 'text': 'I would start things off by talking about why do we need Azure Data Factory and what it is exactly.', 'start': 29.564, 'duration': 5.023}, {'end': 38.993, 'text': 'Then I would talk about various concepts that surround this particular term,', 'start': 35.051, 'duration': 3.942}, {'end': 47.357, 'text': 'would also understand something called as data Lake not into depth but to some extent and would also understand various concepts that surround this term as well.', 'start': 38.993, 'duration': 8.364}, {'end': 53.34, 'text': 'I would discuss two other terms, that is, data Lake and data warehouse, and what is the difference between these two terms.', 'start': 47.797, 'duration': 5.543}, {'end': 55.881, 'text': 'and finally, I would finish things off with the demo part.', 'start': 53.34, 'duration': 2.541}, {'end': 57.002, 'text': 'as far as this demo goes,', 'start': 55.881, 'duration': 1.121}], 'summary': 'Vishal introduces azure data factory, covering its importance and agenda, including discussing data lake and data warehouse concepts, and concluding with a demo.', 'duration': 49.67, 'max_score': 7.332, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA7332.jpg'}], 'start': 7.332, 'title': 'Azure data factory overview', 'summary': 'Provides an overview of azure data factory, highlighting its significance in data integration and management, discussing concepts, differences between data lake and data warehouse, and concluding with a demo.', 'chapters': [{'end': 57.002, 'start': 7.332, 'title': 'Azure data factory overview', 'summary': 'Provides an overview of azure data factory, emphasizing its importance in data integration and management, with a discussion on its concepts, the difference between data lake and data warehouse, and concluding with a demo.', 'duration': 49.67, 'highlights': ['The chapter emphasizes the importance of Azure Data Factory in data integration and data management.', 'The speaker outlines the agenda, which includes discussing the concepts of Azure Data Factory, data Lake, data warehouse, and concluding with a demo.', 'The speaker plans to discuss the difference between data Lake and data warehouse in the session.', 'The session will include a demo of Azure Data Factory.']}], 'duration': 49.67, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA7332.jpg', 'highlights': ['The chapter emphasizes the importance of Azure Data Factory in data integration and data management.', 'The speaker outlines the agenda, which includes discussing the concepts of Azure Data Factory, data Lake, data warehouse, and concluding with a demo.', 'The speaker plans to discuss the difference between data Lake and data warehouse in the session.', 'The session will include a demo of Azure Data Factory.']}, {'end': 327.915, 'segs': [{'end': 117.429, 'src': 'embed', 'start': 57.162, 'weight': 0, 'content': [{'end': 63.165, 'text': 'I would be talking about creating a pipeline and moving data from one source to the other but that is for the later part.', 'start': 57.162, 'duration': 6.003}, {'end': 70.96, 'text': "Let's start with the first topic of discussion that is Azure Data Factory.", 'start': 67.697, 'duration': 3.263}, {'end': 73.262, 'text': 'So why do we need Azure Data Factory??', 'start': 71.32, 'duration': 1.942}, {'end': 79.847, 'text': 'Well, first and foremost, we need to understand the fact that the amount of data that is being generated these days is huge,', 'start': 73.582, 'duration': 6.265}, {'end': 81.909, 'text': 'and this data comes from different sources.', 'start': 79.847, 'duration': 2.062}, {'end': 88.594, 'text': 'Now when we move this particular data to cloud, there are quite a few things that needs to be taken care of now this data.', 'start': 82.509, 'duration': 6.085}, {'end': 92.598, 'text': 'It can come in any form, because we are talking about different sources,', 'start': 88.855, 'duration': 3.743}, {'end': 96.601, 'text': 'and these different sources would transfer or channelize this data in different ways.', 'start': 92.598, 'duration': 4.003}, {'end': 98.318, 'text': 'and it can be in different formats.', 'start': 96.997, 'duration': 1.321}, {'end': 106.122, 'text': 'So when you do decide to bring this data on cloud or at a particular storage place, you need to make sure that this data is well and truly managed.', 'start': 98.798, 'duration': 7.324}, {'end': 110.185, 'text': 'Now, what do I mean by this? Well, you need to transform this data.', 'start': 106.402, 'duration': 3.783}, {'end': 114.607, 'text': 'You need to delete unnecessary part or get rid of all the things that is not needed.', 'start': 110.225, 'duration': 4.382}, {'end': 117.429, 'text': 'Now that is an after part or pre-processing part.', 'start': 115.027, 'duration': 2.402}], 'summary': 'Azure data factory helps manage and transform diverse, large-scale data from various sources for cloud storage and processing.', 'duration': 60.267, 'max_score': 57.162, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA57162.jpg'}, {'end': 190.749, 'src': 'embed', 'start': 158.153, 'weight': 1, 'content': [{'end': 160.854, 'text': 'This is where data Factory steps in what it does.', 'start': 158.153, 'duration': 2.701}, {'end': 163.894, 'text': 'is it kind of helps you automate this complete process?', 'start': 160.854, 'duration': 3.04}, {'end': 170.476, 'text': 'instead of saying automate, I would say it helps you orchestrate this process into more manageable or Organizable manner.', 'start': 163.894, 'duration': 6.582}, {'end': 177.177, 'text': 'Now that is something that is needed and that is why we need something called as data Factory which lets you automate all these processes.', 'start': 170.876, 'duration': 6.301}, {'end': 184.738, 'text': 'Let us move further and try to understand exactly what data Factory is.', 'start': 181.577, 'duration': 3.161}, {'end': 190.749, 'text': 'So if I just go ahead and talk about its definition, I would say it is nothing but a cloud-based integration service,', 'start': 185.305, 'duration': 5.444}], 'summary': 'Data factory orchestrates and automates cloud-based integration processes.', 'duration': 32.596, 'max_score': 158.153, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA158153.jpg'}, {'end': 244.178, 'src': 'embed', 'start': 214.971, 'weight': 5, 'content': [{'end': 218.532, 'text': 'It also helps you go ahead and do analytics, which is very important these days,', 'start': 214.971, 'duration': 3.561}, {'end': 225.674, 'text': 'because we know that the amount of data that is being generated it can be helpful and it can help you take various good business decisions.', 'start': 218.532, 'duration': 7.142}, {'end': 228.314, 'text': 'And this is where analytics comes into picture.', 'start': 226.054, 'duration': 2.26}, {'end': 229.734, 'text': 'Now what data factory does?', 'start': 228.634, 'duration': 1.1}, {'end': 234.596, 'text': 'is it lets you transform this data and make it ready for something like data Lake to use it.', 'start': 229.734, 'duration': 4.862}, {'end': 240.957, 'text': 'now, data Lake is something that lets you use various analytical tools or methods, like you have something called as Azure HD insight.', 'start': 234.596, 'duration': 6.361}, {'end': 244.178, 'text': 'Maybe a Hadoop, spa, Azure data like analytics and all these things.', 'start': 241.097, 'duration': 3.081}], 'summary': 'Data factory prepares data for analytics, enabling informed business decisions with tools like azure hd insight and azure data lake analytics.', 'duration': 29.207, 'max_score': 214.971, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA214971.jpg'}, {'end': 274.984, 'src': 'embed', 'start': 249.581, 'weight': 4, 'content': [{'end': 256.384, 'text': 'just understand one thing that these are nothing but platforms or tools which basically let you go ahead and do various analytical operations.', 'start': 249.581, 'duration': 6.803}, {'end': 258.555, 'text': 'So that is what data Factory lets you do.', 'start': 256.974, 'duration': 1.581}, {'end': 260.558, 'text': 'It lets you get in all the data,', 'start': 258.935, 'duration': 1.623}, {'end': 267.827, 'text': 'arrange it in a particular manner or order and then supplied further for various processing or various other things that can be done with that data.', 'start': 260.558, 'duration': 7.269}, {'end': 274.984, 'text': 'So if you talk about particular steps, what does it do exactly? Well first and foremost what it does is it helps you collect and connect data.', 'start': 268.167, 'duration': 6.817}], 'summary': 'Data factory is a platform for analytical operations, helping collect and connect data.', 'duration': 25.403, 'max_score': 249.581, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA249581.jpg'}, {'end': 312.71, 'src': 'embed', 'start': 282.69, 'weight': 2, 'content': [{'end': 286.914, 'text': 'you can connect to all these resources by using your data factory and then, once you have this data,', 'start': 282.69, 'duration': 4.224}, {'end': 291.478, 'text': 'you can collect it and store it at a central place, say, for example, your data warehouse.', 'start': 286.914, 'duration': 4.564}, {'end': 293.9, 'text': 'then comes the process of transforming and enriching it.', 'start': 291.478, 'duration': 2.422}, {'end': 298.944, 'text': 'Now when I say transforming it, I mean running various patterns on it creating schemas and all those things.', 'start': 294.04, 'duration': 4.904}, {'end': 301.403, 'text': 'You can then actually go ahead and publish this data.', 'start': 299.522, 'duration': 1.881}, {'end': 305.165, 'text': 'Now We all know that Microsoft Azure supports various other tools as well.', 'start': 301.443, 'duration': 3.722}, {'end': 308.608, 'text': 'We have one more popular data visualization tool called as power bi,', 'start': 305.466, 'duration': 3.142}, {'end': 312.71, 'text': 'which is very good when you talk about data integration and various data visualization capabilities.', 'start': 308.608, 'duration': 4.102}], 'summary': 'Use data factory to connect, collect, transform, and publish data to azure resources, including power bi for visualization.', 'duration': 30.02, 'max_score': 282.69, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA282690.jpg'}], 'start': 57.162, 'title': 'Azure data factory and its introduction', 'summary': 'Discusses the importance of azure data factory in managing and transforming large volumes of data to the cloud from different sources, and introduces it as a cloud-based integration service on microsoft azure that automates data-driven workflows, orchestrates data sources, and enables data analytics and visualization.', 'chapters': [{'end': 137.263, 'start': 57.162, 'title': 'Azure data factory', 'summary': 'Discusses the need for azure data factory in managing and transforming large volumes of data from different sources to the cloud, emphasizing the importance of data management and the limitations of traditional data warehouse.', 'duration': 80.101, 'highlights': ['The need to manage and transform large volumes of data from different sources to the cloud is emphasized, highlighting the challenges posed by the diverse formats and structures of the data.', 'The importance of data management and transformation is stressed, indicating the necessity to eliminate unnecessary components and ensure efficient storage and transformation of data.', 'The limitations of traditional data warehouse in effectively managing and transforming diverse data sources are highlighted, suggesting the need for a more efficient solution like Azure Data Factory.']}, {'end': 327.915, 'start': 137.263, 'title': 'Introduction to data factory', 'summary': 'Introduces data factory as a cloud-based integration service that automates the process of creating data-driven workflows, orchestrating data sources, and enabling data analytics and visualization on microsoft azure.', 'duration': 190.652, 'highlights': ['Data Factory automates the process of creating data-driven workflows and orchestrating data sources. Data Factory helps automate the process of creating data-driven workflows, reducing time and effort in managing individual processes.', 'It enables data analytics and visualization, allowing for better business decisions based on the generated data. Data Factory supports data analytics, enabling better business decisions through insights derived from processed data.', 'Data Factory integrates with Microsoft Azure tools like Power BI for data visualization capabilities. Data Factory integrates with Microsoft Azure tools like Power BI, enhancing data visualization and business intelligence capabilities.', 'It facilitates the collection, connection, transformation, and enrichment of data from various sources. Data Factory enables the collection, connection, transformation, and enrichment of data from diverse sources, streamlining the data management process.', 'Data Factory supports the creation of logical pipelines to support various data processes. Data Factory supports the creation of logical pipelines to streamline and manage various data processes efficiently.']}], 'duration': 270.753, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA57162.jpg', 'highlights': ['The need to manage and transform large volumes of data from different sources to the cloud is emphasized, highlighting the challenges posed by the diverse formats and structures of the data.', 'Data Factory automates the process of creating data-driven workflows and orchestrating data sources, reducing time and effort in managing individual processes.', 'It facilitates the collection, connection, transformation, and enrichment of data from various sources, streamlining the data management process.', 'Data Factory integrates with Microsoft Azure tools like Power BI for data visualization capabilities.', 'Data Factory supports the creation of logical pipelines to support various data processes.', 'It enables data analytics and visualization, allowing for better business decisions based on the generated data.', 'The importance of data management and transformation is stressed, indicating the necessity to eliminate unnecessary components and ensure efficient storage and transformation of data.', 'Data Factory supports data analytics, enabling better business decisions through insights derived from processed data.', 'The limitations of traditional data warehouse in effectively managing and transforming diverse data sources are highlighted, suggesting the need for a more efficient solution like Azure Data Factory.']}, {'end': 801.648, 'segs': [{'end': 358.464, 'src': 'embed', 'start': 327.935, 'weight': 0, 'content': [{'end': 334.081, 'text': 'That means you can take a look at all the data that you have and you can actually go ahead and analyze it in real time as well.', 'start': 327.935, 'duration': 6.146}, {'end': 341.068, 'text': 'So these are the processes which we are talking about that is connect and collect a data transform and enrich it publish it and monitor it.', 'start': 334.441, 'duration': 6.627}, {'end': 343.05, 'text': 'Now, that is what a data Factory lets you do.', 'start': 341.168, 'duration': 1.882}, {'end': 349.717, 'text': 'So what are the concepts that surround this particular term?', 'start': 347.554, 'duration': 2.163}, {'end': 358.464, 'text': 'So when we talk about data factory concepts, we have quite a few terms to discuss, like we have pipelines, data sets, activities and linked services.', 'start': 350.538, 'duration': 7.926}], 'summary': 'Azure data factory allows real-time analysis of collected data with concepts like pipelines, data sets, activities, and linked services.', 'duration': 30.529, 'max_score': 327.935, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA327935.jpg'}, {'end': 404.471, 'src': 'embed', 'start': 371.895, 'weight': 3, 'content': [{'end': 374.557, 'text': 'Now this individual process is nothing but an activity.', 'start': 371.895, 'duration': 2.662}, {'end': 379.101, 'text': 'If you take a look at this side, you would understand that activities represent processing step in a pipeline.', 'start': 374.998, 'duration': 4.103}, {'end': 383.354, 'text': 'That means your pipeline can have one or multiple activities.', 'start': 379.551, 'duration': 3.803}, {'end': 386.477, 'text': 'Now, when you talk about this activity, it can be anything.', 'start': 383.814, 'duration': 2.663}, {'end': 393.402, 'text': 'it can be a single process like querying a particular data set, or it can be something like moving data from one source to the other.', 'start': 386.477, 'duration': 6.925}, {'end': 395.464, 'text': 'then you have something called as data sets.', 'start': 393.402, 'duration': 2.062}, {'end': 398.086, 'text': 'now data sets are nothing but sources of data.', 'start': 395.464, 'duration': 2.622}, {'end': 404.471, 'text': 'say, for example, my data is stored at an end location and that end location is nothing but my data set.', 'start': 398.086, 'duration': 6.385}], 'summary': 'Individual process is an activity representing a step in a pipeline. activities can include querying data or moving data between sources. data sets are sources of data.', 'duration': 32.576, 'max_score': 371.895, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA371895.jpg'}, {'end': 533.387, 'src': 'embed', 'start': 504.392, 'weight': 1, 'content': [{'end': 506.394, 'text': 'We have our Azure data Lake analytics.', 'start': 504.392, 'duration': 2.002}, {'end': 509.236, 'text': 'We have HD insights are spark and machine learning.', 'start': 506.434, 'duration': 2.802}, {'end': 513.458, 'text': 'Now these are some of the commonly used and very popular tools which are there.', 'start': 509.556, 'duration': 3.902}, {'end': 516.582, 'text': 'what Azure data Lake does is it supports all these tools.', 'start': 513.458, 'duration': 3.124}, {'end': 521.772, 'text': 'that means You can store data from any source and in any form at your Azure data lake,', 'start': 516.582, 'duration': 5.19}, {'end': 525.957, 'text': 'and then you can use these tools and terms to actually analyze this data.', 'start': 521.772, 'duration': 4.185}, {'end': 528.06, 'text': 'So that is two in one kind of an operation.', 'start': 526.318, 'duration': 1.742}, {'end': 533.387, 'text': 'In fact, it is multiple in one kind of an option basically because it lets you do so many other things.', 'start': 528.34, 'duration': 5.047}], 'summary': 'Azure data lake supports popular tools like hdinsight, spark, and machine learning, allowing storage and analysis of diverse data sources, offering a versatile, multipurpose solution.', 'duration': 28.995, 'max_score': 504.392, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA504392.jpg'}, {'end': 699.547, 'src': 'embed', 'start': 673.115, 'weight': 2, 'content': [{'end': 678.579, 'text': 'first and foremost, what it does is it stores structured data, semi-structured data and unstructured data.', 'start': 673.115, 'duration': 5.464}, {'end': 683.343, 'text': 'That means all the types of data that can possibly be there as far as storing is concerned.', 'start': 678.64, 'duration': 4.703}, {'end': 684.844, 'text': 'Everything is taken care of.', 'start': 683.723, 'duration': 1.121}, {'end': 688.747, 'text': "now, when I talk about structured data, I'm talking about data that has a particular schema.", 'start': 684.844, 'duration': 3.903}, {'end': 691.504, 'text': 'then you have something called as your semi-structured data.', 'start': 688.747, 'duration': 2.757}, {'end': 695.366, 'text': 'Now, this is nothing but your JSON files or maybe your XML data.', 'start': 691.584, 'duration': 3.782}, {'end': 699.547, 'text': "even that kind of data can be handled, and when I'm talking about particular unstructured data,", 'start': 695.366, 'duration': 4.181}], 'summary': 'The system can store structured, semi-structured, and unstructured data, including json and xml files.', 'duration': 26.432, 'max_score': 673.115, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA673115.jpg'}, {'end': 814.412, 'src': 'embed', 'start': 785.75, 'weight': 4, 'content': [{'end': 792.772, 'text': 'So yeah, these are the terms that is extent and vertices now what happens here is since you are replicating this data and you adding it into chunks.', 'start': 785.75, 'duration': 7.022}, {'end': 798.027, 'text': 'You can actually go ahead and process this data parallelly saving a lot of time and giving you high computation.', 'start': 793.206, 'duration': 4.821}, {'end': 800.188, 'text': 'It also supports replication of data.', 'start': 798.568, 'duration': 1.62}, {'end': 801.648, 'text': "This is something that I've already discussed.", 'start': 800.208, 'duration': 1.44}, {'end': 804.769, 'text': 'So I would just move further and try to talk about other concepts as well.', 'start': 801.668, 'duration': 3.101}, {'end': 814.412, 'text': 'Next we are going to understand what a data Lake is and what a data warehouse is now.', 'start': 809.551, 'duration': 4.861}], 'summary': 'Data replication allows parallel processing, saving time and enabling high computation.', 'duration': 28.662, 'max_score': 785.75, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA785750.jpg'}], 'start': 327.935, 'title': 'Data factory and data lake', 'summary': 'Delves into azure data factory concepts such as connect, collect, transform, enrich, publish, and monitor, and key components like pipelines, data sets, activities, and linked services. it also covers the azure data lake, a hyperscale repository supporting various analytical tools, enabling storage of data from any source and form, parallel processing, and replication, saving time and providing high computation.', 'chapters': [{'end': 447.05, 'start': 327.935, 'title': 'Understanding data factory concepts', 'summary': 'Discusses the concepts of azure data factory, including the processes of connect, collect, transform, enrich, publish, and monitor, as well as the key components - pipelines, data sets, activities, and linked services.', 'duration': 119.115, 'highlights': ['Azure Data Factory enables real-time data analysis and processes like connect, collect, transform, enrich, publish, and monitor. Azure Data Factory allows real-time data analysis and facilitates processes such as connect, collect, transform, enrich, publish, and monitor.', 'Key concepts of Azure Data Factory include pipelines, data sets, activities, and linked services. The key concepts of Azure Data Factory include pipelines, data sets, activities, and linked services.', 'A pipeline acts as a carrier for various processes, while activities represent processing steps within a pipeline. A pipeline serves as a carrier for processes, and activities represent individual processing steps within a pipeline.', 'Data sets are sources of data and can be considered as data structures that hold the data. Data sets are sources of data and can be viewed as data structures that store the data.', 'Linked Services connect different data sources and provide the necessary information for data movement. Linked Services connect different data sources and furnish the required information for data movement.']}, {'end': 801.648, 'start': 447.19, 'title': 'Azure data lake overview', 'summary': 'Provides an overview of azure data lake, a hyperscale repository supporting various analytical tools, allowing storage of data from any source and in any form, and enabling parallel processing and replication of data, thus saving time and providing high computation.', 'duration': 354.458, 'highlights': ['Azure Data Lake is a hyperscale repository supporting various analytical tools Azure Data Lake supports tools like Azure Data Lake analytics, HDInsights, Apache Spark, and machine learning, enabling analytics on huge amounts of data.', 'Allows storage of data from any source and in any form Azure Data Lake allows storage of structured, semi-structured, and unstructured data, thus removing barriers for data processing and enabling users to store and process data of any kind and size.', 'Enables parallel processing and replication of data Azure Data Lake breaks data into chunks, replicates it at different locations, and supports parallel processing using a language called U-SQL, resulting in high computation and time-saving.']}], 'duration': 473.713, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA327935.jpg', 'highlights': ['Azure Data Factory enables real-time data analysis and facilitates processes such as connect, collect, transform, enrich, publish, and monitor.', 'Azure Data Lake supports tools like Azure Data Lake analytics, HDInsights, Apache Spark, and machine learning, enabling analytics on huge amounts of data.', 'Allows storage of structured, semi-structured, and unstructured data, thus removing barriers for data processing and enabling users to store and process data of any kind and size.', 'A pipeline serves as a carrier for processes, and activities represent individual processing steps within a pipeline.', 'Enables parallel processing and replication of data, resulting in high computation and time-saving.']}, {'end': 1278.184, 'segs': [{'end': 830.638, 'src': 'embed', 'start': 801.668, 'weight': 5, 'content': [{'end': 804.769, 'text': 'So I would just move further and try to talk about other concepts as well.', 'start': 801.668, 'duration': 3.101}, {'end': 814.412, 'text': 'Next we are going to understand what a data Lake is and what a data warehouse is now.', 'start': 809.551, 'duration': 4.861}, {'end': 821.834, 'text': 'Why did I bring this topic up? Well data warehouse is something that is a traditional approach towards storing data and it is still used widely.', 'start': 814.752, 'duration': 7.082}, {'end': 824.193, 'text': 'But then why do we need data Lake?', 'start': 822.352, 'duration': 1.841}, {'end': 826.515, 'text': 'and why am I comparing these two terms?', 'start': 824.193, 'duration': 2.322}, {'end': 830.638, 'text': 'will quite a few people confuse these two terms, as in what is the difference between them exactly??', 'start': 826.515, 'duration': 4.123}], 'summary': 'Exploring the differences between data lake and data warehouse.', 'duration': 28.97, 'max_score': 801.668, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA801668.jpg'}, {'end': 879.482, 'src': 'embed', 'start': 841.086, 'weight': 0, 'content': [{'end': 844.808, 'text': 'Well, this is a clear differentiation between these two things, not as first of all,', 'start': 841.086, 'duration': 3.722}, {'end': 850.272, 'text': 'you need to understand that your data Lake is nothing but something that is complementary to data warehouse.', 'start': 844.808, 'duration': 5.464}, {'end': 859.369, 'text': 'That means If you have your data at a data Lake that can be stored at data warehouse as well, but there are certain rules that need to be followed.', 'start': 850.792, 'duration': 8.577}, {'end': 866.214, 'text': 'Now when you talk about your data warehouse, what happens is you can use again your data Lake to bring in data.', 'start': 859.809, 'duration': 6.405}, {'end': 872.218, 'text': 'But the main difference is when you talk about a data Lake the data is detailed data or raw data.', 'start': 866.514, 'duration': 5.704}, {'end': 873.999, 'text': 'That means it can be in any particular form.', 'start': 872.258, 'duration': 1.741}, {'end': 876.841, 'text': 'You just need to take the data and dump it into your data Lake.', 'start': 874.439, 'duration': 2.402}, {'end': 879.482, 'text': 'That is not the case with your data warehouse here.', 'start': 877.141, 'duration': 2.341}], 'summary': 'Data lake complements data warehouse, handles detailed/raw data, while data warehouse stores structured data.', 'duration': 38.396, 'max_score': 841.086, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA841086.jpg'}, {'end': 926.919, 'src': 'embed', 'start': 898.039, 'weight': 3, 'content': [{'end': 900.359, 'text': 'This might not be the best of options to deal with.', 'start': 898.039, 'duration': 2.32}, {'end': 907.241, 'text': 'why? well, your data Lake has something called a schema on read and your data warehouse has a schema on right kind of an approach.', 'start': 900.359, 'duration': 6.882}, {'end': 913.576, 'text': 'What this does is, if I talk about it from a data warehouse perspective, where you have schema on right.', 'start': 907.694, 'duration': 5.882}, {'end': 919.977, 'text': "that means when you're writing your data to your data warehouse, it is written in a structured form or in a particular schema.", 'start': 913.576, 'duration': 6.401}, {'end': 926.919, 'text': "So when I'm reading this data, I have to read it in that schema only but when you talk about a data Lake here, you just dump your data.", 'start': 920.377, 'duration': 6.542}], 'summary': 'Data warehouse has schema on write, while data lake allows dumping data.', 'duration': 28.88, 'max_score': 898.039, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA898039.jpg'}, {'end': 974.395, 'src': 'embed', 'start': 944.488, 'weight': 4, 'content': [{'end': 949.81, 'text': 'Basically it works in SQL that is it uses SQL to choir or question your data.', 'start': 944.488, 'duration': 5.322}, {'end': 954.032, 'text': 'But when you talk about your data Lake no matter the data comes from different sources.', 'start': 950.19, 'duration': 3.842}, {'end': 956.964, 'text': 'It can be quiet by using one single language.', 'start': 954.542, 'duration': 2.422}, {'end': 958.665, 'text': 'that is your usql.', 'start': 956.964, 'duration': 1.701}, {'end': 963.288, 'text': "and again, that actually helps you reduce various barriers, because you're talking about different data sources,", 'start': 958.665, 'duration': 4.623}, {'end': 967.05, 'text': 'but still something that lets you access the data is one single programming language.', 'start': 963.288, 'duration': 3.762}, {'end': 968.911, 'text': 'So these are some of the concepts.', 'start': 967.55, 'duration': 1.361}, {'end': 974.395, 'text': 'I felt that you all should know that as what a data Factory is and what a data Lake is again.', 'start': 968.951, 'duration': 5.444}], 'summary': 'Data factory uses usql to query data from different sources, reducing barriers to access data.', 'duration': 29.907, 'max_score': 944.488, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA944488.jpg'}], 'start': 801.668, 'title': 'Data lake vs data warehouse', 'summary': 'Discusses and explains the differences between data lake and data warehouse, emphasizing the need for data lake in handling large, unstructured data and highlighting the confusion around the two terms. it emphasizes differences in data storage, schema, and querying language, and the importance of data lake in contrast to the structured data approach of data warehouse.', 'chapters': [{'end': 840.585, 'start': 801.668, 'title': 'Data lake vs data warehouse', 'summary': 'Discusses the differences between data lake and data warehouse, highlighting the need for data lake and the confusion around the two terms.', 'duration': 38.917, 'highlights': ['Data warehouse is a traditional approach towards storing data and is still widely used.', 'The need for a data lake arises due to the differences and slight variations from a data warehouse.', 'The confusion between data lake and data warehouse is a common concern among many people.']}, {'end': 1278.184, 'start': 841.086, 'title': 'Data lake vs data warehouse', 'summary': 'Explains the distinction between data lake and data warehouse, emphasizing their differences in data storage, schema, and querying language, and highlights the importance of data lake in handling large, unstructured data from diverse sources, in contrast to the structured data approach of data warehouse.', 'duration': 437.098, 'highlights': ['Data Lake stores raw, unstructured data, whereas Data Warehouse stores filtered, summarized, and refined data. The data stored in a data lake is detailed and raw, in contrast to the filtered, summarized, and refined data stored in a data warehouse.', 'Data Lake allows flexible schema on read, while Data Warehouse follows a structured schema on write approach. Data lake offers the benefit of schema flexibility, allowing users to define their schema as needed, as opposed to the structured schema approach of data warehouse.', 'Data Lake uses a single querying language USQL, while Data Warehouse uses SQL for querying data. Data lake uses a single querying language, USQL, to query data from different sources, reducing barriers, whereas data warehouse uses SQL for querying data.', 'Data Lake is crucial for handling large, unstructured data from diverse sources, which is not efficiently managed by Data Warehouse. The chapter emphasizes the importance of data lake in handling large and unstructured data from diverse sources, which might not be efficiently managed by a data warehouse.']}], 'duration': 476.516, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA801668.jpg', 'highlights': ['The confusion between data lake and data warehouse is a common concern among many people.', 'Data Lake is crucial for handling large, unstructured data from diverse sources, which is not efficiently managed by Data Warehouse.', 'Data Lake stores raw, unstructured data, whereas Data Warehouse stores filtered, summarized, and refined data.', 'Data Lake allows flexible schema on read, while Data Warehouse follows a structured schema on write approach.', 'Data Lake uses a single querying language USQL, while Data Warehouse uses SQL for querying data.', 'Data warehouse is a traditional approach towards storing data and is still widely used.', 'The need for a data lake arises due to the differences and slight variations from a data warehouse.']}, {'end': 1587.351, 'segs': [{'end': 1339.743, 'src': 'embed', 'start': 1314.966, 'weight': 2, 'content': [{'end': 1322.088, 'text': 'And what password should I set let it be strong and let it have a number and symbol basically.', 'start': 1314.966, 'duration': 7.122}, {'end': 1326.576, 'text': 'Location South India that does not matter a lot.', 'start': 1324.335, 'duration': 2.241}, {'end': 1328.577, 'text': 'You can choose the location that is there in here.', 'start': 1326.596, 'duration': 1.981}, {'end': 1329.878, 'text': 'You have quite a few options.', 'start': 1328.817, 'duration': 1.061}, {'end': 1333.78, 'text': 'They go out stick to what it is there and I would say select.', 'start': 1330.778, 'duration': 3.002}, {'end': 1337.702, 'text': 'Yes, so I have a demo server as well.', 'start': 1335.001, 'duration': 2.701}, {'end': 1339.743, 'text': 'performance level now it depends.', 'start': 1337.702, 'duration': 2.041}], 'summary': 'Recommend setting a strong password with a number and symbol. various location options available for selection.', 'duration': 24.777, 'max_score': 1314.966, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1314966.jpg'}, {'end': 1399.532, 'src': 'embed', 'start': 1357.512, 'weight': 0, 'content': [{'end': 1362.676, 'text': 'So the price to me is an INR that is Indian rupees and it is 2 to 3.88 per hour.', 'start': 1357.512, 'duration': 5.164}, {'end': 1365.198, 'text': "All I would do is I'll just go ahead and apply.", 'start': 1363.236, 'duration': 1.962}, {'end': 1369.961, 'text': 'I would say pin to dashboard and I would say create.', 'start': 1365.218, 'duration': 4.743}, {'end': 1379.408, 'text': "Now the deploying might take a little longer than normal because yes, as I've already mentioned that it depends on the internet speed.", 'start': 1373.083, 'duration': 6.325}, {'end': 1384.106, 'text': 'and even if your internet is faster at times it might take a longer while for this thing to happen.', 'start': 1379.964, 'duration': 4.142}, {'end': 1389.908, 'text': "So what I'll do is I'll just pause the video for a minute and once the deployment is done, I'll get back to the demo part again.", 'start': 1384.526, 'duration': 5.382}, {'end': 1397.971, 'text': "So yes guys have gone ahead and I've done the process that is we have our so-called demo DB already.", 'start': 1393.269, 'duration': 4.702}, {'end': 1399.532, 'text': 'It took a while.', 'start': 1398.652, 'duration': 0.88}], 'summary': 'Deploying the demo took 2 to 3.88 inr per hour and a while due to internet speed.', 'duration': 42.02, 'max_score': 1357.512, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1357512.jpg'}, {'end': 1474.867, 'src': 'embed', 'start': 1415.279, 'weight': 3, 'content': [{'end': 1419.562, 'text': "This is how it would look like now since we've already created a database we can log into it.", 'start': 1415.279, 'duration': 4.283}, {'end': 1421.643, 'text': 'You can go ahead and given your credentials here.', 'start': 1420.062, 'duration': 1.581}, {'end': 1433.492, 'text': "That is I would be putting in the credentials which I've mentioned that is demo server 1 1 0 0 and then the extension that is database.", 'start': 1421.843, 'duration': 11.649}, {'end': 1437.203, 'text': 'not windows.net.', 'start': 1435.242, 'duration': 1.961}, {'end': 1441.945, 'text': 'Yes, and the name here is admin demo.', 'start': 1438.003, 'duration': 3.942}, {'end': 1453.03, 'text': 'I can go to options here connect to what database we have something called as demo DB and that is three to one TCP IP.', 'start': 1443.646, 'duration': 9.384}, {'end': 1455.791, 'text': 'I say connect.', 'start': 1455.151, 'duration': 0.64}, {'end': 1462.434, 'text': 'It wants me to sign into my so-called Azure account.', 'start': 1459.433, 'duration': 3.001}, {'end': 1474.867, 'text': "I hate this each time it happens because I've already logged in I should have stayed signed in but I did not and that is costing me now.", 'start': 1467.318, 'duration': 7.549}], 'summary': "Logging into the database 'demo db' using 'demo server 1100' credentials and facing sign-in issues with azure account.", 'duration': 59.588, 'max_score': 1415.279, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1415279.jpg'}, {'end': 1547.357, 'src': 'embed', 'start': 1523.968, 'weight': 5, 'content': [{'end': 1531.673, 'text': "So what query should I put in what table should I create I'll just go ahead and pick up some query from Microsoft Azure website.", 'start': 1523.968, 'duration': 7.705}, {'end': 1536.154, 'text': 'Now this is something that lets you create a table with this name,', 'start': 1531.933, 'duration': 4.221}, {'end': 1540.796, 'text': 'and these are the so-called variables or column names that your so-called table will have.', 'start': 1536.154, 'duration': 4.642}, {'end': 1543.436, 'text': "So I'm just going to copy this piece of code.", 'start': 1541.216, 'duration': 2.22}, {'end': 1547.357, 'text': 'If you want it, you can go to Microsoft Azure website and you can have this piece of code as well.', 'start': 1543.856, 'duration': 3.501}], 'summary': 'Instructions for creating a table in microsoft azure with specific column names.', 'duration': 23.389, 'max_score': 1523.968, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1523968.jpg'}], 'start': 1278.644, 'title': 'Creating and managing azure databases', 'summary': 'Covers creating a demo server and database, configuring the server, setting login credentials, selecting performance level, and deploying the server, with slow internet speed affecting deployment time, then logging into an azure database, connecting to a specific database, adding a table, and dealing with challenges of signing in and changing firewall rules for accessibility.', 'chapters': [{'end': 1415.119, 'start': 1278.644, 'title': 'Creating demo server and database', 'summary': 'Outlines the process of creating a demo server and database, configuring the server, setting login credentials, selecting performance level, and deploying the server, with the deployment taking longer due to slow internet speed and the subsequent installation of ssms.', 'duration': 136.475, 'highlights': ['The deployment of the demo server took longer due to slow internet speed. Slow internet speed affected the deployment process, requiring additional time.', 'The price for the demo server in South India is 2 to 3.88 INR per hour. The hourly cost of the demo server in South India is 2 to 3.88 INR.', 'The process included configuring a new demo server and blank database, setting login credentials, and selecting performance level. The process involved server and database configuration, login credential setup, and performance level selection.']}, {'end': 1587.351, 'start': 1415.279, 'title': 'Azure database management', 'summary': 'Outlines the process of logging into an azure database, connecting to a specific database, and adding a table, along with the challenges of signing in and changing firewall rules for accessibility.', 'duration': 172.072, 'highlights': ['Logging into the Azure database using credentials like demo server, extension, and account name, to access the demo DB and make the server accessible for database management.', 'Challenges faced during sign-in, emphasizing the inconvenience of repetitive sign-ins and the impact on productivity.', 'Process of adding a table to the database by running a query from the Microsoft Azure website, including making necessary changes and inserting records, with an example of the affected rows and the successful execution of the query.']}], 'duration': 308.707, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1278644.jpg', 'highlights': ['The deployment of the demo server took longer due to slow internet speed.', 'The hourly cost of the demo server in South India is 2 to 3.88 INR.', 'The process involved server and database configuration, login credential setup, and performance level selection.', 'Logging into the Azure database using credentials like demo server, extension, and account name.', 'Challenges faced during sign-in, emphasizing the inconvenience of repetitive sign-ins and the impact on productivity.', 'Process of adding a table to the database by running a query from the Microsoft Azure website.']}, {'end': 2379.288, 'segs': [{'end': 1630.793, 'src': 'embed', 'start': 1607.805, 'weight': 0, 'content': [{'end': 1615.822, 'text': 'So how do we do that? First thing you need to understand is in order to have a resource group or have all these resources there on your system.', 'start': 1607.805, 'duration': 8.017}, {'end': 1617.403, 'text': 'You need to have a storage account.', 'start': 1615.882, 'duration': 1.521}, {'end': 1621.666, 'text': 'Now this account is nothing but more or less a compliance or a law.', 'start': 1617.763, 'duration': 3.903}, {'end': 1626.63, 'text': 'Basically what that law does is it lets you have your storage entities on Microsoft Azure.', 'start': 1621.726, 'duration': 4.904}, {'end': 1628.671, 'text': 'So let us go ahead and create that account first.', 'start': 1626.91, 'duration': 1.761}, {'end': 1629.492, 'text': 'It is very easy.', 'start': 1628.771, 'duration': 0.721}, {'end': 1630.793, 'text': 'You come to this portal.', 'start': 1629.852, 'duration': 0.941}], 'summary': 'To create resources in a resource group, you need a storage account on microsoft azure, which acts as a compliance law, allowing storage entities. the process of creating the account is straightforward through the portal.', 'duration': 22.988, 'max_score': 1607.805, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1607805.jpg'}, {'end': 1764.852, 'src': 'embed', 'start': 1725.192, 'weight': 1, 'content': [{'end': 1726.393, 'text': 'since I have a storage account,', 'start': 1725.192, 'duration': 1.201}, {'end': 1734.665, 'text': 'I can go ahead and create again containers and then I can use data Factory to basically move data between various sources.', 'start': 1727, 'duration': 7.665}, {'end': 1736.626, 'text': 'as you can see, it has been deployed.', 'start': 1734.665, 'duration': 1.961}, {'end': 1738.807, 'text': 'Do I want to go there? Yes, I do.', 'start': 1736.946, 'duration': 1.861}, {'end': 1742.85, 'text': 'Now when you come here, you see this so-called app here.', 'start': 1740.348, 'duration': 2.502}, {'end': 1744.891, 'text': 'You have something called as access keys.', 'start': 1742.87, 'duration': 2.021}, {'end': 1750.054, 'text': 'What you can do is you can actually come here and copy some of the data that is here.', 'start': 1746.792, 'duration': 3.262}, {'end': 1754.567, 'text': 'This is the name of your storage account, which you would be needing,', 'start': 1750.785, 'duration': 3.782}, {'end': 1759.189, 'text': 'and the quite a few terminologies that you need to remember when you are using or going through this demo.', 'start': 1754.567, 'duration': 4.622}, {'end': 1761.53, 'text': 'Like I mentioned you have created a database.', 'start': 1759.709, 'duration': 1.821}, {'end': 1763.491, 'text': 'You need to know its name or remember its name.', 'start': 1761.55, 'duration': 1.941}, {'end': 1764.852, 'text': 'You need to know the server name.', 'start': 1763.511, 'duration': 1.341}], 'summary': 'Using a storage account to create containers and move data with data factory, deploying and accessing app with access keys, and managing database and server names.', 'duration': 39.66, 'max_score': 1725.192, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1725192.jpg'}, {'end': 1858.416, 'src': 'embed', 'start': 1829.283, 'weight': 3, 'content': [{'end': 1832.164, 'text': 'So I have a blob storage or a container with me now.', 'start': 1829.283, 'duration': 2.881}, {'end': 1834.005, 'text': 'I need to go ahead and create a data factory here.', 'start': 1832.204, 'duration': 1.801}, {'end': 1838.682, 'text': 'So how do we do that? Come here search for data factory directly.', 'start': 1834.286, 'duration': 4.396}, {'end': 1840.784, 'text': 'There you go.', 'start': 1840.344, 'duration': 0.44}, {'end': 1843.066, 'text': 'No, this is not where I want it to be.', 'start': 1841.665, 'duration': 1.401}, {'end': 1844.467, 'text': 'So I close it.', 'start': 1843.626, 'duration': 0.841}, {'end': 1846.228, 'text': 'I click on analytics.', 'start': 1845.127, 'duration': 1.101}, {'end': 1849.37, 'text': 'And here I have the option of data factory.', 'start': 1847.649, 'duration': 1.721}, {'end': 1850.13, 'text': 'I click on it.', 'start': 1849.45, 'duration': 0.68}, {'end': 1853.693, 'text': 'Yes, so I can create a new data factory.', 'start': 1851.291, 'duration': 2.402}, {'end': 1858.416, 'text': "So what should I call this? Let's say demo DF.", 'start': 1854.073, 'duration': 4.343}], 'summary': "Creating a new data factory named 'demo df' in blob storage.", 'duration': 29.133, 'max_score': 1829.283, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1829283.jpg'}, {'end': 1936.111, 'src': 'heatmap', 'start': 1905.159, 'weight': 1, 'content': [{'end': 1907, 'text': 'Okay, so I have a data factory with me now.', 'start': 1905.159, 'duration': 1.841}, {'end': 1911.262, 'text': 'I need to come here and click on this because I need to go ahead and move my data.', 'start': 1907.24, 'duration': 4.022}, {'end': 1912.863, 'text': 'So I come here.', 'start': 1911.783, 'duration': 1.08}, {'end': 1917.305, 'text': 'When I click on this thing, I would be given quite a few options to deal with.', 'start': 1914.324, 'duration': 2.981}, {'end': 1919.446, 'text': 'So I need to copy my data.', 'start': 1918.186, 'duration': 1.26}, {'end': 1920.627, 'text': 'So I click on this icon.', 'start': 1919.466, 'duration': 1.161}, {'end': 1921.547, 'text': 'There you go.', 'start': 1921.107, 'duration': 0.44}, {'end': 1924.809, 'text': 'I need to give a name to the task that is there.', 'start': 1922.388, 'duration': 2.421}, {'end': 1932.91, 'text': "Let's call it say copy from DB to block.", 'start': 1925.629, 'duration': 7.281}, {'end': 1936.111, 'text': 'And I go to next.', 'start': 1935.231, 'duration': 0.88}], 'summary': 'Using data factory to copy data from db to block', 'duration': 30.952, 'max_score': 1905.159, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1905159.jpg'}, {'end': 2281.43, 'src': 'embed', 'start': 2254.386, 'weight': 5, 'content': [{'end': 2257.388, 'text': 'And there you can see the data has been deployed, That is,', 'start': 2254.386, 'duration': 3.002}, {'end': 2266.196, 'text': "I've actually connected my skill server and I've gone ahead and I've moved the data to my so-called blob storage, which you can actually also do.", 'start': 2257.388, 'duration': 8.808}, {'end': 2270.7, 'text': 'then you can go ahead and edit the pipeline if you want, as you want, or you can just go ahead and monitor this data as well.', 'start': 2266.196, 'duration': 4.504}, {'end': 2275.385, 'text': 'It depends on your needs as in what do you want to do? You can go ahead and do quite a few things with it.', 'start': 2271.341, 'duration': 4.044}, {'end': 2281.43, 'text': 'That is you can go ahead and move this data to your so-called power bi and implement various other things on it as well.', 'start': 2275.425, 'duration': 6.005}], 'summary': 'Data deployed to blob storage, editable pipeline, and monitorable data for various uses.', 'duration': 27.044, 'max_score': 2254.386, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA2254386.jpg'}, {'end': 2352.797, 'src': 'embed', 'start': 2324.04, 'weight': 4, 'content': [{'end': 2327.741, 'text': 'This was the blob where we moved the file and then this was the file that we moved.', 'start': 2324.04, 'duration': 3.701}, {'end': 2335.062, 'text': "So yeah, our deployment has succeeded that is we've moved our data from a so-called database to our show called blob storage.", 'start': 2328.041, 'duration': 7.021}, {'end': 2338.293, 'text': 'And as far as this demo goes that is what our aim was.', 'start': 2335.772, 'duration': 2.521}, {'end': 2342.374, 'text': 'I hope that I did throw a sufficient light on following concepts.', 'start': 2338.573, 'duration': 3.801}, {'end': 2348.616, 'text': 'that is, our data Factory, and how do you go ahead and create a data Factory and use it as a pipeline?', 'start': 2342.374, 'duration': 6.242}, {'end': 2352.797, 'text': 'basically, as far as this session goes, I would be resting my session here and here.', 'start': 2348.616, 'duration': 4.181}], 'summary': 'Successful deployment of data from database to blob storage as part of data factory demo.', 'duration': 28.757, 'max_score': 2324.04, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA2324040.jpg'}], 'start': 1587.552, 'title': 'Azure data factory and data movement', 'summary': 'Covers creating azure data factory, specifying storage account details, and enabling data movement between sources. it also explores moving data from a database to blob storage, emphasizing best practices and naming conventions.', 'chapters': [{'end': 1782.453, 'start': 1587.552, 'title': 'Creating azure data factory', 'summary': 'Explains the process of creating a storage account in azure, specifying details such as location, replication, performance, and resource group, with a focus on enabling data movement between various sources using azure data factory.', 'duration': 194.901, 'highlights': ['The process of creating a storage account in Azure is explained, emphasizing the necessity of having a resource group and a storage account for storing entities on Microsoft Azure. Creation of a storage account in Azure, necessity of resource group and storage account for storing entities on Microsoft Azure.', 'Details such as location, replication, performance, and resource group are specified during the creation of the storage account, with a focus on specifying the replication, performance, and resource group details. Location, replication, performance, and resource group details specified during storage account creation.', 'The deployment process of the storage account is briefly explained, mentioning the quick creation of the storage account and the potential longer deployment time. Brief explanation of the deployment process of the storage account.', 'The possibility of creating containers and using Azure Data Factory to move data between various sources after creating the storage account is highlighted. Opportunity to create containers and use Azure Data Factory for data movement after creating the storage account.']}, {'end': 2379.288, 'start': 1785.575, 'title': 'Moving data from database to blob storage', 'summary': 'Demonstrates the process of creating a container in azure blob storage, setting up a data factory, moving data from a database to blob storage, and monitoring the data deployment, emphasizing the importance of naming conventions and best practices in the process.', 'duration': 593.713, 'highlights': ['The process of creating a container in Azure Blob Storage and setting up a data factory is demonstrated. Demonstration of creating a container in Azure Blob Storage and setting up a data factory.', 'The process of moving data from a database to blob storage is explained, emphasizing the importance of naming conventions and writing down all details as a best practice. Explanation of moving data from a database to blob storage, importance of naming conventions and best practices.', 'The importance of monitoring the data deployment and the ability to edit the pipeline or implement various actions on the data is highlighted. Emphasis on monitoring data deployment and the flexibility to edit the pipeline or implement actions on the data.']}], 'duration': 791.736, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/nWM_7iql9bA/pics/nWM_7iql9bA1587552.jpg', 'highlights': ['Creation of a storage account in Azure, necessity of resource group and storage account for storing entities on Microsoft Azure.', 'Details such as location, replication, performance, and resource group specified during storage account creation.', 'Opportunity to create containers and use Azure Data Factory for data movement after creating the storage account.', 'Demonstration of creating a container in Azure Blob Storage and setting up a data factory.', 'Explanation of moving data from a database to blob storage, importance of naming conventions and best practices.', 'Emphasis on monitoring data deployment and the flexibility to edit the pipeline or implement actions on the data.']}], 'highlights': ['Azure Data Factory is crucial for data integration and management.', 'Data Factory automates data-driven workflows and orchestrates data sources.', 'Data Factory facilitates data collection, connection, transformation, and enrichment.', 'Data Factory integrates with Microsoft Azure tools like Power BI for data visualization.', 'Data Factory supports the creation of logical pipelines for various data processes.', 'Azure Data Factory enables real-time data analysis and supports various data processes.', 'Azure Data Lake supports tools like Azure Data Lake analytics, HDInsights, Apache Spark, and machine learning.', 'Data Lake allows storage of structured, semi-structured, and unstructured data.', 'Data Lake is crucial for handling large, unstructured data from diverse sources.', 'Data Lake stores raw, unstructured data, whereas Data Warehouse stores filtered, summarized, and refined data.', 'The deployment of the demo server took longer due to slow internet speed.', 'The hourly cost of the demo server in South India is 2 to 3.88 INR.', 'Creation of a storage account in Azure is necessary for storing entities on Microsoft Azure.']}