title
Understanding OneLake within Microsoft Fabric

description
The key to Microsoft Fabric is definitely OneLake! OneLake enables the reuse of data within Fabric to help drive all of the engines. One copy of the data! Josh Caplan joins us to help get an understanding of the power of OneLake. OneLake, the OneDrive for data https://learn.microsoft.com/fabric/onelake/onelake-overview OneLake shortcuts https://learn.microsoft.com/fabric/onelake/onelake-shortcuts OneLake File Explorer https://learn.microsoft.com/fabric/onelake/onelake-file-explorer Josh Caplan: https://twitter.com/JoshCaplan1984 https://www.linkedin.com/in/josh-caplan-2139a3a6/ 📢 Become a member: https://guyinacu.be/membership ******************* Want to take your Power BI skills to the next level? We have training courses available to help you with your journey. 🎓 Guy in a Cube courses: https://guyinacu.be/courses ******************* LET'S CONNECT! ******************* -- http://twitter.com/guyinacube -- http://twitter.com/awsaxton -- http://twitter.com/patrickdba -- http://www.facebook.com/guyinacube -- https://www.instagram.com/guyinacube/ -- https://guyinacube.com ***Gear*** 🛠 Check out my Tools page - https://guyinacube.com/tools/ #MicrosoftFabric #OneLake #GuyInACube

detail
{'title': 'Understanding OneLake within Microsoft Fabric', 'heatmap': [], 'summary': 'Introduces onelake, a data lake for the entire organization, similar to onedrive, enabling simplified data management, collaboration, and workspace creation, while also covering the setup of a transactional data warehouse using sql and the integration of data from various sources through onelake data hub.', 'chapters': [{'end': 244.841, 'segs': [{'end': 143.621, 'src': 'embed', 'start': 114.636, 'weight': 0, 'content': [{'end': 116.057, 'text': "You don't think about these things with OneDrive.", 'start': 114.636, 'duration': 1.421}, {'end': 119.279, 'text': "It's there, you'll have now a data lake as a service.", 'start': 116.637, 'duration': 2.642}, {'end': 121.501, 'text': 'that whole solution built out of the box for you.', 'start': 119.699, 'duration': 1.802}, {'end': 124.603, 'text': 'And you can just start putting your data in there, collaborating over it and using it.', 'start': 121.641, 'duration': 2.962}, {'end': 127.166, 'text': "So for my tenant, there's just one, one lake.", 'start': 124.844, 'duration': 2.322}, {'end': 128.187, 'text': 'There is one, one lake.', 'start': 127.286, 'duration': 0.901}, {'end': 130.529, 'text': 'Not never a two lake or a zero lake.', 'start': 128.747, 'duration': 1.782}, {'end': 132.591, 'text': "It'll be one, one lake for the entire tenant.", 'start': 130.908, 'duration': 1.683}, {'end': 133.872, 'text': "And you didn't have to set it up.", 'start': 133.011, 'duration': 0.861}, {'end': 134.632, 'text': "You didn't have to provision it.", 'start': 133.892, 'duration': 0.74}, {'end': 135.353, 'text': "It'll be there.", 'start': 134.693, 'duration': 0.66}, {'end': 138.716, 'text': "No matter what you do, when you start loading data to Fabric, it'll be going into one lake.", 'start': 135.433, 'duration': 3.283}, {'end': 140.278, 'text': 'Nice Enough of all this talking.', 'start': 138.976, 'duration': 1.302}, {'end': 141.939, 'text': 'You know, we like to do it here on Guy in a Cube.', 'start': 140.558, 'duration': 1.381}, {'end': 143.621, 'text': "Let's do what? Let's head over to your machine.", 'start': 141.959, 'duration': 1.662}], 'summary': 'Onedrive offers a data lake as a service, providing a single lake for entire tenant without setup or provisioning.', 'duration': 28.985, 'max_score': 114.636, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg114636.jpg'}, {'end': 189.546, 'src': 'embed', 'start': 164.09, 'weight': 1, 'content': [{'end': 168.752, 'text': "And it's the workspace that actually allows multiple teams to collaborate over the same data lake.", 'start': 164.09, 'duration': 4.662}, {'end': 174.315, 'text': "Is it fair to think of a workspace as kind of like a folder structure within OneLake? And that's what it's going to ultimately translate over to.", 'start': 168.912, 'duration': 5.403}, {'end': 175.516, 'text': "It's a place in OneLake.", 'start': 174.375, 'duration': 1.141}, {'end': 178.919, 'text': "You see in the workspace I'm in, I have a few of these fabric data items here.", 'start': 175.616, 'duration': 3.303}, {'end': 180.139, 'text': 'I have a data warehouse.', 'start': 178.939, 'duration': 1.2}, {'end': 181.04, 'text': 'I have a lake house.', 'start': 180.179, 'duration': 0.861}, {'end': 183.642, 'text': "And let's actually open up one of these warehouses real quick.", 'start': 181.4, 'duration': 2.242}, {'end': 187.945, 'text': "Inside the warehouse, you'll see I have a schema in here, and I have a table in here.", 'start': 184.242, 'duration': 3.703}, {'end': 189.546, 'text': 'This is the UI view of the world.', 'start': 188.205, 'duration': 1.341}], 'summary': 'Onelake workspace facilitates collaboration, including data warehouse and lake house.', 'duration': 25.456, 'max_score': 164.09, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg164090.jpg'}, {'end': 224.132, 'src': 'embed', 'start': 196.111, 'weight': 2, 'content': [{'end': 199.673, 'text': "So because we're the OneDrive for data, you can explore your files right from Windows.", 'start': 196.111, 'duration': 3.562}, {'end': 201.854, 'text': "You'll be able to do that here with OneLake as well.", 'start': 199.813, 'duration': 2.041}, {'end': 203.595, 'text': 'So I can just open up File Explorer actually.', 'start': 201.874, 'duration': 1.721}, {'end': 206.796, 'text': "And in File Explorer, you'll see a OneLake option here.", 'start': 204.075, 'duration': 2.721}, {'end': 210.238, 'text': "And I see all my workspaces, the same workspaces you saw in the UI, but here they're folders.", 'start': 207.156, 'duration': 3.082}, {'end': 215.824, 'text': "And going into that workspace, you'll see those two data items that you saw in the UI, the warehouse that I had in the lake house.", 'start': 210.598, 'duration': 5.226}, {'end': 218.206, 'text': "And since we were in the warehouse before, I'll go into the warehouse.", 'start': 216.084, 'duration': 2.122}, {'end': 219.828, 'text': "I'll see a folder for tables.", 'start': 218.326, 'duration': 1.502}, {'end': 222.411, 'text': "Under that folder for tables, I'll see a schema.", 'start': 219.888, 'duration': 2.523}, {'end': 224.132, 'text': 'And the one table we saw before.', 'start': 222.611, 'duration': 1.521}], 'summary': 'Onelake allows exploring files from windows, with workspaces appearing as folders, containing data items and tables.', 'duration': 28.021, 'max_score': 196.111, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg196111.jpg'}], 'start': 0.189, 'title': 'Onelake at microsoft', 'summary': 'Introduces onelake, aiming to provide a single data lake for the entire organization, simplifying data management and collaboration, ultimately offering a familiar and user-friendly experience similar to onedrive, with the capability to create workspaces and manage data items.', 'chapters': [{'end': 244.841, 'start': 0.189, 'title': 'Introduction to onelake at microsoft', 'summary': 'Introduces onelake as the onedrive for data, aiming to provide a single data lake for the entire organization, simplifying data management and collaboration, ultimately offering a familiar and user-friendly experience similar to onedrive, with the capability to create workspaces and manage data items.', 'duration': 244.652, 'highlights': ['OneLake is positioned as the OneDrive for data, offering a single data lake for the entire organization, simplifying data management and collaboration. OneLake aims to simplify data management by providing a single data lake for the entire organization, enabling seamless collaboration and reducing the need for multiple siloed lakes, ultimately streamlining data governance and accessibility.', 'The concept of workspaces in OneLake allows multiple teams to collaborate over the same data lake, resembling a folder structure within OneLake. Workspaces in OneLake enable multiple teams to collaborate over the same data lake, offering a familiar folder structure within the platform, enhancing team collaboration and data organization.', 'OneLake provides a user-friendly experience similar to OneDrive, allowing users to explore files directly from Windows and interact naturally with the platform. OneLake offers a user-friendly experience akin to OneDrive, enabling users to explore files directly from Windows and interact seamlessly with the platform, enhancing accessibility and user experience.']}], 'duration': 244.652, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg189.jpg', 'highlights': ['OneLake is positioned as the OneDrive for data, offering a single data lake for the entire organization, simplifying data management and collaboration.', 'The concept of workspaces in OneLake allows multiple teams to collaborate over the same data lake, resembling a folder structure within OneLake.', 'OneLake provides a user-friendly experience similar to OneDrive, allowing users to explore files directly from Windows and interact naturally with the platform.']}, {'end': 627.939, 'segs': [{'end': 272.183, 'src': 'embed', 'start': 244.861, 'weight': 2, 'content': [{'end': 248.142, 'text': "So we're in the warehouse right now, and data warehouse is going to be fully transactional.", 'start': 244.861, 'duration': 3.281}, {'end': 250.262, 'text': "You're going to work with it through SQL.", 'start': 248.202, 'duration': 2.06}, {'end': 252.583, 'text': "So I want to bring data to a warehouse, I'm going to do it through SQL.", 'start': 250.282, 'duration': 2.301}, {'end': 255.784, 'text': 'We have our one table in our small warehouse at the moment.', 'start': 252.683, 'duration': 3.101}, {'end': 256.985, 'text': "Let's create a second table.", 'start': 256.024, 'duration': 0.961}, {'end': 260.168, 'text': "And we'll use this one to track the Guy in the Cube merchandise sales.", 'start': 257.546, 'duration': 2.622}, {'end': 260.689, 'text': 'Yeah, there we go.', 'start': 260.189, 'duration': 0.5}, {'end': 263.052, 'text': "So I'll create a table here, all T-SQL.", 'start': 260.81, 'duration': 2.242}, {'end': 264.254, 'text': "Let's insert one row.", 'start': 263.373, 'duration': 0.881}, {'end': 265.815, 'text': 'I know those banana shirts are very popular.', 'start': 264.294, 'duration': 1.521}, {'end': 267.117, 'text': 'So popular, and I buy 50 of them.', 'start': 265.956, 'duration': 1.161}, {'end': 269.079, 'text': 'Oh Insert that one row.', 'start': 267.577, 'duration': 1.502}, {'end': 270.621, 'text': 'All right, we created a table, created the one row.', 'start': 269.099, 'duration': 1.522}, {'end': 272.183, 'text': 'So load it through T-SQL.', 'start': 270.841, 'duration': 1.342}], 'summary': 'Setting up a fully transactional data warehouse using sql with one table and tracking merchandise sales.', 'duration': 27.322, 'max_score': 244.861, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg244861.jpg'}, {'end': 327.582, 'src': 'embed', 'start': 295.411, 'weight': 0, 'content': [{'end': 296.671, 'text': 'It means this is open source.', 'start': 295.411, 'duration': 1.26}, {'end': 301.052, 'text': 'By keeping it in open format, you can use it anywhere with Fabric and not just Fabric.', 'start': 296.911, 'duration': 4.141}, {'end': 302.172, 'text': 'This is an open data lake.', 'start': 301.272, 'duration': 0.9}, {'end': 306.093, 'text': 'So any application that knows how to talk to AWS Gen2 can talk to One Lake and work with this data lake.', 'start': 302.192, 'duration': 3.901}, {'end': 312.414, 'text': "Typically, a data lake, you can put any kind of data in there, not just structured data, and you don't have to necessarily do it through SQL.", 'start': 306.153, 'duration': 6.261}, {'end': 313.835, 'text': 'We go back to the UI for one second.', 'start': 312.634, 'duration': 1.201}, {'end': 315.175, 'text': "Let's see the lake house we had in here.", 'start': 313.935, 'duration': 1.24}, {'end': 316.315, 'text': "Let's ignore tables for a moment.", 'start': 315.355, 'duration': 0.96}, {'end': 317.215, 'text': "Let's look at the file section.", 'start': 316.335, 'duration': 0.88}, {'end': 319.256, 'text': 'File section lets you put anything you want in it.', 'start': 317.395, 'duration': 1.861}, {'end': 320.636, 'text': "So let's actually get some data in there.", 'start': 319.396, 'duration': 1.24}, {'end': 323.738, 'text': 'You can browse, go back and browse my workspace in the file explorer here.', 'start': 320.916, 'duration': 2.822}, {'end': 327.582, 'text': "See our lake has, I'm gonna see the same folder structure, including the file section.", 'start': 324.179, 'duration': 3.403}], 'summary': 'One lake is an open data lake compatible with aws gen2, enabling storage and easy access to various types of data.', 'duration': 32.171, 'max_score': 295.411, 'thumbnail': ''}, {'end': 389.853, 'src': 'embed', 'start': 357.226, 'weight': 3, 'content': [{'end': 359.809, 'text': "But how do we get this in here? I didn't go and upload every table from Windows.", 'start': 357.226, 'duration': 2.583}, {'end': 361.29, 'text': 'Data got in here a few different ways.', 'start': 359.989, 'duration': 1.301}, {'end': 364.052, 'text': "These tables here, we're building these through Databricks.", 'start': 361.57, 'duration': 2.482}, {'end': 366.454, 'text': 'We actually switched Databricks to go ahead and use one link.', 'start': 364.232, 'duration': 2.222}, {'end': 369.296, 'text': 'Because behind the scenes, like I said, actually look, these are all files.', 'start': 366.474, 'duration': 2.822}, {'end': 372.678, 'text': 'And to access these files, we support the same ADLS Gen2 APIs.', 'start': 369.756, 'duration': 2.922}, {'end': 376.081, 'text': "And if you right click on any of these locations, bring up the properties, you'll see the path to those files.", 'start': 372.698, 'duration': 3.383}, {'end': 378.883, 'text': 'Yep Databricks uses the ABFS driver to go and connect.', 'start': 376.121, 'duration': 2.762}, {'end': 381.445, 'text': 'You can actually take the ABFS path directly from here, copy it.', 'start': 378.903, 'duration': 2.542}, {'end': 382.906, 'text': 'If I flip over to Databricks real quick.', 'start': 381.645, 'duration': 1.261}, {'end': 383.967, 'text': "Oh man, that's easy.", 'start': 383.126, 'duration': 0.841}, {'end': 389.853, 'text': 'The thing I like, like from the Databricks side, if you already have like some implementation in Databricks, but you wanted to switch it over, Yep.', 'start': 384.207, 'duration': 5.646}], 'summary': 'Data uploaded to databricks through one link, supporting adls gen2 apis and abfs driver for connection.', 'duration': 32.627, 'max_score': 357.226, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg357226.jpg'}, {'end': 618.52, 'src': 'embed', 'start': 588.408, 'weight': 4, 'content': [{'end': 589.969, 'text': "they're coming from the pure data engineering.", 'start': 588.408, 'duration': 1.561}, {'end': 594.372, 'text': "Let them work with SQL, let them work with Spark, let them work with their engine of choice if they're not even coming from within Fabric.", 'start': 589.989, 'duration': 4.383}, {'end': 597.539, 'text': 'Everybody here builds the same data lake.', 'start': 595.137, 'duration': 2.402}, {'end': 602.843, 'text': 'And that data resides in OneLake and it can be referenced in other spots, including Power BI.', 'start': 597.899, 'duration': 4.944}, {'end': 605.445, 'text': 'All right, Josh, thank you so much for walking us through that.', 'start': 602.883, 'duration': 2.562}, {'end': 610.429, 'text': "OneLake is amazing and I'm sure there's a lot more that we can cover in other videos.", 'start': 605.745, 'duration': 4.684}, {'end': 614.572, 'text': 'Let us know in the comments below what you want to know about OneLake or what questions that you have.', 'start': 610.629, 'duration': 3.943}, {'end': 618.52, 'text': "we'll get those answered and or like spin up some more videos on it.", 'start': 615.072, 'duration': 3.448}], 'summary': 'Data engineers work with sql, spark, and onelake for unified data lake, accessible in power bi.', 'duration': 30.112, 'max_score': 588.408, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg588408.jpg'}], 'start': 244.861, 'title': 'Data warehouses and onelake data hub', 'summary': 'Covers setting up a transactional data warehouse using sql and exploring delta lake format, as well as introducing onelake data hub to store and integrate data from various sources, reducing duplication and simplifying data management.', 'chapters': [{'end': 312.414, 'start': 244.861, 'title': 'Working with data warehouses and delta lake format', 'summary': 'Involves setting up a fully transactional data warehouse using sql, creating and loading tables, and exploring the open-source delta lake format, which enables data to be accessed by any application that can talk to aws gen2.', 'duration': 67.553, 'highlights': ['The data warehouse is fully transactional and can be accessed through SQL, allowing for the creation and loading of tables.', 'The data in the warehouse is stored in Delta Lake format, which is open source and accessible to any application that can talk to AWS Gen2.', 'The open data lake allows for storage of any kind of data, not just structured data, and can be accessed without necessarily using SQL.']}, {'end': 627.939, 'start': 312.634, 'title': 'Onelake data hub overview', 'summary': 'Introduces onelake data hub, explaining its capability to store unstructured and structured data, allowing the integration of data from different sources such as databricks and amazon s3, and the use of shortcuts to virtualize data, reducing data duplication and simplifying data management, enabling teams to work with their preferred data processing engines.', 'duration': 315.305, 'highlights': ['OneLake Data Hub allows the storage of unstructured data, supporting any file format, and structured data in Delta-like format in the table section, enabling automatic integration with data processing engines like SQL and Spark, and direct report building with Power BI.', 'Data from Databricks can be accessed in OneLake using the same ADLS Gen2 APIs, simplifying the process of switching data location and enabling direct data reading and writing.', 'The use of shortcuts in OneLake allows virtualization of data from different sources such as Amazon S3, simplifying data management, reducing data duplication, and keeping the data connected to the original source, enabling data exploration and report building without the need for data copying or ETL management.', 'Teams within Fabric can work with their preferred data processing engines, such as SQL and Spark, while accessing the same data lake in OneLake, promoting data consistency and collaboration among teams.', 'The chapter concludes by inviting viewers to explore more about OneLake and Microsoft Fabric and encourages them to share their questions and feedback.']}], 'duration': 383.078, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/wEcRTSNhtLg/pics/wEcRTSNhtLg244861.jpg', 'highlights': ['OneLake Data Hub supports storage of unstructured and structured data in Delta-like format, enabling automatic integration with data processing engines like SQL and Spark.', 'The open data lake allows for storage of any kind of data, not just structured data, and can be accessed without necessarily using SQL.', 'The data warehouse is fully transactional and can be accessed through SQL, allowing for the creation and loading of tables.', 'Data from Databricks can be accessed in OneLake using the same ADLS Gen2 APIs, simplifying the process of switching data location and enabling direct data reading and writing.', 'Teams within Fabric can work with their preferred data processing engines, such as SQL and Spark, while accessing the same data lake in OneLake, promoting data consistency and collaboration among teams.']}], 'highlights': ['OneLake is positioned as the OneDrive for data, offering a single data lake for the entire organization, simplifying data management and collaboration.', 'The concept of workspaces in OneLake allows multiple teams to collaborate over the same data lake, resembling a folder structure within OneLake.', 'OneLake provides a user-friendly experience similar to OneDrive, allowing users to explore files directly from Windows and interact naturally with the platform.', 'OneLake Data Hub supports storage of unstructured and structured data in Delta-like format, enabling automatic integration with data processing engines like SQL and Spark.', 'The open data lake allows for storage of any kind of data, not just structured data, and can be accessed without necessarily using SQL.', 'The data warehouse is fully transactional and can be accessed through SQL, allowing for the creation and loading of tables.', 'Data from Databricks can be accessed in OneLake using the same ADLS Gen2 APIs, simplifying the process of switching data location and enabling direct data reading and writing.', 'Teams within Fabric can work with their preferred data processing engines, such as SQL and Spark, while accessing the same data lake in OneLake, promoting data consistency and collaboration among teams.']}