title
Python Pandas Tutorial 8. Concat Dataframes

description
This tutorial goes over how to use pandas concat function to join or append dataframes. Topics that are covered in this Python Pandas Video: 0:00 What is concat? 3:24 Concat two dataframe using concat() function 4:22 ignore_index argument in concat() function 4:42 List of arguments for concat() function 5:16 What is "keys"? pass "keys" to concat() function 8:53 "axis" argument in concat() function 12:26 Join dataframe with series() function Link for code used in this tutorial: https://github.com/codebasics/py/blob/master/pandas/8_concat/pandas_concat.ipynb To download csv and code for all tutorials: go to https://github.com/codebasics/py, click on a green button to clone or download the entire repository and then go to relevant folder to get access to that specific file. Do you want to learn technology from me? Check https://codebasics.io/ for my affordable video courses. Next Video: Python Pandas Tutorial 9. Merge Dataframes: https://www.youtube.com/watch?v=h4hOPGo4UVU&list=PLeo1K3hjS3uuASpe-1LjfG5f14Bnozjwy&index=9 Popular Playlist: Complete python course: https://www.youtube.com/playlist?list=PLeo1K3hjS3uv5U-Lmlnucd7gqF-3ehIh0 Data science course: https://www.youtube.com/playlist?list=PLeo1K3hjS3us_ELKYSj_Fth2tIEkdKXvV Machine learning tutorials: https://www.youtube.com/playlist?list=PLeo1K3hjS3uvCeTYTeyfe0-rN5r8zn9rw Pandas tutorials: https://www.youtube.com/playlist?list=PLeo1K3hjS3uuASpe-1LjfG5f14Bnozjwy Git github tutorials: https://www.youtube.com/playlist?list=PLeo1K3hjS3usJuxZZUBdjAcilgfQHkRzW Matplotlib course: https://www.youtube.com/playlist?list=PLeo1K3hjS3uu4Lr8_kro2AqaO6CFYgKOl Data structures course: https://www.youtube.com/playlist?list=PLeo1K3hjS3uu_n_a__MI_KktGTLYopZ12 Website: https://codebasics.io/ Facebook: https://www.facebook.com/codebasicshub Twitter: https://twitter.com/codebasicshub

detail
{'title': 'Python Pandas Tutorial 8. Concat Dataframes', 'heatmap': [{'end': 293.693, 'start': 272.794, 'weight': 0.88}, {'end': 424.953, 'start': 403.306, 'weight': 0.73}, {'end': 680.472, 'start': 579.898, 'weight': 0.749}], 'summary': 'This tutorial covers pandas concatenation, creating data frames for weather analysis in india, joining multiple data frames using pd.concat function, and various dataframe operations such as retrieving subsets, appending columns, aligning rows, and joining with a series.', 'chapters': [{'end': 50.931, 'segs': [{'end': 50.931, 'src': 'embed', 'start': 0.583, 'weight': 0, 'content': [{'end': 3.324, 'text': 'dear friends, welcome to code basics coding tutorial.', 'start': 0.583, 'duration': 2.741}, {'end': 7.766, 'text': 'in this tutorial we are going to learn about pandas concatenate.', 'start': 3.324, 'duration': 4.442}, {'end': 14.228, 'text': 'now, concatenate is an operation that you do when you want to join two or more data frames.', 'start': 7.766, 'duration': 6.462}, {'end': 16.088, 'text': 'without talking much about it.', 'start': 14.228, 'duration': 1.86}, {'end': 18.89, 'text': "let's jump straight into it.", 'start': 16.088, 'duration': 2.802}, {'end': 22.351, 'text': "so I'm going to run Jupyter notebook as usual.", 'start': 18.89, 'duration': 3.461}, {'end': 31.74, 'text': 'Now, if you have not watched my Jupyter notebook tutorial before, then I would say you should pause here and watch that tutorial first.', 'start': 23.291, 'duration': 8.449}, {'end': 36.802, 'text': 'Okay, so I have my notebook up and running.', 'start': 33.9, 'duration': 2.902}, {'end': 41.725, 'text': "I'm going to click on new and start a new notebook.", 'start': 36.862, 'duration': 4.863}, {'end': 47.609, 'text': 'And the first thing that we do always is import pandas as pd.', 'start': 42.126, 'duration': 5.483}, {'end': 50.931, 'text': "Now I'm going to create my data frame first.", 'start': 48.069, 'duration': 2.862}], 'summary': 'Learn about pandas concatenate to join data frames.', 'duration': 50.348, 'max_score': 0.583, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA583.jpg'}], 'start': 0.583, 'title': 'Pandas concatenate', 'summary': 'Focuses on learning about pandas concatenate, which involves joining two or more data frames and involves creating a new notebook and importing pandas as pd.', 'chapters': [{'end': 50.931, 'start': 0.583, 'title': 'Pandas concatenate tutorial', 'summary': 'Focuses on learning about pandas concatenate, which involves joining two or more data frames and involves creating a new notebook and importing pandas as pd.', 'duration': 50.348, 'highlights': ['The chapter introduces the topic of pandas concatenate, explaining it as an operation to join two or more data frames.', 'The chapter emphasizes the initial steps of the tutorial, including creating a new notebook and importing pandas as pd.']}], 'duration': 50.348, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA583.jpg', 'highlights': ['The chapter introduces pandas concatenate to join data frames.', 'Emphasizes creating a new notebook and importing pandas as pd.']}, {'end': 256.725, 'segs': [{'end': 150.545, 'src': 'embed', 'start': 51.071, 'weight': 0, 'content': [{'end': 57.656, 'text': 'Okay, so we are going to have weather data stored in two different data frames.', 'start': 51.071, 'duration': 6.585}, {'end': 60.898, 'text': "So let's say I have India weather.", 'start': 58.156, 'duration': 2.742}, {'end': 64.343, 'text': 'for India weather.', 'start': 63.101, 'duration': 1.242}, {'end': 66.327, 'text': 'I have name of the city.', 'start': 64.343, 'duration': 1.984}, {'end': 79.478, 'text': "okay, so let's say I have three different cities in India and for these cities what i have is a temperature data.", 'start': 66.327, 'duration': 13.151}, {'end': 87.983, 'text': 'so this represents an average temperature throughout the year in these three cities.', 'start': 79.478, 'duration': 8.505}, {'end': 91.925, 'text': 'and another thing i have is humidity.', 'start': 87.983, 'duration': 3.942}, {'end': 96.647, 'text': 'so the humidity levels in Mumbai?', 'start': 93.265, 'duration': 3.382}, {'end': 99.429, 'text': 'I know they are pretty high.', 'start': 96.647, 'duration': 2.782}, {'end': 101.51, 'text': "so I'm just putting some numbers here.", 'start': 99.429, 'duration': 2.081}, {'end': 111.875, 'text': 'Delhi is pretty dry, so the humidity would be low, and again Bangalore it should have a high humidity.', 'start': 101.51, 'duration': 10.365}, {'end': 115.617, 'text': "and now you're going to.", 'start': 111.875, 'duration': 3.742}, {'end': 119.359, 'text': "so what I'm going to do is just create a data frame out of it.", 'start': 115.617, 'duration': 3.742}, {'end': 133.619, 'text': 'okay, and so this is how you create can create a data frame by giving a json object as an input and when you say control, enter,', 'start': 123.775, 'duration': 9.844}, {'end': 135.699, 'text': "it's gonna show you that data frame.", 'start': 133.619, 'duration': 2.08}, {'end': 137.44, 'text': 'so i have this data frame all right.', 'start': 135.699, 'duration': 1.741}, {'end': 144.083, 'text': "now i'm going to create second data frame, which will be, let's say, weather in usa.", 'start': 137.44, 'duration': 6.643}, {'end': 150.545, 'text': "so this will be us weather and i'm going to put some dummy data here.", 'start': 144.083, 'duration': 6.462}], 'summary': 'Weather data for india and usa stored in separate data frames, including temperature and humidity for multiple cities.', 'duration': 99.474, 'max_score': 51.071, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA51071.jpg'}, {'end': 220.975, 'src': 'embed', 'start': 190.916, 'weight': 2, 'content': [{'end': 203.285, 'text': 'And I want to now join these two so that I can get a single data frame which has weather data for all the cities in India and in the USA.', 'start': 190.916, 'duration': 12.369}, {'end': 208.387, 'text': 'So pandas provide pd.concat function.', 'start': 204.085, 'duration': 4.302}, {'end': 216.912, 'text': 'So in the pd.concat function, the first argument you pass is the data frames that you want to join together.', 'start': 208.928, 'duration': 7.984}, {'end': 220.975, 'text': 'Now I have two data frames here, IndiaWeather and USWeather.', 'start': 217.413, 'duration': 3.562}], 'summary': 'Join india and usa weather data using pd.concat to create a single data frame.', 'duration': 30.059, 'max_score': 190.916, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA190916.jpg'}], 'start': 51.071, 'title': 'Weather data analysis in india and creating data frames in pandas', 'summary': 'Covers storing weather data for three cities in india, average temperature and humidity levels, and creating data frames for analysis. it also demonstrates creating data frames from json objects, joining multiple data frames using pd.concat function, and using original data frame index for the joined data frame.', 'chapters': [{'end': 119.359, 'start': 51.071, 'title': 'Weather data analysis in india', 'summary': 'Discusses storing weather data for three cities in india, including average temperature and humidity levels, and creating a data frame for analysis.', 'duration': 68.288, 'highlights': ['Creating data frames for weather data of three cities in India, including average temperature and humidity levels.', 'Mumbai has high humidity levels, Delhi has low humidity levels, and Bangalore has high humidity levels.']}, {'end': 256.725, 'start': 123.775, 'title': 'Creating and joining data frames in pandas', 'summary': 'Demonstrates creating data frames from json objects, joining multiple data frames using pd.concat function, and using original data frame index for the joined data frame.', 'duration': 132.95, 'highlights': ['The pd.concat function in Pandas allows joining multiple data frames to create a single data frame, facilitating the combination of data from different sources.', "Creating a data frame from a JSON object and displaying it using 'control, enter' command enables visualization of the data frame directly in the interface.", 'Demonstrating the creation of two data frames, IndiaWeather and USWeather, and joining them using the pd.concat function, showcasing the practical application of the function in combining data sets from different sources.']}], 'duration': 205.654, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA51071.jpg', 'highlights': ['Creating data frames for weather data of three cities in India, including average temperature and humidity levels.', 'Mumbai has high humidity levels, Delhi has low humidity levels, and Bangalore has high humidity levels.', 'The pd.concat function in Pandas allows joining multiple data frames to create a single data frame, facilitating the combination of data from different sources.', "Creating a data frame from a JSON object and displaying it using 'control, enter' command enables visualization of the data frame directly in the interface.", 'Demonstrating the creation of two data frames, IndiaWeather and USWeather, and joining them using the pd.concat function, showcasing the practical application of the function in combining data sets from different sources.']}, {'end': 913.251, 'segs': [{'end': 315.045, 'src': 'heatmap', 'start': 272.794, 'weight': 4, 'content': [{'end': 282.824, 'text': 'ignore index is equal to true, and when you do that, you will see that now you got a continuous index.', 'start': 272.794, 'duration': 10.03}, {'end': 293.693, 'text': "okay. so if you want to know more about all the arguments that panda's concat function can take, then just go to panda's website, the documentation,", 'start': 282.824, 'duration': 10.869}, {'end': 303.778, 'text': 'and here type in concat and it should give you the documentation on various arguments that it can take.', 'start': 293.693, 'duration': 10.085}, {'end': 308.121, 'text': 'we just used ignore index, which was by default false.', 'start': 303.778, 'duration': 4.343}, {'end': 315.045, 'text': 'we made it true and then it will just ignore the index in your original data frame.', 'start': 308.121, 'duration': 6.924}], 'summary': 'By setting ignore index to true in pandas concat function, a continuous index is obtained, and for more information on available arguments, refer to pandas documentation.', 'duration': 58.32, 'max_score': 272.794, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA272794.jpg'}, {'end': 380.271, 'src': 'embed', 'start': 350.333, 'weight': 0, 'content': [{'end': 353.496, 'text': 'let me increase the font size a little bit so that you can see better.', 'start': 350.333, 'duration': 3.163}, {'end': 363.341, 'text': 'okay, so what this keys is gonna do is you can associate a key with each of these data frames that you have passed in this list here.', 'start': 354.256, 'duration': 9.085}, {'end': 367.083, 'text': 'okay, so my first data frame is India weather.', 'start': 363.341, 'duration': 3.742}, {'end': 378.509, 'text': 'so I can pass India as a key and US as a key for the second data frame, and when I run it, all right.', 'start': 367.083, 'duration': 11.426}, {'end': 380.271, 'text': 'so this is what I get.', 'start': 378.509, 'duration': 1.762}], 'summary': 'Demonstrating key association with data frames for india and us.', 'duration': 29.938, 'max_score': 350.333, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA350333.jpg'}, {'end': 424.953, 'src': 'heatmap', 'start': 403.306, 'weight': 0.73, 'content': [{'end': 413.948, 'text': 'and the way you can use this is index is you can simply say DF dot, log of allosystems for location, and if you say India,', 'start': 403.306, 'duration': 10.642}, {'end': 418.35, 'text': 'then now you can retrieve a subset of your data frame.', 'start': 413.948, 'duration': 4.402}, {'end': 422.311, 'text': 'similarly, if you say US, you will get back your US data.', 'start': 418.35, 'duration': 3.961}, {'end': 424.953, 'text': 'so having this index is useful.', 'start': 422.311, 'duration': 2.642}], 'summary': 'Using the index, you can retrieve subsets of data by location, such as india or us.', 'duration': 21.647, 'max_score': 403.306, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA403306.jpg'}, {'end': 566.143, 'src': 'embed', 'start': 532.325, 'weight': 3, 'content': [{'end': 538.644, 'text': 'so now you have these two data frames and when you these two data frames,', 'start': 532.325, 'duration': 6.319}, {'end': 546.986, 'text': 'ideally what you want to see is you want to see wind speed appear as a column in your original data frame.', 'start': 538.644, 'duration': 8.342}, {'end': 555.267, 'text': 'okay, so you want to get a final data frame which has city, temperature and wind speed.', 'start': 546.986, 'duration': 8.281}, {'end': 556.428, 'text': 'so how do you do that?', 'start': 555.267, 'duration': 1.161}, {'end': 566.143, 'text': 'if you to do PD dot, concat and pass in the argument these two data frames?', 'start': 556.428, 'duration': 9.715}], 'summary': 'Merge two data frames to include wind speed in original data frame.', 'duration': 33.818, 'max_score': 532.325, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA532325.jpg'}, {'end': 684.554, 'src': 'heatmap', 'start': 579.898, 'weight': 1, 'content': [{'end': 585.764, 'text': 'then what happens is is gonna just append the second data frame as rows here.', 'start': 579.898, 'duration': 5.866}, {'end': 592.591, 'text': 'what you want is wind speed to appear as a column, so that this part right here should go here.', 'start': 585.764, 'duration': 6.827}, {'end': 594.373, 'text': 'okay, it should not create additional rows.', 'start': 592.591, 'duration': 1.782}, {'end': 598.974, 'text': 'In order to do that, you can use axis argument.', 'start': 595.512, 'duration': 3.462}, {'end': 603.496, 'text': "So when you say x is equal to one, okay? Now let's look at the documentation.", 'start': 599.314, 'duration': 4.182}, {'end': 610.459, 'text': 'So when you see the documentation by default axis is zero, means it will append second data frame as rows.', 'start': 603.996, 'duration': 6.463}, {'end': 616.382, 'text': "But when you change axis to be one, now it's gonna append them as rows.", 'start': 611.12, 'duration': 5.262}, {'end': 624.647, 'text': 'Column so you can see that now you have temperature of Mumbai is 32 and wind speed is 7.', 'start': 617.743, 'duration': 6.904}, {'end': 633.752, 'text': "Okay, now What might happen if, let's say, the order of these cities is different?", 'start': 624.647, 'duration': 9.105}, {'end': 639.201, 'text': "so what I'm going to do is data is always not perfect.", 'start': 633.752, 'duration': 5.449}, {'end': 645.807, 'text': "so let's say you're missing data from bangalore and the order of city is different here.", 'start': 639.201, 'duration': 6.606}, {'end': 650.552, 'text': 'so first you have delhi and then you have mumbai.', 'start': 645.807, 'duration': 4.745}, {'end': 656.417, 'text': 'okay, and now, when you execute this, see what happened is it just went by the rows.', 'start': 650.552, 'duration': 5.865}, {'end': 659.218, 'text': 'so first row was Mumbai and first row was Delhi here.', 'start': 656.417, 'duration': 2.801}, {'end': 661.78, 'text': 'so Mumbai and Delhi here.', 'start': 659.218, 'duration': 2.562}, {'end': 664.402, 'text': 'so it just appended them.', 'start': 661.78, 'duration': 2.622}, {'end': 666.183, 'text': "now this doesn't look correct.", 'start': 664.402, 'duration': 1.781}, {'end': 669.125, 'text': 'you want Mumbai to be here right.', 'start': 666.183, 'duration': 2.942}, {'end': 673.347, 'text': 'so in order to do that, you can use index arguments.', 'start': 669.125, 'duration': 4.222}, {'end': 680.472, 'text': 'so in pandas data frame you can, while creating a data frame, you can always pass an index.', 'start': 673.347, 'duration': 7.125}, {'end': 684.554, 'text': "okay. so let's say my index here is zero, one, two.", 'start': 680.472, 'duration': 4.082}], 'summary': 'Appending data frames, adjusting column and row orientation, and handling missing data in pandas dataframe.', 'duration': 28.137, 'max_score': 579.898, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA579898.jpg'}, {'end': 782.149, 'src': 'embed', 'start': 746.109, 'weight': 2, 'content': [{'end': 755.236, 'text': 'last thing we are going to cover is we can also join our data frame with a series.', 'start': 746.109, 'duration': 9.127}, {'end': 759.099, 'text': "so let's say again, i have this temperature data frame.", 'start': 755.236, 'duration': 3.863}, {'end': 764.182, 'text': "as you can see here, if you have, let's say, a series, okay.", 'start': 759.099, 'duration': 5.083}, {'end': 768.363, 'text': "so let's say you have pandas series here.", 'start': 764.182, 'duration': 4.181}, {'end': 782.149, 'text': "okay, and pandas series, let's say, the name of the series is, let's say, event, okay, and let's say you have.", 'start': 768.363, 'duration': 13.786}], 'summary': 'Join data frame with a series to integrate temperature data.', 'duration': 36.04, 'max_score': 746.109, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA746109.jpg'}], 'start': 256.725, 'title': 'Pandas dataframe operations', 'summary': 'Covers using keys to retrieve subsets of data frames, appending data frames as columns, using axis argument to append data frames as columns, using index to align rows from different data frames, and joining a data frame with a series. it also hints at the next tutorial on a better way of joining data frames using merge.', 'chapters': [{'end': 315.045, 'start': 256.725, 'title': 'Pandas concat function', 'summary': 'Explains how to use the ignore index argument in the pandas concat function to create a continuous index, and suggests referring to the pandas website for more information on other arguments.', 'duration': 58.32, 'highlights': ['By setting ignore index to true in the concat function, a continuous index can be obtained.', 'For more information on the arguments of the pandas concat function, the documentation on the pandas website can be referred to.']}, {'end': 913.251, 'start': 315.045, 'title': 'Pandas dataframe operations', 'summary': 'Covers using keys to retrieve subsets of data frames, appending data frames as columns, using axis argument to append data frames as columns, using index to align rows from different data frames, and joining a data frame with a series. it also hints at the next tutorial on a better way of joining data frames using merge.', 'duration': 598.206, 'highlights': ["Using keys to retrieve subsets of data frames: Associating a key with each data frame to retrieve subsets of data frames, such as retrieving data for India and US, can be done by passing an additional argument called 'keys'.", 'Appending data frames as columns using concat operation: Using axis argument to append data frames as columns instead of rows, resulting in a final data frame with city, temperature, and wind speed.', 'Using index to align rows from different data frames: The index argument in pandas data frame can be used to align rows from different data frames while using concat operation, ensuring correct alignment of city data.', 'Joining a data frame with a series using concat operation: Appending a series into a data frame as a new column using pandas concat operation, resulting in the addition of the series as a new column into the data frame.']}], 'duration': 656.526, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/WGOEFok1szA/pics/WGOEFok1szA256725.jpg', 'highlights': ["Using keys to retrieve subsets of data frames: Associating a key with each data frame to retrieve subsets of data frames, such as retrieving data for India and US, can be done by passing an additional argument called 'keys'.", 'Using index to align rows from different data frames: The index argument in pandas data frame can be used to align rows from different data frames while using concat operation, ensuring correct alignment of city data.', 'Joining a data frame with a series using concat operation: Appending a series into a data frame as a new column using pandas concat operation, resulting in the addition of the series as a new column into the data frame.', 'Appending data frames as columns using concat operation: Using axis argument to append data frames as columns instead of rows, resulting in a final data frame with city, temperature, and wind speed.', 'By setting ignore index to true in the concat function, a continuous index can be obtained.', 'For more information on the arguments of the pandas concat function, the documentation on the pandas website can be referred to.']}], 'highlights': ['The pd.concat function in Pandas allows joining multiple data frames to create a single data frame, facilitating the combination of data from different sources.', 'Creating data frames for weather data of three cities in India, including average temperature and humidity levels.', "Using keys to retrieve subsets of data frames: Associating a key with each data frame to retrieve subsets of data frames, such as retrieving data for India and US, can be done by passing an additional argument called 'keys'.", 'Joining a data frame with a series using concat operation: Appending a series into a data frame as a new column using pandas concat operation, resulting in the addition of the series as a new column into the data frame.', 'Using index to align rows from different data frames: The index argument in pandas data frame can be used to align rows from different data frames while using concat operation, ensuring correct alignment of city data.', 'The chapter introduces pandas concatenate to join data frames.', "Creating a data frame from a JSON object and displaying it using 'control, enter' command enables visualization of the data frame directly in the interface.", 'By setting ignore index to true in the concat function, a continuous index can be obtained.', 'Emphasizes creating a new notebook and importing pandas as pd.', 'For more information on the arguments of the pandas concat function, the documentation on the pandas website can be referred to.', "Creating a data frame from a JSON object and displaying it using 'control, enter' command enables visualization of the data frame directly in the interface.", 'Demonstrating the creation of two data frames, IndiaWeather and USWeather, and joining them using the pd.concat function, showcasing the practical application of the function in combining data sets from different sources.', 'Appending data frames as columns using concat operation: Using axis argument to append data frames as columns instead of rows, resulting in a final data frame with city, temperature, and wind speed.']}