title
Python Pandas Tutorial (Part 10): Working with Dates and Time Series Data

description
In this video, we will be learning how to work with DateTime and Time Series data in Pandas. This video is sponsored by Brilliant. Go to https://brilliant.org/cms to sign up for free. Be one of the first 200 people to sign up with this link and get 20% off your premium subscription. In this Python Programming video, we will be learning several different concepts about working with DateTimes and Time Series data in Pandas. We will learn how to convert values to datetimes, how to filter by dates, how to resample our dates to do some more in-depth analysis, and more. Let's get started... The code for this video can be found at: http://bit.ly/Pandas-10 StackOverflow Survey Download Page - http://bit.ly/SO-Survey-Download Datetime Formatting Codes - http://bit.ly/python-dt-fmt Pandas Date Offset Codes - http://bit.ly/pandas-dt-fmt ✅ Support My Channel Through Patreon: https://www.patreon.com/coreyms ✅ Become a Channel Member: https://www.youtube.com/channel/UCCezIgC97PvUuR4_gbFUs5g/join ✅ One-Time Contribution Through PayPal: https://goo.gl/649HFY ✅ Cryptocurrency Donations: Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3 Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33 Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot ✅ Corey's Public Amazon Wishlist http://a.co/inIyro1 ✅ Equipment I Use and Books I Recommend: https://www.amazon.com/shop/coreyschafer ▶️ You Can Find Me On: My Website - http://coreyms.com/ My Second Channel - https://www.youtube.com/c/coreymschafer Facebook - https://www.facebook.com/CoreyMSchafer Twitter - https://twitter.com/CoreyMSchafer Instagram - https://www.instagram.com/coreymschafer/ #Python #Pandas

detail
{'title': 'Python Pandas Tutorial (Part 10): Working with Dates and Time Series Data', 'heatmap': [{'end': 1202.336, 'start': 1132.381, 'weight': 0.748}], 'summary': 'The tutorial covers techniques in pandas for working with date and time series data, analyzing historical cryptocurrency data with nearly 24,000 rows, converting strings to date time objects, filtering date-time data, slicing data for time analysis, resampling time series data, and visualizing data using matplotlib with a specific example showing the highest trade value for january 1st, 2020 as 132.68.', 'chapters': [{'end': 32.183, 'segs': [{'end': 32.183, 'src': 'embed', 'start': 0.189, 'weight': 0, 'content': [{'end': 0.429, 'text': 'Hey there.', 'start': 0.189, 'duration': 0.24}, {'end': 5.691, 'text': "How's it going, everybody? In this video, we're going to be learning how to work with date and time series data within Pandas.", 'start': 0.449, 'duration': 5.242}, {'end': 8.832, 'text': "Now, there's a ton of interesting stuff that we can do with date time data.", 'start': 6.071, 'duration': 2.761}, {'end': 10.653, 'text': "And we'll be learning about that here.", 'start': 9.232, 'duration': 1.421}, {'end': 15.955, 'text': "So we'll learn how to properly read in our data so that we can use date time functionality.", 'start': 11.133, 'duration': 4.822}, {'end': 22.157, 'text': "We'll also see how to filter by date times, how to group dates by resampling the time frames.", 'start': 16.355, 'duration': 5.802}, {'end': 27.099, 'text': "And we'll also take a look at doing some simple plotting with our time series data as well.", 'start': 22.577, 'duration': 4.522}, {'end': 32.183, 'text': "Now I'd like to mention that we do have a sponsor for the series of videos and that is Brilliant.", 'start': 27.699, 'duration': 4.484}], 'summary': 'Learn how to work with date and time series data in pandas, including filtering, resampling, and plotting.', 'duration': 31.994, 'max_score': 0.189, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc189.jpg'}], 'start': 0.189, 'title': 'Working with date and time in pandas', 'summary': 'Covers techniques in pandas for working with date and time series data including data reading, filtering, grouping by resampling, and simple plotting, with a mention of the sponsor brilliant.', 'chapters': [{'end': 32.183, 'start': 0.189, 'title': 'Working with date and time in pandas', 'summary': 'Covers working with date and time series data in pandas, including reading in data, filtering by date times, grouping dates by resampling the time frames, and simple plotting, with a mention of the sponsor brilliant.', 'duration': 31.994, 'highlights': ["We'll learn how to properly read in our data so that we can use date time functionality.", "We'll see how to filter by date times and how to group dates by resampling the time frames.", "We'll also take a look at doing some simple plotting with our time series data.", 'The series of videos is sponsored by Brilliant.']}], 'duration': 31.994, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc189.jpg', 'highlights': ["We'll learn how to properly read in our data for date time functionality.", "We'll see how to filter by date times and group dates by resampling.", "We'll take a look at doing simple plotting with time series data.", 'The series of videos is sponsored by Brilliant.']}, {'end': 513.503, 'segs': [{'end': 58.227, 'src': 'embed', 'start': 32.505, 'weight': 1, 'content': [{'end': 39.874, 'text': 'So I really want to thank Brilliant for sponsoring this series and it would be great if you all could check them out using the link in the description section below and support the sponsors.', 'start': 32.505, 'duration': 7.369}, {'end': 42.357, 'text': "And I'll talk more about their services in just a bit.", 'start': 40.214, 'duration': 2.143}, {'end': 44.599, 'text': "So with that said, let's go ahead and get started.", 'start': 42.777, 'duration': 1.822}, {'end': 51.123, 'text': "Okay, so first of all, I've been using the Stack Overflow survey data for this entire series so far,", 'start': 45.16, 'duration': 5.963}, {'end': 58.227, 'text': "but that data set doesn't actually have any date or time series data, so I had to choose a different data set for this video.", 'start': 51.123, 'duration': 7.104}], 'summary': 'Sponsored by brilliant, using stack overflow survey data, switched data set for this video.', 'duration': 25.722, 'max_score': 32.505, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc32505.jpg'}, {'end': 114.194, 'src': 'embed', 'start': 69.412, 'weight': 0, 'content': [{'end': 77.48, 'text': "So I've got my notebook opened up here where I'm reading in this CSV file of data and let's go ahead and take a look at what this looks like.", 'start': 69.412, 'duration': 8.068}, {'end': 88.77, 'text': "So we can see here that I'm loading in this CSV file and I called this ETH underscore 1H, and that's because this is historical data for Ethereum,", 'start': 77.78, 'duration': 10.99}, {'end': 89.891, 'text': 'which is a cryptocurrency.', 'start': 88.77, 'duration': 1.121}, {'end': 94.275, 'text': 'And this data is broken down on one hour segments.', 'start': 90.471, 'duration': 3.804}, {'end': 99.2, 'text': 'So if we look down here at the head of this data, we can see that we have some columns here.', 'start': 94.696, 'duration': 4.504}, {'end': 103.564, 'text': 'The first one is a date column and these are broken down by the hour.', 'start': 99.34, 'duration': 4.224}, {'end': 109.911, 'text': 'We also have some other information here like the symbols, the open and closing values for these hours.', 'start': 104.145, 'duration': 5.766}, {'end': 114.194, 'text': 'the highs and lows and also the volume, so we can.', 'start': 110.291, 'duration': 3.903}], 'summary': 'Reading historical ethereum data from csv file in one-hour segments.', 'duration': 44.782, 'max_score': 69.412, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc69412.jpg'}], 'start': 32.505, 'title': 'Analyzing cryptocurrency and date time data', 'summary': 'Discusses the analysis of historical cryptocurrency data, emphasizing ethereum, and working with date time data in pandas, with a dataset containing almost 24,000 rows.', 'chapters': [{'end': 114.194, 'start': 32.505, 'title': 'Analyzing historical cryptocurrency data', 'summary': 'Discusses the analysis of historical cryptocurrency data, including the use of stack overflow survey data, the need for a different dataset, and the key attributes of the cryptocurrency data set, with a focus on ethereum, presented in one-hour segments.', 'duration': 81.689, 'highlights': ['The data set used for the series so far is the Stack Overflow survey data, which lacks date or time series data.', 'A different data set, comprising historical cryptocurrency data, specifically for Ethereum, broken down into one-hour segments, was chosen for analysis.', 'The cryptocurrency data includes columns for date, symbols, open and closing values, highs and lows, and volume, providing comprehensive attributes for analysis.']}, {'end': 513.503, 'start': 114.194, 'title': 'Working with date time data in pandas', 'summary': 'Covers working with date time data in pandas, including accessing rows and columns, converting data to date time objects, and parsing date time formats, with an example dataset containing almost 24,000 rows.', 'duration': 399.309, 'highlights': ['The chapter explains accessing rows and columns using df.shape, revealing that the dataset contains almost 24,000 rows, providing a good amount of data for analysis.', 'It demonstrates the process of converting a column to a date time using the pandas to_datetime method, highlighting the need to specify the format string to correctly parse the date time data.', 'It showcases the option to convert data to date time objects during the data loading process by using the parse_dates argument in the read_csv method, enabling the parsing of specific columns as date time as the data is read in.']}], 'duration': 480.998, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc32505.jpg', 'highlights': ['The dataset contains almost 24,000 rows, providing a good amount of data for analysis.', 'A different dataset, comprising historical cryptocurrency data for Ethereum, was chosen for analysis.', 'The cryptocurrency data includes columns for date, symbols, open and closing values, highs and lows, and volume.']}, {'end': 789.073, 'segs': [{'end': 636.924, 'src': 'embed', 'start': 611.337, 'weight': 2, 'content': [{'end': 617.885, 'text': "And I'm going to set that equal to that D parser variable there that is set to our lambda function.", 'start': 611.337, 'duration': 6.548}, {'end': 623.471, 'text': "OK, so now if I run this cell here, then we can see that we didn't get any errors.", 'start': 618.205, 'duration': 5.266}, {'end': 624.332, 'text': "So that's good.", 'start': 623.671, 'duration': 0.661}, {'end': 627.796, 'text': 'And now if I run this DF dot head here.', 'start': 624.913, 'duration': 2.883}, {'end': 633.962, 'text': 'then we can see that now our data frame was already loaded in as a date time.', 'start': 628.917, 'duration': 5.045}, {'end': 636.924, 'text': "So we didn't have to do any conversions later on.", 'start': 634.522, 'duration': 2.402}], 'summary': 'A lambda function was successfully applied to a data frame, resulting in no errors and the data frame being loaded as a datetime.', 'duration': 25.587, 'max_score': 611.337, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc611337.jpg'}, {'end': 789.073, 'src': 'embed', 'start': 749.948, 'weight': 0, 'content': [{'end': 759.052, 'text': "So to do that, we could just grab what we have here and I could simply create a new column by simply like I'm accessing a column.", 'start': 749.948, 'duration': 9.104}, {'end': 763.054, 'text': 'So I could call this column day of week and set this equal to.', 'start': 759.652, 'duration': 3.402}, {'end': 766.241, 'text': 'and paste in that date time method there.', 'start': 763.92, 'duration': 2.321}, {'end': 771.464, 'text': 'If I run this and then we look at our data frame, then we can see that.', 'start': 766.762, 'duration': 4.702}, {'end': 779.688, 'text': 'now we can quickly see over here on the right that okay, the 13th was a Friday and then we have these dates down here towards the end.', 'start': 771.464, 'duration': 8.224}, {'end': 780.769, 'text': 'This was a Saturday.', 'start': 779.708, 'duration': 1.061}, {'end': 785.671, 'text': "So it's nice to see, or be able to see what days these trades actually took place.", 'start': 781.289, 'duration': 4.382}, {'end': 789.073, 'text': "So now let's look at how we can explore our data a bit.", 'start': 786.151, 'duration': 2.922}], 'summary': "Creating a new column 'day of week' using date time method to visualize trade days in the data frame.", 'duration': 39.125, 'max_score': 749.948, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc749948.jpg'}], 'start': 513.503, 'title': 'Working with date time in python', 'summary': 'Discusses converting strings to date time objects using lambda functions, accessing the dt class on the series object to perform date time methods, and creating a new column to display the day of the week for each date.', 'chapters': [{'end': 789.073, 'start': 513.503, 'title': 'Working with date time in python', 'summary': 'Discusses converting strings to date time objects using lambda functions, accessing the dt class on the series object to perform date time methods, and creating a new column to display the day of the week for each date.', 'duration': 275.57, 'highlights': ["Creating a lambda function, 'D_parser', to convert each string to a date time object, ensuring the entire series is passed through the function, resulting in the successful conversion as verified by running the cell and checking the data frame.", "Accessing the DT class on the series object to apply the 'day_name' method to the entire date column, allowing for the retrieval of the day of the week for each date in the series, showcasing the similarity to accessing the str class for string methods on an entire series.", "Creating a new column, 'day_of_week', to display the day of the week for each date, providing a quick reference for the days on which the trades took place, facilitating better understanding and analysis of the trade data."]}], 'duration': 275.57, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc513503.jpg', 'highlights': ["Creating a new column, 'day_of_week', to display the day of the week for each date, facilitating better understanding and analysis of the trade data.", "Accessing the DT class on the series object to apply the 'day_name' method to the entire date column, allowing for the retrieval of the day of the week for each date in the series.", "Creating a lambda function, 'D_parser', to convert each string to a date time object, ensuring the entire series is passed through the function, resulting in the successful conversion."]}, {'end': 1039.079, 'segs': [{'end': 904.334, 'src': 'embed', 'start': 856.548, 'weight': 1, 'content': [{'end': 866.452, 'text': 'So to get the amount of time that spans between these two dates here, then I could simply say, take the max value and then subtract the min value.', 'start': 856.548, 'duration': 9.904}, {'end': 871.654, 'text': 'And if I run this, then we can see that we get this time delta.', 'start': 867.573, 'duration': 4.081}, {'end': 877.237, 'text': 'that says that there were almost 1000 days between the earliest date in our data set and the most recent.', 'start': 871.654, 'duration': 5.583}, {'end': 885.08, 'text': 'So we have 986 days in this entire data set of cryptocurrency data, almost 1000.', 'start': 877.577, 'duration': 7.503}, {'end': 890.064, 'text': 'So that would definitely be a lot of days to look through if we want to find some specific ranges.', 'start': 885.08, 'duration': 4.984}, {'end': 897.669, 'text': "So what if we wanted to do some filters by date? So for example, let's say that we just wanted to view the data for 2020.", 'start': 890.464, 'duration': 7.205}, {'end': 904.334, 'text': 'Now that we have these converted to date times, we can create filters just like we have in previous videos.', 'start': 897.669, 'duration': 6.665}], 'summary': 'Data spans almost 1000 days, 986 days in cryptocurrency dataset, with potential for date filters.', 'duration': 47.786, 'max_score': 856.548, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc856548.jpg'}, {'end': 1006.667, 'src': 'embed', 'start': 953.939, 'weight': 0, 'content': [{'end': 958.12, 'text': "okay, so now that I have that filter, let's just do a df.loc.", 'start': 953.939, 'duration': 4.181}, {'end': 961.102, 'text': "Again. we've seen this in previous videos.", 'start': 959.401, 'duration': 1.701}, {'end': 963.865, 'text': "And then I'll pass in that filter.", 'start': 961.643, 'duration': 2.222}, {'end': 971.45, 'text': 'So if I run this, then my bottom row here should be January 1 of 2020.', 'start': 964.425, 'duration': 7.025}, {'end': 976.234, 'text': "And it is and we can see that we have 17, 000 hours here of 2020 data, or I'm sorry, that's 1700 hours of 2020 data.", 'start': 971.45, 'duration': 4.784}, {'end': 987.46, 'text': "okay. so the reason that this doesn't go above 2020 is simply because you know our latest data runs out.", 'start': 981.278, 'duration': 6.182}, {'end': 992.821, 'text': "so we're not getting 2021, since 2021 hasn't happened yet.", 'start': 987.46, 'duration': 5.361}, {'end': 994.462, 'text': 'but what if we wanted data for 2019?', 'start': 992.821, 'duration': 1.641}, {'end': 1000.463, 'text': "well, in order to do that, we'd also have to put in an upper bound as well.", 'start': 994.462, 'duration': 6.001}, {'end': 1006.667, 'text': "so to do that, i'm going to say okay, we want data to be greater than or equal to 2019,", 'start': 1000.463, 'duration': 6.204}], 'summary': 'Filtering and locating data for 2019, showing 1700 hours of 2020 data.', 'duration': 52.728, 'max_score': 953.939, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc953939.jpg'}], 'start': 789.633, 'title': 'Analyzing and filtering date-time data', 'summary': 'Illustrates analyzing cryptocurrency data, extracting earliest and latest dates, determining a time span of 986 days, and creating filters for specific date ranges, as well as demonstrates filtering date-time data in pandas for 2020 and 2019 with quantifiable results.', 'chapters': [{'end': 904.334, 'start': 789.633, 'title': 'Analyzing date time data in python', 'summary': 'Illustrates how to analyze a dataset containing over 20,000 rows of cryptocurrency data, including extracting the earliest and latest dates, determining the time span between them, which is 986 days, and creating filters for specific date ranges such as in 2020.', 'duration': 114.701, 'highlights': ['The dataset contains over 20,000 rows of cryptocurrency data, with the earliest date being 2017-07-01 and the most recent date being March 13th, 2020.', 'The time span between the earliest and latest dates in the dataset is 986 days, indicating a significant duration to analyze.', 'The demonstration includes creating filters for specific date ranges, such as extracting data for the year 2020.']}, {'end': 1039.079, 'start': 904.735, 'title': 'Filtering date-time data in pandas', 'summary': 'Demonstrates filtering date-time data in pandas using string formatted date-times and actual date-time objects, showcasing examples of filtering data for 2020 and 2019 with quantifiable results.', 'duration': 134.344, 'highlights': ['Filtering date-time data using string formatted date-times and actual date-time objects, showcasing examples of filtering data for 2020 and 2019 with quantifiable results', 'Creating a filter to extract rows greater than or equal to 2020, resulting in 1700 hours of 2020 data', 'Implementing an upper bound filter to extract data for 2019, resulting in all the rows of data available for 2019']}], 'duration': 249.446, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc789633.jpg', 'highlights': ['The time span between the earliest and latest dates in the dataset is 986 days, indicating a significant duration to analyze.', 'Creating a filter to extract rows greater than or equal to 2020, resulting in 1700 hours of 2020 data', 'The dataset contains over 20,000 rows of cryptocurrency data, with the earliest date being 2017-07-01 and the most recent date being March 13th, 2020.', 'Implementing an upper bound filter to extract data for 2019, resulting in all the rows of data available for 2019', 'The demonstration includes creating filters for specific date ranges, such as extracting data for the year 2020.', 'Filtering date-time data using string formatted date-times and actual date-time objects, showcasing examples of filtering data for 2020 and 2019 with quantifiable results']}, {'end': 1277.598, 'segs': [{'end': 1122.975, 'src': 'embed', 'start': 1095.537, 'weight': 4, 'content': [{'end': 1099.899, 'text': "So let's do PD dot to date time for both of those and run this,", 'start': 1095.537, 'duration': 4.362}, {'end': 1105.723, 'text': 'and now we can see that we get those same results as before for all of the rows in 2019.', 'start': 1099.899, 'duration': 5.824}, {'end': 1114.649, 'text': 'now one nice feature about dates is that if we set our index so that it uses the date which would actually be a good idea for this data set,', 'start': 1105.723, 'duration': 8.926}, {'end': 1122.975, 'text': 'since all of these date or timestamps are unique then we can actually do this same thing by using slicing instead.', 'start': 1114.649, 'duration': 8.326}], 'summary': 'Using pd.to_datetime for date index, enabling slicing for unique timestamps.', 'duration': 27.438, 'max_score': 1095.537, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1095537.jpg'}, {'end': 1202.336, 'src': 'heatmap', 'start': 1132.381, 'weight': 0.748, 'content': [{'end': 1137.124, 'text': "So here at the bottom I'm going to say df.set, underscore index.", 'start': 1132.381, 'duration': 4.743}, {'end': 1141.327, 'text': "And then I'm going to pass in that we want to set the index to date.", 'start': 1137.585, 'duration': 3.742}, {'end': 1145.21, 'text': 'And if I run this, then that looks good.', 'start': 1142.007, 'duration': 3.203}, {'end': 1147.931, 'text': 'We have set our index to use date here.', 'start': 1145.37, 'duration': 2.561}, {'end': 1151.554, 'text': "And now that that looks good, it actually didn't make that change.", 'start': 1148.432, 'duration': 3.122}, {'end': 1156.238, 'text': 'I want to say in place is equal to true to make that change permanent.', 'start': 1152.054, 'duration': 4.184}, {'end': 1165.087, 'text': "so I'll run that and if we look at our data frame again, then now we have that date as our index and now, with that date index,", 'start': 1156.238, 'duration': 8.849}, {'end': 1169.972, 'text': 'we can actually filter our dates just by passing them into our brackets.', 'start': 1165.087, 'duration': 4.885}, {'end': 1180.521, 'text': 'so if we wanted the data for 2019, then I could literally just say that I want the data here for 2019, pass that into my brackets.', 'start': 1169.972, 'duration': 10.549}, {'end': 1185.124, 'text': 'If I run that, then we can see that we get the same thing here.', 'start': 1180.961, 'duration': 4.163}, {'end': 1192.789, 'text': 'We get this value for January 1st, and then the top value here is for December 31st.', 'start': 1185.164, 'duration': 7.625}, {'end': 1202.336, 'text': "So it's a bit easier to just access these within brackets when these are our indexes, rather than creating a filter.", 'start': 1193.27, 'duration': 9.066}], 'summary': "Using df.set_index('date') and inplace=true, we set the date as the index, enabling easy filtering by passing dates into brackets.", 'duration': 69.955, 'max_score': 1132.381, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1132381.jpg'}, {'end': 1169.972, 'src': 'embed', 'start': 1142.007, 'weight': 0, 'content': [{'end': 1145.21, 'text': 'And if I run this, then that looks good.', 'start': 1142.007, 'duration': 3.203}, {'end': 1147.931, 'text': 'We have set our index to use date here.', 'start': 1145.37, 'duration': 2.561}, {'end': 1151.554, 'text': "And now that that looks good, it actually didn't make that change.", 'start': 1148.432, 'duration': 3.122}, {'end': 1156.238, 'text': 'I want to say in place is equal to true to make that change permanent.', 'start': 1152.054, 'duration': 4.184}, {'end': 1165.087, 'text': "so I'll run that and if we look at our data frame again, then now we have that date as our index and now, with that date index,", 'start': 1156.238, 'duration': 8.849}, {'end': 1169.972, 'text': 'we can actually filter our dates just by passing them into our brackets.', 'start': 1165.087, 'duration': 4.885}], 'summary': 'Setting date as index allows easy date filtering in the data frame.', 'duration': 27.965, 'max_score': 1142.007, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1142007.jpg'}], 'start': 1039.079, 'title': 'Filtering and slicing data in pandas for time analysis', 'summary': 'Covers filtering data using actual datetime values, comparing dates, converting strings to datetime, setting an index, filtering data by passing dates into brackets, utilizing slicing for specific date ranges, and calculating the average closing price for ethereum in pandas.', 'chapters': [{'end': 1114.649, 'start': 1039.079, 'title': 'Filtering data by date in pandas', 'summary': 'Demonstrates how to filter data using actual datetime values in pandas, providing an example of comparing dates and converting strings to datetime, resulting in the retrieval of data for a specific time range.', 'duration': 75.57, 'highlights': ['Converting strings to datetime using pd.to_datetime allows for comparing dates and filtering data effectively.', 'Using actual datetime values in filtering data helps in retrieving data for specific time ranges and can be particularly useful when setting the index to use dates in a dataset.', 'The process involves replacing string dates with actual datetime values to perform comparisons and retrieve specific data, contributing to improved data analysis and insights.']}, {'end': 1277.598, 'start': 1114.649, 'title': 'Using indexing and slicing in data analysis', 'summary': 'Demonstrates how to set an index using a date column, filter data by passing dates into brackets, utilize slicing for specific date ranges, and calculate the average closing price for ethereum.', 'duration': 162.949, 'highlights': ['Setting the index to use the date column allows for easy filtering of data just by passing dates into brackets, simplifying the process of accessing data within indexes.', 'Using slicing for specific date ranges, such as for January and February of 2020, is inclusive of the second value, enabling efficient analysis of data within a specified time frame.', 'Utilizing the average closing price for Ethereum can be achieved by accessing the close column and calculating the mean, providing a method for statistical analysis of the data.']}], 'duration': 238.519, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1039079.jpg', 'highlights': ['Setting the index to use the date column simplifies data filtering and access', 'Converting strings to datetime enables effective date comparisons and filtering', 'Utilizing slicing for specific date ranges allows efficient analysis of data', 'Using actual datetime values in filtering data helps retrieve specific time ranges', 'Calculating the average closing price for Ethereum provides statistical analysis']}, {'end': 2127.775, 'segs': [{'end': 1302.145, 'src': 'embed', 'start': 1278.119, 'weight': 6, 'content': [{'end': 1284.59, 'text': 'If I run that, then we can see that we get all of those closing values on each of those hours for all of those days.', 'start': 1278.119, 'duration': 6.471}, {'end': 1287.575, 'text': 'And now to get the mean of that, I can just say .', 'start': 1285.211, 'duration': 2.364}, {'end': 1294.502, 'text': 'mean And that gives us the average closing price for all of those rows within that time frame.', 'start': 1287.575, 'duration': 6.927}, {'end': 1298.103, 'text': 'And remember, each of those days is reporting by the hour.', 'start': 1295.002, 'duration': 3.101}, {'end': 1302.145, 'text': 'But what if we wanted to see this data in a different way?', 'start': 1298.983, 'duration': 3.162}], 'summary': 'Analyzing closing values by hour for multiple days to calculate average closing price within the specified time frame.', 'duration': 24.026, 'max_score': 1278.119, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1278119.jpg'}, {'end': 1876.041, 'src': 'embed', 'start': 1828.443, 'weight': 2, 'content': [{'end': 1833.666, 'text': 'But the point of a high and low value is to know the high for that time period and the low for that time period.', 'start': 1828.443, 'duration': 5.223}, {'end': 1836.668, 'text': "So we probably don't want mean here either.", 'start': 1834.447, 'duration': 2.221}, {'end': 1840.311, 'text': 'So how can we resample this to where we can?', 'start': 1837.068, 'duration': 3.243}, {'end': 1845.434, 'text': 'you know, resample and use multiple columns but also use multiple aggregation methods?', 'start': 1840.311, 'duration': 5.123}, {'end': 1849.658, 'text': "Now, we've actually seen this in previous videos and used this method.", 'start': 1846.074, 'duration': 3.584}, {'end': 1854.242, 'text': 'But what we want to use here is the AGG, the ag method.', 'start': 1850.198, 'duration': 4.044}, {'end': 1862.069, 'text': 'And the ag method also accepts a map of columns and the aggregation functions that we want to run on that column.', 'start': 1854.782, 'duration': 7.287}, {'end': 1868.855, 'text': "So, for example, let's do this with the values for let's see, we'll do the closing column.", 'start': 1862.529, 'duration': 6.326}, {'end': 1872.458, 'text': "we'll do the high and low columns and then we'll also do the volume here.", 'start': 1869.235, 'duration': 3.223}, {'end': 1876.041, 'text': "so I'm going to grab this from up here.", 'start': 1872.458, 'duration': 3.583}], 'summary': 'Resample and aggregate multiple columns using agg method for high, low, closing, and volume values.', 'duration': 47.598, 'max_score': 1828.443, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1828443.jpg'}, {'end': 1934.02, 'src': 'embed', 'start': 1901.973, 'weight': 0, 'content': [{'end': 1917.386, 'text': "let's say that for the closing value I do want to grab the mean of that And then I'll say for the high column I want to use the max aggregation function for that,", 'start': 1901.973, 'duration': 15.413}, {'end': 1918.688, 'text': 'since we want the max value.', 'start': 1917.386, 'duration': 1.302}, {'end': 1924.394, 'text': 'For the low column, I want to get the min value.', 'start': 1919.108, 'duration': 5.286}, {'end': 1934.02, 'text': "And for volume, I'll go ahead and just sum up all of the volume for that entire time period.", 'start': 1926.215, 'duration': 7.805}], 'summary': 'Calculate mean closing value, max high, min low, and sum of volume.', 'duration': 32.047, 'max_score': 1901.973, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1901973.jpg'}, {'end': 2084.324, 'src': 'embed', 'start': 2051.795, 'weight': 3, 'content': [{'end': 2054.177, 'text': 'And you can find that link in the description section below.', 'start': 2051.795, 'duration': 2.382}, {'end': 2058.418, 'text': "Again, that's brilliant.org forward slash CMS.", 'start': 2054.737, 'duration': 3.681}, {'end': 2062.022, 'text': "Okay, so I think that's going to do it for this pandas video.", 'start': 2059.62, 'duration': 2.402}, {'end': 2067.568, 'text': 'I hope you feel like you got a good idea for how to work with date and time series data within pandas.', 'start': 2062.042, 'duration': 5.526}, {'end': 2071.952, 'text': "And like I said, there's a lot more that we can cover with date time data.", 'start': 2067.967, 'duration': 3.985}, {'end': 2077.618, 'text': 'But I feel like what we did here should definitely provide you with the basics of being able to convert,', 'start': 2072.553, 'duration': 5.065}, {'end': 2084.324, 'text': 'Analyze and resample your data so that you can do the exact analysis that you need.', 'start': 2078.138, 'duration': 6.186}], 'summary': 'Learn how to work with date and time series data in pandas, covering basics of data conversion, analysis, and resampling.', 'duration': 32.529, 'max_score': 2051.795, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc2051795.jpg'}, {'end': 2127.775, 'src': 'embed', 'start': 2108.965, 'weight': 1, 'content': [{'end': 2112.827, 'text': 'And if you enjoy these tutorials and would like to support them, then there are several ways you can do that.', 'start': 2108.965, 'duration': 3.862}, {'end': 2115.868, 'text': 'The easiest way is to simply like the video and give it a thumbs up.', 'start': 2113.247, 'duration': 2.621}, {'end': 2119.751, 'text': "And also it's a huge help to share these videos with anyone who you think would find them useful.", 'start': 2116.229, 'duration': 3.522}, {'end': 2124.693, 'text': "And if you have the means, you can contribute to Patreon and there's a link to that page in the description section below.", 'start': 2120.151, 'duration': 4.542}, {'end': 2127.775, 'text': 'Be sure to subscribe for future videos and thank you all for watching.', 'start': 2125.193, 'duration': 2.582}], 'summary': 'Support tutorials by liking, sharing, and contributing to patreon. subscribe for future videos.', 'duration': 18.81, 'max_score': 2108.965, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc2108965.jpg'}], 'start': 1278.119, 'title': 'Resampling and plotting time series data', 'summary': 'Demonstrates how to resample time series data by day, week, and aggregate multiple columns with functions like max, min, mean, and sum, then visualizes the data using matplotlib in jupyter notebooks. includes a specific example where the highest trade value for january 1st, 2020 was 132.68.', 'chapters': [{'end': 1430.197, 'start': 1278.119, 'title': 'Resampling data for daily highs', 'summary': 'Demonstrates how to resample hourly stock data to obtain daily high values, including a specific example where the highest trade value for january 1st, 2020 was 132.68, and the process of resampling data to view it on a daily, weekly, or monthly basis.', 'duration': 152.078, 'highlights': ['The highest trade value for January 1st, 2020 was 132.68, obtained by resampling the data on an hourly basis to a daily basis and using the max function to find the highest trade value for each day.', 'The demonstration includes the process of resampling data to view it on a daily, weekly, or monthly basis, providing flexibility in analyzing stock data at different time intervals.', 'The chapter explains how to obtain the average closing price for a specific time frame by using the mean function on the stock data, showcasing the ability to calculate meaningful statistical measures from the given dataset.']}, {'end': 2127.775, 'start': 1430.637, 'title': 'Resampling and plotting time series data in pandas', 'summary': 'Demonstrates how to resample time series data by day, week, and aggregate multiple columns with functions like max, min, mean, and sum, then visualizes the data using matplotlib in jupyter notebooks.', 'duration': 697.138, 'highlights': ['The chapter shows how to resample time series data by day and week, using functions like max, min, mean, and sum to aggregate multiple columns, providing a comprehensive overview of the data. For example, resampling by week allows for obtaining the mean closing costs, max highs, min lows, and the sum of volume for each week.', 'The tutorial provides a practical demonstration of visualizing resampled time series data using matplotlib in Jupyter Notebooks, making it extremely easy to plot out information and generate insightful visualizations. The demonstration includes a simple line plot of the resampled data, showcasing the power of pandas in data visualization.', 'The chapter also introduces the use of Brilliant for supplementing learning with hands-on courses in statistics and machine learning, offering a deep dive into data analysis fundamentals and advanced techniques, providing an excellent resource for further learning and skill development in data analysis and machine learning.']}], 'duration': 849.656, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/UFuo7EHI8zc/pics/UFuo7EHI8zc1278119.jpg', 'highlights': ['The highest trade value for January 1st, 2020 was 132.68, obtained by resampling the data on an hourly basis to a daily basis and using the max function to find the highest trade value for each day.', 'The chapter shows how to resample time series data by day and week, using functions like max, min, mean, and sum to aggregate multiple columns, providing a comprehensive overview of the data.', 'The demonstration includes the process of resampling data to view it on a daily, weekly, or monthly basis, providing flexibility in analyzing stock data at different time intervals.', 'The tutorial provides a practical demonstration of visualizing resampled time series data using matplotlib in Jupyter Notebooks, making it extremely easy to plot out information and generate insightful visualizations.', 'The chapter explains how to obtain the average closing price for a specific time frame by using the mean function on the stock data, showcasing the ability to calculate meaningful statistical measures from the given dataset.', 'For example, resampling by week allows for obtaining the mean closing costs, max highs, min lows, and the sum of volume for each week.', 'The chapter also introduces the use of Brilliant for supplementing learning with hands-on courses in statistics and machine learning, offering a deep dive into data analysis fundamentals and advanced techniques, providing an excellent resource for further learning and skill development in data analysis and machine learning.']}], 'highlights': ['The highest trade value for January 1st, 2020 was 132.68', 'The dataset contains almost 24,000 rows, providing a good amount of data for analysis', "Creating a new column, 'day_of_week', to display the day of the week for each date", 'The time span between the earliest and latest dates in the dataset is 986 days', 'Setting the index to use the date column simplifies data filtering and access', 'The series of videos is sponsored by Brilliant']}