title
Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module
description
In this video, we will be learning how to use multiprocessing in Python.
This video is sponsored by Brilliant. Go to https://brilliant.org/cms to sign up for free. Be one of the first 200 people to sign up with this link and get 20% off your premium subscription.
In this Python Programming video, we will be learning how to run code in parallel using the multiprocessing module. We will also look at how to process multiple high-resolution images at the same time using a ProcessPoolExecutor from the concurrent.futures module. Let's get started...
The code from this video can be found at:
http://bit.ly/multiprocess-code
Threading Tutorial - https://youtu.be/IEEhzQoKtQU
List Comprehensions Tutorial - https://youtu.be/3dt4OGnU5sM
Exception Handling Tutorial - https://youtu.be/NIWwJbo-9_8
Pillow (PIL) Tutorial - https://youtu.be/6Qs3wObeWwc
✅ Support My Channel Through Patreon:
https://www.patreon.com/coreyms
✅ Become a Channel Member:
https://www.youtube.com/channel/UCCezIgC97PvUuR4_gbFUs5g/join
✅ One-Time Contribution Through PayPal:
https://goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
http://a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
https://www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - http://coreyms.com/
My Second Channel - https://www.youtube.com/c/coreymschafer
Facebook - https://www.facebook.com/CoreyMSchafer
Twitter - https://twitter.com/CoreyMSchafer
Instagram - https://www.instagram.com/coreymschafer/
#Python
detail
{'title': 'Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module', 'heatmap': [{'end': 1092.871, 'start': 1058.437, 'weight': 1}], 'summary': 'Learn about parallel code execution using the multiprocessing module, achieving substantial speed improvement, and the implementation of multiprocessing in python for both cpu-bound and i.o.-bound tasks. explore the creation, start, and join methods for running processes in parallel, efficient multiprocessing using process pool executor, and usage of concurrent.futures module for optimizing performance in python scripts.', 'chapters': [{'end': 234.496, 'segs': [{'end': 77.882, 'src': 'embed', 'start': 0.229, 'weight': 0, 'content': [{'end': 5.796, 'text': "Hey there, how's it going everybody? In this video, we're going to be learning how to run code in parallel using the multiprocessing module.", 'start': 0.229, 'duration': 5.567}, {'end': 13.064, 'text': "Now, if you'd also like to learn about running code concurrently using the threading module, then I did recently put out a video on that as well.", 'start': 6.156, 'duration': 6.908}, {'end': 16.168, 'text': "So I'll be sure to leave a link to that video in the description section below.", 'start': 13.405, 'duration': 2.763}, {'end': 19.749, 'text': "Now, if you don't know the difference between threading and multiprocessing,", 'start': 16.588, 'duration': 3.161}, {'end': 22.59, 'text': "then you should have a grasp on the difference between those once we're finished.", 'start': 19.749, 'duration': 2.841}, {'end': 27.372, 'text': 'Now, I would like to mention that we do have a sponsor for this video, and that is Brilliant.org.', 'start': 23.05, 'duration': 4.322}, {'end': 35.415, 'text': 'So I really want to thank Brilliant for sponsoring the video and it would be great if you all could go and check them out using the link in the description section below and support the sponsors.', 'start': 27.712, 'duration': 7.703}, {'end': 37.916, 'text': "And I'll talk more about their services in just a bit.", 'start': 35.815, 'duration': 2.101}, {'end': 40.197, 'text': "So with that said, let's go ahead and get started.", 'start': 38.296, 'duration': 1.901}, {'end': 44.099, 'text': 'Okay, so first, why would we want to use multiprocessing?', 'start': 40.777, 'duration': 3.322}, {'end': 50.022, 'text': "So, basically, we want to use multiprocessing whenever it's going to significantly speed up our program.", 'start': 44.519, 'duration': 5.503}, {'end': 54.084, 'text': 'Now the speed up comes from different tasks running in parallel.', 'start': 50.422, 'duration': 3.662}, {'end': 60.448, 'text': "Now in this video, we're going to start off with a basic example of where we learn how to run some simple sleep methods in parallel.", 'start': 54.505, 'duration': 5.943}, {'end': 67.994, 'text': "But then we'll finish up with a real world example where we do some image processing on a directory of high resolution images.", 'start': 60.888, 'duration': 7.106}, {'end': 75.06, 'text': 'I want to show that real world example, because personally, when I watch tutorials that only show how it works on basic examples,', 'start': 68.434, 'duration': 6.626}, {'end': 77.882, 'text': "then I always feel like I don't really walk away with any useful knowledge.", 'start': 75.06, 'duration': 2.822}], 'summary': 'Learn about running code in parallel using multiprocessing for speed improvement and real-world examples.', 'duration': 77.653, 'max_score': 0.229, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso229.jpg'}], 'start': 0.229, 'title': 'Multiprocessing for speeding up programs', 'summary': 'Explores parallel code execution using the multiprocessing module, highlighting the significant speed improvement achieved through parallel execution, from a simple sleep method to image processing on high-resolution images.', 'chapters': [{'end': 44.099, 'start': 0.229, 'title': 'Parallel code execution with multiprocessing', 'summary': 'Explores the concept of running code in parallel using the multiprocessing module and mentions a previous video on running code concurrently using the threading module. it also introduces a sponsor for the video, brilliant.org. the chapter begins with an explanation of the difference between threading and multiprocessing and the purpose of using multiprocessing.', 'duration': 43.87, 'highlights': ['The chapter explains the concept of running code in parallel using the multiprocessing module and mentions a previous video on running code concurrently using the threading module.', 'The chapter introduces a sponsor for the video, Brilliant.org, and encourages viewers to support the sponsors.', 'The chapter begins with an explanation of the difference between threading and multiprocessing and the purpose of using multiprocessing.']}, {'end': 234.496, 'start': 44.519, 'title': 'Multiprocessing for speeding up programs', 'summary': 'Discusses the use of multiprocessing to significantly speed up programs through parallel execution, starting with a simple sleep method and progressing to a real-world example of image processing on high-resolution images.', 'duration': 189.977, 'highlights': ['Multiprocessing provides speedup from parallel tasks', 'Real-world example of image processing', 'Example of running simple sleep methods in parallel']}], 'duration': 234.267, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso229.jpg', 'highlights': ['Multiprocessing provides speedup from parallel tasks', 'Real-world example of image processing', 'Example of running simple sleep methods in parallel', 'The chapter explains the concept of running code in parallel using the multiprocessing module and mentions a previous video on running code concurrently using the threading module', 'The chapter begins with an explanation of the difference between threading and multiprocessing and the purpose of using multiprocessing', 'The chapter introduces a sponsor for the video, Brilliant.org, and encourages viewers to support the sponsors']}, {'end': 441.452, 'segs': [{'end': 300.922, 'src': 'embed', 'start': 234.916, 'weight': 0, 'content': [{'end': 238.618, 'text': 'And running everything in order like this is called running it synchronously.', 'start': 234.916, 'duration': 3.702}, {'end': 242.059, 'text': "Now, if you have some tasks that don't need to be run synchronously,", 'start': 238.998, 'duration': 3.061}, {'end': 248.801, 'text': 'then we can use the multiprocessing module to split these tasks up onto other CPUs and run them at the same time.', 'start': 242.059, 'duration': 6.742}, {'end': 255.183, 'text': 'Now if you watched my last video on threading, then I mentioned that tasks were going to either be I.O.', 'start': 249.301, 'duration': 5.882}, {'end': 256.803, 'text': 'bound or CPU bound.', 'start': 255.223, 'duration': 1.58}, {'end': 262.245, 'text': 'So CPU bound tasks are things that are crunching a lot of numbers and using the CPU.', 'start': 257.224, 'duration': 5.021}, {'end': 263.246, 'text': 'And I.O.', 'start': 262.745, 'duration': 0.501}, {'end': 270.568, 'text': "bound tasks are things that are waiting for input and output operations to be completed and they're not really using the CPU all that much.", 'start': 263.266, 'duration': 7.302}, {'end': 272.709, 'text': 'So some other examples of I.O.', 'start': 271.028, 'duration': 1.681}, {'end': 278.752, 'text': 'bound tasks include file system operations and network operations like downloading stuff online.', 'start': 272.749, 'duration': 6.003}, {'end': 285.535, 'text': "Now, in that threading video I mentioned that we wouldn't get much of a speed up when using threading on CPU bound tasks,", 'start': 279.112, 'duration': 6.423}, {'end': 288.576, 'text': 'because those threads are still only running one process.', 'start': 285.535, 'duration': 3.041}, {'end': 296.56, 'text': "But with multiprocessing, we're going to actually spread the work out on the multiple processors on our machine and run those tasks at the same time.", 'start': 288.976, 'duration': 7.584}, {'end': 300.922, 'text': 'So we can use this with both IO-bound tasks and CPU-bound tasks.', 'start': 297.04, 'duration': 3.882}], 'summary': 'Using multiprocessing for io-bound and cpu-bound tasks, running them at the same time on multiple processors.', 'duration': 66.006, 'max_score': 234.916, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso234916.jpg'}, {'end': 386.123, 'src': 'embed', 'start': 358.076, 'weight': 5, 'content': [{'end': 362.339, 'text': "Okay. so now that we've talked about multiprocessing and what it looks like to run code in parallel,", 'start': 358.076, 'duration': 4.263}, {'end': 367.584, 'text': "Now let's see how to actually do this with our current script.", 'start': 363.22, 'duration': 4.364}, {'end': 370.428, 'text': "First, let's import the multiprocessing module.", 'start': 367.785, 'duration': 2.643}, {'end': 374.492, 'text': "This is in the standard library, so we don't need to install anything.", 'start': 371.148, 'duration': 3.344}, {'end': 380.559, 'text': "Up here at the top, I'm just going to say import multiprocessing.", 'start': 375.453, 'duration': 5.106}, {'end': 386.123, 'text': "Now I'm going to show an older way of how to do multiprocessing first so that we can get a good idea of what's going on.", 'start': 381.079, 'duration': 5.044}], 'summary': 'Introduction to using multiprocessing in python for parallel code execution.', 'duration': 28.047, 'max_score': 358.076, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso358076.jpg'}], 'start': 234.916, 'title': 'Implementing multiprocessing in python', 'summary': 'Covers the implementation of the multiprocessing module to run asynchronous tasks, distinguishing between cpu-bound and i.o.-bound tasks. it also discusses the difference between threading and multiprocessing, highlighting how multiprocessing spreads work across multiple processors and demonstrates its implementation in python.', 'chapters': [{'end': 278.752, 'start': 234.916, 'title': 'Multiprocessing for asynchronous tasks', 'summary': 'Discusses the use of the multiprocessing module to run tasks asynchronously, distinguishing between cpu-bound and i.o.-bound tasks, with examples including file system operations and network operations.', 'duration': 43.836, 'highlights': ['The multiprocessing module is used to split tasks onto other CPUs and run them simultaneously, allowing for asynchronous execution.', 'Tasks are categorized as either CPU-bound, involving heavy number crunching and CPU usage, or I.O.-bound, which entail waiting for input/output operations to be completed without extensively using the CPU.', 'Examples of I.O.-bound tasks include file system operations and network operations such as downloading content online.']}, {'end': 441.452, 'start': 279.112, 'title': 'Multiprocessing in python', 'summary': 'Discusses the difference between threading and multiprocessing, highlighting how multiprocessing spreads work across multiple processors, allowing tasks to run at the same time, and demonstrates how to implement multiprocessing in python using the multiprocessing module.', 'duration': 162.34, 'highlights': ['Multiprocessing spreads work across multiple processors, allowing tasks to run at the same time', 'Difference between threading and multiprocessing', 'Demonstration of implementing multiprocessing in Python using the multiprocessing module']}], 'duration': 206.536, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso234916.jpg', 'highlights': ['The multiprocessing module is used to split tasks onto other CPUs and run them simultaneously, allowing for asynchronous execution.', 'Tasks are categorized as either CPU-bound or I.O.-bound, involving heavy number crunching and CPU usage, or waiting for input/output operations to be completed without extensively using the CPU.', 'Multiprocessing spreads work across multiple processors, allowing tasks to run at the same time', 'Examples of I.O.-bound tasks include file system operations and network operations such as downloading content online.', 'Difference between threading and multiprocessing', 'Demonstration of implementing multiprocessing in Python using the multiprocessing module']}, {'end': 1001.134, 'segs': [{'end': 639.178, 'src': 'embed', 'start': 613.288, 'weight': 0, 'content': [{'end': 618.253, 'text': 'And then our script continued on to print that our script had finished in about one second.', 'start': 613.288, 'duration': 4.965}, {'end': 625.375, 'text': 'Now. if using multiprocessing seems a bit complicated right now, then definitely stick around until the end of the video,', 'start': 618.693, 'duration': 6.682}, {'end': 628.935, 'text': "because we're going to see an easier way of how to do what we're doing here.", 'start': 625.375, 'duration': 3.56}, {'end': 633.437, 'text': "But I think it's important to understand what this is doing so far,", 'start': 629.316, 'duration': 4.121}, {'end': 639.178, 'text': "even if we use other methods where we don't actually manually call these start methods and join methods.", 'start': 633.437, 'duration': 5.741}], 'summary': 'Script finished in about one second, exploring simpler methods for multiprocessing.', 'duration': 25.89, 'max_score': 613.288, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso613288.jpg'}], 'start': 441.972, 'title': 'Multiprocessing in python', 'summary': 'Explains the concept of multiprocessing in python, demonstrating the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes.', 'chapters': [{'end': 1001.134, 'start': 441.972, 'title': 'Multiprocessing in python', 'summary': 'Explains the concept of multiprocessing in python, demonstrating the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes.', 'duration': 559.162, 'highlights': ['The chapter demonstrates the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about 1.5 seconds (10 times) as compared to running synchronously, which would take 10 seconds.', 'The chapter explains the concept of multiprocessing in Python, demonstrating the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about one second, showing the efficiency of multiprocessing even with more processes than available cores.', 'The chapter demonstrates the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about one second, showing the efficiency of multiprocessing even with more processes than available cores.', 'The chapter explains the concept of multiprocessing in Python, demonstrating the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about one second, showing the efficiency of multiprocessing even with more processes than available cores.', 'The chapter explains the concept of multiprocessing in Python, demonstrating the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about one second, showing the efficiency of multiprocessing even with more processes than available cores.']}], 'duration': 559.162, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso441972.jpg', 'highlights': ['The chapter demonstrates the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about 1.5 seconds (10 times) as compared to running synchronously, which would take 10 seconds.', 'The chapter explains the concept of multiprocessing in Python, demonstrating the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about one second, showing the efficiency of multiprocessing even with more processes than available cores.']}, {'end': 1337.311, 'segs': [{'end': 1043.308, 'src': 'embed', 'start': 1002.834, 'weight': 0, 'content': [{'end': 1010.838, 'text': "Okay, so I said before that I was going to show you the older way of doing multi-processing and then I'd show you what I believe is a faster,", 'start': 1002.834, 'duration': 8.004}, {'end': 1012.058, 'text': 'easier way of doing this.', 'start': 1010.838, 'duration': 1.22}, {'end': 1019.702, 'text': "And I still wanted to show you the manual way of creating these processes because I think this can still be useful depending on what you're doing.", 'start': 1012.499, 'duration': 7.203}, {'end': 1026.005, 'text': "And also I think it's better to learn this manual way first to understand a bit better what's going on in the background.", 'start': 1020.162, 'duration': 5.843}, {'end': 1031.445, 'text': 'But in Python 3.2, they added something called a process pull executor.', 'start': 1026.545, 'duration': 4.9}, {'end': 1036.666, 'text': 'And in a lot of cases, this will be an easier and more efficient way to run multiple processes.', 'start': 1031.886, 'duration': 4.78}, {'end': 1043.308, 'text': 'And it also allows us to easily switch over to using multiple threads instead of processes as well,', 'start': 1037.146, 'duration': 6.162}], 'summary': 'Python 3.2 introduced process pool executor for easier and more efficient multiprocessing.', 'duration': 40.474, 'max_score': 1002.834, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1002834.jpg'}, {'end': 1092.871, 'src': 'heatmap', 'start': 1058.437, 'weight': 1, 'content': [{'end': 1063.823, 'text': "So up here at the top, let's instead import concurrent dot futures.", 'start': 1058.437, 'duration': 5.386}, {'end': 1066.586, 'text': "And I actually don't think I need multiprocessing anymore.", 'start': 1064.184, 'duration': 2.402}, {'end': 1069.89, 'text': "So I'm just going to say import concurrent dot futures.", 'start': 1066.967, 'duration': 2.923}, {'end': 1076.676, 'text': "futures Now I'm going to leave everything else that I have here for now, so that we can see the difference between these.", 'start': 1070.791, 'duration': 5.885}, {'end': 1083.722, 'text': "Now when we use this process pull executor, it's usually best to use this with a context manager.", 'start': 1077.797, 'duration': 5.925}, {'end': 1092.871, 'text': "So above our processes list here, I'm going to do the same thing that we already have, but just with our concurrent futures module instead.", 'start': 1084.123, 'duration': 8.748}], 'summary': 'Import concurrent.futures instead of multiprocessing for better efficiency.', 'duration': 34.434, 'max_score': 1058.437, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1058437.jpg'}, {'end': 1144.369, 'src': 'embed', 'start': 1110.725, 'weight': 1, 'content': [{'end': 1117.53, 'text': 'and now within our or with our executor here there are a couple of different methods that we can use now.', 'start': 1110.725, 'duration': 6.805}, {'end': 1122.314, 'text': 'if we want to execute the function once at a time, then we can use the submit method.', 'start': 1117.53, 'duration': 4.784}, {'end': 1127.778, 'text': 'So the submit method schedules a function to be executed and returns a future object.', 'start': 1122.754, 'duration': 5.024}, {'end': 1131.02, 'text': "So let's add this in and then I'll explain this a bit more.", 'start': 1128.178, 'duration': 2.842}, {'end': 1144.369, 'text': "So I'm going to say f1 is equal to executor.submit and I will submit that do something function and let's also pass in an argument of 1.", 'start': 1131.44, 'duration': 12.929}], 'summary': 'Using the submit method to schedule function execution and return future object.', 'duration': 33.644, 'max_score': 1110.725, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1110725.jpg'}, {'end': 1296.111, 'src': 'embed', 'start': 1234.15, 'weight': 2, 'content': [{'end': 1237.612, 'text': "And that's a lot less code than we had down here that's commented out.", 'start': 1234.15, 'duration': 3.462}, {'end': 1241.554, 'text': "But we're still not running this multiple times yet like we were down here.", 'start': 1237.992, 'duration': 3.562}, {'end': 1246.918, 'text': 'So if we wanted to run this multiple times, then we could just run submit multiple times.', 'start': 1241.995, 'duration': 4.923}, {'end': 1251.541, 'text': 'So I could say, let me go above our result here.', 'start': 1247.398, 'duration': 4.143}, {'end': 1255.884, 'text': "I'm going to add in another execution of this do something function.", 'start': 1251.941, 'duration': 3.943}, {'end': 1263.009, 'text': "So I'm going to call this F2 is equal to executor.submit do something with one second.", 'start': 1256.304, 'duration': 6.705}, {'end': 1267.572, 'text': 'And then I will also print out the F2 result.', 'start': 1263.509, 'duration': 4.063}, {'end': 1272.556, 'text': "So, if I run this, then we can see that it's the same thing.", 'start': 1268.132, 'duration': 4.424}, {'end': 1276.579, 'text': 'It kicks both of these off at the same time, and we finished in about one second.', 'start': 1272.636, 'duration': 3.943}, {'end': 1283.084, 'text': "And if we wanted to run this 10 times, like we did below, then we likely wouldn't want to run submit 10 different times,", 'start': 1276.959, 'duration': 6.125}, {'end': 1285.186, 'text': 'so we could use a loop like we did before.', 'start': 1283.084, 'duration': 2.102}, {'end': 1292.169, 'text': "so instead of running one at a time, i'm going to use a loop and we could use a regular loop like we did below.", 'start': 1285.766, 'duration': 6.403}, {'end': 1296.111, 'text': "but i'm going to go ahead and use a list comprehension to create these instead.", 'start': 1292.169, 'duration': 3.942}], 'summary': 'Using executor.submit to run code multiple times, finishing in about one second. loop and list comprehension used for efficiency.', 'duration': 61.961, 'max_score': 1234.15, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1234150.jpg'}], 'start': 1002.834, 'title': 'Python multiprocessing and asynchronous execution', 'summary': 'Explains the process pool executor for efficient multiprocessing in python 3.2 and demonstrates using python executor for asynchronous execution, resulting in more efficient code with reduced time and lines, and employing list comprehension for multiple tasks.', 'chapters': [{'end': 1160.738, 'start': 1002.834, 'title': 'Python multiprocessing: process pool executor', 'summary': 'Explains the process pool executor which is a more efficient way to run multiple processes in python 3.2, allowing easy switching to multiple threads, and the submit method schedules a function to be executed and returns a future object.', 'duration': 157.904, 'highlights': ['The process pool executor in Python 3.2 is a more efficient way to run multiple processes, allowing easy switching to multiple threads.', "The submit method schedules a function to be executed and returns a future object, encapsulating the function's execution.", 'Understanding the manual way of creating processes can be useful and helps in better comprehending background operations.']}, {'end': 1337.311, 'start': 1161.118, 'title': 'Using python executor for asynchronous execution', 'summary': 'Discusses using python executor to run multiple tasks asynchronously, demonstrating the process of submitting and running tasks concurrently, resulting in efficient code with reduced lines and time taken, as well as employing list comprehension for creating multiple tasks.', 'duration': 176.193, 'highlights': ['Demonstrating the process of submitting and running tasks concurrently', 'Reduced lines and time taken for execution', 'Employing list comprehension for creating multiple tasks']}], 'duration': 334.477, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1002834.jpg', 'highlights': ['The process pool executor in Python 3.2 is a more efficient way to run multiple processes, allowing easy switching to multiple threads.', "The submit method schedules a function to be executed and returns a future object, encapsulating the function's execution.", 'Demonstrating the process of submitting and running tasks concurrently', 'Reduced lines and time taken for execution', 'Employing list comprehension for creating multiple tasks', 'Understanding the manual way of creating processes can be useful and helps in better comprehending background operations.']}, {'end': 1707.789, 'segs': [{'end': 1382.491, 'src': 'embed', 'start': 1337.752, 'weight': 0, 'content': [{'end': 1340.454, 'text': "Okay, so now we've created a list comprehension.", 'start': 1337.752, 'duration': 2.702}, {'end': 1347.82, 'text': "that's running our submit method with this do something function and an argument of one second, 10 different times.", 'start': 1340.454, 'duration': 7.366}, {'end': 1355.387, 'text': 'Now, in order to get these results, we can actually use another function from the concurrent futures module called as completed.', 'start': 1348.28, 'duration': 7.107}, {'end': 1361.953, 'text': "And this will give us an iterator that we can loop over that will yield the results of our processes as they're completed.", 'start': 1355.727, 'duration': 6.226}, {'end': 1368.539, 'text': "So I think this is really useful and it's one of the good things about using these processing pool executors.", 'start': 1362.394, 'duration': 6.145}, {'end': 1382.491, 'text': 'So to use this, we can say just for f in concurrent, oops, sorry about my typing there, concurrent dot futures dot as underscore completed.', 'start': 1368.94, 'duration': 13.551}], 'summary': 'Using list comprehension, submit method, and concurrent futures to process tasks efficiently.', 'duration': 44.739, 'max_score': 1337.752, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1337752.jpg'}, {'end': 1465.876, 'src': 'embed', 'start': 1435.492, 'weight': 2, 'content': [{'end': 1437.755, 'text': "So that's why it might take longer.", 'start': 1435.492, 'duration': 2.263}, {'end': 1451.168, 'text': 'But even though it took longer in our simple little example here I still like to use these processing pool executors most of the time because I trust it to allot the processes a lot more than I trust myself.', 'start': 1438.135, 'duration': 13.033}, {'end': 1456.634, 'text': 'So I pass that off to the process pool executor to do and to make that decision for me.', 'start': 1451.529, 'duration': 5.105}, {'end': 1460.875, 'text': "Now to prove that these results are actually coming in as they're completed.", 'start': 1457.014, 'duration': 3.861}, {'end': 1465.876, 'text': 'let me actually pass in a different range of seconds for our processes to sleep.', 'start': 1460.875, 'duration': 5.001}], 'summary': 'Using processing pool executors for better process allocation and decision-making.', 'duration': 30.384, 'max_score': 1435.492, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1435492.jpg'}, {'end': 1559.067, 'src': 'embed', 'start': 1534.043, 'weight': 3, 'content': [{'end': 1543.831, 'text': 'Okay, so now, if I run this, then we can see that it actually started our five second process first, and then our four, then our three, then our two,', 'start': 1534.043, 'duration': 9.788}, {'end': 1544.392, 'text': 'then our one.', 'start': 1543.831, 'duration': 0.561}, {'end': 1548.315, 'text': 'But it finished these in the order that they came in.', 'start': 1545.292, 'duration': 3.023}, {'end': 1551.999, 'text': 'And the lowered number seconds are towards the top.', 'start': 1548.875, 'duration': 3.124}, {'end': 1559.067, 'text': "Now I'm not sure why the one second process took so much longer than the two and the three second processes.", 'start': 1552.119, 'duration': 6.948}], 'summary': 'Processes ran in descending order, but one-second process took longer than others.', 'duration': 25.024, 'max_score': 1534.043, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1534043.jpg'}, {'end': 1650.606, 'src': 'embed', 'start': 1625.806, 'weight': 1, 'content': [{'end': 1633.692, 'text': 'then there is actually something similar that we can do with processes where we can use the map method to run our function over a list of values.', 'start': 1625.806, 'duration': 7.886}, {'end': 1640.157, 'text': "So if you're familiar with the built-in map method, then this is very similar, except it uses processes instead.", 'start': 1634.112, 'duration': 6.045}, {'end': 1646.262, 'text': 'So it runs the function every time or with every item of the interval that we pass in.', 'start': 1640.757, 'duration': 5.505}, {'end': 1650.606, 'text': "So let's say that I want to map our function to our list of seconds.", 'start': 1646.602, 'duration': 4.004}], 'summary': "Using the 'map' method with processes to run functions over a list of values.", 'duration': 24.8, 'max_score': 1625.806, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1625806.jpg'}], 'start': 1337.752, 'title': 'Using concurrent.futures and processing pool executors in python', 'summary': 'Explains the usage of concurrent.futures module to run a submit method with a do something function and an argument of one second, 10 times, and then retrieve the results using the completed function, and explores the usage of processing pool executors in python, demonstrating how to submit tasks, manage their execution, and optimize performance, with emphasis on the order of completion and the time taken for execution.', 'chapters': [{'end': 1409.127, 'start': 1337.752, 'title': 'Using concurrent.futures for parallel processing', 'summary': 'Explains how to use the concurrent.futures module to run a submit method with a do something function and an argument of one second, 10 times, and then retrieve the results using the completed function, showcasing the advantages of processing pool executors.', 'duration': 71.375, 'highlights': ['Using list comprehension to run submit method 10 times with do something function and an argument of one second.', "Utilizing completed function from concurrent futures module to retrieve results as they're completed.", 'Demonstrating the advantage of using processing pool executors for parallel processing.']}, {'end': 1707.789, 'start': 1409.527, 'title': 'Processing pool executors in python', 'summary': 'Explores the usage of processing pool executors in python, demonstrating how to submit tasks, manage their execution, and optimize performance, with emphasis on the order of completion and the time taken for execution.', 'duration': 298.262, 'highlights': ['The processing pool executor took 3 seconds to complete the tasks, potentially due to hardware constraints, showing the trust in its ability to allocate processes effectively.', 'Tasks were completed in the order in which they finished, showcasing the order of completion and the processing efficiency, with the script finishing in about 5 seconds.', 'The map method in processing pool executors runs the function with every item of the iterable passed in, providing a similar functionality to the built-in map method in Python.']}], 'duration': 370.037, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1337752.jpg', 'highlights': ["Utilizing completed function from concurrent futures module to retrieve results as they're completed.", 'The map method in processing pool executors runs the function with every item of the iterable passed in, providing a similar functionality to the built-in map method in Python.', 'Demonstrating the advantage of using processing pool executors for parallel processing.', 'Tasks were completed in the order in which they finished, showcasing the order of completion and the processing efficiency, with the script finishing in about 5 seconds.', 'Using list comprehension to run submit method 10 times with do something function and an argument of one second.', 'The processing pool executor took 3 seconds to complete the tasks, potentially due to hardware constraints, showing the trust in its ability to allocate processes effectively.']}, {'end': 2642.202, 'segs': [{'end': 1781.157, 'src': 'embed', 'start': 1754.122, 'weight': 2, 'content': [{'end': 1761.466, 'text': 'But when you loop over your results using map like we did here, then it returns the results in the order that they were started.', 'start': 1754.122, 'duration': 7.344}, {'end': 1769.11, 'text': 'So since we slept for five seconds first, then we waited for that one to finish before printing out the other results.', 'start': 1762.086, 'duration': 7.024}, {'end': 1771.071, 'text': "But it still didn't slow us down.", 'start': 1769.59, 'duration': 1.481}, {'end': 1774.913, 'text': 'We can see that our entire script still finished in five seconds here.', 'start': 1771.131, 'duration': 3.782}, {'end': 1781.157, 'text': 'But it looks like our five seconds was done sleeping first and then our four, then three, then two, then one.', 'start': 1775.333, 'duration': 5.824}], 'summary': 'Using map, the script finished in 5 seconds, despite varying sleep times.', 'duration': 27.035, 'max_score': 1754.122, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1754122.jpg'}, {'end': 1819.645, 'src': 'embed', 'start': 1794.028, 'weight': 4, 'content': [{'end': 1801.513, 'text': "Now, another thing to point out here is that if our function raises an exception, it won't actually raise that exception while running the process.", 'start': 1794.028, 'duration': 7.485}, {'end': 1807.137, 'text': 'The exception will be raised when its value is retrieved from the results iterator.', 'start': 1801.893, 'duration': 5.244}, {'end': 1813.181, 'text': "So if you need to handle any exceptions, then you can do that here within the iterator if you'd like.", 'start': 1807.537, 'duration': 5.644}, {'end': 1819.645, 'text': "And if you'd like to learn more about handling exceptions, then I do have a more in-depth video if you'd like to learn more about that.", 'start': 1813.561, 'duration': 6.084}], 'summary': 'Exceptions in function are raised when retrieving results iterator.', 'duration': 25.617, 'max_score': 1794.028, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1794028.jpg'}, {'end': 1943.434, 'src': 'embed', 'start': 1917.383, 'weight': 1, 'content': [{'end': 1922.166, 'text': 'So for this multiprocessing example, we want something that is more CPU bound.', 'start': 1917.383, 'duration': 4.783}, {'end': 1927.808, 'text': "So in this script, I'm doing some image processing on the images that we downloaded in the threading video.", 'start': 1922.606, 'duration': 5.202}, {'end': 1934.551, 'text': "And I'll have these images available in the description section below as well in case anyone didn't follow along with that threading video.", 'start': 1928.268, 'duration': 6.283}, {'end': 1940.113, 'text': 'So let me go over this script to show you how someone might do this image processing normally.', 'start': 1935.031, 'duration': 5.082}, {'end': 1943.434, 'text': "So I'm using the pillow library here.", 'start': 1940.533, 'duration': 2.901}], 'summary': 'Demonstrating cpu-bound multiprocessing for image processing using the pillow library.', 'duration': 26.051, 'max_score': 1917.383, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1917383.jpg'}, {'end': 2364.629, 'src': 'embed', 'start': 2341.001, 'weight': 0, 'content': [{'end': 2347.522, 'text': 'So we could see all of those kick off in our activity monitor and we could see that they were all returning faster.', 'start': 2341.001, 'duration': 6.521}, {'end': 2349.223, 'text': "That's because they were running in parallel.", 'start': 2347.602, 'duration': 1.621}, {'end': 2355.144, 'text': 'And now instead of taking 22 seconds like it did before, now it finished in seven seconds.', 'start': 2349.663, 'duration': 5.481}, {'end': 2357.364, 'text': 'So more than a third of the time.', 'start': 2355.224, 'duration': 2.14}, {'end': 2359.885, 'text': "So that's a pretty significant speed up there.", 'start': 2357.644, 'duration': 2.241}, {'end': 2364.629, 'text': 'And this would be even more significant if we were processing even more images.', 'start': 2360.405, 'duration': 4.224}], 'summary': 'Running processes in parallel reduced time from 22 to 7 seconds, a more than a third reduction.', 'duration': 23.628, 'max_score': 2341.001, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso2341001.jpg'}, {'end': 2417.266, 'src': 'embed', 'start': 2380.942, 'weight': 3, 'content': [{'end': 2384.305, 'text': "And it's hard to tell exactly what you should be using without some benchmarks.", 'start': 2380.942, 'duration': 3.363}, {'end': 2385.206, 'text': 'But again,', 'start': 2384.665, 'duration': 0.541}, {'end': 2393.232, 'text': "a good rule of thumb is that you want to use threads for things that are IO bound and you'll want to use processes for things that are CPU bound.", 'start': 2385.206, 'duration': 8.026}, {'end': 2401.698, 'text': 'But you may actually see significant speedups using threads with this example, since this is reading and writing to disk,', 'start': 2393.652, 'duration': 8.046}, {'end': 2403.4, 'text': 'which is an IO bound operation.', 'start': 2401.698, 'duration': 1.702}, {'end': 2417.266, 'text': 'Now, one nice thing about using the concurrent futures library like this is that switching between threads and processes is just as simple as changing this process pull executor and just using a thread pull executor instead.', 'start': 2403.84, 'duration': 13.426}], 'summary': 'Use threads for io bound tasks and processes for cpu bound tasks; consider concurrent futures library for easy switching.', 'duration': 36.324, 'max_score': 2380.942, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso2380942.jpg'}], 'start': 1708.309, 'title': 'Speeding up python scripts', 'summary': 'Covers using map function for parallel processing, demonstrating script execution in five seconds; leveraging multiprocessing for cpu-bound operations, yielding reduced processing time for image processing; and implementing concurrent futures library for more than one-third reduction in processing time, advising on thread and process selection for io and cpu bound tasks.', 'chapters': [{'end': 1793.626, 'start': 1708.309, 'title': 'Using map to process in parallel', 'summary': 'Explains how the map function processes tasks in parallel, returning results in the order they were started, enabling efficient loop iteration and demonstrating a script that finished executing in five seconds.', 'duration': 85.317, 'highlights': ['The map function returns results in the order they were started, allowing efficient loop iteration and avoiding slowdowns, as demonstrated by the script finishing in five seconds.', "Processes kicked off at the same time, except for a one-second delay at the start and didn't all complete simultaneously when using the map function for loop iteration.", 'The script finished executing in five seconds, with the order of results being printed in the order they were started, not in the order they were finished.']}, {'end': 2340.601, 'start': 1794.028, 'title': 'Using multiprocessing in python', 'summary': 'Discusses the use of multiprocessing in python, demonstrating how it can be leveraged for cpu-bound operations and how to implement it for image processing, resulting in a reduction in processing time.', 'duration': 546.573, 'highlights': ["The chapter explains that exceptions raised within a function won't be raised during the process but when its value is retrieved from the results iterator, allowing for exception handling within the iterator.", 'It demonstrates the automatic joining of processes and their completion after the context manager ends, ensuring that the processes are completed before the context manager finishes.', 'The chapter provides a real-world example of using multiple processes for CPU-bound operations like image processing, showcasing a reduction in processing time from 22 seconds to a faster, parallel processing method using multiprocessing.', 'It outlines the transformation of a script using a for loop for processing images to using a function and a process pool executor to leverage multiprocessing for parallel processing, resulting in faster processing times and the utilization of multiple Python processes.']}, {'end': 2642.202, 'start': 2341.001, 'title': 'Speeding up scripts with threading and multiprocessing', 'summary': 'Demonstrates the use of concurrent futures library to speed up scripts, achieving a more than one-third reduction in processing time by running tasks in parallel, and advises on the selection of threads for io bound tasks and processes for cpu bound tasks, while highlighting the benefits of experimenting with different approaches to maximize performance.', 'duration': 301.201, 'highlights': ['The use of concurrent futures library reduced the processing time from 22 seconds to 7 seconds, achieving a speedup of more than one-third, by running tasks in parallel.', 'The advice to use threads for IO bound tasks and processes for CPU bound tasks is emphasized, with the recommendation to experiment with different approaches for better performance.', 'A demonstration of switching between thread and process pool executors using concurrent futures library showcases the ease of transitioning between threads and processes for different tasks.', 'The mention of Brilliant.org as a sponsor and its relevance to threading, multiprocessing, and data science is highlighted, along with an offer for free sign up and a discount for premium subscription through a provided link.', 'The chapter concludes by summarizing the key learnings and offering support through various means, including subscribing, sharing, and contributing through Patreon.']}], 'duration': 933.893, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/fKl2JW_qrso/pics/fKl2JW_qrso1708309.jpg', 'highlights': ['The use of concurrent futures library reduced processing time from 22 seconds to 7 seconds, achieving a speedup of more than one-third.', 'The chapter provides a real-world example of using multiple processes for CPU-bound operations like image processing, showcasing a reduction in processing time from 22 seconds to a faster, parallel processing method using multiprocessing.', 'The map function returns results in the order they were started, allowing efficient loop iteration and avoiding slowdowns, as demonstrated by the script finishing in five seconds.', 'The advice to use threads for IO bound tasks and processes for CPU bound tasks is emphasized, with the recommendation to experiment with different approaches for better performance.', "The chapter explains that exceptions raised within a function won't be raised during the process but when its value is retrieved from the results iterator, allowing for exception handling within the iterator."]}], 'highlights': ['The use of concurrent futures library reduced processing time from 22 seconds to 7 seconds, achieving a speedup of more than one-third.', 'The chapter demonstrates the creation, start, and join methods to run processes in parallel, achieving a significant speed improvement, and passing arguments to multiprocessing processes, allowing the script to finish in about 1.5 seconds (10 times) as compared to running synchronously, which would take 10 seconds.', 'The processing pool executor took 3 seconds to complete the tasks, potentially due to hardware constraints, showing the trust in its ability to allocate processes effectively.', 'The chapter explains the concept of running code in parallel using the multiprocessing module and mentions a previous video on running code concurrently using the threading module.', 'The multiprocessing module is used to split tasks onto other CPUs and run them simultaneously, allowing for asynchronous execution.', 'The process pool executor in Python 3.2 is a more efficient way to run multiple processes, allowing easy switching to multiple threads.', "Utilizing completed function from concurrent futures module to retrieve results as they're completed.", 'The map method in processing pool executors runs the function with every item of the iterable passed in, providing a similar functionality to the built-in map method in Python.', 'The chapter provides a real-world example of using multiple processes for CPU-bound operations like image processing, showcasing a reduction in processing time from 22 seconds to a faster, parallel processing method using multiprocessing.', 'The advice to use threads for IO bound tasks and processes for CPU bound tasks is emphasized, with the recommendation to experiment with different approaches for better performance.']}