title
Cyber Python 2077 - Using computer vision to read and walk from Cyberpunk 2077 map

description
Using some basic computer vision techniques with #Python and #OpenCV to have a simple AI walk around in #cyberpunk2077. Series playlist: https://www.youtube.com/playlist?list=PLQVvvaa0QuDdcI_Z6S1nbWyOfqY48SKnM Sample code: https://github.com/Sentdex/cyberpython2077 Keys file: https://github.com/Sentdex/pygta5/blob/master/keys.py Grabscreen file: https://github.com/Sentdex/pygta5/blob/master/grabscreen.py Channel membership: https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ/join Discord: https://discord.gg/sentdex Support the content: https://pythonprogramming.net/support-donate/ Twitter: https://twitter.com/sentdex Instagram: https://instagram.com/sentdex Facebook: https://www.facebook.com/pythonprogramming.net/ Twitch: https://www.twitch.tv/sentdex

detail
{'title': 'Cyber Python 2077 - Using computer vision to read and walk from Cyberpunk 2077 map', 'heatmap': [{'end': 1727.02, 'start': 1683.83, 'weight': 0.839}], 'summary': 'Explores ai and automation in cyberpunk 2077, using computer vision and python scripts for automated walking, gaming path detection with opencv, color matching, hue calculation, path detection, image processing logic optimization, and game navigation automation with insights for cyberpunk 2077, emphasizing the need for high frame rates and discussing game pathing techniques.', 'chapters': [{'end': 287.273, 'segs': [{'end': 222.877, 'src': 'embed', 'start': 195.367, 'weight': 1, 'content': [{'end': 200.128, 'text': 'And then keys, this was written by Daniel just to let us control both the mouse and the keys.', 'start': 195.367, 'duration': 4.761}, {'end': 201.668, 'text': "So I'm going to grab these two.", 'start': 200.788, 'duration': 0.88}, {'end': 203.549, 'text': 'I will put links to both in the description.', 'start': 201.768, 'duration': 1.781}, {'end': 205.049, 'text': 'So if you want to follow along, you can.', 'start': 203.629, 'duration': 1.42}, {'end': 207.709, 'text': 'Also, if you want to follow along in some other game, you totally can.', 'start': 205.149, 'duration': 2.56}, {'end': 209.35, 'text': "You don't have to run Cyberpunk 2077 to follow along.", 'start': 207.729, 'duration': 1.621}, {'end': 213.832, 'text': 'lots of games have mini maps, lots of games have pathing.', 'start': 211.09, 'duration': 2.742}, {'end': 219.795, 'text': "on the map, you should be able to use all of the same techniques that we're going to use here with, like any game that you want.", 'start': 213.832, 'duration': 5.963}, {'end': 222.877, 'text': 'um, just hopefully the pathing is is decent.', 'start': 219.795, 'duration': 3.082}], 'summary': 'Daniel offers techniques for controlling mouse and keys in games, applicable to various games with mini maps and pathing.', 'duration': 27.51, 'max_score': 195.367, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ195367.jpg'}, {'end': 302.322, 'src': 'embed', 'start': 271.081, 'weight': 0, 'content': [{'end': 274.143, 'text': "but anyway, what we're gonna try to do is we're gonna try to follow this yellow path.", 'start': 271.081, 'duration': 3.062}, {'end': 280.468, 'text': 'but yeah, you could in theory like, if you look at the mini map here, all the roads are marked with this like blue color,', 'start': 274.143, 'duration': 6.325}, {'end': 282.129, 'text': "and that's the same across the game.", 'start': 280.468, 'duration': 1.661}, {'end': 287.273, 'text': 'so different games are going to have different kind of methodologies for showing paths and things on the mini map,', 'start': 282.129, 'duration': 5.144}, {'end': 291.977, 'text': 'but in general they tend to be pretty standardized in that specific game.', 'start': 287.273, 'duration': 4.704}, {'end': 296.699, 'text': "so anyway, what we're going to try to do is follow that yellow path.", 'start': 291.977, 'duration': 4.722}, {'end': 302.322, 'text': 'so the first order of business we need to see the game, so we grab the screen for the game, then we need to see just the mini map.', 'start': 296.699, 'duration': 5.623}], 'summary': 'Following the yellow path on the standardized mini map in the game.', 'duration': 31.241, 'max_score': 271.081, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ271081.jpg'}], 'start': 2.43, 'title': 'Ai and automation in cyberpunk 2077', 'summary': 'Explores achieving ai functionality and automated walking in cyberpunk 2077, using computer vision techniques for navigation and python scripts for automatic movement, emphasizing the need for high frame rates and discussing game pathing techniques.', 'chapters': [{'end': 101.05, 'start': 2.43, 'title': 'Ai functionality in cyberpunk 2077', 'summary': 'Explores the possibility of achieving ai functionality in cyberpunk 2077 using computer vision techniques, particularly for navigating on foot through the city, by following the minimap.', 'duration': 98.62, 'highlights': ['The chapter discusses the potential of developing AI in Cyberpunk 2077 using computer vision techniques, particularly for navigating on foot through the city.', 'The focus is on the use of the minimap for navigating on foot in the game, as it appears to have pretty good pathing.', 'The chapter mentions the comparison with GTA 5 and highlights the difference in focus between the two games, emphasizing the significance of being on foot in Cyberpunk 2077.']}, {'end': 195.307, 'start': 101.591, 'title': 'Automated walking in cyberpunk 2077', 'summary': 'Outlines a methodology for automatic movement in cyberpunk 2077 using python scripts for grabbing the screen and direct input, emphasizing the need for high frame rates and providing links to the necessary scripts.', 'duration': 93.716, 'highlights': ['The chapter discusses the methodology for automatic walking in Cyberpunk 2077 using Python scripts for grabbing the screen and direct input, with an emphasis on achieving high frame rates.', 'Emphasizes the need for grabbing the screen and direct input for automatic walking in Cyberpunk 2077, highlighting the importance of high frame rates for efficient reading of frames.', 'Mentions the use of Python scripts for grabbing the screen and direct input as the suitable method for automatic walking in Cyberpunk 2077, due to the requirement of high frame rates.']}, {'end': 287.273, 'start': 195.367, 'title': 'Game pathing techniques', 'summary': 'Discusses techniques for navigating game paths using cyberpunk 2077 as an example, including controlling mouse and keys, navigating paths on foot and by car, and challenges faced in demonstrating the game due to cpu heat.', 'duration': 91.906, 'highlights': ['The chapter discusses techniques for navigating game paths using Cyberpunk 2077 as an example, including controlling mouse and keys, navigating paths on foot and by car, and challenges faced in demonstrating the game due to CPU heat.', "The game's CPU gets very hot during gameplay, causing the fans to generate significant noise.", 'Different games may have varying methodologies for displaying paths and landmarks on the mini-map, often using distinct colors for roads and paths.']}], 'duration': 284.843, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ2430.jpg', 'highlights': ['The chapter discusses the potential of developing AI in Cyberpunk 2077 using computer vision techniques, particularly for navigating on foot through the city.', 'The chapter discusses the methodology for automatic walking in Cyberpunk 2077 using Python scripts for grabbing the screen and direct input, with an emphasis on achieving high frame rates.', 'The chapter discusses techniques for navigating game paths using Cyberpunk 2077 as an example, including controlling mouse and keys, navigating paths on foot and by car, and challenges faced in demonstrating the game due to CPU heat.']}, {'end': 622.772, 'segs': [{'end': 313.548, 'src': 'embed', 'start': 287.273, 'weight': 1, 'content': [{'end': 291.977, 'text': 'but in general they tend to be pretty standardized in that specific game.', 'start': 287.273, 'duration': 4.704}, {'end': 296.699, 'text': "so anyway, what we're going to try to do is follow that yellow path.", 'start': 291.977, 'duration': 4.722}, {'end': 302.322, 'text': 'so the first order of business we need to see the game, so we grab the screen for the game, then we need to see just the mini map.', 'start': 296.699, 'duration': 5.623}, {'end': 309.666, 'text': "so then we're going to use a region of interest with just opencv slice, get just the mini map, then we want to see the path right.", 'start': 302.322, 'duration': 7.344}, {'end': 313.548, 'text': "so then we're going to mask and find that yellow color and then we'll go from there.", 'start': 309.666, 'duration': 3.882}], 'summary': 'Standardized game paths; extract mini-map, find yellow path using opencv.', 'duration': 26.275, 'max_score': 287.273, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ287273.jpg'}, {'end': 502.848, 'src': 'embed', 'start': 458.605, 'weight': 3, 'content': [{'end': 460.606, 'text': 'Actually, let me think about this.', 'start': 458.605, 'duration': 2.001}, {'end': 461.567, 'text': "I think it's actually YX.", 'start': 460.626, 'duration': 0.941}, {'end': 468.792, 'text': "Sorry, so so there's only make sense, because why start to the tip at the top?", 'start': 461.647, 'duration': 7.145}, {'end': 470.213, 'text': "so it's actually what your exo.", 'start': 468.792, 'duration': 1.421}, {'end': 481.813, 'text': 'anyone to 377 and then on for the X coordinates, its 2181, 224, 69 on, To figure out what your coordinates are.', 'start': 470.213, 'duration': 11.6}, {'end': 483.474, 'text': 'just trial and error it honestly.', 'start': 481.813, 'duration': 1.661}, {'end': 484.835, 'text': 'Just paste in the ranges.', 'start': 483.514, 'duration': 1.321}, {'end': 487.197, 'text': 'Take your best guess to start and then hone it in.', 'start': 484.855, 'duration': 2.342}, {'end': 490.299, 'text': "So at least in this game, let's pull up the mini map.", 'start': 487.617, 'duration': 2.682}, {'end': 496.083, 'text': "As you can see, the mini map in Cyberpunk 2077 is like not a perfect square, right? It's a little distorted.", 'start': 490.719, 'duration': 5.364}, {'end': 498.385, 'text': "It's not like a tilted square.", 'start': 496.583, 'duration': 1.802}, {'end': 501.467, 'text': "It's like, I don't know what to describe it besides distorted.", 'start': 498.445, 'duration': 3.022}, {'end': 502.848, 'text': "It's not a perfect thing.", 'start': 501.607, 'duration': 1.241}], 'summary': 'In cyberpunk 2077, coordinates are 2181, 224, 69 for x, y, z respectively, and the mini map is distorted.', 'duration': 44.243, 'max_score': 458.605, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ458605.jpg'}, {'end': 570.923, 'src': 'embed', 'start': 526.603, 'weight': 0, 'content': [{'end': 531.766, 'text': "So I'm going to I think we'll just..", 'start': 526.603, 'duration': 5.163}, {'end': 536.871, 'text': "I'm going to copy this over and explain what this is.", 'start': 531.766, 'duration': 5.105}, {'end': 540.655, 'text': "Because, again, your values are going to change if you're using some other game or something like that.", 'start': 536.891, 'duration': 3.764}, {'end': 545.279, 'text': "I'm just assuming probably most people watching are probably just simply watching out of curiosity.", 'start': 540.675, 'duration': 4.604}, {'end': 548.162, 'text': 'And then some people are going to probably apply it to another game.', 'start': 545.7, 'duration': 2.462}, {'end': 551.025, 'text': 'And then maybe 1% of you are actually following along in Cyberpunk.', 'start': 548.182, 'duration': 2.843}, {'end': 554.51, 'text': "So.. I think we'll just do it this way.", 'start': 551.065, 'duration': 3.445}, {'end': 560.921, 'text': "So for masking, again, kind of like selecting this, I'm going to be honest with you, the way I go about doing it..", 'start': 555.572, 'duration': 5.349}, {'end': 563.021, 'text': 'is the following.', 'start': 562.341, 'duration': 0.68}, {'end': 568.003, 'text': "So first of all, when you mask, you're actually first going to convert to HSV.", 'start': 563.421, 'duration': 4.582}, {'end': 570.923, 'text': 'Now HSV is hue saturation value.', 'start': 568.583, 'duration': 2.34}], 'summary': 'Demonstrating masking technique with hsv conversion for cyberpunk game, with 1% audience participation.', 'duration': 44.32, 'max_score': 526.603, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ526603.jpg'}], 'start': 287.273, 'title': 'Gaming path detection with opencv and screen grab and pathing in python', 'summary': 'Discusses using opencv to detect and follow the yellow path in a game, involving steps such as grabbing the game screen, isolating the mini map, finding and following the yellow path, while also covering the process of grabbing the screen for 100 frames, adjusting the region depending on game resolution, converting to rgb, resizing, and defining pathing function for mini-map analysis in python, with insights for cyberpunk 2077.', 'chapters': [{'end': 323.702, 'start': 287.273, 'title': 'Gaming path detection with opencv', 'summary': 'Discusses the process of using opencv to detect and follow the yellow path in a game, involving steps such as grabbing the game screen, isolating the mini map, and finding and following the yellow path.', 'duration': 36.429, 'highlights': ['Using OpenCV to grab the game screen and isolate the mini map for path detection', 'Applying a region of interest and mask to find the yellow color and follow the path', 'Implementing a technique to walk along the detected path']}, {'end': 622.772, 'start': 323.702, 'title': 'Screen grab and pathing in python', 'summary': 'Covers the process of grabbing the screen for 100 frames, adjusting the region depending on game resolution, converting to rgb, resizing, and defining pathing function for mini-map analysis in python, sharing insights for cyberpunk 2077.', 'duration': 299.07, 'highlights': ['The process involves grabbing the screen for 100 frames, adjusting the region depending on game resolution, converting to RGB, and resizing for display, achieving a decent frame rate sufficient for the task at hand.', 'The chapter demonstrates the procedure for defining the pathing function for mini-map analysis, including the process of selecting the region of interest and explaining the X and Y coordinates for the mini-map in Cyberpunk 2077.', 'Insights are shared on the approach to mask selection, including converting to HSV and adjusting the HSV color picker values to fit the OpenCV range of 0-255, with a note for alternative methods and the utilization of NumPy for additional functionality.']}], 'duration': 335.499, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ287273.jpg', 'highlights': ['Using OpenCV to grab the game screen and isolate the mini map for path detection', 'Applying a region of interest and mask to find the yellow color and follow the path', 'Implementing a technique to walk along the detected path', 'The process involves grabbing the screen for 100 frames, adjusting the region depending on game resolution, converting to RGB, and resizing for display, achieving a decent frame rate sufficient for the task at hand', 'The chapter demonstrates the procedure for defining the pathing function for mini-map analysis, including the process of selecting the region of interest and explaining the X and Y coordinates for the mini-map in Cyberpunk 2077', 'Insights are shared on the approach to mask selection, including converting to HSV and adjusting the HSV color picker values to fit the OpenCV range of 0-255, with a note for alternative methods and the utilization of NumPy for additional functionality']}, {'end': 813.583, 'segs': [{'end': 737.232, 'src': 'embed', 'start': 694.864, 'weight': 1, 'content': [{'end': 696.546, 'text': 'We can run that again.', 'start': 694.864, 'duration': 1.682}, {'end': 699.068, 'text': 'And sure enough, what we see is, okay, great.', 'start': 697.106, 'duration': 1.962}, {'end': 700.81, 'text': 'So we filtered out quite a bit.', 'start': 699.168, 'duration': 1.642}, {'end': 702.532, 'text': 'We can still see our character color.', 'start': 700.87, 'duration': 1.662}, {'end': 706.796, 'text': 'And then we can see some of the road it looked like.', 'start': 703.713, 'duration': 3.083}, {'end': 710.54, 'text': 'And just some general noise.', 'start': 708.939, 'duration': 1.601}, {'end': 712.641, 'text': 'We still want to kind of like filter those out.', 'start': 710.92, 'duration': 1.721}, {'end': 714.341, 'text': "So again, we'll just do the same thing again.", 'start': 712.801, 'duration': 1.54}, {'end': 718.163, 'text': 'Okay, we do 150.', 'start': 714.702, 'duration': 3.461}, {'end': 718.964, 'text': "And then we'll run that again.", 'start': 718.163, 'duration': 0.801}, {'end': 725.867, 'text': "And then, okay, at this point, it looks like we've actually filtered out basically everything except for the character on the map.", 'start': 719.684, 'duration': 6.183}, {'end': 728.328, 'text': 'And then we have the path that we want.', 'start': 726.427, 'duration': 1.901}, {'end': 731.169, 'text': 'Well, the character color is like this like blue.', 'start': 728.428, 'duration': 2.741}, {'end': 733.07, 'text': 'Let me pull it up real quick for you.', 'start': 731.349, 'duration': 1.721}, {'end': 737.232, 'text': 'Bloop So the character color on this map is a blue and the path is a yellow.', 'start': 733.31, 'duration': 3.922}], 'summary': 'Filtering process successfully eliminates noise, leaving a blue character and yellow path on the map.', 'duration': 42.368, 'max_score': 694.864, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ694864.jpg'}, {'end': 813.583, 'src': 'embed', 'start': 789.87, 'weight': 0, 'content': [{'end': 798.014, 'text': 'Okay, now that we have that, we have to talk about what is the actual logic that we intend to use.', 'start': 789.87, 'duration': 8.144}, {'end': 800.516, 'text': 'So pulling up the game.', 'start': 798.555, 'duration': 1.961}, {'end': 811.242, 'text': 'So what we really want to do is we want to follow this path, but we need to somehow formulate programming logic that will follow that path for us,', 'start': 801.016, 'duration': 10.226}, {'end': 813.583, 'text': 'based visually on what the map looks like.', 'start': 811.242, 'duration': 2.341}], 'summary': 'Discussion on formulating programming logic to follow a specific path visually on the map.', 'duration': 23.713, 'max_score': 789.87, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ789870.jpg'}], 'start': 622.792, 'title': 'Color matching, hue calculation, path detection, and color filtering', 'summary': 'Covers color matching and hue calculation with a range of 0 to 255 for hue, saturation, and value, including a preliminary calculation using approximately 127.5. it also explores path detection by color filtering to identify and follow paths while excluding unwanted colors.', 'chapters': [{'end': 662.501, 'start': 622.792, 'title': 'Color matching and hue calculation', 'summary': 'Discusses color matching and hue calculation, detailing the range of 0 to 255 for hue, saturation, and value, and a preliminary hue calculation using a large number of approximately 127.5.', 'duration': 39.709, 'highlights': ['The range of 0 to 255 for hue, saturation, and value is discussed, indicating a comprehensive color matching system.', 'A preliminary hue calculation approach is described, using a large number of approximately 127.5 for color range evaluation.']}, {'end': 813.583, 'start': 665.083, 'title': 'Path detection and color filtering', 'summary': 'Discusses the process of color filtering to detect a path, filtering out unwanted colors and determining the logic to follow the detected path.', 'duration': 148.5, 'highlights': ['The path detection process involved filtering colors in the range of 0 to 150 for both hue and saturation, effectively isolating the character color and the desired path. By filtering colors in the range of 0 to 150 for hue and saturation, the chapter successfully isolated the character color and the desired path, effectively detecting the path.', 'The adjustment of hue values, specifically in the range of 75 to 150, played a crucial role in further refining the color filtering process to isolate the desired path. The adjustment of hue values, particularly in the range of 75 to 150, significantly contributed to further refining the color filtering process, enhancing the isolation of the desired path.', 'The discussion also revolves around formulating programming logic to visually follow the detected path within the game. The chapter delves into formulating programming logic to visually follow the detected path within the game, addressing the practical application of the color filtering process.']}], 'duration': 190.791, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ622792.jpg', 'highlights': ['The range of 0 to 255 for hue, saturation, and value is discussed, indicating a comprehensive color matching system.', 'The adjustment of hue values, specifically in the range of 75 to 150, played a crucial role in further refining the color filtering process to isolate the desired path.', 'The path detection process involved filtering colors in the range of 0 to 150 for both hue and saturation, effectively isolating the character color and the desired path.']}, {'end': 1193.461, 'segs': [{'end': 1005.91, 'src': 'embed', 'start': 951.42, 'weight': 0, 'content': [{'end': 952.341, 'text': "Let's run that real quick.", 'start': 951.42, 'duration': 0.921}, {'end': 954.462, 'text': "And now it's so small.", 'start': 952.821, 'duration': 1.641}, {'end': 957.184, 'text': "Oh no, I can't get it.", 'start': 956.284, 'duration': 0.9}, {'end': 960.206, 'text': "It's like up in the corner.", 'start': 959.126, 'duration': 1.08}, {'end': 962.368, 'text': 'I want to make that way bigger.', 'start': 961.147, 'duration': 1.221}, {'end': 965.825, 'text': "Okay, maybe I've got it that way.", 'start': 964.144, 'duration': 1.681}, {'end': 967.967, 'text': 'So anyway, now we have the mini map.', 'start': 966.426, 'duration': 1.541}, {'end': 973.472, 'text': 'And now if we zoom in enough, we get just like the few dots right ahead of us.', 'start': 968.267, 'duration': 5.205}, {'end': 976.114, 'text': "So I think that's probably better.", 'start': 974.272, 'duration': 1.842}, {'end': 978.116, 'text': 'So as we approach, hopefully I can make it in time.', 'start': 976.174, 'duration': 1.942}, {'end': 984.321, 'text': 'As we approach this turn, like if we centered those, we would probably..', 'start': 978.376, 'duration': 5.945}, {'end': 987.526, 'text': "Like I'm trying to just walk based on what I'm looking at on the thing.", 'start': 984.801, 'duration': 2.725}, {'end': 989.951, 'text': "I think that that's like good enough to get around there.", 'start': 987.847, 'duration': 2.104}, {'end': 993.858, 'text': "So now we'll find out.", 'start': 991.674, 'duration': 2.184}, {'end': 995.06, 'text': "Oh, now it's gone.", 'start': 994.419, 'duration': 0.641}, {'end': 1002.748, 'text': 'I legit lost the path on the game for some reason.', 'start': 999.406, 'duration': 3.342}, {'end': 1005.91, 'text': "Anyway, so we'll have to handle for that kind of stuff at a later date.", 'start': 1002.968, 'duration': 2.942}], 'summary': 'Struggling to adjust and navigate the mini map to reach the destination in time while encountering issues with path visibility.', 'duration': 54.49, 'max_score': 951.42, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ951420.jpg'}, {'end': 1179.047, 'src': 'embed', 'start': 1142.148, 'weight': 4, 'content': [{'end': 1145.93, 'text': "But as it goes this way, right, the x's will be, you know, to the left.", 'start': 1142.148, 'duration': 3.782}, {'end': 1148.491, 'text': "So it'll be less than.", 'start': 1145.97, 'duration': 2.521}, {'end': 1151.352, 'text': 'And then this would be perfectly center.', 'start': 1149.051, 'duration': 2.301}, {'end': 1153.633, 'text': 'And then these would be like greater than center values.', 'start': 1151.372, 'duration': 2.261}, {'end': 1155.534, 'text': 'So we want it to be in the perfect center.', 'start': 1154.114, 'duration': 1.42}, {'end': 1158.176, 'text': 'So first we want to know what is perfect center.', 'start': 1156.295, 'duration': 1.881}, {'end': 1160.537, 'text': 'what are the current locations?', 'start': 1159.076, 'duration': 1.461}, {'end': 1162.119, 'text': 'and then, like what is the delta?', 'start': 1160.537, 'duration': 1.582}, {'end': 1164.101, 'text': 'essentially, and that will inform us.', 'start': 1162.119, 'duration': 1.982}, {'end': 1167.364, 'text': 'which way do we want the mouse to turn?', 'start': 1164.101, 'duration': 3.263}, {'end': 1179.047, 'text': "so, coming back to sublime text, and i think we probably want, We'll want this window, but I'm going to get rid of it for now.", 'start': 1167.364, 'duration': 11.683}], 'summary': 'Discussing positioning and alignment for mouse movement in the context of the code editor.', 'duration': 36.899, 'max_score': 1142.148, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ1142148.jpg'}], 'start': 815.942, 'title': 'Optimizing character pathing and image processing logic', 'summary': 'Discusses zooming in the mini-map for better navigation and avoiding obstacles, and the process of centering pixels to achieve perfect centering.', 'chapters': [{'end': 1005.91, 'start': 815.942, 'title': 'Optimizing character pathing in mini-map', 'summary': 'Discusses optimizing character pathing in the mini-map by zooming in to focus on a smaller area, leading to better navigation and avoiding obstacles.', 'duration': 189.968, 'highlights': ["By zooming in on the mini-map and focusing on a smaller area, the character's pathing can be optimized, leading to better navigation and avoidance of obstacles.", "The character's movement is improved by zooming in on the mini-map, allowing for better pathing and navigation around obstacles.", "The approach of zooming in on the mini-map and focusing on a smaller area improves the character's ability to navigate and avoid obstacles effectively."]}, {'end': 1193.461, 'start': 1007.19, 'title': 'Image processing logic for centering pixels', 'summary': 'Discusses the process of finding and analyzing the coordinate locations of white pixels in an image array to determine the off-center distance and direction, aiming to achieve perfect centering.', 'duration': 186.271, 'highlights': ['The process involves finding the coordinate location of all the white pixels in the image mask, represented by a value of 255, using np.argwhere, resulting in an array of matches.', 'The coordinate locations obtained are in the format y, x, and the focus is on determining the x values to achieve perfect centering, with the delta informing the mouse movement direction.', 'The next step is to calculate the average match, referred to as meanY, to further inform the adjustment needed for centering the pixels.']}], 'duration': 377.519, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ815942.jpg', 'highlights': ["By zooming in on the mini-map, the character's pathing can be optimized for better navigation and obstacle avoidance.", "The character's movement is improved by zooming in on the mini-map, allowing for better pathing and navigation around obstacles.", "The approach of zooming in on the mini-map improves the character's ability to navigate and avoid obstacles effectively.", 'The process involves finding the coordinate location of all the white pixels in the image mask, represented by a value of 255, using np.argwhere, resulting in an array of matches.', 'The coordinate locations obtained are in the format y, x, and the focus is on determining the x values to achieve perfect centering, with the delta informing the mouse movement direction.', 'The next step is to calculate the average match, referred to as meanY, to further inform the adjustment needed for centering the pixels.']}, {'end': 1997.667, 'segs': [{'end': 1247.304, 'src': 'embed', 'start': 1193.521, 'weight': 3, 'content': [{'end': 1196.444, 'text': "And in this case, it's going to be a numpy.mean.", 'start': 1193.521, 'duration': 2.923}, {'end': 1200.184, 'text': 'of the matches.', 'start': 1198.342, 'duration': 1.842}, {'end': 1205.871, 'text': "but we want just the x elements and to do this with like a 2d array, for example, it's going to be colon comma.", 'start': 1200.184, 'duration': 5.687}, {'end': 1208.455, 'text': 'whoops, colon comma comma.', 'start': 1205.871, 'duration': 2.584}, {'end': 1213.622, 'text': "um, the index that we want to use, so it's gonna be colon comma one.", 'start': 1208.455, 'duration': 5.167}, {'end': 1224.349, 'text': 'Okay, so once we have that, that will be the mean of all of the x values of those matches.', 'start': 1215.745, 'duration': 8.604}, {'end': 1232.992, 'text': "So this will tell us what the mean is, and then we just want to know what is our goal value, right? So we're going to say target.", 'start': 1224.969, 'duration': 8.023}, {'end': 1234.353, 'text': 'Is that what I call it? Yeah, I call it target.', 'start': 1233.132, 'duration': 1.221}, {'end': 1244.161, 'text': "Equals and in this case it will be whatever the mini map is dot shape so the shape of the mini map and again It's going to be in I think YX.", 'start': 1234.933, 'duration': 9.228}, {'end': 1247.304, 'text': 'Yeah, YX, and so we want to know the first.', 'start': 1244.281, 'duration': 3.023}], 'summary': 'Using numpy.mean to find mean of x values in 2d array for target in mini map shape yx.', 'duration': 53.783, 'max_score': 1193.521, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ1193521.jpg'}, {'end': 1576.183, 'src': 'embed', 'start': 1546.058, 'weight': 6, 'content': [{'end': 1548.059, 'text': 'I think I need to make that timer a little longer.', 'start': 1546.058, 'duration': 2.001}, {'end': 1551.16, 'text': 'Okay, so we got an error.', 'start': 1549.86, 'duration': 1.3}, {'end': 1555.343, 'text': 'Okay, a couple of things happened.', 'start': 1554.422, 'duration': 0.921}, {'end': 1558.224, 'text': 'So let me pull up the console.', 'start': 1555.383, 'duration': 2.841}, {'end': 1559.004, 'text': 'So this is what we got.', 'start': 1558.244, 'duration': 0.76}, {'end': 1561.606, 'text': 'It was walking, I assure you.', 'start': 1560.105, 'duration': 1.501}, {'end': 1563.867, 'text': 'This is the error that we saw.', 'start': 1562.826, 'duration': 1.041}, {'end': 1569.05, 'text': "I think it's because we actually ended up walking beyond the path that we intended to take.", 'start': 1564.007, 'duration': 5.043}, {'end': 1571.497, 'text': 'Um, so there was no path.', 'start': 1570.095, 'duration': 1.402}, {'end': 1573.76, 'text': 'So we would want to have some sort of handling for that.', 'start': 1571.978, 'duration': 1.782}, {'end': 1576.183, 'text': "There's going to be times where who knows what happens.", 'start': 1573.8, 'duration': 2.383}], 'summary': 'Debugging process revealed error due to walking beyond intended path, highlighting need for better handling.', 'duration': 30.125, 'max_score': 1546.058, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ1546058.jpg'}, {'end': 1727.02, 'src': 'heatmap', 'start': 1683.83, 'weight': 0.839, 'content': [{'end': 1688.355, 'text': 'Let me pull up close sublime, pull up this.', 'start': 1683.83, 'duration': 4.525}, {'end': 1693.93, 'text': "And I think that's good enough.", 'start': 1693.149, 'duration': 0.781}, {'end': 1694.671, 'text': "Let's run it.", 'start': 1694.291, 'duration': 0.38}, {'end': 1697.716, 'text': "Okay So I'll start like this too.", 'start': 1696.254, 'duration': 1.462}, {'end': 1698.317, 'text': "Let's just see.", 'start': 1697.756, 'duration': 0.561}, {'end': 1700.06, 'text': 'And also see like that path is not.', 'start': 1698.577, 'duration': 1.483}, {'end': 1700.981, 'text': "Okay That's good enough.", 'start': 1700.24, 'duration': 0.741}, {'end': 1701.281, 'text': 'I guess.', 'start': 1701.021, 'duration': 0.26}, {'end': 1703.365, 'text': "Let's see if he moves quickly enough.", 'start': 1702.203, 'duration': 1.162}, {'end': 1705.368, 'text': "It's so juddery.", 'start': 1704.606, 'duration': 0.762}, {'end': 1709.651, 'text': 'Okay Yeah, all right.', 'start': 1707.391, 'duration': 2.26}, {'end': 1711.132, 'text': 'That appears to work.', 'start': 1710.032, 'duration': 1.1}, {'end': 1712.893, 'text': 'We could probably run it a little longer.', 'start': 1711.192, 'duration': 1.701}, {'end': 1721.077, 'text': 'We might even want to move a larger error, but that at least took the turn decently.', 'start': 1712.913, 'duration': 8.164}, {'end': 1722.398, 'text': 'Again, pretty juddery.', 'start': 1721.457, 'duration': 0.941}, {'end': 1723.839, 'text': 'We would want to smooth that mouse down.', 'start': 1722.418, 'duration': 1.421}, {'end': 1727.02, 'text': 'But other than that, I think it actually worked pretty well.', 'start': 1724.759, 'duration': 2.261}], 'summary': 'Tested motion, encountered juddery movement, but overall successful.', 'duration': 43.19, 'max_score': 1683.83, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ1683830.jpg'}, {'end': 1953.518, 'src': 'embed', 'start': 1911.445, 'weight': 0, 'content': [{'end': 1917.769, 'text': 'so you could probably get around it doing that and i i think you would be able to probably navigate with a really high percentage.', 'start': 1911.445, 'duration': 6.324}, {'end': 1920.33, 'text': 'you know like 70 or 95 percent of the time.', 'start': 1917.769, 'duration': 2.561}, {'end': 1924.233, 'text': "i think that's enough to to do everything on foot that you would need to do on this game.", 'start': 1920.33, 'duration': 3.903}, {'end': 1926.815, 'text': "So that's pretty cool.", 'start': 1926.155, 'duration': 0.66}, {'end': 1932.743, 'text': "I think the only major downside is like if you happen to be in an area that doesn't have a sidewalk,", 'start': 1927.236, 'duration': 5.507}, {'end': 1936.387, 'text': 'it does put the path for you to walk like literally in the middle of the street.', 'start': 1932.743, 'duration': 3.644}, {'end': 1937.288, 'text': "That's kind of a problem.", 'start': 1936.407, 'duration': 0.881}, {'end': 1941.153, 'text': "So yeah, but there's really nowhere else for you to walk.", 'start': 1939.13, 'duration': 2.023}, {'end': 1941.834, 'text': "So I don't know.", 'start': 1941.213, 'duration': 0.621}, {'end': 1942.515, 'text': "I'm not really sure.", 'start': 1941.934, 'duration': 0.581}, {'end': 1945.616, 'text': "Okay, I think that's a good stopping point.", 'start': 1944.075, 'duration': 1.541}, {'end': 1946.896, 'text': 'Hopefully you guys enjoyed.', 'start': 1945.936, 'duration': 0.96}, {'end': 1953.518, 'text': 'If you have any ideas or requests for something that we could do in this game, feel free to let me know.', 'start': 1946.956, 'duration': 6.562}], 'summary': 'A high navigation success rate of 70-95% on foot in the game, but potential issues with sidewalks and path placement.', 'duration': 42.073, 'max_score': 1911.445, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ1911445.jpg'}], 'start': 1193.521, 'title': 'Image processing error calculation and game navigation automation', 'summary': 'Demonstrates using numpy.mean to calculate error in image processing with a 2d array, determining target value and calculating error, resulting in directional movements and corresponding error values. it also details the process of automating mouse and keyboard movements to navigate a game using python and opencv, including setting delays, handling errors, and potential improvements in object detection.', 'chapters': [{'end': 1342.438, 'start': 1193.521, 'title': 'Calculating error in image processing', 'summary': 'Demonstrates using numpy.mean to calculate the error in image processing with a 2d array, then determining the target value and calculating the error, resulting in directional movements and corresponding error values.', 'duration': 148.917, 'highlights': ['Using numpy.mean to calculate the mean of the x values of matches in image processing.', 'Determining the target value by finding the shape of the mini map and calculating the first element divided by 2.', 'Calculating the error by subtracting the mean value from the target value for directional movement indication.', 'Observing and interpreting the error values to determine the required directional movements.', 'Adjusting the keyboard used for the task to a non-mechanical one for better video recording.']}, {'end': 1997.667, 'start': 1342.678, 'title': 'Automating mouse and keyboard movement', 'summary': 'Details the process of automating mouse and keyboard movements to navigate a game, using python and opencv, including setting delays, handling errors, and potential improvements in object detection, with the goal of achieving efficient navigation in the game.', 'duration': 654.989, 'highlights': ['The script involves setting a delay of one second to allow time for setup before the mouse and keyboard control takes effect, which can be crucial for avoiding unintended actions (e.g., moving beyond intended path) (relevance score: 5)', 'The code includes error handling to address situations where the intended path is not present or when there are inconsistencies in the game environment (e.g., transparent objects affecting HSV values), which can impact the accuracy of the navigation (relevance score: 4)', 'The chapter emphasizes the potential for further improvement through object detection to avoid obstacles like people and cars, increasing the navigation success rate to 70-95%, showcasing future potential enhancements to the automation process (relevance score: 3)']}], 'duration': 804.146, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/dUU6ZsJlZKQ/pics/dUU6ZsJlZKQ1193521.jpg', 'highlights': ['Using numpy.mean to calculate the mean of the x values of matches in image processing.', 'Determining the target value by finding the shape of the mini map and calculating the first element divided by 2.', 'Calculating the error by subtracting the mean value from the target value for directional movement indication.', 'The script involves setting a delay of one second to allow time for setup before the mouse and keyboard control takes effect, which can be crucial for avoiding unintended actions (e.g., moving beyond intended path)', 'The code includes error handling to address situations where the intended path is not present or when there are inconsistencies in the game environment (e.g., transparent objects affecting HSV values), which can impact the accuracy of the navigation', 'The chapter emphasizes the potential for further improvement through object detection to avoid obstacles like people and cars, increasing the navigation success rate to 70-95%, showcasing future potential enhancements to the automation process', 'Observing and interpreting the error values to determine the required directional movements.', 'Adjusting the keyboard used for the task to a non-mechanical one for better video recording.']}], 'highlights': ['The chapter emphasizes the potential for further improvement through object detection to avoid obstacles like people and cars, increasing the navigation success rate to 70-95%, showcasing future potential enhancements to the automation process', 'The chapter discusses the potential of developing AI in Cyberpunk 2077 using computer vision techniques, particularly for navigating on foot through the city', 'The chapter discusses the methodology for automatic walking in Cyberpunk 2077 using Python scripts for grabbing the screen and direct input, with an emphasis on achieving high frame rates', 'The chapter discusses techniques for navigating game paths using Cyberpunk 2077 as an example, including controlling mouse and keys, navigating paths on foot and by car, and challenges faced in demonstrating the game due to CPU heat', 'The process involves finding the coordinate location of all the white pixels in the image mask, represented by a value of 255, using np.argwhere, resulting in an array of matches', 'The chapter demonstrates the procedure for defining the pathing function for mini-map analysis, including the process of selecting the region of interest and explaining the X and Y coordinates for the mini-map in Cyberpunk 2077', 'The range of 0 to 255 for hue, saturation, and value is discussed, indicating a comprehensive color matching system', 'The adjustment of hue values, specifically in the range of 75 to 150, played a crucial role in further refining the color filtering process to isolate the desired path', 'The path detection process involved filtering colors in the range of 0 to 150 for both hue and saturation, effectively isolating the character color and the desired path', 'The script involves setting a delay of one second to allow time for setup before the mouse and keyboard control takes effect, which can be crucial for avoiding unintended actions (e.g., moving beyond intended path)']}