title
Lecture 41 — Overview of Recommender Systems | Stanford University

description
Check out the following interesting papers. Happy learning! Paper Title: "On the Role of Reviewer Expertise in Temporal Review Helpfulness Prediction" Paper: https://aclanthology.org/2023.findings-eacl.125/ Dataset: https://huggingface.co/datasets/tafseer-nayeem/review_helpfulness_prediction Paper Title: "Abstractive Unsupervised Multi-Document Summarization using Paraphrastic Sentence Fusion" Paper: https://aclanthology.org/C18-1102/ Paper Title: "Extract with Order for Coherent Multi-Document Summarization" Paper: https://aclanthology.org/W17-2407.pdf Paper Title: "Paraphrastic Fusion for Abstractive Multi-Sentence Compression Generation" Paper: https://dl.acm.org/doi/abs/10.1145/3132847.3133106 Paper Title: "Neural Diverse Abstractive Sentence Compression Generation" Paper: https://link.springer.com/chapter/10.1007/978-3-030-15719-7_14

detail
{'title': 'Lecture 41 — Overview of Recommender Systems | Stanford University', 'heatmap': [{'end': 634.12, 'start': 609.707, 'weight': 1}, {'end': 668.685, 'start': 647.967, 'weight': 0.831}, {'end': 823.023, 'start': 795.615, 'weight': 0.798}], 'summary': 'The lecture provides an overview of recommender systems, discussing common types like content-based and collaborative filtering, the need for evaluation in large catalogs, the long tail phenomenon, and the impact of recommendation systems in transforming overlooked items into bestsellers.', 'chapters': [{'end': 202.956, 'segs': [{'end': 36.717, 'src': 'embed', 'start': 0.609, 'weight': 3, 'content': [{'end': 2.25, 'text': 'Welcome back to Mining of Massive Data Sets.', 'start': 0.609, 'duration': 1.641}, {'end': 4.372, 'text': "Today's topic is recommender systems.", 'start': 2.871, 'duration': 1.501}, {'end': 8.154, 'text': "We're going to start with an overview of recommendation systems and why they are necessary.", 'start': 5.033, 'duration': 3.121}, {'end': 14.08, 'text': "Then we're going to look at the two most common types of recommender systems, content-based systems and collaborative filtering.", 'start': 8.796, 'duration': 5.284}, {'end': 18.383, 'text': "And finally, we're going to look at how to evaluate recommender systems to make sure they're doing a good job.", 'start': 14.66, 'duration': 3.723}, {'end': 20.205, 'text': "Let's start with an overview.", 'start': 19.344, 'duration': 0.861}, {'end': 27.911, 'text': 'Imagine any situation where a user interacts with a really large catalog of items.', 'start': 22.768, 'duration': 5.143}, {'end': 32.473, 'text': 'Now these items could be products at Amazon, they could be movies at Netflix,', 'start': 28.512, 'duration': 3.961}, {'end': 36.717, 'text': "they could be music from Pandora's catalog or they could be news items on Google News.", 'start': 32.473, 'duration': 4.244}], 'summary': 'Overview of recommender systems and their necessity, types, and evaluation methods.', 'duration': 36.108, 'max_score': 0.609, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM609.jpg'}, {'end': 87.044, 'src': 'embed', 'start': 51.21, 'weight': 0, 'content': [{'end': 56.034, 'text': "The user knows what they're looking for, and they go and they search the catalog for the precise item that they're looking for.", 'start': 51.21, 'duration': 4.824}, {'end': 61.518, 'text': "Now, when you have a really large catalog of items, very often the user doesn't know exactly what they're looking for.", 'start': 56.754, 'duration': 4.764}, {'end': 63.26, 'text': 'And this is where recommendations come in.', 'start': 61.939, 'duration': 1.321}, {'end': 70.646, 'text': 'The system recommends to the user certain items that they think the user will be interested in based on what they know about the user.', 'start': 64.361, 'duration': 6.285}, {'end': 73.529, 'text': 'Now, why do we really need such recommendations?', 'start': 71.767, 'duration': 1.762}, {'end': 82.881, 'text': 'The key that made recommendation so important and why recommendation system developed so much in the last 10 or 20 years,', 'start': 75.376, 'duration': 7.505}, {'end': 85.823, 'text': 'is that we moved from an era of scarcity to an era of abundance.', 'start': 82.881, 'duration': 2.942}, {'end': 87.044, 'text': 'What do I mean by this?', 'start': 86.203, 'duration': 0.841}], 'summary': 'Recommendation systems help users find items in large catalogs, adapting to abundance over scarcity.', 'duration': 35.834, 'max_score': 51.21, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM51210.jpg'}, {'end': 196.883, 'src': 'embed', 'start': 167.32, 'weight': 2, 'content': [{'end': 168.361, 'text': "Let's examine what this is.", 'start': 167.32, 'duration': 1.041}, {'end': 174.429, 'text': "Now imagine a graph where on the x-axis, we've taken the items in the catalog.", 'start': 169.946, 'duration': 4.483}, {'end': 178.651, 'text': 'Remember, items might be books, or music, or video, or news articles.', 'start': 174.489, 'duration': 4.162}, {'end': 180.973, 'text': "And you've ranked these items by popularity.", 'start': 178.992, 'duration': 1.981}, {'end': 186.856, 'text': 'So the most popular items are on the left, and as we move towards the right, the items become less and less popular.', 'start': 181.473, 'duration': 5.383}, {'end': 188.518, 'text': 'What do I mean by popular??', 'start': 186.877, 'duration': 1.641}, {'end': 195.522, 'text': 'Well, I mean the number of times the item is purchased in a week, or it could be the number of times a movie is viewed in a week or a month,', 'start': 188.798, 'duration': 6.724}, {'end': 196.883, 'text': 'or some some fixed time period.', 'start': 195.522, 'duration': 1.361}], 'summary': 'Analyzing popularity of items in a catalog based on weekly purchases or views.', 'duration': 29.563, 'max_score': 167.32, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM167320.jpg'}], 'start': 0.609, 'title': 'Recommender systems', 'summary': 'Covers an overview of recommender systems, including common types like content-based and collaborative filtering, and the need for evaluation to ensure effectiveness in large catalogs such as amazon, netflix, pandora, and google news. it also discusses the importance of recommendation systems in an era of abundance, where the internet enables zero cost dissemination of information about products, leading to a large catalog of items, and the shift from an era of scarcity to an era of abundance, leading to the long tail phenomenon.', 'chapters': [{'end': 36.717, 'start': 0.609, 'title': 'Recommender systems overview', 'summary': 'Covers an overview of recommender systems, including the common types, content-based and collaborative filtering, and the need for evaluation to ensure effectiveness in large catalogs such as amazon, netflix, pandora, and google news.', 'duration': 36.108, 'highlights': ['The need for recommender systems in scenarios with large item catalogs such as Amazon, Netflix, Pandora, and Google News.', 'Overview of the two most common types of recommender systems: content-based systems and collaborative filtering.', 'The importance of evaluating recommender systems to ensure they are effectively serving their purpose in providing recommendations for users.']}, {'end': 202.956, 'start': 37.438, 'title': 'The power of recommendations in an era of abundance', 'summary': 'Discusses the importance of recommendation systems in an era of abundance, where the internet enables zero cost dissemination of information about products, leading to a large catalog of items, and the shift from an era of scarcity to an era of abundance, leading to the long tail phenomenon.', 'duration': 165.518, 'highlights': ['The internet enables zero cost dissemination of information about products, leading to a large catalog of items. The web enables zero cost dissemination of information about products, allowing for a much larger number of products than ever before, without the limitation of shelf space.', 'Shift from an era of scarcity to an era of abundance, leading to the long tail phenomenon. The shift from a scarcity of shelf space to an abundance of products due to the internet has given rise to the long tail phenomenon, where less popular items in the catalog become more accessible and relevant.', 'Importance of recommendation systems in an era of abundance. Recommendation systems are crucial in helping users discover items in a large catalog when they do not know exactly what they are looking for, and are based on what the system knows about the user to enhance user experience.']}], 'duration': 202.347, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM609.jpg', 'highlights': ['The internet enables zero cost dissemination of information about products, leading to a large catalog of items.', 'The shift from an era of scarcity to an era of abundance, leading to the long tail phenomenon.', 'Importance of recommendation systems in an era of abundance.', 'The need for recommender systems in scenarios with large item catalogs such as Amazon, Netflix, Pandora, and Google News.', 'Overview of the two most common types of recommender systems: content-based systems and collaborative filtering.', 'The importance of evaluating recommender systems to ensure they are effectively serving their purpose in providing recommendations for users.']}, {'end': 414.975, 'segs': [{'end': 256.291, 'src': 'embed', 'start': 228.831, 'weight': 0, 'content': [{'end': 231.932, 'text': 'You can see that this curve, you know, has a very steep fall initially.', 'start': 228.831, 'duration': 3.101}, {'end': 236.353, 'text': 'The, the, you know, you have a really, really, a few really, really popular items.', 'start': 232.832, 'duration': 3.521}, {'end': 243.355, 'text': 'And then as you move towards the right as the, you know, as the item rank becomes greater, the popularity falls off very steeply.', 'start': 237.053, 'duration': 6.302}, {'end': 249.256, 'text': 'But at a certain point, you can see that this popularity stops, you know, you know, falls off less and less steeply.', 'start': 244.195, 'duration': 5.061}, {'end': 252.277, 'text': 'And, you know, it, it never quite reaches the x-axis.', 'start': 249.836, 'duration': 2.441}, {'end': 256.291, 'text': 'The interesting thing here is that there is a cutoff point.', 'start': 253.951, 'duration': 2.34}], 'summary': 'The popularity of items falls steeply initially, then less steeply, with a noticeable cutoff point.', 'duration': 27.46, 'max_score': 228.831, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM228831.jpg'}, {'end': 301.868, 'src': 'embed', 'start': 264.714, 'weight': 1, 'content': [{'end': 269.816, 'text': "If you're a physical retailer like a Walmart, it's not economic to stock this item,", 'start': 264.714, 'duration': 5.102}, {'end': 274.457, 'text': 'because the rent cost of stocking the item is more than you make by when you sell the item.', 'start': 269.816, 'duration': 4.641}, {'end': 280.803, 'text': "And therefore, a retailer, any right thinking retailer, doesn't stock items that are unpopular.", 'start': 275.202, 'duration': 5.601}, {'end': 283.804, 'text': 'They, you know, they only stock the, the head of the distribution.', 'start': 281.183, 'duration': 2.621}, {'end': 288.085, 'text': "So, there's this cutoff point that I show on this graph here.", 'start': 284.404, 'duration': 3.681}, {'end': 294.466, 'text': 'And items that are more popular than this, the, the more popular items, are available at a retail store.', 'start': 289.125, 'duration': 5.341}, {'end': 300.408, 'text': 'But the less popular items, the items that are to the right of the cutoff point, are not available at any retail store.', 'start': 294.986, 'duration': 5.422}, {'end': 301.868, 'text': "They're only available online.", 'start': 300.768, 'duration': 1.1}], 'summary': 'Physical retailers avoid stocking unpopular items due to economic reasons, leading to availability only online.', 'duration': 37.154, 'max_score': 264.714, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM264714.jpg'}, {'end': 369.496, 'src': 'embed', 'start': 338.954, 'weight': 2, 'content': [{'end': 344.157, 'text': 'In fact, in some cases, the area under the curve on the right is about as large, or could be even larger,', 'start': 338.954, 'duration': 5.203}, {'end': 346.919, 'text': 'than the area of the curve under the curve on the, on the left.', 'start': 344.157, 'duration': 2.762}, {'end': 354.185, 'text': 'So you have all these items that could never be found in a physical store, but that can be only found online.', 'start': 349.201, 'duration': 4.984}, {'end': 362.15, 'text': "But there are so many of them that it's very hard for any user to find all these items right?", 'start': 355.305, 'duration': 6.845}, {'end': 369.496, 'text': 'So when you have this era of abundance and you have so many items and many of them are only found online, how you know,', 'start': 362.19, 'duration': 7.306}], 'summary': 'Online abundance makes finding items challenging.', 'duration': 30.542, 'max_score': 338.954, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM338954.jpg'}, {'end': 408.55, 'src': 'embed', 'start': 382.303, 'weight': 4, 'content': [{'end': 386.225, 'text': "The user doesn't even know where to start looking, and that's where recommendations engines come in.", 'start': 382.303, 'duration': 3.922}, {'end': 393.769, 'text': 'So recommendation engines work in the case of many, many kinds of items, books, music, movies, news articles.', 'start': 388.346, 'duration': 5.423}, {'end': 395.911, 'text': 'Interestingly, they even work in the case of people.', 'start': 394.15, 'duration': 1.761}, {'end': 402.344, 'text': "For example, when you go to Facebook or LinkedIn or Twitter, there are so many people that you don't know who to follow or who to friend.", 'start': 396.519, 'duration': 5.825}, {'end': 408.55, 'text': 'And so Facebook or LinkedIn or Twitter makes recommendations to you on the people that you know that you could follow or friend.', 'start': 402.745, 'duration': 5.805}], 'summary': 'Recommendation engines aid in finding items and people, like on facebook, linkedin, and twitter.', 'duration': 26.247, 'max_score': 382.303, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM382303.jpg'}], 'start': 203.316, 'title': 'Popularity distribution and long tail phenomenon in retail', 'summary': 'Discusses the steep fall in popularity distribution of items in a large catalog, indicating uneconomical items for physical retailers, and explains the long tail phenomenon, emphasizing the need for recommendation engines in introducing users to the abundance of online items.', 'chapters': [{'end': 283.804, 'start': 203.316, 'title': 'Popularity distribution in retail', 'summary': 'Discusses the popularity distribution of items in a large catalog, revealing a steep fall in popularity as item rank increases, with a cutoff point indicating items that are uneconomical for physical retailers to stock.', 'duration': 80.488, 'highlights': ['The popularity distribution curve shows a steep fall in popularity for items, with a few highly popular items and a sharp decline in popularity as item rank increases.', 'There is a cutoff point indicating items that are less popular and uneconomical for physical retailers to stock due to high rent costs.', 'Physical retailers like Walmart prioritize stocking only the most popular items to ensure economic viability.']}, {'end': 414.975, 'start': 284.404, 'title': 'The long tail phenomenon', 'summary': 'Explains the long tail phenomenon, illustrating that less popular items are only available online, and the significant area under the curve on the right can be as large as or larger than the left side. it also emphasizes the need for recommendation engines in introducing users to the abundance of online items.', 'duration': 130.571, 'highlights': ['The significant area under the curve on the right can be as large as or larger than the left side, indicating the abundance of online items.', 'Less popular items are only available online, creating a phenomenon called the long tail.', 'The need for recommendation engines arises due to the era of abundance with millions of items available only online, making it challenging for users to find new items.', 'Recommendation engines are crucial for helping users discover new items, as they work for various kinds of items and even in the case of people on social media platforms like Facebook, LinkedIn, and Twitter.']}], 'duration': 211.659, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM203316.jpg', 'highlights': ['The popularity distribution curve shows a steep fall in popularity for items, with a few highly popular items and a sharp decline in popularity as item rank increases.', 'Physical retailers like Walmart prioritize stocking only the most popular items to ensure economic viability.', 'The significant area under the curve on the right can be as large as or larger than the left side, indicating the abundance of online items.', 'Less popular items are only available online, creating a phenomenon called the long tail.', 'The need for recommendation engines arises due to the era of abundance with millions of items available only online, making it challenging for users to find new items.', 'Recommendation engines are crucial for helping users discover new items, as they work for various kinds of items and even in the case of people on social media platforms like Facebook, LinkedIn, and Twitter.']}, {'end': 1009.24, 'segs': [{'end': 485.57, 'src': 'embed', 'start': 454.602, 'weight': 0, 'content': [{'end': 457.503, 'text': 'And lo and behold, those people started buying Touching the Void as well.', 'start': 454.602, 'duration': 2.901}, {'end': 461.105, 'text': 'The interesting point is, this made Touching the Void a bestseller.', 'start': 458.223, 'duration': 2.882}, {'end': 467.508, 'text': 'In fact, it became a bigger bestseller even than Into Thin Air, even though a few years ago it had sank without a trace.', 'start': 461.765, 'duration': 5.743}, {'end': 472.019, 'text': 'So this example should show you the power of recommendation systems.', 'start': 468.376, 'duration': 3.643}, {'end': 479.325, 'text': "There are these items, these sort of gems, like touching the void, you know, that people don't know because they don't know to look for them.", 'start': 472.56, 'duration': 6.765}, {'end': 485.57, 'text': "But a good recommendation system can expose people to these hidden gems that they wouldn't have known about otherwise.", 'start': 479.926, 'duration': 5.644}], 'summary': 'Recommendation systems made touching the void a bigger bestseller than into thin air, showcasing their power to expose hidden gems.', 'duration': 30.968, 'max_score': 454.602, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM454602.jpg'}, {'end': 544.265, 'src': 'embed', 'start': 515.809, 'weight': 1, 'content': [{'end': 521.373, 'text': "And another place where you'll see these editorial recommendations is often on the home pages of websites.", 'start': 515.809, 'duration': 5.564}, {'end': 531.868, 'text': "For example, if you go to the home page of most popular websites, including product websites, you'll see editorial picks.", 'start': 521.753, 'duration': 10.115}, {'end': 535.594, 'text': 'These are products that have been picked by the editorial staff to feature on the home page.', 'start': 531.888, 'duration': 3.706}, {'end': 544.265, 'text': "The drawback with editorial or hand curated recommendations is that it's done entirely by, you know,", 'start': 536.678, 'duration': 7.587}], 'summary': "Editorial recommendations appear on popular websites' home pages, including product websites, showcasing products picked by the editorial staff.", 'duration': 28.456, 'max_score': 515.809, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM515808.jpg'}, {'end': 604.829, 'src': 'embed', 'start': 560.397, 'weight': 2, 'content': [{'end': 565.12, 'text': 'For example, if you go to YouTube, you can see the most popular videos, for instance, right?', 'start': 560.397, 'duration': 4.723}, {'end': 572.205, 'text': 'So these are simple aggregates which sort of take into account user activity to make recommendations to other users.', 'start': 565.701, 'duration': 6.504}, {'end': 576.007, 'text': "But these recommendations don't depend on the user.", 'start': 572.605, 'duration': 3.402}, {'end': 580.23, 'text': 'They only depend on, you know, the, the aggregate activity of a lot of other users.', 'start': 576.588, 'duration': 3.642}, {'end': 587.953, 'text': 'The third and most interesting kind of recommendation to us is recommendations that are tailored to individual users.', 'start': 582.008, 'duration': 5.945}, {'end': 592.738, 'text': 'Right? For example, book recommendations tailored to your taste.', 'start': 588.894, 'duration': 3.844}, {'end': 596.301, 'text': 'Or movie recommendation based on the movies that you watched previously.', 'start': 593.158, 'duration': 3.143}, {'end': 599.244, 'text': 'Or music recommendation based on your music interests.', 'start': 596.661, 'duration': 2.583}, {'end': 601.366, 'text': 'And this is our focus here.', 'start': 600.024, 'duration': 1.342}, {'end': 604.829, 'text': 'Recommendations that are tailored to individual users.', 'start': 602.367, 'duration': 2.462}], 'summary': "Youtube uses aggregate user activity for recommendations, but the focus is on tailoring recommendations to individual users' tastes in books, movies, and music.", 'duration': 44.432, 'max_score': 560.397, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM560397.jpg'}, {'end': 639.783, 'src': 'heatmap', 'start': 609.707, 'weight': 1, 'content': [{'end': 610.948, 'text': "So let's look at a formal model.", 'start': 609.707, 'duration': 1.241}, {'end': 614.711, 'text': 'Let C be a set of customers and S a set of items.', 'start': 611.368, 'duration': 3.343}, {'end': 620.495, 'text': "We're going to create a function called a utility function or a utility matrix.", 'start': 616.532, 'duration': 3.963}, {'end': 630.262, 'text': 'The utility function is a function that looks at every pair of customer and item and maps it to a rating.', 'start': 621.355, 'duration': 8.907}, {'end': 634.12, 'text': 'Okay? R in this case is a set of ratings.', 'start': 630.282, 'duration': 3.838}, {'end': 639.783, 'text': 'And for example, R could be a star rating from one star to five star.', 'start': 635.001, 'duration': 4.782}], 'summary': 'Formal model: c set of customers, s set of items, utility function maps pairs to ratings. r: 1-5 stars.', 'duration': 30.076, 'max_score': 609.707, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM609707.jpg'}, {'end': 675.049, 'src': 'heatmap', 'start': 647.967, 'weight': 0.831, 'content': [{'end': 654.991, 'text': 'So that, you know, a lower value indicates that the user liked the product less, and a higher value indicates that the user liked the product more.', 'start': 647.967, 'duration': 7.024}, {'end': 658.813, 'text': "Let's look at an example of a utility matrix.", 'start': 656.872, 'duration': 1.941}, {'end': 662.581, 'text': 'Now on the top we have four movies here.', 'start': 660.68, 'duration': 1.901}, {'end': 665.643, 'text': 'Avatar, Lord of the Rings, Matrix, and Pirates of the Caribbean.', 'start': 662.601, 'duration': 3.042}, {'end': 668.685, 'text': 'And down here we have four users, Alice, Bob, Carol, and David.', 'start': 666.204, 'duration': 2.481}, {'end': 675.049, 'text': 'And the utility matrix gives you ratings for certain movies and certain users.', 'start': 669.966, 'duration': 5.083}], 'summary': 'Utility matrix shows user ratings for movies; higher value indicates more liked product.', 'duration': 27.082, 'max_score': 647.967, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM647967.jpg'}, {'end': 782.403, 'src': 'embed', 'start': 738.95, 'weight': 4, 'content': [{'end': 740.951, 'text': 'So this is the key problem for recommender systems.', 'start': 738.95, 'duration': 2.001}, {'end': 749.416, 'text': 'Once we find out for each user certain movies that they would have rated highly or which the system thinks they might have rated highly,', 'start': 741.472, 'duration': 7.944}, {'end': 751.697, 'text': 'then we can recommend those movies to those users.', 'start': 749.416, 'duration': 2.281}, {'end': 756.159, 'text': 'So there are three key problems in the space of recommender systems.', 'start': 753.258, 'duration': 2.901}, {'end': 760.361, 'text': 'The first is gathering the known ratings in the matrix.', 'start': 756.899, 'duration': 3.462}, {'end': 765.724, 'text': 'Now in the previous slide when you look at the utility matrix it was already filled in with certain values.', 'start': 760.961, 'duration': 4.763}, {'end': 771.94, 'text': "But how do you get, gather those values in the first place? So that's the key that the first problem that you need to tackle.", 'start': 766.479, 'duration': 5.461}, {'end': 777.762, 'text': 'The second problem is to extrapolate unknown rating from known ratings.', 'start': 774.041, 'duration': 3.721}, {'end': 782.403, 'text': "But we're mainly interested in the high unknown ratings.", 'start': 779.702, 'duration': 2.701}], 'summary': 'Recommender systems face three key problems: gathering known ratings, extrapolating unknown ratings, and recommending highly rated movies to users.', 'duration': 43.453, 'max_score': 738.95, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM738950.jpg'}, {'end': 823.023, 'src': 'heatmap', 'start': 795.615, 'weight': 0.798, 'content': [{'end': 799.676, 'text': 'And finally, the third key problem is evaluating extrapolation methods.', 'start': 795.615, 'duration': 4.061}, {'end': 804.097, 'text': 'Once you have a recommendation system that can extrapolate unknown ratings from known ratings,', 'start': 800.136, 'duration': 3.961}, {'end': 806.638, 'text': 'how do you know that the recommender system is doing well?', 'start': 804.097, 'duration': 2.541}, {'end': 809.419, 'text': 'This is where you know the evaluation methodologies come in.', 'start': 807.018, 'duration': 2.401}, {'end': 814.54, 'text': "Let's start with the first problem, that of gathering ratings.", 'start': 812.02, 'duration': 2.52}, {'end': 820.762, 'text': "The first and simplest way of gathering ratings is, is what, what I'll call the explicit method.", 'start': 815.961, 'duration': 4.801}, {'end': 823.023, 'text': 'Simply ask people to rate items.', 'start': 821.483, 'duration': 1.54}], 'summary': 'Challenges in evaluating extrapolation methods for recommender systems and gathering ratings using explicit method.', 'duration': 27.408, 'max_score': 795.615, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM795615.jpg'}, {'end': 878.794, 'src': 'embed', 'start': 847.068, 'weight': 6, 'content': [{'end': 853.31, 'text': 'So the explicit method has the advantage of simplicity and of getting direct responses from users.', 'start': 847.068, 'duration': 6.242}, {'end': 856.223, 'text': "The problem though is that it doesn't scale.", 'start': 854.642, 'duration': 1.581}, {'end': 863.947, 'text': 'Only a small fraction of users who viewed a movie or listened to a you know piece of music or bought a product,', 'start': 857.944, 'duration': 6.003}, {'end': 865.688, 'text': 'actually bother to leave a rating or review.', 'start': 863.947, 'duration': 1.741}, {'end': 869.289, 'text': "Most users don't actually leave ratings or reviews.", 'start': 866.828, 'duration': 2.461}, {'end': 878.794, 'text': "So, while the data that's explicitly gathered excellent data, it, it's not sufficient in most cases for recommendations,", 'start': 870.03, 'duration': 8.764}], 'summary': "Explicit method gathers data but doesn't scale due to low user input for ratings and reviews.", 'duration': 31.726, 'max_score': 847.068, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM847068.jpg'}, {'end': 947.749, 'src': 'embed', 'start': 916.327, 'weight': 5, 'content': [{'end': 921.33, 'text': "The problem though is that it's very hard using implicit ratings to learn low ratings.", 'start': 916.327, 'duration': 5.003}, {'end': 926.554, 'text': "It's quite easy to learn high ratings because you, you might have a rule that a purchase implies a high rating.", 'start': 922.171, 'duration': 4.383}, {'end': 932.058, 'text': 'But you, you can never learn a rating that a user disliked a product implicitly.', 'start': 926.974, 'duration': 5.084}, {'end': 941.324, 'text': 'In practice, Most recommender systems on most websites use a combination of explicit and implicit ratings.', 'start': 934.639, 'duration': 6.685}, {'end': 947.749, 'text': 'When explicit ratings are available, they use them, but they supplement them with implicit ratings when needed.', 'start': 941.884, 'duration': 5.865}], 'summary': 'Most recommender systems use a mix of explicit and implicit ratings for learning user preferences.', 'duration': 31.422, 'max_score': 916.327, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM916327.jpg'}, {'end': 1009.24, 'src': 'embed', 'start': 996.069, 'weight': 8, 'content': [{'end': 998.251, 'text': 'There are three approaches to building recommendation systems.', 'start': 996.069, 'duration': 2.182}, {'end': 1001.213, 'text': 'The first is content based recommendations.', 'start': 999.372, 'duration': 1.841}, {'end': 1003.735, 'text': 'The second is collaborative filtering.', 'start': 1001.874, 'duration': 1.861}, {'end': 1006.237, 'text': 'And the third is latent factor based models.', 'start': 1004.276, 'duration': 1.961}, {'end': 1009.24, 'text': "Let's start with content based approaches.", 'start': 1007.038, 'duration': 2.202}], 'summary': 'Three recommendation system approaches: content-based, collaborative filtering, and latent factor models.', 'duration': 13.171, 'max_score': 996.069, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM996069.jpg'}], 'start': 415.956, 'title': 'The power of recommendation systems', 'summary': 'Demonstrates the impact of recommendation systems by highlighting the transformation of overlooked books into bestsellers, and explores various types of recommendation systems including editorial, simple aggregates, and tailored user recommendations. it also provides an overview of utility matrix, sparse data, challenges in extrapolating unknown ratings, methods for gathering ratings, and different approaches to building recommendation systems.', 'chapters': [{'end': 604.829, 'start': 415.956, 'title': 'Power of recommendation systems', 'summary': "Showcases the power of recommendation systems through the example of 'touching the void' and 'into thin air', demonstrating how recommendation systems can turn a previously overlooked book into a bestseller, and it further explores types of recommendation systems including editorial or hand curated, simple aggregates, and recommendations tailored to individual users.", 'duration': 188.873, 'highlights': ['Touching the Void became a bigger bestseller than Into Thin Air, demonstrating the power of recommendation systems. Touching the Void, previously overlooked, became a bigger bestseller than Into Thin Air due to the influence of recommendation systems.', 'Editorial or hand curated recommendations are built by hand and often found on the home pages of most popular websites. Editorial recommendations are hand curated picks often featured on the home pages of popular websites.', 'Simple aggregates like top ten lists and most popular recommendations are based on user activity but do not depend on individual user preferences. Simple aggregates like top ten lists and most popular recommendations are based on user activity, not individual preferences.', 'Recommendations tailored to individual users are the most interesting kind of recommendation system. Recommendations tailored to individual users, such as book or movie recommendations based on individual preferences, are the most interesting kind of recommendation system.']}, {'end': 1009.24, 'start': 609.707, 'title': 'Recommendation systems overview', 'summary': 'Explains the concept of utility matrix, sparse data in recommendation systems, and the challenges in extrapolating unknown ratings. it also discusses the methods for gathering ratings, explicit and implicit, and the three approaches to building recommendation systems: content based, collaborative filtering, and latent factor based models.', 'duration': 399.533, 'highlights': ['Sparse Data in Recommendation Systems The utility matrix in recommendation systems is typically sparse, as most users have not rated most items, leading to the challenge of extrapolating unknown values and the cold start problem for new items or users.', 'Implicit Ratings in Recommendation Systems Implicit ratings, learned from user actions such as purchases, are more scalable than explicit ratings, but struggle to learn low ratings, leading most recommender systems to use a combination of explicit and implicit ratings.', 'Gathering Ratings: Explicit and Implicit Methods The explicit method involves directly asking users to rate items, providing excellent data but suffering from scalability issues, while the implicit method learns ratings from user actions, such as purchases, and is more scalable but struggles to learn low ratings.', 'Challenges in Extrapolating Unknown Ratings The key problem in recommendation systems is to figure out unknown values in the utility matrix, particularly high unknown ratings, and evaluating extrapolation methods to ensure the recommender system is performing well.', 'Approaches to Building Recommendation Systems The three approaches to building recommendation systems are content-based recommendations, collaborative filtering, and latent factor-based models, each offering distinct methods for generating recommendations.']}], 'duration': 593.284, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/1JRrCEgiyHM/pics/1JRrCEgiyHM415956.jpg', 'highlights': ['Touching the Void became a bigger bestseller than Into Thin Air, demonstrating the power of recommendation systems.', 'Editorial recommendations are hand curated picks often featured on the home pages of popular websites.', 'Simple aggregates like top ten lists and most popular recommendations are based on user activity, not individual preferences.', 'Recommendations tailored to individual users, such as book or movie recommendations based on individual preferences, are the most interesting kind of recommendation system.', 'The utility matrix in recommendation systems is typically sparse, as most users have not rated most items, leading to the challenge of extrapolating unknown values and the cold start problem for new items or users.', 'Implicit ratings, learned from user actions such as purchases, are more scalable than explicit ratings, but struggle to learn low ratings, leading most recommender systems to use a combination of explicit and implicit ratings.', 'The explicit method involves directly asking users to rate items, providing excellent data but suffering from scalability issues, while the implicit method learns ratings from user actions, such as purchases, and is more scalable but struggles to learn low ratings.', 'The key problem in recommendation systems is to figure out unknown values in the utility matrix, particularly high unknown ratings, and evaluating extrapolation methods to ensure the recommender system is performing well.', 'The three approaches to building recommendation systems are content-based recommendations, collaborative filtering, and latent factor-based models, each offering distinct methods for generating recommendations.']}], 'highlights': ['The need for recommender systems in scenarios with large item catalogs such as Amazon, Netflix, Pandora, and Google News.', 'The importance of evaluating recommender systems to ensure they are effectively serving their purpose in providing recommendations for users.', 'The shift from an era of scarcity to an era of abundance, leading to the long tail phenomenon.', 'The internet enables zero cost dissemination of information about products, leading to a large catalog of items.', 'The popularity distribution curve shows a steep fall in popularity for items, with a few highly popular items and a sharp decline in popularity as item rank increases.', 'Recommendation engines are crucial for helping users discover new items, as they work for various kinds of items and even in the case of people on social media platforms like Facebook, LinkedIn, and Twitter.', 'The significant area under the curve on the right can be as large as or larger than the left side, indicating the abundance of online items.', 'Less popular items are only available online, creating a phenomenon called the long tail.', 'Touching the Void became a bigger bestseller than Into Thin Air, demonstrating the power of recommendation systems.', 'Recommendations tailored to individual users, such as book or movie recommendations based on individual preferences, are the most interesting kind of recommendation system.', 'The utility matrix in recommendation systems is typically sparse, as most users have not rated most items, leading to the challenge of extrapolating unknown values and the cold start problem for new items or users.', 'Implicit ratings, learned from user actions such as purchases, are more scalable than explicit ratings, but struggle to learn low ratings, leading most recommender systems to use a combination of explicit and implicit ratings.', 'The three approaches to building recommendation systems are content-based recommendations, collaborative filtering, and latent factor-based models, each offering distinct methods for generating recommendations.']}