title
WE GOT ACCESS TO GPT-3! [Epic Special Edition]

description
In this special edition, Dr. Tim Scarfe, Yannic Kilcher and Dr. Keith Duggar speak with Professor Gary Marcus, Dr. Walid Saba and Connor Leahy about GPT-3. We have all had a significant amount of time to experiment with GPT-3 and show you demos of it in use and the considerations. Do you think GPT-3 is a step towards AGI? Answer in the comments! 00:00:00 Connor's take on LinkedIn 00:00:47 Show teaser 00:20:02 Tim Introduction 00:26:55 First look at GPT-3, python sorting 00:31:05 Search strategy in LMs 00:38:28 Character analogies and Melanie Mitchell 00:44:27 Substitution cipher 00:47:21 Database prompt 00:53:00 Broader Impact Generation 01:02:47 Gary Marcus Interview (Robust.AI) 01:29:11 Connor Leahy Interview (Eleuther.AI) 01:32:29 Connor -- Tabular data 01:33:41 Connor -- other surprising examples? 01:34:54 Connor -- Is interpolated stuff new? 01:37:43 Connor -- structure of the brain / How GPT works 01:41:21 Connor -- Why cant GPT-3 reason? 01:46:30 Connor -- Missing information problem and ideas on our our brains work 01:54:28 Connor -- Topology of brain/models 01:58:49 Connor -- Hardware lottery / LSTM / Transformer 02:01:41 Connor -- NNs are just matrix program search 02:10:32 Connor -- Google -- information retrieval, the new paradigm, how to extract info from GPT-3, RL controller on top? 02:19:38 Connor -- Database example / "pattern matching is Turing complete" 02:23:55 Connor -- Did gpt3 understand? 02:26:30 Connor -- Are the GOFAI people right? 02:27:40 Walid Saba on GPT-3 02:30:41 Walid -- What is understanding and pattern recognition 02:35:56 Walid -- Chomsky would be happy 02:42:13 Walid -- Redefining success 02:46:05 Walid on Hinton 02:47:34 Walid on software 3.0 02:53:11 Keith -- We use machine learning because we cant write code to do the same thing 02:59:36 Keith -- What is pattern recognition and understanding 03:14:06 GPT-3 trials -- Turing Dialog 03:15:35 GPT-3 trials -- Mary Enjoyed a Sandwich 03:16:19 GPT-3 trials -- BBC has five offices in Germany. 03:16:55 GPT-3 trials -- Database prompt 03:20:23 GPT-3 trials -- Python 03:20:31 GPT-3 trials -- Patterns 03:21:01 GPT-3 trials -- Database again 03:25:11 GPT-3 trials -- GPT-3 experiment -- the trophy doesn’t fit in the suitcase 03:27:32 GPT-3 trials -- Scrambling words 03:30:41 GPT-3 trials -- PDF cleanup example (Gwern) 03:35:03 GPT-3 trials -- Word breaking and simple text patterns 03:37:16 GPT-3 trials -- Typing of entities 03:38:30 GPT-3 trials -- Basic Python append 03:39:07 GPT-3 trials -- Automatic programming? 03:42:31 GPT-3 trials -- Passive aggressive dialog input 03:44:39 GPT-3 trials -- symptoms of depression 03:45:43 GPT-3 trials -- Red shirts reasoning challenge 03:49:59 GPT-3 trials -- Binary encoding 03:50:36 Concluding statements from Walid, Tim and Yannic Pod version: https://anchor.fm/machinelearningstreettalk/episodes/031-WE-GOT-ACCESS-TO-GPT-3--With-Gary-Marcus--Walid-Saba-and-Connor-Leahy-en2h1k Connor Leahy: https://www.linkedin.com/in/connor-j-leahy/ https://twitter.com/NPCollapse Eleuther.AI Discord -- https://discord.com/invite/vtRgjbM Gary Marcus: https://www.linkedin.com/in/gary-marcus-b6384b4/ https://twitter.com/GaryMarcus https://www.robust.ai Walid Saba: https://www.linkedin.com/in/walidsaba/ https://medium.com/ontologik https://ontologik.ai

detail
{'title': 'WE GOT ACCESS TO GPT-3! [Epic Special Edition]', 'heatmap': [{'end': 4420.417, 'start': 4271.291, 'weight': 1}], 'summary': "Discusses gpt-3's limitations, implications, language model enhancements, predictive applications, ai challenges, consciousness, neural networks, future integration, language understanding, machine learning and reasoning, and capabilities with insights into its potential impact on ai development and the need for diverse problem-solving approaches.", 'chapters': [{'end': 1095.748, 'segs': [{'end': 147.378, 'src': 'embed', 'start': 107.755, 'weight': 0, 'content': [{'end': 114.86, 'text': "It's an impressive piece of engineering, but it's actually a distraction from what we all want, which is artificial intelligence that we can trust,", 'start': 107.755, 'duration': 7.105}, {'end': 115.781, 'text': 'that we can count on.', 'start': 114.86, 'duration': 0.921}, {'end': 118.263, 'text': "that's reliable and that understands the world around it.", 'start': 115.781, 'duration': 2.482}, {'end': 126.109, 'text': "What GPT-3 does is it takes a whole lot of statistical data generated by humans, and it's parasitic on all that data.", 'start': 118.983, 'duration': 7.126}, {'end': 132.57, 'text': 'And the data is of humans having conversations or just text in things like Reddit.', 'start': 126.887, 'duration': 5.683}, {'end': 136.112, 'text': 'And that text is correlated with the world.', 'start': 133.151, 'duration': 2.961}, {'end': 137.453, 'text': "People don't just talk at random.", 'start': 136.172, 'duration': 1.281}, {'end': 139.054, 'text': 'They talk about the things in the world around it.', 'start': 137.473, 'duration': 1.581}, {'end': 142.576, 'text': 'But all that GPT-3 is really learning is correlations between texts.', 'start': 139.374, 'duration': 3.202}, {'end': 145.837, 'text': 'Gary comments on the hype around GPT-3.', 'start': 142.936, 'duration': 2.901}, {'end': 147.378, 'text': 'There is mass hysteria.', 'start': 146.058, 'duration': 1.32}], 'summary': 'Gpt-3 is impressive, but parasitic on human-generated data, causing mass hysteria.', 'duration': 39.623, 'max_score': 107.755, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w107755.jpg'}, {'end': 327.545, 'src': 'embed', 'start': 302.967, 'weight': 2, 'content': [{'end': 309.793, 'text': 'And so part of the language problem is to do a bunch of inference relative to an understanding of the world.', 'start': 302.967, 'duration': 6.826}, {'end': 312.995, 'text': 'So common ground is a huge part of language.', 'start': 310.433, 'duration': 2.562}, {'end': 317.238, 'text': "Having that background knowledge is important in interpreting what's going on.", 'start': 313.295, 'duration': 3.943}, {'end': 320.7, 'text': 'And GPT is just not helping with that problem.', 'start': 317.758, 'duration': 2.942}, {'end': 327.545, 'text': 'Gary asserts that GPT-free fans are just discarding decades of knowledge in computational linguistics.', 'start': 321.12, 'duration': 6.425}], 'summary': 'Gpt struggles with inference, common ground, and background knowledge, hindering interpretation. gary criticizes gpt-free fans for discarding decades of computational linguistics knowledge.', 'duration': 24.578, 'max_score': 302.967, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w302967.jpg'}, {'end': 376.89, 'src': 'embed', 'start': 346.569, 'weight': 3, 'content': [{'end': 348.05, 'text': "We're just going to reinvent everything.", 'start': 346.569, 'duration': 1.481}, {'end': 351.411, 'text': "We're going to make our sentence predictor do everything.", 'start': 348.43, 'duration': 2.981}, {'end': 356.433, 'text': 'Gary also comments on how hard it is to impute fresh information into GPT-3.', 'start': 351.671, 'duration': 4.762}, {'end': 361, 'text': 'For GPT-3, if you find out something new, you have to retrain the entire system.', 'start': 357.458, 'duration': 3.542}, {'end': 363.562, 'text': 'You want to be able to find out one new fact.', 'start': 361.4, 'duration': 2.162}, {'end': 370.086, 'text': 'We also asked Gary, what does he think about prompt engineering? So what GPT-3 does is basically auto-complete.', 'start': 363.862, 'duration': 6.224}, {'end': 376.89, 'text': "And by playing around with prompting, you're changing what the auto-complete problem is, but you're using the human knowledge to do that.", 'start': 370.146, 'duration': 6.744}], 'summary': 'Reinventing sentence predictor, gpt-3 requires entire system retraining for new information, prompt engineering alters auto-complete using human knowledge.', 'duration': 30.321, 'max_score': 346.569, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w346569.jpg'}, {'end': 694.373, 'src': 'embed', 'start': 660.869, 'weight': 4, 'content': [{'end': 663.971, 'text': "What's exciting is that Google is fundamentally a UX company.", 'start': 660.869, 'duration': 3.102}, {'end': 668.775, 'text': 'They found a way to make it easy for us humans,', 'start': 664.272, 'duration': 4.503}, {'end': 674.138, 'text': 'to make it simple for us to communicate what we want from the machine and get answers that are useful to us.', 'start': 668.775, 'duration': 5.363}, {'end': 682.084, 'text': "It's because there's so much brainpower and engineering effort went into making a Google that is aligned to me as a human.", 'start': 674.738, 'duration': 7.346}, {'end': 684.585, 'text': "If an alien used Google, it wouldn't work at all.", 'start': 682.384, 'duration': 2.201}, {'end': 687.508, 'text': 'Google is this hierarchy of systems.', 'start': 684.986, 'duration': 2.522}, {'end': 694.373, 'text': 'I think what really made Google overtake AltaVista in the early 2000s is the PageRank algorithm, which is a ranker.', 'start': 687.828, 'duration': 6.545}], 'summary': "Google's success is attributed to its human-centered ux approach and the pagerank algorithm, leading to overtaking altavista in the early 2000s.", 'duration': 33.504, 'max_score': 660.869, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w660869.jpg'}, {'end': 951.331, 'src': 'embed', 'start': 925.379, 'weight': 5, 'content': [{'end': 931.863, 'text': "It's such a complex problem, right? I spoke with Keith Duggar about why we use machine learning.", 'start': 925.379, 'duration': 6.484}, {'end': 940.888, 'text': 'What is understanding and some potential architectures for artificial general intelligence? I think the future lies in hybrid systems.', 'start': 932.363, 'duration': 8.525}, {'end': 951.331, 'text': "using programming and things like logic and mathematics and whatever in the domain where they're useful,", 'start': 942.709, 'duration': 8.622}], 'summary': 'Discussing the potential of hybrid systems for artificial general intelligence with keith duggar.', 'duration': 25.952, 'max_score': 925.379, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w925379.jpg'}], 'start': 21.608, 'title': 'Gpt-3 and future of language models', 'summary': "Discusses the limitations and challenges of gpt-3, including its reliance on statistical data, lack of true understanding, and the potential for agi. it also explores google's ux success, gpt-3's significance in ai development, and the need for a hybrid system of machine learning and logic.", 'chapters': [{'end': 281.526, 'start': 21.608, 'title': 'Gpt-3: hype vs reality', 'summary': 'Discusses the limitations of gpt-3 and its lack of true understanding, highlighting its reliance on statistical data and the illusion of reasoning, while questioning its potential for agi and the dichotomy between pattern matching and reasoning.', 'duration': 259.918, 'highlights': ["GPT-3's limitations and lack of true understanding, relying on statistical data and the illusion of reasoning.", 'The discussion on the potential for GPT-4 to incorporate inputs beyond text, such as images, touch, smell, and gravity, raising questions about its potential as AGI.', 'The distinction between reasoning and pattern matching, highlighting the dichotomy and questioning the practicality in the real world.', "The skepticism towards GPT-3's capabilities, emphasizing its reliance on mass hysteria and cherry-picked examples, leading to a distorted perception of its actual performance.", "Yannick and Gary Marcus's skepticism towards the broader impact statements, questioning their informational value in practice."]}, {'end': 636.991, 'start': 281.526, 'title': 'Challenges of gpt-3 and future of language models', 'summary': 'Highlights the fundamental challenges facing gpt-3 in achieving natural language understanding, including the lack of common ground in language, the need for background knowledge, and the limitations in imputing fresh information. it also discusses the disregard for decades of knowledge in computational linguistics by gpt-3 fans, the potential of prompt engineering, and the debate about scaling up language models leading to artificial general intelligence.', 'duration': 355.465, 'highlights': ['The real problem in language is the lack of common ground and the need for background knowledge for interpretation. Lack of common ground in language, need for background knowledge for interpretation', 'GPT-3 faces challenges in imputing fresh information and requires retraining the entire system to accommodate new facts. Challenges in imputing fresh information, need for retraining the entire system', 'GPT-3 fans have disregarded decades of knowledge in computational linguistics, showing intellectual arrogance. Disregard for decades of knowledge in computational linguistics', 'The debate about the potential of prompt engineering and its reliance on human knowledge. Debate about prompt engineering, reliance on human knowledge', 'The discussion on whether scaling up language models will lead to artificial general intelligence, with a 30% possibility according to one perspective. Debate about scaling up language models leading to artificial general intelligence, 30% possibility']}, {'end': 1095.748, 'start': 637.812, 'title': "Google's ux and gpt-3: the future of ai", 'summary': "Discusses google's success as a ux company, the significance of gpt-3 in ai development, and debates around natural language understanding and reasoning, emphasizing the need for a hybrid system of machine learning and logic. key points include google's focus on ux, gpt-3's potential as a foundation for the next trillion-dollar company, and the distinction between pattern recognition and reasoning.", 'duration': 457.936, 'highlights': ["Google's success is attributed to its UX focus, making it easy for humans to communicate with machines and obtain useful answers, with the PageRank algorithm being a key factor in overtaking AltaVista in the early 2000s. Google's success is linked to its UX approach, simplifying human-machine communication; PageRank algorithm's role in surpassing AltaVista.", "GPT-3 is positioned as a foundational element for the next major company, likened to Google's early days overtaking AltaVista, and it is suggested that GPT-3 prompts should be tested with high schoolers to gauge its capabilities. GPT-3's potential as the basis for a future major company; proposal to test GPT-3 prompts with high schoolers for assessment.", "Debates arise around GPT-3's capabilities, with a call to anthropomorphize it more, concerns over its lack of conversational structure similar to Facebook's Blender chatbot, and comparisons to human writing processes highlighting the need for improvement. Discussions on GPT-3's capabilities and improvements, proposing anthropomorphization, lack of conversational structure, and comparisons to human writing processes.", 'The distinction between pattern recognition and natural language understanding is emphasized, with the assertion that language understanding is a binary decision and separate from processing text, leading to debates on the complexities of language understanding and computational linguistics. Emphasis on the difference between pattern recognition and language understanding; debates on language understanding complexities and computational linguistics.', 'The need for a hybrid system incorporating machine learning and logic for artificial general intelligence, along with discussions on defining intelligence, reasoning, and pattern recognition, is stressed, suggesting a collective agreement on these definitions. Advocacy for a hybrid system of machine learning and logic for artificial general intelligence; discussions on defining intelligence, reasoning, and pattern recognition.']}], 'duration': 1074.14, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w21608.jpg', 'highlights': ["GPT-3's limitations and lack of true understanding, relying on statistical data and the illusion of reasoning.", "The skepticism towards GPT-3's capabilities, emphasizing its reliance on mass hysteria and cherry-picked examples, leading to a distorted perception of its actual performance.", 'The real problem in language is the lack of common ground and the need for background knowledge for interpretation.', 'GPT-3 faces challenges in imputing fresh information and requires retraining the entire system to accommodate new facts.', "Google's success is attributed to its UX focus, making it easy for humans to communicate with machines and obtain useful answers, with the PageRank algorithm being a key factor in overtaking AltaVista in the early 2000s.", 'The need for a hybrid system incorporating machine learning and logic for artificial general intelligence, along with discussions on defining intelligence, reasoning, and pattern recognition, is stressed, suggesting a collective agreement on these definitions.']}, {'end': 2114.129, 'segs': [{'end': 1124.877, 'src': 'embed', 'start': 1096.348, 'weight': 0, 'content': [{'end': 1102.372, 'text': 'In this part of the show, you can see our visceral reaction in real time when we first played with GPT-3.', 'start': 1096.348, 'duration': 6.024}, {'end': 1107.876, 'text': "Apparently, because of the byte pair encoding, it's really bad if you don't put spaces.", 'start': 1102.852, 'duration': 5.024}, {'end': 1115.171, 'text': "How does the database respond when maybe not? Booyah! It's better than before.", 'start': 1108.036, 'duration': 7.135}, {'end': 1120.555, 'text': 'The green in squatting are squatting, and limbled, they are squatting.', 'start': 1116.552, 'duration': 4.003}, {'end': 1124.877, 'text': "If I were GPT-3, I'd just be like, fuck you.", 'start': 1121.255, 'duration': 3.622}], 'summary': 'Initial reaction to gpt-3 was mixed, but it improved database responses.', 'duration': 28.529, 'max_score': 1096.348, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1096348.jpg'}, {'end': 1195.026, 'src': 'embed', 'start': 1165.616, 'weight': 1, 'content': [{'end': 1166.436, 'text': "you'll get something.", 'start': 1165.616, 'duration': 0.82}, {'end': 1168.636, 'text': 'you should get something on the manuscript.', 'start': 1166.436, 'duration': 2.2}, {'end': 1173.297, 'text': 'we have now added additional data to the manuscript to address the reviewers concern.', 'start': 1168.636, 'duration': 4.661}, {'end': 1176.358, 'text': 'this is revolutionary.', 'start': 1173.297, 'duration': 3.061}, {'end': 1177.758, 'text': 'this is gonna.', 'start': 1176.358, 'duration': 1.4}, {'end': 1179.779, 'text': 'this is gonna save you a lot of time yelling.', 'start': 1177.758, 'duration': 2.021}, {'end': 1186.38, 'text': 'this is you have any idea how much you optimize the stupid rebuttals and they never do anything.', 'start': 1179.779, 'duration': 6.601}, {'end': 1189.041, 'text': 'the author state in the main findings of the study is to increase.', 'start': 1186.38, 'duration': 2.661}, {'end': 1192.921, 'text': 'They basically look at that.', 'start': 1190.717, 'duration': 2.204}, {'end': 1195.026, 'text': 'Even two is a typical one.', 'start': 1193.322, 'duration': 1.704}], 'summary': "Manuscript updated with additional data to address reviewers' concern, offering revolutionary time-saving optimization.", 'duration': 29.41, 'max_score': 1165.616, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1165616.jpg'}, {'end': 1399.561, 'src': 'embed', 'start': 1372.331, 'weight': 2, 'content': [{'end': 1377.873, 'text': "A lot of people say that he's a grifter, that he's just taking stabs at the tech industry continuously.", 'start': 1372.331, 'duration': 5.542}, {'end': 1385.775, 'text': "He famously wasn't given access to GPT-3 and maybe it's just because they hate Gary Marcus and just out of spite they didn't want to give him access.", 'start': 1378.253, 'duration': 7.522}, {'end': 1393.578, 'text': 'But I personally think in the spirit of science, all the researchers that are working in this area should have access to GPT-3.', 'start': 1386.295, 'duration': 7.283}, {'end': 1399.561, 'text': "I think it's fostered a bit of a hype bubble, which isn't particularly good for AI going forwards.", 'start': 1393.758, 'duration': 5.803}], 'summary': "Some feel he's a grifter, denied access to gpt-3, fostering hype bubble for ai.", 'duration': 27.23, 'max_score': 1372.331, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1372331.jpg'}, {'end': 1561.228, 'src': 'embed', 'start': 1531.648, 'weight': 3, 'content': [{'end': 1534.289, 'text': 'I think the examples demonstrating that have been cherry-picked.', 'start': 1531.648, 'duration': 2.641}, {'end': 1544.613, 'text': 'Do I think that GPT-3 could be used for generating articles, text, creative fiction? Definitely.', 'start': 1536.41, 'duration': 8.203}, {'end': 1548.102, 'text': 'It seems to be very good at pattern matching.', 'start': 1546.081, 'duration': 2.021}, {'end': 1558.486, 'text': 'Is it possible that GPT-3 could do something beyond my wildest dreams? Maybe all of this is just a failure of imagination on our part.', 'start': 1548.542, 'duration': 9.944}, {'end': 1561.228, 'text': 'And I have to accept that is a possibility.', 'start': 1559.027, 'duration': 2.201}], 'summary': 'Gpt-3 can be used for generating articles, text, and creative fiction, with potential beyond imagination.', 'duration': 29.58, 'max_score': 1531.648, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1531648.jpg'}, {'end': 1911.342, 'src': 'embed', 'start': 1882.479, 'weight': 5, 'content': [{'end': 1887.642, 'text': 'Now, they are transductive autoregressive language models, which means they just predict the next word and the next word.', 'start': 1882.479, 'duration': 5.163}, {'end': 1890.244, 'text': 'So that means you put a prompt in here.', 'start': 1888.103, 'duration': 2.141}, {'end': 1896.01, 'text': 'And it might be, hello, my name is.', 'start': 1892.287, 'duration': 3.723}, {'end': 1898.052, 'text': 'So these are all tokens.', 'start': 1896.831, 'duration': 1.221}, {'end': 1900.373, 'text': 'So this is your initial prompt.', 'start': 1898.652, 'duration': 1.721}, {'end': 1906.098, 'text': 'You put this into the language model and then what happens is it will predict the next word.', 'start': 1900.834, 'duration': 5.264}, {'end': 1911.342, 'text': 'So Tim, and then it will predict the next word after Tim has been predicted.', 'start': 1906.558, 'duration': 4.784}], 'summary': 'Transductive autoregressive language models predict next words based on prompts.', 'duration': 28.863, 'max_score': 1882.479, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1882479.jpg'}, {'end': 1978.792, 'src': 'embed', 'start': 1950.477, 'weight': 4, 'content': [{'end': 1954.381, 'text': "So let's assume that we take the top three.", 'start': 1950.477, 'duration': 3.904}, {'end': 1963.107, 'text': 'So we might have my, we might have oranges and this might be probability 0.8, 0.9, 0.1.', 'start': 1954.681, 'duration': 8.426}, {'end': 1964.989, 'text': 'And there might actually be some other ones over there.', 'start': 1963.109, 'duration': 1.88}, {'end': 1967.85, 'text': "There'll be many, but let's just say we take the top three.", 'start': 1965.409, 'duration': 2.441}, {'end': 1973.951, 'text': 'What we then can do is put these words into GPT-3 and we can see where this goes.', 'start': 1968.23, 'duration': 5.721}, {'end': 1976.272, 'text': 'We can actually develop this into a kind of tree structure.', 'start': 1973.991, 'duration': 2.281}, {'end': 1978.792, 'text': "So let's go through the thought process here.", 'start': 1976.752, 'duration': 2.04}], 'summary': 'Using top three choices with probabilities, inputting into gpt-3, and developing a tree structure', 'duration': 28.315, 'max_score': 1950.477, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1950477.jpg'}, {'end': 2122.872, 'src': 'embed', 'start': 2093.297, 'weight': 6, 'content': [{'end': 2095.478, 'text': "It's still technically a greedy search,", 'start': 2093.297, 'duration': 2.181}, {'end': 2101.122, 'text': "but it's a heuristic search algorithm that explores a graph by expanding the most promising node in a limited set.", 'start': 2095.478, 'duration': 5.644}, {'end': 2103.724, 'text': "It's an optimization of the best first search.", 'start': 2101.523, 'duration': 2.201}, {'end': 2108.847, 'text': 'So BFS, you might remember that from your computer science classes that reduces its memory requirements.', 'start': 2103.764, 'duration': 5.083}, {'end': 2114.129, 'text': "It's a graph search which orders all the partial solutions according to some heuristic.", 'start': 2109.347, 'duration': 4.782}, {'end': 2119.191, 'text': 'But in beam search, only a predetermined number of the best partial solutions are kept as candidates.', 'start': 2114.409, 'duration': 4.782}, {'end': 2121.032, 'text': "So thus, it's a greedy algorithm.", 'start': 2119.531, 'duration': 1.501}, {'end': 2122.872, 'text': 'Okay, so coming back to this.', 'start': 2121.492, 'duration': 1.38}], 'summary': 'Heuristic beam search is a greedy algorithm that explores a graph by expanding the most promising node in a limited set, optimizing memory requirements and keeping a predetermined number of best partial solutions as candidates.', 'duration': 29.575, 'max_score': 2093.297, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2093297.jpg'}], 'start': 1096.348, 'title': 'Gpt-3 and its implications', 'summary': 'Delves into real-time reactions to gpt-3, manuscript optimization, controversy surrounding gpt-3, its potential for agi, limitations, and querying language models strategies, potentially saving significant time and offering insights into its capabilities and future developments.', 'chapters': [{'end': 1186.38, 'start': 1096.348, 'title': 'Gpt-3 reaction and manuscript optimization', 'summary': "Reveals our real-time reaction to gpt-3 and discusses the optimization of a manuscript through the addition of data to address reviewers' concerns, potentially saving a significant amount of time in the process.", 'duration': 90.032, 'highlights': ["The optimization of a manuscript through the addition of data to address reviewers' concerns, potentially saving a significant amount of time in the process.", 'Real-time reaction to GPT-3, highlighting the impact of byte pair encoding on text generation.', 'Acknowledgement of constructive comments from reviewers and the value of their feedback in a paper.']}, {'end': 1847.921, 'start': 1186.38, 'title': 'Gpt-3 and agi controversy', 'summary': 'Discusses the controversy surrounding gpt-3 and its potential for artificial general intelligence (agi), featuring viewpoints from critics and advocates, while showcasing interactive experiences with gpt-3. it also addresses the limitations of gpt-3 in non-interactive processes and prompt engineering, offering insights into its capabilities and potential future developments.', 'duration': 661.541, 'highlights': ["GPT-3 controversy and viewpoints The chapter presents contrasting viewpoints from critics and advocates of GPT-3, including Gary Marcus, Walid Sabah, Connor Leahy, and Keith Duggar, highlighting the controversy and diverse perspectives surrounding GPT-3's potential for artificial general intelligence.", 'Interactive experiences with GPT-3 It showcases interactive experiences with GPT-3, demonstrating its capabilities in generating articles, text, and creative fiction, while also exploring the limitations of GPT-3 in non-interactive processes and prompt engineering.', 'Limitations and potential of GPT-3 The chapter addresses the limitations of GPT-3 in non-interactive processes and prompt engineering, offering insights into its capabilities and potential future developments, while emphasizing the need for further exploration and creative applications.']}, {'end': 2114.129, 'start': 1848.361, 'title': 'Querying language models strategy', 'summary': 'Discusses the strategy for querying language models, explaining the deterministic nature of language models and the use of transductive autoregressive and generative models, along with greedy algorithm and beam search strategies.', 'duration': 265.768, 'highlights': ["GPT-3 provides a probability distribution for the next word based on a given prompt, allowing exploration of potential word options. GPT-3 offers a probability distribution for the next word, enabling the exploration of potential word options, such as 'my' and 'oranges', with associated probabilities like 0.8 and 0.9.", 'Explanation of the transductive autoregressive and generative model nature of language models, predicting the next word based on the initial prompt and the previously predicted words. Elaboration on the transductive autoregressive and generative model nature of language models, involving the prediction of the next word based on the initial prompt and the previously predicted words.', 'Discussion of the greedy algorithm in the context of exploring the tree of possibilities in language models, emphasizing the strategy of making locally optimal choices at each stage. Exploration of the greedy algorithm for traversing the tree of possibilities in language models, highlighting the approach of making locally optimal choices at each stage.', 'Introduction of beam search as an improvement on the greedy search strategy, involving the exploration of a limited set of promising nodes in a graph. Introduction of beam search as an enhancement to the greedy search strategy, which involves exploring a limited set of promising nodes in a graph, optimizing the best first search and reducing memory requirements.']}], 'duration': 1017.781, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w1096348.jpg', 'highlights': ['Real-time reaction to GPT-3, highlighting the impact of byte pair encoding on text generation.', "The optimization of a manuscript through the addition of data to address reviewers' concerns, potentially saving a significant amount of time in the process.", "GPT-3 controversy and viewpoints The chapter presents contrasting viewpoints from critics and advocates of GPT-3, including Gary Marcus, Walid Sabah, Connor Leahy, and Keith Duggar, highlighting the controversy and diverse perspectives surrounding GPT-3's potential for artificial general intelligence.", 'Interactive experiences with GPT-3 It showcases interactive experiences with GPT-3, demonstrating its capabilities in generating articles, text, and creative fiction, while also exploring the limitations of GPT-3 in non-interactive processes and prompt engineering.', "GPT-3 provides a probability distribution for the next word based on a given prompt, allowing exploration of potential word options. GPT-3 offers a probability distribution for the next word, enabling the exploration of potential word options, such as 'my' and 'oranges', with associated probabilities like 0.8 and 0.9.", 'Explanation of the transductive autoregressive and generative model nature of language models, predicting the next word based on the initial prompt and the previously predicted words. Elaboration on the transductive autoregressive and generative model nature of language models, involving the prediction of the next word based on the initial prompt and the previously predicted words.', 'Introduction of beam search as an improvement on the greedy search strategy, involving the exploration of a limited set of promising nodes in a graph. Introduction of beam search as an enhancement to the greedy search strategy, which involves exploring a limited set of promising nodes in a graph, optimizing the best first search and reducing memory requirements.']}, {'end': 3154.691, 'segs': [{'end': 2155.347, 'src': 'embed', 'start': 2130.415, 'weight': 0, 'content': [{'end': 2137.198, 'text': 'But the problem is when you use this deterministic approach in language models, you get these repeating cycles right,', 'start': 2130.415, 'duration': 6.783}, {'end': 2144.522, 'text': 'so it might produce a little block of text here and then it will just produce the same block of text and the same block of text and the same block of text,', 'start': 2137.198, 'duration': 7.324}, {'end': 2149.924, 'text': "and it's essentially getting itself stuck in a loop and it doesn't produce very good quality text.", 'start': 2144.522, 'duration': 5.402}, {'end': 2155.347, 'text': 'and the way to improve that is to introduce some randomness into the way that we query these models.', 'start': 2149.924, 'duration': 5.423}], 'summary': 'Deterministic language models cause repeating text cycles, introducing randomness improves quality.', 'duration': 24.932, 'max_score': 2130.415, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2130415.jpg'}, {'end': 2338.564, 'src': 'embed', 'start': 2309.689, 'weight': 1, 'content': [{'end': 2315.471, 'text': "So we've seen some examples online of GPT-3 solving some basic character analogies.", 'start': 2309.689, 'duration': 5.782}, {'end': 2322.614, 'text': 'The first thing we notice is that it helps to put spaces between the characters because of the byte pair encoding issue.', 'start': 2316.732, 'duration': 5.882}, {'end': 2324.875, 'text': 'And this is interesting, actually.', 'start': 2323.514, 'duration': 1.361}, {'end': 2327.217, 'text': 'Melanie Mitchell posted on Twitter.', 'start': 2325.135, 'duration': 2.082}, {'end': 2332.14, 'text': 'She actually wrote an article about this, but one of the things she posted here was analogies.', 'start': 2327.877, 'duration': 4.263}, {'end': 2338.564, 'text': "I'm quite interested using GPT-3 to demonstrate something that requires reasoning, not memorization.", 'start': 2332.54, 'duration': 6.024}], 'summary': 'Gpt-3 can solve character analogies, but requires spaces due to byte pair encoding issue. melanie mitchell shared this on twitter, emphasizing the need for reasoning over memorization.', 'duration': 28.875, 'max_score': 2309.689, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2309689.jpg'}, {'end': 2480.497, 'src': 'embed', 'start': 2418.99, 'weight': 2, 'content': [{'end': 2425.818, 'text': "So there are combinatorially many paths you can take through this tree if we're doing stochastic sampling at every single iteration.", 'start': 2418.99, 'duration': 6.828}, {'end': 2430.303, 'text': 'So in my opinion, doing it five times is nowhere near good enough.', 'start': 2426.258, 'duration': 4.045}, {'end': 2441.108, 'text': "I would go further and say it's not possible to come up with any measure of effectiveness unless you are prepared to have a very large sampling of that tree.", 'start': 2431.244, 'duration': 9.864}, {'end': 2445.329, 'text': 'so Melanie starts off by doing zero shot analogy making, which is here.', 'start': 2441.108, 'duration': 4.221}, {'end': 2448.771, 'text': 'so we just put the question in once and then she tried it a bunch of times.', 'start': 2445.329, 'duration': 3.442}, {'end': 2451.372, 'text': 'she was expecting pqs, but she never got pqs.', 'start': 2448.771, 'duration': 2.601}, {'end': 2457.454, 'text': 'so she concluded that GPT-3 cannot perform zero shot analogy making in this domain.', 'start': 2451.372, 'duration': 6.082}, {'end': 2463.558, 'text': 'next Melanie gave it one training example, So giving it an example of what the solution is to the problem.', 'start': 2457.454, 'duration': 6.104}, {'end': 2467.283, 'text': 'And this time it answered it correctly on every single trial.', 'start': 2464.119, 'duration': 3.164}, {'end': 2471.327, 'text': 'So one-shot learning appears to work for these simple analogies.', 'start': 2467.883, 'duration': 3.444}, {'end': 2475.772, 'text': 'Then she tried to see if GPT-3 can generalize to strings of different lengths.', 'start': 2471.708, 'duration': 4.064}, {'end': 2480.497, 'text': 'And with one training example, GPT-3 could not generalize to the longest string.', 'start': 2476.112, 'duration': 4.385}], 'summary': "Stochastic sampling needs extensive tree sampling for effectiveness, demonstrated by gpt-3's zero and one shot analogy making and string generalization capabilities.", 'duration': 61.507, 'max_score': 2418.99, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2418990.jpg'}, {'end': 2578.983, 'src': 'embed', 'start': 2550.188, 'weight': 5, 'content': [{'end': 2551.989, 'text': "The program's performance was mixed.", 'start': 2550.188, 'duration': 1.801}, {'end': 2560.033, 'text': "GPT-3 was not designed to make analogies per se, and it's surprising that it's able to do so reasonably well on some of these problems,", 'start': 2552.509, 'duration': 7.524}, {'end': 2562.394, 'text': "although in many cases it's not able to generalize well.", 'start': 2560.033, 'duration': 2.361}, {'end': 2568.318, 'text': 'Moreover, when it does succeed, it does so only after being shown some number of training examples.', 'start': 2563.055, 'duration': 5.263}, {'end': 2574.601, 'text': 'To my mind, this defeats the purpose of analogy making, which is perhaps the only zero-shot learning mechanism in human cognition.', 'start': 2568.678, 'duration': 5.923}, {'end': 2578.983, 'text': 'That is, you adapt knowledge you have about one situation to a new situation.', 'start': 2574.981, 'duration': 4.002}], 'summary': "Gpt-3's mixed performance in making analogies challenges zero-shot learning.", 'duration': 28.795, 'max_score': 2550.188, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2550188.jpg'}, {'end': 2633.387, 'src': 'embed', 'start': 2609.834, 'weight': 8, 'content': [{'end': 2616.376, 'text': "So many people speculated that GPT-3's training data included papers or books that discussed copycat analogy problems.", 'start': 2609.834, 'duration': 6.542}, {'end': 2618.736, 'text': 'And it could be using that by memorization.', 'start': 2616.736, 'duration': 2}, {'end': 2627.142, 'text': "She tried different letter strings and she concluded that the inclusion in GPT-3's training data is not likely to be responsible for its performance.", 'start': 2619.196, 'duration': 7.946}, {'end': 2633.387, 'text': 'Others also noted that there might be some kind of successorship, either alphabetically or numerically,', 'start': 2627.642, 'duration': 5.745}], 'summary': "Gpt-3's training data not likely responsible for its performance, speculated successorship.", 'duration': 23.553, 'max_score': 2609.834, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2609834.jpg'}, {'end': 2896.415, 'src': 'embed', 'start': 2871.272, 'weight': 7, 'content': [{'end': 2878.579, 'text': "saying that it's a database prompt where GPT-3 seems to do some sort of reasoning and understanding of its ignorance.", 'start': 2871.272, 'duration': 7.307}, {'end': 2880.721, 'text': 'So the database knows nothing.', 'start': 2879.18, 'duration': 1.541}, {'end': 2882.883, 'text': "The database knows everything that's added to it.", 'start': 2880.901, 'duration': 1.982}, {'end': 2884.845, 'text': 'The database knows nothing else.', 'start': 2883.264, 'duration': 1.581}, {'end': 2890.11, 'text': 'When asked a question, if the answer has been added to the database, the database says the answer.', 'start': 2885.426, 'duration': 4.684}, {'end': 2896.415, 'text': 'When asked a question, if the answer has not been added to the database, the database says it does not know.', 'start': 2890.771, 'duration': 5.644}], 'summary': 'Gpt-3 operates as a database prompt, providing answers if data is present, otherwise admitting ignorance.', 'duration': 25.143, 'max_score': 2871.272, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2871272.jpg'}, {'end': 3064.603, 'src': 'embed', 'start': 3040.419, 'weight': 6, 'content': [{'end': 3049.981, 'text': 'So the key question is is this an example of us building an application, or is this an example of us being led by GPT-3 to do what it wants to do,', 'start': 3040.419, 'duration': 9.562}, {'end': 3050.981, 'text': 'not what we want to do?', 'start': 3049.981, 'duration': 1}, {'end': 3057.322, 'text': "Now it's tempting to believe that this is an example of natural language understanding, but we can start to play with this right?", 'start': 3051.421, 'duration': 5.901}, {'end': 3062.723, 'text': "So why don't we change database to be the gym till?", 'start': 3057.342, 'duration': 5.381}, {'end': 3064.603, 'text': "Okay, so we'll change all of that.", 'start': 3063.183, 'duration': 1.42}], 'summary': 'Discussion on potential influence of gpt-3 on application development.', 'duration': 24.184, 'max_score': 3040.419, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3040419.jpg'}], 'start': 2114.409, 'title': "Gpt-3's language model enhancements", 'summary': "Delves into enhancing language models with randomness via temperature scaling and top-k sampling, showcasing gpt-3's performance in analogy making and analyzing its behavior with database prompts.", 'chapters': [{'end': 2441.108, 'start': 2114.409, 'title': 'Improving language models with randomness', 'summary': 'Discusses the limitations of deterministic language models, proposes the introduction of randomness through temperature scaling and top-k sampling, and provides examples of using gpt-3 for character analogies with insights into its stochastic behavior and byte pair encoding issue.', 'duration': 326.699, 'highlights': ['Introduction of randomness through temperature scaling and top-k sampling to improve deterministic language models, addressing issues such as repeating cycles and low text quality.', 'Insights into using GPT-3 for character analogies and the impact of byte pair encoding issue, highlighting the need for stochastic sampling and the impact of temperature on model performance.', 'Discussion on the impact of stochastic sampling and branching factor in GPT-3, emphasizing the necessity for extensive sampling to assess effectiveness.']}, {'end': 2818.075, 'start': 2441.108, 'title': "Gpt-3's analogy making performance", 'summary': "Highlights melanie's experiments testing gpt-3's ability in analogy making, showing its performance in zero-shot, one-shot, and multi-shot learning, as well as its struggles with generalization and abstract similarity, ultimately questioning its effectiveness in capturing human-like analogy making abilities.", 'duration': 376.967, 'highlights': ["Melanie tested GPT-3's performance in zero-shot, one-shot, and multi-shot learning for analogy making, showing its struggles with generalization and abstract similarity. Melanie experimented with zero-shot, one-shot, and multi-shot learning to test GPT-3's ability in analogy making, demonstrating its challenges in generalization and abstract similarity.", 'GPT-3 showed correct answers in one-shot learning for simple analogies but struggled to generalize to longer strings, even with multiple training examples. GPT-3 demonstrated correct answers in one-shot learning for simple analogies, but faced challenges in generalizing to longer strings, even with multiple training examples.', "The program's performance in letter string analogy problems was mixed, with instances of accurate responses and struggles in generalization, leading to questions about its effectiveness in capturing human-like analogy making abilities. GPT-3's performance in letter string analogy problems was varied, with instances of accurate responses and struggles in generalization, prompting doubts about its ability to capture human-like analogy making abilities."]}, {'end': 3154.691, 'start': 2820.845, 'title': 'Gpt-3 database prompt analysis', 'summary': 'Discusses an analysis of a database prompt using gpt-3, highlighting its influence in guiding the conversation and the limitations of natural language understanding, with a focus on the impact of different word substitutions on the prompt.', 'duration': 333.846, 'highlights': ["The application pathway was influenced by GPT-3, raising questions on whether the outcome is a result of GPT-3's influence or the intended application, indicating a potential limitation of GPT-3's natural language understanding.", "Despite altering key words like 'database', 'no', 'question', and 'answer', GPT-3's responses remained consistent, implying a reliance on pattern matching rather than genuine language understanding.", "The database prompt, when manipulated with different word substitutions, did not significantly impact GPT-3's responses, suggesting that its learning from training corpus did not influence its pattern matching capabilities."]}], 'duration': 1040.282, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w2114409.jpg', 'highlights': ['Introduction of randomness through temperature scaling and top-k sampling to improve deterministic language models, addressing issues such as repeating cycles and low text quality.', 'Insights into using GPT-3 for character analogies and the impact of byte pair encoding issue, highlighting the need for stochastic sampling and the impact of temperature on model performance.', 'Discussion on the impact of stochastic sampling and branching factor in GPT-3, emphasizing the necessity for extensive sampling to assess effectiveness.', "Melanie tested GPT-3's performance in zero-shot, one-shot, and multi-shot learning for analogy making, showing its struggles with generalization and abstract similarity.", 'GPT-3 showed correct answers in one-shot learning for simple analogies but struggled to generalize to longer strings, even with multiple training examples.', "The program's performance in letter string analogy problems was mixed, with instances of accurate responses and struggles in generalization, leading to questions about its effectiveness in capturing human-like analogy making abilities.", "The application pathway was influenced by GPT-3, raising questions on whether the outcome is a result of GPT-3's influence or the intended application, indicating a potential limitation of GPT-3's natural language understanding.", "Despite altering key words like 'database', 'no', 'question', and 'answer', GPT-3's responses remained consistent, implying a reliance on pattern matching rather than genuine language understanding.", "The database prompt, when manipulated with different word substitutions, did not significantly impact GPT-3's responses, suggesting that its learning from training corpus did not influence its pattern matching capabilities."]}, {'end': 3988.387, 'segs': [{'end': 3219.985, 'src': 'embed', 'start': 3182.107, 'weight': 0, 'content': [{'end': 3189.534, 'text': 'In this segment of the show, I use GPT-3 to generate the broader impact statements, which are now mandatory on the NeurIPS papers.', 'start': 3182.107, 'duration': 7.427}, {'end': 3193.858, 'text': "So what's another example of an application that we can write using GPT-3?", 'start': 3190.454, 'duration': 3.404}, {'end': 3198.984, 'text': 'One interesting thing that we thought of was generating the broader impact statements.', 'start': 3194.259, 'duration': 4.725}, {'end': 3203.95, 'text': 'You might have heard of these broader impact statements now, which have become mandatory,', 'start': 3199.004, 'duration': 4.946}, {'end': 3212.618, 'text': "and yannick said that he felt that in practice these broader impact statements they don't really carry any information, they're completely formulaic,", 'start': 3203.95, 'duration': 8.668}, {'end': 3216.702, 'text': 'they tend to target some high and level research area that the paper is broadly a part of,', 'start': 3212.618, 'duration': 4.084}, {'end': 3219.985, 'text': "but don't really talk about the contents of the paper itself.", 'start': 3216.702, 'duration': 3.283}], 'summary': 'Using gpt-3 to generate mandatory broader impact statements for neurips papers, which are seen as formulaic and lacking in specific information.', 'duration': 37.878, 'max_score': 3182.107, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3182107.jpg'}, {'end': 3520.48, 'src': 'embed', 'start': 3490.234, 'weight': 2, 'content': [{'end': 3494.179, 'text': 'It is the responsibility of the scientist to explicitly try to minimize such biases.', 'start': 3490.234, 'duration': 3.945}, {'end': 3494.66, 'text': 'Of course it is.', 'start': 3494.199, 'duration': 0.461}, {'end': 3501.308, 'text': 'Given that the larger data set is harder to properly investigate the biases in the corpus,', 'start': 3495.601, 'duration': 5.707}, {'end': 3505.032, 'text': 'we believe that notions of fairness need to be explicitly tracked.', 'start': 3501.308, 'duration': 3.724}, {'end': 3508.176, 'text': 'Regulating fairness, bias, fairness, bias, okay, yeah.', 'start': 3505.373, 'duration': 2.803}, {'end': 3513.598, 'text': 'The third one is backpropagating linearly improves transferability of adversarial examples.', 'start': 3508.777, 'duration': 4.821}, {'end': 3520.48, 'text': 'So this is talking about the vulnerability of deep neural networks to adversarial examples has drawn great attention from the community.', 'start': 3514.739, 'duration': 5.741}], 'summary': 'Scientists must minimize biases; fairness needs explicit tracking. vulnerability of deep neural networks to adversarial examples draws community attention.', 'duration': 30.246, 'max_score': 3490.234, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3490234.jpg'}, {'end': 3642.366, 'src': 'embed', 'start': 3615.644, 'weight': 3, 'content': [{'end': 3625.335, 'text': "So I still say that the number one reason to use GPT-3 is to plagiarize work and to generate text because no one will ever know that you've done it.", 'start': 3615.644, 'duration': 9.691}, {'end': 3632.935, 'text': "It's amazing how easy it is to just change text so it doesn't look like it's being plagiarized.", 'start': 3627.008, 'duration': 5.927}, {'end': 3641.505, 'text': 'Anyway, the second completion is this work exploits the transferability of adversarial examples to effectively.', 'start': 3633.836, 'duration': 7.669}, {'end': 3642.366, 'text': "that's quite clever, isn't it?", 'start': 3641.505, 'duration': 0.861}], 'summary': 'Gpt-3 is used for plagiarism and text generation, exploiting adversarial examples.', 'duration': 26.722, 'max_score': 3615.644, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3615644.jpg'}, {'end': 3873.995, 'src': 'embed', 'start': 3849.158, 'weight': 4, 'content': [{'end': 3856.264, 'text': "It's an impressive piece of engineering, but it's actually a distraction from what we all want, which is artificial intelligence that we can trust,", 'start': 3849.158, 'duration': 7.106}, {'end': 3857.205, 'text': 'that we can count on.', 'start': 3856.264, 'duration': 0.941}, {'end': 3859.687, 'text': "that's reliable and that understands the world around it.", 'start': 3857.205, 'duration': 2.482}, {'end': 3867.571, 'text': "What GPT-3 does is it takes a whole lot of statistical data generated by humans, and it's parasitic on all that data.", 'start': 3860.367, 'duration': 7.204}, {'end': 3873.995, 'text': 'And the data is of humans having conversations or just text in things like Reddit.', 'start': 3868.312, 'duration': 5.683}], 'summary': 'Gpt-3 uses statistical data from human conversations to generate text, but distracts from the goal of trustworthy artificial intelligence.', 'duration': 24.837, 'max_score': 3849.158, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3849158.jpg'}], 'start': 3154.691, 'title': "Gpt-3's predictive application", 'summary': "Discusses gpt-3's use in generating broader impact statements, highlighting current limitations, potential impact on scientific progress, and specific examples, while exploring its capabilities, plagiarism potential, reliance on statistical correlations, and critique on achieving artificial general intelligence.", 'chapters': [{'end': 3600.39, 'start': 3154.691, 'title': 'Gpt-3 application in predicting broader impact statements', 'summary': 'Discusses the use of gpt-3 to generate broader impact statements, highlighting the limitations of current broader impact statements and their potential impact on scientific progress, while showcasing specific examples of broader impact statements and gpt-3 predictions.', 'duration': 445.699, 'highlights': ['The chapter discusses the use of GPT-3 to generate broader impact statements GPT-3 is used to generate broader impact statements, showcasing its application in predicting the potential impact of research papers.', 'The limitations of current broader impact statements and their potential impact on scientific progress Current broader impact statements are deemed formulaic and carry little information, potentially stifling innovation and scientific progress.', "Specific examples of broader impact statements and GPT-3 predictions Specific examples of broader impact statements are analyzed, highlighting biases and the responsibility of scientists, while GPT-3 predictions showcase deviations from the standard 'technology good, technology bad, technology biased' script."]}, {'end': 3988.387, 'start': 3600.45, 'title': 'Gpt-3: understanding and critique', 'summary': 'Explores the capabilities and limitations of gpt-3, highlighting its potential for plagiarism, its reliance on statistical correlations rather than true understanding, and the critique from gary marcus on its limitations in achieving artificial general intelligence.', 'duration': 387.937, 'highlights': ["The number one reason to use GPT-3 is to plagiarize work and to generate text because no one will ever know that you've done it, showcasing its potential for deceptive text generation.", 'GPT-3 is criticized for its reliance on statistical correlations rather than true understanding, leading to cases where it produces nonsensical or dangerous outputs, such as suggesting death in a sentence continuation, highlighting the limitations of its knowledge depth.', 'Gary Marcus criticizes GPT-3 for being a distraction from the goal of achieving reliable artificial intelligence that truly understands the world, emphasizing its reliance on statistical data and correlations rather than genuine understanding.']}], 'duration': 833.696, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3154691.jpg', 'highlights': ['GPT-3 used to generate broader impact statements, predicting potential impact of research papers.', 'Current broader impact statements deemed formulaic, potentially stifling innovation and scientific progress.', 'Specific examples of broader impact statements analyzed, highlighting biases and responsibility of scientists.', 'GPT-3 criticized for potential use in plagiarizing work and generating deceptive text.', "Criticism of GPT-3's reliance on statistical correlations rather than true understanding.", 'Gary Marcus criticizes GPT-3 for being a distraction from achieving reliable artificial intelligence.']}, {'end': 4738.927, 'segs': [{'end': 4017.459, 'src': 'embed', 'start': 3988.487, 'weight': 0, 'content': [{'end': 3991.13, 'text': "And so it's really like driverless cars, right?", 'start': 3988.487, 'duration': 2.643}, {'end': 3996.556, 'text': 'Anybody can build a prototype of a driverless car, but building one that you can count on is really hard.', 'start': 3991.35, 'duration': 5.206}, {'end': 4002.463, 'text': 'You can build a half-assed, if I can use that word, on the air chatbot by looking at a lot of prior data.', 'start': 3996.957, 'duration': 5.506}, {'end': 4004.845, 'text': "That doesn't mean the chatbot has a clue what's going on.", 'start': 4002.503, 'duration': 2.342}, {'end': 4013.595, 'text': "So I actually have a thread on Twitter right now because somebody's told me that GPT-3 was a step towards AGI, artificial general intelligence.", 'start': 4005.566, 'duration': 8.029}, {'end': 4015.457, 'text': 'And I was like are you serious??', 'start': 4014.035, 'duration': 1.422}, {'end': 4017.459, 'text': "And if so, what's that step?", 'start': 4015.637, 'duration': 1.822}], 'summary': "Building reliable driverless cars is hard. gpt-3's role in agi is questionable.", 'duration': 28.972, 'max_score': 3988.487, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3988487.jpg'}, {'end': 4097.126, 'src': 'embed', 'start': 4065.408, 'weight': 3, 'content': [{'end': 4066.969, 'text': 'But there is mass hysteria.', 'start': 4065.408, 'duration': 1.561}, {'end': 4070.111, 'text': "So everybody, The Guardian, The New York Times, they've all got to run these.", 'start': 4067.009, 'duration': 3.102}, {'end': 4073.013, 'text': 'And The Economist all have to run articles about how cool GPT-3 is.', 'start': 4070.151, 'duration': 2.862}, {'end': 4074.854, 'text': 'Often, they produce a sample.', 'start': 4073.533, 'duration': 1.321}, {'end': 4077.735, 'text': 'Frequently, they cherry pick the sample to make it sound better than it is.', 'start': 4074.914, 'duration': 2.821}, {'end': 4082.478, 'text': 'Hats off to The Times the other day actually included some of its cases about dating,', 'start': 4078.115, 'duration': 4.363}, {'end': 4087.34, 'text': 'one of which said and we go for dinner and drinks and dinner and drinks, and dinner and drinks, and dinner and drinks.', 'start': 4082.478, 'duration': 4.862}, {'end': 4093.564, 'text': "If you actually read it, you realize this is nonsense, but there's a lot of press hype around it.", 'start': 4088.761, 'duration': 4.803}, {'end': 4097.126, 'text': "We'll see if there's any really good commercial application.", 'start': 4094.125, 'duration': 3.001}], 'summary': 'Media hyping gpt-3 with cherry-picked samples; commercial potential uncertain.', 'duration': 31.718, 'max_score': 4065.408, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4065408.jpg'}, {'end': 4420.417, 'src': 'heatmap', 'start': 4271.291, 'weight': 1, 'content': [{'end': 4276.397, 'text': "And so if you just rely on what you've seen before, you can get yourself into all kinds of trouble.", 'start': 4271.291, 'duration': 5.106}, {'end': 4280, 'text': "What you really want to be able to do is to say, here's a novel situation.", 'start': 4276.417, 'duration': 3.583}, {'end': 4283.224, 'text': "Here's what I know about relevant knowledge about the world.", 'start': 4280.361, 'duration': 2.863}, {'end': 4285.005, 'text': 'how viruses work, whatever.', 'start': 4283.604, 'duration': 1.401}, {'end': 4287.245, 'text': "And I'm going to try to make some inferences.", 'start': 4285.445, 'duration': 1.8}, {'end': 4293.448, 'text': "Reasoning sometimes can be deductive, like the Socrates is mortal kind of thing, where it's absolutely guaranteed.", 'start': 4287.526, 'duration': 5.922}, {'end': 4295.849, 'text': "Sometimes it's inductive, so you make your best guesses.", 'start': 4293.488, 'duration': 2.361}, {'end': 4303.032, 'text': 'But you make those best guesses with respect to an understanding of the world and its processes, how physics works, how medicine works, et cetera.', 'start': 4296.249, 'duration': 6.783}, {'end': 4310.375, 'text': "And systems like GPT-3 give the illusion of doing that some of the time, but they're not really doing it any of the time.", 'start': 4303.552, 'duration': 6.823}, {'end': 4316.078, 'text': 'relying on kind of the closest example in some corpus where people have done the reasoning.', 'start': 4311.095, 'duration': 4.983}, {'end': 4320.92, 'text': "And these systems can't themselves do the reasoning, cannot be trusted to do it.", 'start': 4316.558, 'duration': 4.362}, {'end': 4325.943, 'text': "So we've spoken to Conor Leahy of Eleuther AI.", 'start': 4321.781, 'duration': 4.162}, {'end': 4332.506, 'text': "They're trying to replicate GPT-3 in an open source variant, and they are, let's say, big fans.", 'start': 4326.263, 'duration': 6.243}, {'end': 4339.23, 'text': 'And what comes up over and over as you talk is this notion of understanding the physical world and so on.', 'start': 4333.087, 'duration': 6.143}, {'end': 4343.332, 'text': 'And Connor would say, this is a distraction.', 'start': 4339.95, 'duration': 3.382}, {'end': 4350.497, 'text': 'This argument is irrelevant because GPT-3, all the input it gets is text.', 'start': 4343.352, 'duration': 7.145}, {'end': 4355.08, 'text': 'So its world consists of only text.', 'start': 4350.837, 'duration': 4.243}, {'end': 4364.786, 'text': "And to say that GPT-3 doesn't do things like understand physics is obvious, because GPT-3 has no notion of physics,", 'start': 4355.76, 'duration': 9.026}, {'end': 4370.509, 'text': 'because its generating function that it tries to approximate includes no physics.', 'start': 4364.786, 'duration': 5.723}, {'end': 4376.833, 'text': 'Would you generally agree with that? You can run that argument in two different ways.', 'start': 4371.129, 'duration': 5.704}, {'end': 4378.874, 'text': 'So it is true.', 'start': 4377.173, 'duration': 1.701}, {'end': 4381.055, 'text': 'It knows nothing about physics.', 'start': 4378.894, 'duration': 2.161}, {'end': 4385.198, 'text': 'Its input does not directly refer to physics or psychology or anything else.', 'start': 4381.496, 'duration': 3.702}, {'end': 4387.619, 'text': 'Yet, it gives the illusion that it does.', 'start': 4385.958, 'duration': 1.661}, {'end': 4394.883, 'text': "The real problem is that GPT-3 is almost like a magician, but you shouldn't believe that the magician is actually doing the trick.", 'start': 4388.179, 'duration': 6.704}, {'end': 4398.165, 'text': 'You need to understand the difference between a magic trick and reality.', 'start': 4395.503, 'duration': 2.662}, {'end': 4405.309, 'text': "If a magician makes it look like they're sawing somebody in half, you've got to know that they're not actually sawing somebody in half.", 'start': 4398.645, 'duration': 6.664}, {'end': 4411.552, 'text': "It's a trick and that you wouldn't want to put an actual person with an actual saw next to the actual magician.", 'start': 4405.609, 'duration': 5.943}, {'end': 4418.976, 'text': 'So GPT-3 is a magic trick that conveys an illusion of understanding something about physics, psychology, and so forth.', 'start': 4411.752, 'duration': 7.224}, {'end': 4420.417, 'text': "It absolutely doesn't.", 'start': 4419.296, 'duration': 1.121}], 'summary': 'Gpt-3 gives the illusion of understanding but lacks reasoning and knowledge of the physical world.', 'duration': 149.126, 'max_score': 4271.291, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4271291.jpg'}, {'end': 4332.506, 'src': 'embed', 'start': 4303.552, 'weight': 1, 'content': [{'end': 4310.375, 'text': "And systems like GPT-3 give the illusion of doing that some of the time, but they're not really doing it any of the time.", 'start': 4303.552, 'duration': 6.823}, {'end': 4316.078, 'text': 'relying on kind of the closest example in some corpus where people have done the reasoning.', 'start': 4311.095, 'duration': 4.983}, {'end': 4320.92, 'text': "And these systems can't themselves do the reasoning, cannot be trusted to do it.", 'start': 4316.558, 'duration': 4.362}, {'end': 4325.943, 'text': "So we've spoken to Conor Leahy of Eleuther AI.", 'start': 4321.781, 'duration': 4.162}, {'end': 4332.506, 'text': "They're trying to replicate GPT-3 in an open source variant, and they are, let's say, big fans.", 'start': 4326.263, 'duration': 6.243}], 'summary': 'Gpt-3 relies on examples, lacks reasoning. eleuther ai replicates gpt-3 in open source.', 'duration': 28.954, 'max_score': 4303.552, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4303552.jpg'}, {'end': 4411.552, 'src': 'embed', 'start': 4381.496, 'weight': 4, 'content': [{'end': 4385.198, 'text': 'Its input does not directly refer to physics or psychology or anything else.', 'start': 4381.496, 'duration': 3.702}, {'end': 4387.619, 'text': 'Yet, it gives the illusion that it does.', 'start': 4385.958, 'duration': 1.661}, {'end': 4394.883, 'text': "The real problem is that GPT-3 is almost like a magician, but you shouldn't believe that the magician is actually doing the trick.", 'start': 4388.179, 'duration': 6.704}, {'end': 4398.165, 'text': 'You need to understand the difference between a magic trick and reality.', 'start': 4395.503, 'duration': 2.662}, {'end': 4405.309, 'text': "If a magician makes it look like they're sawing somebody in half, you've got to know that they're not actually sawing somebody in half.", 'start': 4398.645, 'duration': 6.664}, {'end': 4411.552, 'text': "It's a trick and that you wouldn't want to put an actual person with an actual saw next to the actual magician.", 'start': 4405.609, 'duration': 5.943}], 'summary': 'Gpt-3 is like a magician, creating illusions without actual capabilities, highlighting the difference between magic and reality.', 'duration': 30.056, 'max_score': 4381.496, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4381496.jpg'}, {'end': 4563.774, 'src': 'embed', 'start': 4539.054, 'weight': 5, 'content': [{'end': 4544.879, 'text': "the problem is usually that you get in trouble when you need to go to a set of data that you haven't seen before.", 'start': 4539.054, 'duration': 5.825}, {'end': 4546.52, 'text': "that's different from the data that you have.", 'start': 4544.879, 'duration': 1.641}, {'end': 4552.545, 'text': "The neural networks are actually fine if you're just dealing with data that's exactly like what we've seen before.", 'start': 4546.54, 'duration': 6.005}, {'end': 4559.07, 'text': "There's a technical thing about IID, but basically it means that if your new data are in the same space as the old data, you're good to go.", 'start': 4552.585, 'duration': 6.485}, {'end': 4563.774, 'text': "But, as I've been arguing for 30 years and as Yoshua Bengio has been arguing for the last year,", 'start': 4559.17, 'duration': 4.604}], 'summary': 'Neural networks work well with familiar data in same space as old data.', 'duration': 24.72, 'max_score': 4539.054, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4539054.jpg'}, {'end': 4733.464, 'src': 'embed', 'start': 4701.029, 'weight': 6, 'content': [{'end': 4704.71, 'text': 'So maybe there is a reason to use these language models in that context.', 'start': 4701.029, 'duration': 3.681}, {'end': 4707.991, 'text': "I don't think they help at all in the context of language.", 'start': 4705.33, 'duration': 2.661}, {'end': 4713.053, 'text': 'They might help with problems about synonyms and conceivably they can help with things like parts of speech.', 'start': 4708.331, 'duration': 4.722}, {'end': 4717.034, 'text': "But the real problem in language is that you don't say everything that you mean.", 'start': 4713.113, 'duration': 3.921}, {'end': 4721.436, 'text': 'It would be incredibly tedious if you said everything you mean.', 'start': 4717.814, 'duration': 3.622}, {'end': 4727, 'text': "If I have a ball on the table, I'm probably not going to say, I took measurements of it.", 'start': 4721.697, 'duration': 5.303}, {'end': 4733.464, 'text': "It's actually an exercise ball and it has little grooves in it and it's manufactured by MuscleMax by fitness.", 'start': 4727.08, 'duration': 6.384}], 'summary': 'Language models may assist with synonyms and parts of speech, but the real challenge is conveying meaning efficiently.', 'duration': 32.435, 'max_score': 4701.029, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4701029.jpg'}], 'start': 3988.487, 'title': 'Ai challenges and gpt-3 limitations', 'summary': 'Covers challenges in developing reliable ai technology, citing examples of driverless cars and chatbots, and questions the capability of gpt-3 as a step towards artificial general intelligence. it discusses the limitations and hype surrounding gpt-3, including its inability to fulfill natural language generation and reasoning tasks, and emphasizes the shortcomings of neural networks in learning and extrapolation to unfamiliar data.', 'chapters': [{'end': 4044.096, 'start': 3988.487, 'title': 'Challenges in developing advanced ai technology', 'summary': 'Discusses the challenges in creating reliable ai technology, citing examples of driverless cars and chatbots. it also questions the capability of gpt-3 as a step towards artificial general intelligence, emphasizing the limitations of current ai systems in reasoning and understanding.', 'duration': 55.609, 'highlights': ['Building reliable AI technology is challenging, as seen in the difficulty of creating dependable driverless cars and chatbots.', "The limitations of current AI systems are evident, as exemplified by GPT-3's inability to reason and understand beyond generating synonyms for better searches.", "GPT-3's purported advancement towards artificial general intelligence is questioned, highlighting the inability of the system to articulate complex arguments beyond producing good embeddings."]}, {'end': 4738.927, 'start': 4044.156, 'title': 'Gpt-3: hype vs reality', 'summary': 'Discusses the hype surrounding gpt-3, highlighting its limitations, including its inability to fulfill natural language generation and reasoning tasks, and the illusion it creates in understanding complex concepts like physics and language. it also emphasizes the shortcomings of neural networks in learning and the challenges in extrapolating to unfamiliar data.', 'duration': 694.771, 'highlights': ["GPT-3's hype and limitations: GPT-3's mass hysteria and media coverage are critiqued, highlighting its inability to revolutionize chatbots, fulfill natural language generation and reasoning tasks, and the press' cherry-picking of samples to amplify its capabilities.", "Illusion of understanding: GPT-3 creates an illusion of understanding complex concepts like physics and psychology, despite lacking the capacity to reason about the consequences of its actions or anticipate them. It's compared to a magician's trick, conveying an illusion without actual understanding.", 'Neural networks and learning challenges: The limitations of neural networks in learning from unfamiliar data are discussed, emphasizing the difficulties in extrapolating to different scenarios and the necessity of learning rather than theoretical capabilities.', 'Language model limitations: The chapter discusses the limitations of language models in understanding language, highlighting the challenge of not saying everything that is meant and the inability to explicitly program and understand language constructions.']}], 'duration': 750.44, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w3988487.jpg', 'highlights': ['Building reliable AI technology is challenging, as seen in the difficulty of creating dependable driverless cars and chatbots.', "The limitations of current AI systems are evident, as exemplified by GPT-3's inability to reason and understand beyond generating synonyms for better searches.", "GPT-3's purported advancement towards artificial general intelligence is questioned, highlighting the inability of the system to articulate complex arguments beyond producing good embeddings.", "GPT-3's hype and limitations: GPT-3's mass hysteria and media coverage are critiqued, highlighting its inability to revolutionize chatbots, fulfill natural language generation and reasoning tasks, and the press' cherry-picking of samples to amplify its capabilities.", "Illusion of understanding: GPT-3 creates an illusion of understanding complex concepts like physics and psychology, despite lacking the capacity to reason about the consequences of its actions or anticipate them. It's compared to a magician's trick, conveying an illusion without actual understanding.", 'Neural networks and learning challenges: The limitations of neural networks in learning from unfamiliar data are discussed, emphasizing the difficulties in extrapolating to different scenarios and the necessity of learning rather than theoretical capabilities.', 'Language model limitations: The chapter discusses the limitations of language models in understanding language, highlighting the challenge of not saying everything that is meant and the inability to explicitly program and understand language constructions.']}, {'end': 6044.042, 'segs': [{'end': 4855.471, 'src': 'embed', 'start': 4827.244, 'weight': 0, 'content': [{'end': 4831.167, 'text': 'So the common way of talking about this is interpretability.', 'start': 4827.244, 'duration': 3.923}, {'end': 4835.11, 'text': 'GPT-3 is really like nowhere on interpretability.', 'start': 4831.527, 'duration': 3.583}, {'end': 4840.718, 'text': 'Classic symbolic AI, at least you could figure out why particular decisions were made and what the knowledge was.', 'start': 4835.171, 'duration': 5.547}, {'end': 4842.52, 'text': 'GPT-3 is terrible at that.', 'start': 4841.038, 'duration': 1.482}, {'end': 4848.949, 'text': "So you can't point in the network and say where is its representation of potable liquids versus non-potable liquids?", 'start': 4842.56, 'duration': 6.389}, {'end': 4855.471, 'text': 'You can find some correlations and say I noticed that potable liquids were used in these sentences and non-potable in that sentence.', 'start': 4849.269, 'duration': 6.202}], 'summary': 'Gpt-3 lacks interpretability and struggles with explaining decisions and knowledge.', 'duration': 28.227, 'max_score': 4827.244, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4827244.jpg'}, {'end': 5077.822, 'src': 'embed', 'start': 5053.411, 'weight': 1, 'content': [{'end': 5059.813, 'text': "I think that's actually interesting to try to look at how we combine different technologies to solve particular engineering problems.", 'start': 5053.411, 'duration': 6.402}, {'end': 5065.595, 'text': "I think it's actually less dogmatic than GPT-3, and I think that there's more substance to that.", 'start': 5059.873, 'duration': 5.722}, {'end': 5072.198, 'text': 'I think where we want to go is we want to say there are a lot of tools in cognition Look at humans.', 'start': 5065.755, 'duration': 6.443}, {'end': 5077.822, 'text': 'We use System 1 and System 2 from Kahneman, for example, which I call the reflexive and deliberative systems.', 'start': 5072.258, 'duration': 5.564}], 'summary': 'Combining technologies to solve engineering problems with less dogmatism than gpt-3 and leveraging cognitive tools like system 1 and system 2.', 'duration': 24.411, 'max_score': 5053.411, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w5053411.jpg'}, {'end': 5306.988, 'src': 'embed', 'start': 5280.64, 'weight': 5, 'content': [{'end': 5284.761, 'text': 'They were derived purely statistically from language.', 'start': 5280.64, 'duration': 4.121}, {'end': 5289.222, 'text': 'GPT-3 has read the whole internet, human conversations and so on.', 'start': 5284.901, 'duration': 4.321}, {'end': 5296.087, 'text': 'Do you think there is anything about humans that we can learn from GPT-3, from querying it?', 'start': 5289.462, 'duration': 6.625}, {'end': 5304.227, 'text': "Do you think we can learn something about humanity or human psyche from it, given the text it's reading?", 'start': 5296.347, 'duration': 7.88}, {'end': 5306.988, 'text': "Don't forget that it's very corpus-dependent.", 'start': 5304.587, 'duration': 2.401}], 'summary': 'Gpt-3 has read the entire internet and can offer insights into humanity and language.', 'duration': 26.348, 'max_score': 5280.64, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w5280640.jpg'}, {'end': 5431.126, 'src': 'embed', 'start': 5389.44, 'weight': 6, 'content': [{'end': 5399.402, 'text': 'what makes GPT-3 fascinating is the scaling laws that we found that like just make the model bigger and it just gets better in a very predictable way.', 'start': 5389.44, 'duration': 9.962}, {'end': 5402.403, 'text': 'And there seems to be little to no limit to this.', 'start': 5400.002, 'duration': 2.401}, {'end': 5404.463, 'text': "So there's a scaling hypothesis thing.", 'start': 5402.783, 'duration': 1.68}, {'end': 5408.669, 'text': "This is what I think is the whole story is that your GPT-3 It's not new architecture.", 'start': 5404.723, 'duration': 3.946}, {'end': 5410.27, 'text': 'It was new theoretical insight.', 'start': 5408.689, 'duration': 1.581}, {'end': 5412.632, 'text': 'It was just a larger model and still it got much better.', 'start': 5410.29, 'duration': 2.342}, {'end': 5414.233, 'text': 'On the other side.', 'start': 5413.413, 'duration': 0.82}, {'end': 5420.478, 'text': 'what I find interesting, what gets me emotionally, is if you use GPT-3, if you like, play with it.', 'start': 5414.233, 'duration': 6.245}, {'end': 5422.94, 'text': "at least you guys can tell me you're respectable for me.", 'start': 5420.478, 'duration': 2.462}, {'end': 5429.845, 'text': 'It is a subjectively different experience to work with GPT-3 than it is to work with any other kind of like deep learning model,', 'start': 5423.16, 'duration': 6.685}, {'end': 5431.126, 'text': 'GPT-2 or anything else.', 'start': 5429.845, 'duration': 1.281}], 'summary': "Gpt-3's scaling laws make it better predictably with size, offering a distinct user experience compared to other models.", 'duration': 41.686, 'max_score': 5389.44, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w5389440.jpg'}, {'end': 5499.588, 'src': 'embed', 'start': 5472.742, 'weight': 2, 'content': [{'end': 5478.084, 'text': "Are there any things that you've tried to do with GPT-3 which you were underwhelmed with? Math.", 'start': 5472.742, 'duration': 5.342}, {'end': 5480.505, 'text': "So it's bad at math.", 'start': 5478.704, 'duration': 1.801}, {'end': 5482.206, 'text': 'The thing about Mathis, though.', 'start': 5481.145, 'duration': 1.061}, {'end': 5487.908, 'text': 'what I find fascinating is that all the errors it makes in math are similar to the errors humans make, not the kind of errors machines make.', 'start': 5482.206, 'duration': 5.702}, {'end': 5490.929, 'text': "It's like it'll forget to carry a 1 or so, which is really interesting.", 'start': 5487.928, 'duration': 3.001}, {'end': 5492.403, 'text': "But it's pretty bad at math.", 'start': 5491.522, 'duration': 0.881}, {'end': 5495.004, 'text': "That's probably a BPE thing, so the way the text is encoded.", 'start': 5492.423, 'duration': 2.581}, {'end': 5499.588, 'text': 'Also, it actually does poems better than I expected it to because of the BPE thing.', 'start': 5495.605, 'duration': 3.983}], 'summary': 'Gpt-3 is underwhelming in math, makes human-like errors, but excels in poems due to bpe encoding.', 'duration': 26.846, 'max_score': 5472.742, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w5472742.jpg'}, {'end': 5718.632, 'src': 'embed', 'start': 5696.234, 'weight': 3, 'content': [{'end': 5703.9, 'text': "The fascinating thing is every time it's interpolated things together and generated some text, Yannick and I have Googled it and we can never find it.", 'start': 5696.234, 'duration': 7.666}, {'end': 5706.139, 'text': "Yeah, that's the same thing.", 'start': 5704.577, 'duration': 1.562}, {'end': 5714.107, 'text': "Philosophically then, we're asking the question, so it's interpolated between things that already exist, but apparently it's created new content.", 'start': 5706.539, 'duration': 7.568}, {'end': 5718.632, 'text': 'When a writer writes a book or creates new content, is it new content?', 'start': 5714.428, 'duration': 4.204}], 'summary': 'Interpolated text generates new content, challenging the definition of originality.', 'duration': 22.398, 'max_score': 5696.234, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w5696234.jpg'}, {'end': 5896.021, 'src': 'embed', 'start': 5865.43, 'weight': 4, 'content': [{'end': 5866.131, 'text': "That's really interesting.", 'start': 5865.43, 'duration': 0.701}, {'end': 5871.156, 'text': 'But I think from my perspective, there are two things I saw GPT-3 doing really well.', 'start': 5866.171, 'duration': 4.985}, {'end': 5872.978, 'text': 'So one is information retrieval.', 'start': 5871.236, 'duration': 1.742}, {'end': 5879.945, 'text': 'And in that we can include this kind of interpolation or walking through this space or whatever it is.', 'start': 5873.559, 'duration': 6.386}, {'end': 5883.429, 'text': 'Pattern matching is super good on GPT-3.', 'start': 5880.486, 'duration': 2.943}, {'end': 5886.813, 'text': 'The thing that seems to be missing for me is something that we would call reasoning.', 'start': 5883.85, 'duration': 2.963}, {'end': 5891.357, 'text': "And I don't know whether we need to put it in another box, because you were just saying when you do a presentation,", 'start': 5887.273, 'duration': 4.084}, {'end': 5896.021, 'text': "you can think of yourself as being a kind of generator and you're trying lots of different things.", 'start': 5891.357, 'duration': 4.664}], 'summary': 'Gpt-3 excels in information retrieval and pattern matching, lacking in reasoning.', 'duration': 30.591, 'max_score': 5865.43, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w5865430.jpg'}], 'start': 4739.007, 'title': "Gpt-3's limitations and implications", 'summary': 'Discusses the limitations of gpt-3 in language inference, new knowledge reasoning, and creativity, along with its implications for ai development and the need for diverse problem-solving approaches.', 'chapters': [{'end': 4868.736, 'start': 4739.007, 'title': 'Language inference and interpretability', 'summary': 'Discusses the importance of inference and shared knowledge in language understanding, highlighting the limitations of gpt-3 in accessing and representing common ground and knowledge, impacting interpretability.', 'duration': 129.729, 'highlights': ["GPT-3 lacks interpretability, making it challenging to connect actual knowledge with the system's representation, impacting language understanding.", 'Shared understanding and common ground are crucial in language, requiring access to specific and general knowledge for interpretation.', "Skilled writers leave some aspects open for interpretation, enhancing the reader's engagement and reconstruction of the narrative."]}, {'end': 5346.869, 'start': 4868.876, 'title': 'Limitations of gpt-3 and the future of ai', 'summary': 'Discusses the limitations of gpt-3 in understanding and reasoning over new knowledge, the reliance on prompting for minimal impact, and the need to combine different technologies for effective cognitive solutions, predicting a limited future for gpt-3 and emphasizing the importance of considering diverse tools and problem-solving approaches in the development of ai.', 'duration': 477.993, 'highlights': ["GPT-3's limitations in understanding and reasoning over new knowledge GPT-3's inability to effectively incorporate new facts without retraining the entire system, hindering its ability to reason over new knowledge and make new inferences.", "Reliance on prompting for minimal impact The minimal impact of prompting on GPT-3's performance, as it primarily revises the autocomplete problem without an internal system to effectively utilize human knowledge.", "Emphasizing the need to combine different technologies for effective cognitive solutions The importance of integrating diverse technologies, such as pattern perception and symbolic systems, to address specific engineering problems, highlighting the limitations of GPT-3's dogmatic approach and advocating for a more open problem-solving perspective."]}, {'end': 5696.214, 'start': 5346.989, 'title': 'Gpt-3: fascinating insights and limitations', 'summary': "Discusses conor leahy's fascination with gpt-3, highlighting its scaling laws, subjective experience, strengths in storytelling and poetry, limitations in math and structured tasks, and biases in data processing.", 'duration': 349.225, 'highlights': ["GPT-3's scaling laws make the model significantly better in a very predictable way, with little to no limit to improvement.", 'Using GPT-3 provides a subjectively different experience compared to other deep learning models, excelling in storytelling and poetry.', 'GPT-3 performs poorly in math and structured tasks, exhibiting errors similar to those made by humans.', "The model's training data lacks tabular data, and a bias exists in GPT-3 where it consistently only takes the first forum posts of most forums.", 'GPT-3 has impressed with philosophical points and stories, showing potential for generating esoteric content.']}, {'end': 6044.042, 'start': 5696.234, 'title': 'Ai interpolation and the creation of new content', 'summary': 'Discusses the philosophical implications of ai interpolation, comparing it to human creative processes and questioning the distinction between new and existing content, while highlighting the strengths and limitations of gpt-3 in information retrieval and reasoning.', 'duration': 347.808, 'highlights': ['The chapter discusses the philosophical implications of AI interpolation and questions whether the content generated is truly new, comparing it to human creative processes. The discussion delves into the philosophical question of whether AI interpolation truly creates new content, drawing comparisons to human creativity and the generation of new ideas. This challenges the notion of originality and creativity in AI-generated content.', 'The strengths and limitations of GPT-3 in information retrieval and reasoning are highlighted, with emphasis on its pattern matching capabilities and the absence of reasoning abilities. The transcript emphasizes the strengths of GPT-3 in information retrieval and pattern matching, while also highlighting its limitations in reasoning, indicating a gap in its ability to fill in missing information and perform complex reasoning tasks.', 'The discussion touches on the internal processes of GPT-3 and compares it to human cognitive processes, shedding light on the differences in the generation of speech and the underlying mechanisms. The conversation delves into the internal processes of GPT-3 and contrasts them with human cognitive processes, particularly in the generation of speech, highlighting differences in the mechanisms and decision-making processes involved.']}], 'duration': 1305.035, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w4739007.jpg', 'highlights': ["GPT-3's limitations in understanding and reasoning over new knowledge, hindering its ability to reason over new knowledge and make new inferences.", "The importance of integrating diverse technologies, such as pattern perception and symbolic systems, to address specific engineering problems, highlighting the limitations of GPT-3's dogmatic approach and advocating for a more open problem-solving perspective.", 'GPT-3 performs poorly in math and structured tasks, exhibiting errors similar to those made by humans.', 'The chapter discusses the philosophical implications of AI interpolation and questions whether the content generated is truly new, comparing it to human creative processes.', 'The strengths and limitations of GPT-3 in information retrieval and reasoning are highlighted, with emphasis on its pattern matching capabilities and the absence of reasoning abilities.', 'The conversation delves into the internal processes of GPT-3 and contrasts them with human cognitive processes, particularly in the generation of speech, highlighting differences in the mechanisms and decision-making processes involved.', "GPT-3's scaling laws make the model significantly better in a very predictable way, with little to no limit to improvement.", 'Using GPT-3 provides a subjectively different experience compared to other deep learning models, excelling in storytelling and poetry.']}, {'end': 7042.146, 'segs': [{'end': 6143.858, 'src': 'embed', 'start': 6113.082, 'weight': 0, 'content': [{'end': 6115.284, 'text': "GPT-3 doesn't seem to know we're talking about a person.", 'start': 6113.082, 'duration': 2.202}, {'end': 6119.586, 'text': 'There are loads of examples of this, like some kind of reversibility or something that requires reasoning.', 'start': 6115.304, 'duration': 4.282}, {'end': 6121.927, 'text': 'So I might say how many feet fit in a shoe.', 'start': 6119.626, 'duration': 2.301}, {'end': 6126.05, 'text': "And of course we know that one foot fits in a shoe, but GPT-3 doesn't know that.", 'start': 6122.808, 'duration': 3.242}, {'end': 6130.432, 'text': "Why do you think GPT-3 doesn't know that? Because it has no access to the physical universe.", 'start': 6126.31, 'duration': 4.122}, {'end': 6133.114, 'text': "This is just, again, it's the fish climbing a tree argument.", 'start': 6130.633, 'duration': 2.481}, {'end': 6139.716, 'text': "Respect for this guy, but I couldn't finish his interview because I was just like, you're barking up the wrong tree.", 'start': 6133.594, 'duration': 6.122}, {'end': 6143.858, 'text': "Everything you say is true on a technical level, but it's missing the forest for the trees.", 'start': 6139.796, 'duration': 4.062}], 'summary': 'Gpt-3 lacks understanding of simple human concepts, like one foot in a shoe, due to no access to the physical universe.', 'duration': 30.776, 'max_score': 6113.082, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w6113082.jpg'}, {'end': 6388.583, 'src': 'embed', 'start': 6360.57, 'weight': 1, 'content': [{'end': 6363.412, 'text': 'Like the entropy of text as a.', 'start': 6360.57, 'duration': 2.842}, {'end': 6364.032, 'text': 'well, the information.', 'start': 6363.412, 'duration': 0.62}, {'end': 6369.116, 'text': "a single word has very little information, but it has a very high entropy because it's very hard to predict what the next word is.", 'start': 6364.032, 'duration': 5.084}, {'end': 6374.581, 'text': "But if you have a frame of video, it's actually not that hard to predict the next frame of video.", 'start': 6369.337, 'duration': 5.244}, {'end': 6380.274, 'text': 'like the mutual information between two words is very low, but between two video frames very high.', 'start': 6374.581, 'duration': 5.693}, {'end': 6383.237, 'text': 'And it seems to like balance out again, read the paper.', 'start': 6380.294, 'duration': 2.943}, {'end': 6384.478, 'text': "I'm like, yeah.", 'start': 6383.397, 'duration': 1.081}, {'end': 6388.583, 'text': "And so if you, it's like we should taboo the word reasoning.", 'start': 6384.959, 'duration': 3.624}], 'summary': 'Entropy and mutual information differ for text and video, impacting predictability and information content.', 'duration': 28.013, 'max_score': 6360.57, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w6360570.jpg'}, {'end': 6705.071, 'src': 'embed', 'start': 6678.365, 'weight': 2, 'content': [{'end': 6684.207, 'text': "And we are just there to rationalize it, to explain it to other humans, why we did things, whether or not that's actually true.", 'start': 6678.365, 'duration': 5.842}, {'end': 6688.529, 'text': 'So again, moderately divisive controversial disclaimer here.', 'start': 6684.508, 'duration': 4.021}, {'end': 6690.71, 'text': 'I think there is overwhelming evidence.', 'start': 6689.19, 'duration': 1.52}, {'end': 6691.151, 'text': 'This is true.', 'start': 6690.73, 'duration': 0.421}, {'end': 6694.152, 'text': 'I think there is extremely powerful evidence that.', 'start': 6691.491, 'duration': 2.661}, {'end': 6699.029, 'text': 'The primary purpose of talking is to bullshit your rivals and such.', 'start': 6694.847, 'duration': 4.182}, {'end': 6705.071, 'text': 'And like information seeking is like a secondary, like a happy accident that we can also do that.', 'start': 6699.109, 'duration': 5.962}], 'summary': 'Talking is primarily for manipulating rivals, with some information seeking as a secondary purpose.', 'duration': 26.706, 'max_score': 6678.365, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w6678365.jpg'}], 'start': 6044.202, 'title': 'Gpt-3 and consciousness', 'summary': "Delves into gpt-3's potential consciousness, its comparison to human cognition, challenges in modeling physical world reasoning, evolution of language, brain functions, and the role of feedback connections, raising thought-provoking questions about knowledge and learning.", 'chapters': [{'end': 6226.654, 'start': 6044.202, 'title': 'Gba three: consciousness and understanding', 'summary': 'Discusses the proposal that gpt-3 may exhibit consciousness in its processing, its similarities and limitations compared to human cognition, and the distinction between pattern matching and understanding in computer vision, posing thought-provoking questions about the nature of knowledge and learning.', 'duration': 182.452, 'highlights': ["GPT-3's processing resembles the serial processing of making and refining guesses, similar to human cognition. N/A", 'The limitations of GPT-3 in understanding language and reasoning are highlighted, showcasing its differences from human communication and knowledge. Examples provided: language ambiguity, reasoning capabilities, lack of access to the physical universe.', 'The distinction between pattern matching and understanding in computer vision is discussed, raising questions about the nature of innate knowledge and evolutionary learning. Comparison between scene analysis and computer vision, questioning the origin of innate knowledge and evolutionary fitness.']}, {'end': 6678.145, 'start': 6226.875, 'title': 'Learning algorithms and human consciousness', 'summary': 'Explores the relationship between gpt-3 and human consciousness, delving into the challenges of modeling physical world reasoning and the entropy differences between text and video frames.', 'duration': 451.27, 'highlights': ['The chapter delves into the challenges of modeling physical world reasoning and the differences between modeling text and video frames, highlighting the entropy variations and difficulty levels, with video frames being easier to predict than text.', 'The discussion raises questions about human consciousness and the limitations of models like GPT-3 in materializing internal states as language tokens or discrete states, prompting the consideration of consciousness as a CEO and the theory of consciousness as the press secretary.', 'The conversation also touches on the potential of GPT-3 to perform reasoning tasks through parallel guessing and post hoc rationalization, presenting contrasting views on the nature of human thinking and the potential limitations of current architectures.', 'An interesting comparison is drawn between the predictability of text and video frames, emphasizing the higher entropy of text and the challenges involved in predicting the next word compared to predicting the next frame of video.']}, {'end': 7042.146, 'start': 6678.365, 'title': 'Evolution of language and brain functions', 'summary': "Discusses the evolution of language, the brain's functioning, and the role of feedback connections, suggesting that language evolved as a meeting signal and for bullshitting rivals, while the brain consists of parallel gpt-3-like models with feedback connections.", 'duration': 363.781, 'highlights': ['The primary purpose of language is to bullshit rivals and that information seeking is a secondary function, possibly due to language evolving as a meeting signal rather than just for information exchange.', 'The brain is not unified but composed of different parts that communicate and make alliances, leading to humans being extremely inconsistent and having cognitive dissonance.', 'The brain likely consists of parallel GPT-3-like models with feedback connections, and a top K function selects the best outputs from these models, supported by neuroscience evidence.']}], 'duration': 997.944, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w6044202.jpg', 'highlights': ['The limitations of GPT-3 in understanding language and reasoning are highlighted, showcasing its differences from human communication and knowledge.', 'The chapter delves into the challenges of modeling physical world reasoning and the differences between modeling text and video frames, highlighting the entropy variations and difficulty levels, with video frames being easier to predict than text.', 'The primary purpose of language is to bullshit rivals and that information seeking is a secondary function, possibly due to language evolving as a meeting signal rather than just for information exchange.']}, {'end': 8007.58, 'segs': [{'end': 7125.49, 'src': 'embed', 'start': 7094.968, 'weight': 0, 'content': [{'end': 7099.688, 'text': 'Do we need to have this huge heterogeneous architecture to model intelligence?', 'start': 7094.968, 'duration': 4.72}, {'end': 7102.089, 'text': "Or do you think it's simpler than we realize??", 'start': 7100.189, 'duration': 1.9}, {'end': 7110.08, 'text': "I think there is a good possibility that I'm about to say like 100%, but I'd say 30%, 30%.", 'start': 7102.849, 'duration': 7.231}, {'end': 7113.082, 'text': 'we can get superhuman AGI by just making GPT-3 bigger.', 'start': 7110.08, 'duration': 3.002}, {'end': 7115.744, 'text': 'Like literally change nothing else about it.', 'start': 7113.102, 'duration': 2.642}, {'end': 7117.125, 'text': 'Just adding a re okay.', 'start': 7115.984, 'duration': 1.141}, {'end': 7118.846, 'text': 'Adding a reinforcement learning controller on top.', 'start': 7117.285, 'duration': 1.561}, {'end': 7125.49, 'text': 'Is that just GPT-5 that any changes a hundred trillion parameters is just going to spontaneously be smarter than all humans.', 'start': 7119.306, 'duration': 6.184}], 'summary': 'Possibility of achieving superhuman agi by making gpt-3 bigger and adding reinforcement learning controller, with estimated 30% chance.', 'duration': 30.522, 'max_score': 7094.968, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7094968.jpg'}, {'end': 7179.428, 'src': 'embed', 'start': 7153.871, 'weight': 1, 'content': [{'end': 7161.657, 'text': "but I think the reason transformers are currently the top architecture is not because they're better than LSTMs or RNNs or something,", 'start': 7153.871, 'duration': 7.786}, {'end': 7163.418, 'text': "but they're just more efficient on GPUs.", 'start': 7161.657, 'duration': 1.761}, {'end': 7170.921, 'text': 'is you can run a much larger batch size and much more efficiently on Transformer than you can on LSTMs.', 'start': 7164.096, 'duration': 6.825}, {'end': 7179.428, 'text': 'And OpenAI actually checked this in their scaling law paper, where they showed that actually LSTMs just perform a constant worse than Transformers.', 'start': 7171.021, 'duration': 8.407}], 'summary': 'Transformers are top architecture due to efficiency on gpus, allowing larger batch sizes and more efficient processing compared to lstms.', 'duration': 25.557, 'max_score': 7153.871, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7153871.jpg'}, {'end': 7364.57, 'src': 'embed', 'start': 7332.798, 'weight': 2, 'content': [{'end': 7335.58, 'text': 'neural networks is such a misnomer in so many possible ways.', 'start': 7332.798, 'duration': 2.782}, {'end': 7339.662, 'text': 'It would be way less cool if we called it matrix program search or something.', 'start': 7336.161, 'duration': 3.501}, {'end': 7341.422, 'text': "But that's what it is.", 'start': 7340.522, 'duration': 0.9}, {'end': 7348.743, 'text': 'It is a first order optimization algorithm of numerical algorithms in some very large high dimensional space.', 'start': 7341.882, 'duration': 6.861}, {'end': 7349.524, 'text': "That's what they are.", 'start': 7349.064, 'duration': 0.46}, {'end': 7350.544, 'text': "They're not neurons.", 'start': 7349.544, 'duration': 1}, {'end': 7351.904, 'text': 'Maybe neurons do the same thing.', 'start': 7350.624, 'duration': 1.28}, {'end': 7353.064, 'text': 'That would be a coincidence.', 'start': 7352.204, 'duration': 0.86}, {'end': 7357.725, 'text': 'It is program search.', 'start': 7353.584, 'duration': 4.141}, {'end': 7364.57, 'text': 'If you have a large enough space of programs and you search No matter what the programs are made out of,', 'start': 7359.486, 'duration': 5.084}], 'summary': 'Neural networks are first-order optimization algorithms for numerical algorithms in large high-dimensional spaces.', 'duration': 31.772, 'max_score': 7332.798, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7332798.jpg'}, {'end': 7432.366, 'src': 'embed', 'start': 7401.454, 'weight': 3, 'content': [{'end': 7403.575, 'text': 'I expect we will have a theory of.', 'start': 7401.454, 'duration': 2.121}, {'end': 7408.638, 'text': 'And there is some manifold geometrics interpretations of neural networks that are emerging.', 'start': 7403.595, 'duration': 5.043}, {'end': 7409.679, 'text': "I'm not an expert on that.", 'start': 7408.858, 'duration': 0.821}, {'end': 7415.762, 'text': "But I expect that we'll find some mathematically elegant explanation of why this space of programs is particularly fruitful.", 'start': 7409.939, 'duration': 5.823}, {'end': 7422.392, 'text': 'And I expect that we will then see that, oh, transformers happen to give us a bias in this part of the space.', 'start': 7416.781, 'duration': 5.611}, {'end': 7423.715, 'text': "I mean, that's what CNNs do.", 'start': 7422.733, 'duration': 0.982}, {'end': 7426.34, 'text': 'They explore as a regular.', 'start': 7424.636, 'duration': 1.704}, {'end': 7427.782, 'text': "It's a better example than transformers.", 'start': 7426.5, 'duration': 1.282}, {'end': 7432.366, 'text': 'I love this really interesting, this framing of a program search.', 'start': 7428.644, 'duration': 3.722}], 'summary': 'Emerging manifold geometric interpretations of neural networks may lead to mathematically elegant explanations and biases in program space.', 'duration': 30.912, 'max_score': 7401.454, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7401454.jpg'}, {'end': 7571.759, 'src': 'embed', 'start': 7542.691, 'weight': 4, 'content': [{'end': 7548.437, 'text': 'But for some reason, something about the priors that we put into these matrices, about like this thing,', 'start': 7542.691, 'duration': 5.746}, {'end': 7553.782, 'text': 'something about that just biases it towards programs that are useful to humans.', 'start': 7548.437, 'duration': 5.345}, {'end': 7557.545, 'text': 'One of the reasons might be that quantum physics is basically vector algebra.', 'start': 7554.322, 'duration': 3.223}, {'end': 7558.967, 'text': 'That might be the reason.', 'start': 7558.186, 'duration': 0.781}, {'end': 7562.07, 'text': 'Like that might actually be the reason is that the reality is just made of vectors.', 'start': 7559.027, 'duration': 3.043}, {'end': 7571.759, 'text': "maybe you don't think it might have something to do with the fact that humans have to be fairly energy efficient or brains in general, like nature,", 'start': 7562.791, 'duration': 8.968}], 'summary': 'Priors bias matrices towards human-useful programs, possibly due to quantum physics and energy efficiency.', 'duration': 29.068, 'max_score': 7542.691, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7542691.jpg'}, {'end': 7921.06, 'src': 'embed', 'start': 7885.847, 'weight': 5, 'content': [{'end': 7888.328, 'text': 'Do you have any sort of things that you realized?', 'start': 7885.847, 'duration': 2.481}, {'end': 7889.988, 'text': "ah, that's how I have to do it.", 'start': 7888.328, 'duration': 1.66}, {'end': 7893.249, 'text': 'or ah, here is a neat little trick to do something?', 'start': 7889.988, 'duration': 3.261}, {'end': 7899.368, 'text': "I'll give you a bit of a cop-out answer, because I'm actually don't consider myself particularly proficient in GPT-3..", 'start': 7894.025, 'duration': 5.343}, {'end': 7901.989, 'text': "Like I've played with it a lot, but not as much as other people.", 'start': 7899.588, 'duration': 2.401}, {'end': 7903.13, 'text': 'Read for that.', 'start': 7902.23, 'duration': 0.9}, {'end': 7906.052, 'text': 'If you want the expert tips on how to do, how to use GPT-3.', 'start': 7903.37, 'duration': 2.682}, {'end': 7908.693, 'text': "There's a really great essay called what was it called??", 'start': 7906.452, 'duration': 2.241}, {'end': 7913.156, 'text': 'It was like UX as alignment or something like that, as a bottleneck and alignment or something like that.', 'start': 7908.713, 'duration': 4.443}, {'end': 7921.06, 'text': 'They basically make the argument is so they start with this example that two, I think like two Navy ships crashed into each other and.', 'start': 7913.456, 'duration': 7.604}], 'summary': 'Limited expertise in gpt-3, recommends an essay for expert tips on gpt-3 usage.', 'duration': 35.213, 'max_score': 7885.847, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7885847.jpg'}], 'start': 7042.547, 'title': 'Ai scaling, neural networks, and human-like computation bias', 'summary': 'Discusses scaling gpt-3 with reinforcement learning for achieving superhuman agi, efficiency of transformer architectures, neural networks as optimization algorithm, limitations and capabilities of program search, bias of neural networks towards human usefulness, and challenges in interacting with gpt-3.', 'chapters': [{'end': 7246.114, 'start': 7042.547, 'title': 'Agi architecture and scaling', 'summary': 'Discusses the potential of achieving superhuman agi by scaling gpt-3 with reinforcement learning and the efficiency of transformer architectures over lstms in the context of computational resources and performance.', 'duration': 203.567, 'highlights': ['The potential of achieving superhuman AGI by scaling GPT-3 with reinforcement learning The speaker suggests a 30% chance of achieving superhuman AGI by simply scaling GPT-3 with a reinforcement learning controller, without changing any other aspect, potentially reaching 100 trillion parameters.', "Efficiency of transformer architectures over LSTMs in the context of computational resources and performance The speaker explains that transformer architectures are more efficient on GPUs compared to LSTMs, as demonstrated in OpenAI's scaling law paper, indicating a potential for LSTMs to perform similarly with sufficient computational resources.", 'Challenges in training LSTMs and their limitations The discussion touches upon the difficulties of training LSTMs, including a forgetting problem and context fragmentation issues, hinting at the computational constraints being a significant factor in their performance.']}, {'end': 7542.406, 'start': 7246.114, 'title': 'Neural networks and program search', 'summary': 'Discusses the concept of neural networks as a first-order optimization algorithm of numerical algorithms in a high-dimensional space, the potential unifying grand theory of neural networks, and the limitations and capabilities of program search across large spaces of programs.', 'duration': 296.292, 'highlights': ['Neural networks as a first-order optimization algorithm of numerical algorithms in a high-dimensional space The chapter emphasizes neural networks as a first-order optimization algorithm in a high-dimensional space, highlighting their function as numerical algorithm searchers.', 'Potential unifying grand theory of neural networks The discussion speculates on the potential discovery of a unifying grand theory of neural networks in the near future, suggesting that the architecture may become irrelevant in light of a more general algorithm.', 'Limitations and capabilities of program search across large spaces of programs It delves into the limitations and capabilities of program search across large spaces of programs, addressing the challenges of dense sampling and the inability to find useful programs through exhaustive search.']}, {'end': 8007.58, 'start': 7542.691, 'title': 'The bias of neural networks in human-like computation', 'summary': 'Discusses the bias of neural networks towards programs useful to humans, its relation to quantum physics, turing universality, and the challenges in interacting with gpt-3.', 'duration': 464.889, 'highlights': ['The bias of neural networks towards programs useful to humans is potentially related to quantum physics and the energy efficiency of human brains, which impacts their computational complexity and usefulness (e.g. speech recognition).', 'Turing universality underpins the power of neural networks as computational machines, enabling them to simulate any other system with at most a polynomial speed increase, with implications for efficiency in solving specific problems.', 'Interacting with GPT-3 requires understanding its nature as a text continuation model, with a focus on effective communication and alignment between user intent and the machine, emphasizing the importance of UX in achieving useful outcomes.']}], 'duration': 965.033, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w7042547.jpg', 'highlights': ['There is a 30% chance of achieving superhuman AGI by scaling GPT-3 with reinforcement learning, potentially reaching 100 trillion parameters.', "Transformer architectures are more efficient on GPUs compared to LSTMs, as demonstrated in OpenAI's scaling law paper.", 'Neural networks function as first-order optimization algorithms in a high-dimensional space.', 'There is speculation about the potential discovery of a unifying grand theory of neural networks in the near future.', 'The bias of neural networks towards programs useful to humans is potentially related to quantum physics and the energy efficiency of human brains.', 'Interacting with GPT-3 requires understanding its nature as a text continuation model, with a focus on effective communication and alignment between user intent and the machine.']}, {'end': 8874.368, 'segs': [{'end': 8060.43, 'src': 'embed', 'start': 8007.861, 'weight': 0, 'content': [{'end': 8012.963, 'text': 'But for the most part, I could just enter three vaguely related words and it gives me exactly what I want.', 'start': 8007.861, 'duration': 5.102}, {'end': 8016.305, 'text': 'And that is not because my words were particularly great.', 'start': 8013.664, 'duration': 2.641}, {'end': 8017.526, 'text': "It's because Google is great.", 'start': 8016.525, 'duration': 1.001}, {'end': 8025.21, 'text': "It's because there's so much brainpower engineering effort went into making a Google that is aligned to me as a human.", 'start': 8017.866, 'duration': 7.344}, {'end': 8027.733, 'text': "If an alien used Google, it wouldn't work at all.", 'start': 8025.512, 'duration': 2.221}, {'end': 8033.977, 'text': "Even if it spoke our language, it would ask something that Google does not understand and would just spit out garbage because it's an alien.", 'start': 8028.114, 'duration': 5.863}, {'end': 8039.44, 'text': "You're contrasting Google and GPT-3 and Google is this hierarchy of systems.", 'start': 8034.537, 'duration': 4.903}, {'end': 8046.284, 'text': 'I think what really made Google overtake out of Easter and is it the early 2000s is the PageRank algorithm, which is a ranker.', 'start': 8039.76, 'duration': 6.524}, {'end': 8052.266, 'text': 'And now they have a series of rankers that fire in sequence, depending on how many results there are.', 'start': 8046.764, 'duration': 5.502}, {'end': 8055.668, 'text': 'Presumably, they also look at how many people are clicking on the links.', 'start': 8052.687, 'duration': 2.981}, {'end': 8057.369, 'text': 'They do some kind of collaborative filtering.', 'start': 8055.708, 'duration': 1.661}, {'end': 8059.089, 'text': 'They do some deep learning models.', 'start': 8057.409, 'duration': 1.68}, {'end': 8060.43, 'text': 'They do a whole bunch of stuff.', 'start': 8059.289, 'duration': 1.141}], 'summary': "Google's advanced system uses rankers and deep learning models to provide accurate search results aligned with human needs.", 'duration': 52.569, 'max_score': 8007.861, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8007861.jpg'}, {'end': 8209.128, 'src': 'embed', 'start': 8173.738, 'weight': 3, 'content': [{'end': 8179.92, 'text': 'And I expect that in the future with like reinforcement learning, with Google type services, Siri type hard coded rules or whatever,', 'start': 8173.738, 'duration': 6.182}, {'end': 8181.321, 'text': "there's going to be interface like that.", 'start': 8179.92, 'duration': 1.401}, {'end': 8184.302, 'text': 'I expect GPT to be integrated into Siri within the next five years.', 'start': 8181.461, 'duration': 2.841}, {'end': 8187.551, 'text': 'This gets into one of the other points I wanted to make.', 'start': 8184.83, 'duration': 2.721}, {'end': 8190.253, 'text': 'I want to talk about the database example from Matt Brockman.', 'start': 8187.732, 'duration': 2.521}, {'end': 8195.876, 'text': 'Now, the dream here, and you framed this up the other week, is software 3.0.', 'start': 8190.933, 'duration': 4.943}, {'end': 8197.116, 'text': "It's prompt engineering.", 'start': 8195.876, 'duration': 1.24}, {'end': 8204.403, 'text': "And Yannick is making the argument quite a lot that The only problem is that, and you've just made, it's very difficult to design these prompts.", 'start': 8197.517, 'duration': 6.886}, {'end': 8207.085, 'text': "But I must admit, I've got no agenda here.", 'start': 8204.823, 'duration': 2.262}, {'end': 8209.128, 'text': 'I would love it if GPT-3 worked.', 'start': 8207.505, 'duration': 1.623}], 'summary': 'Gpt expected to integrate into siri within 5 years, aiming for prompt engineering and interface like google services.', 'duration': 35.39, 'max_score': 8173.738, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8173738.jpg'}, {'end': 8581.114, 'src': 'embed', 'start': 8558.858, 'weight': 5, 'content': [{'end': 8567.946, 'text': "I think GPT-3 is much closer in its flaws and strengths to whatever the human algorithm is than any other machine we've ever built.", 'start': 8558.858, 'duration': 9.088}, {'end': 8574.867, 'text': "I've already anthropomorphized all the image models when I made them do Rorschach tests and so on.", 'start': 8568.72, 'duration': 6.147}, {'end': 8576.449, 'text': 'Maybe we can do the same.', 'start': 8574.887, 'duration': 1.562}, {'end': 8579.812, 'text': 'Yeah, the hard thing is to ask it questions.', 'start': 8576.889, 'duration': 2.923}, {'end': 8581.114, 'text': "That's what I find the hard thing.", 'start': 8579.892, 'duration': 1.222}], 'summary': 'Gpt-3 is closer to human algorithm than any other machine. asking it questions is challenging.', 'duration': 22.256, 'max_score': 8558.858, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8558858.jpg'}], 'start': 8007.861, 'title': "Google's user alignment and gpt-3's future", 'summary': "Discusses google's alignment to human users, surpassing alien use, with insights into the pagerank algorithm's impact. it also explores gpt-3's potential, limitations, future integration, challenges of communication, and advancements in ai.", 'chapters': [{'end': 8060.43, 'start': 8007.861, 'title': "Google's alignment to human vs alien use", 'summary': "Discusses how google's engineering efforts have made it aligned to human users, surpassing its effectiveness for alien users, and how the pagerank algorithm and series of rankers contributed to its overtaking of other search engines in the early 2000s.", 'duration': 52.569, 'highlights': ["Google's alignment to human users is due to extensive brainpower engineering effort, making it effective in understanding human queries.", "The PageRank algorithm and subsequent series of rankers contributed to Google's dominance in the early 2000s by refining search result relevance and sequence.", "Google's functionality would not work effectively for an alien user, as it is specifically tailored to human queries and understanding.", 'Google employs collaborative filtering, deep learning models, and other techniques to refine search results and user experience.', 'Even with vague input, Google accurately provides desired results due to its alignment with human language and queries.']}, {'end': 8874.368, 'start': 8060.95, 'title': 'Gpt-3 and the future of ai', 'summary': 'Discusses the potential of gpt-3, its limitations, and the future integration of gpt-3 into various platforms, including its use in reinforcement learning and prompt engineering. it also explores the challenges of communicating with gpt-3 and the potential for future advancements in ai.', 'duration': 813.418, 'highlights': ["GPT-3 is likely to be integrated into platforms like Siri and Visual Studio Code, leveraging reinforcement learning for good coding style suggestions. GPT-3's potential integration into platforms like Siri and Visual Studio Code using reinforcement learning for coding style suggestions is highlighted as a significant future development.", 'The importance of prompt engineering and the challenges of communicating with GPT-3 are emphasized. The significance of prompt engineering and the difficulties in communicating effectively with GPT-3 are highlighted, indicating potential areas for improvement in AI interaction.', 'The limitations and challenges of GPT-3 are discussed, including its difficulty in understanding and answering questions accurately. Discussions about the limitations and challenges of GPT-3, particularly in accurately understanding and answering questions, shed light on the current shortcomings of the AI model.']}], 'duration': 866.507, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8007861.jpg', 'highlights': ["Google's alignment to human users is due to extensive brainpower engineering effort, making it effective in understanding human queries.", "The PageRank algorithm and subsequent series of rankers contributed to Google's dominance in the early 2000s by refining search result relevance and sequence.", 'Google employs collaborative filtering, deep learning models, and other techniques to refine search results and user experience.', 'GPT-3 is likely to be integrated into platforms like Siri and Visual Studio Code, leveraging reinforcement learning for good coding style suggestions.', 'The importance of prompt engineering and the challenges of communicating with GPT-3 are emphasized.', 'The limitations and challenges of GPT-3 are discussed, including its difficulty in understanding and answering questions accurately.']}, {'end': 10581.235, 'segs': [{'end': 8903.186, 'src': 'embed', 'start': 8874.928, 'weight': 2, 'content': [{'end': 8878.288, 'text': 'And, as you could tell from the episode that we did with him recently,', 'start': 8874.928, 'duration': 3.36}, {'end': 8884.11, 'text': 'his thesis is that natural language processing is not the same as natural language understanding.', 'start': 8878.288, 'duration': 5.822}, {'end': 8890.491, 'text': 'And after having played with GPT-3 for several hours, we got his take on what he thinks of GPT-3.', 'start': 8884.67, 'duration': 5.821}, {'end': 8894.955, 'text': "we've been playing with this together now for a few hours and in.", 'start': 8891.531, 'duration': 3.424}, {'end': 8896.598, 'text': 'in some ways it surprised us.', 'start': 8894.955, 'duration': 1.643}, {'end': 8903.186, 'text': 'but overall, what we said the other week about natural language processing is not understanding, blah, blah, blah.', 'start': 8896.598, 'duration': 6.588}], 'summary': 'Gpt-3 is assessed after playing with it for several hours, revealing surprises and insights on natural language processing versus understanding.', 'duration': 28.258, 'max_score': 8874.928, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8874928.jpg'}, {'end': 8954.163, 'src': 'embed', 'start': 8924.086, 'weight': 0, 'content': [{'end': 8927.228, 'text': "It's a retrieval engine more than anything.", 'start': 8924.086, 'duration': 3.142}, {'end': 8931.991, 'text': "There's not even NLP, let alone NLU.", 'start': 8928.909, 'duration': 3.082}, {'end': 8932.492, 'text': 'Come on.', 'start': 8932.071, 'duration': 0.421}, {'end': 8934.633, 'text': "There's no understanding whatsoever.", 'start': 8932.792, 'duration': 1.841}, {'end': 8944.74, 'text': 'This is just an engine that has digested millions of textual patterns, and it will spit out something similar to your prompt.', 'start': 8935.097, 'duration': 9.643}, {'end': 8950.762, 'text': 'So the word understanding is even, it should not be on the same planet.', 'start': 8945.74, 'duration': 5.022}, {'end': 8954.163, 'text': 'There is no language capabilities whatsoever.', 'start': 8950.842, 'duration': 3.321}], 'summary': 'The retrieval engine lacks nlp and nlu, with no language capabilities or understanding.', 'duration': 30.077, 'max_score': 8924.086, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8924086.jpg'}, {'end': 9133.997, 'src': 'embed', 'start': 9105.614, 'weight': 1, 'content': [{'end': 9107.654, 'text': 'This is pattern recognition.', 'start': 9105.614, 'duration': 2.04}, {'end': 9109.435, 'text': 'I saw too many of these.', 'start': 9108.275, 'duration': 1.16}, {'end': 9111.796, 'text': 'So understanding is different.', 'start': 9109.595, 'duration': 2.201}, {'end': 9113.557, 'text': 'Understanding is binary.', 'start': 9111.936, 'duration': 1.621}, {'end': 9114.973, 'text': 'I think I understand.', 'start': 9114.313, 'duration': 0.66}, {'end': 9118.314, 'text': 'So are you saying, though, that understanding and knowledge are synonymous?', 'start': 9114.993, 'duration': 3.321}, {'end': 9125.315, 'text': 'Understanding and knowledge not synonymous, but definitely if, and only if, they go together?', 'start': 9118.774, 'duration': 6.541}, {'end': 9129.176, 'text': "But with pattern recognition, there's no understanding.", 'start': 9126.396, 'duration': 2.78}, {'end': 9133.997, 'text': "When I started to recognize that this is a cat, I didn't understand anything.", 'start': 9129.676, 'duration': 4.321}], 'summary': 'Pattern recognition and understanding are distinct, knowledge is not synonymous with understanding.', 'duration': 28.383, 'max_score': 9105.614, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w9105614.jpg'}, {'end': 9501.728, 'src': 'embed', 'start': 9478.65, 'weight': 3, 'content': [{'end': 9490.793, 'text': 'What he was talking about is, look, you and people speak of domain-specific NLU and if we circumscribe the domain and we constrain it,', 'start': 9478.65, 'duration': 12.143}, {'end': 9496.947, 'text': "and The thing about language or any infinite domain, and that's why game playing is not a test for AI.", 'start': 9490.793, 'duration': 6.154}, {'end': 9501.728, 'text': 'Alan Turing was genius to consider language the test for AI.', 'start': 9497.267, 'duration': 4.461}], 'summary': 'Nlu and domain constraints make game playing not a test for ai; turing considered language the true ai test.', 'duration': 23.078, 'max_score': 9478.65, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w9478650.jpg'}], 'start': 8874.928, 'title': 'Gpt-3 limitations and language understanding', 'summary': "Explores gpt-3's limitations as a retrieval engine, discusses the distinction between pattern recognition and understanding, and highlights chomsky's hypothesis on language understanding. it emphasizes the binary nature of understanding and the importance of reasoning in ai.", 'chapters': [{'end': 9000.6, 'start': 8874.928, 'title': 'Gpt-3 natural language understanding', 'summary': 'Explores the limitations of gpt-3, highlighting that it is more of a retrieval engine than a natural language processing or understanding tool, with no language capabilities and a heavy reliance on the input prompt to determine the output.', 'duration': 125.672, 'highlights': ['GPT-3 is described as a retrieval engine more than anything, lacking NLP and NLU capabilities.', 'The engine is observed to be heavily influenced by the input prompt, which overwhelmingly determines the output.', 'The speaker emphasizes that GPT-3 has no language capabilities or understanding, merely digesting textual patterns and producing similar outputs to the prompt.']}, {'end': 9348.358, 'start': 9002.077, 'title': 'Pattern recognition vs. understanding', 'summary': 'Discusses the distinction between pattern recognition and understanding, emphasizing the binary nature of understanding and the need for a base for reasoning, with examples and explanations of the differences between the two concepts.', 'duration': 346.281, 'highlights': ['Understanding is binary, while pattern recognition is a skill based on what is seen and experienced, following the machine learning paradigm. Understanding is portrayed as a binary concept, while pattern recognition is described as a skill based on observation and experience.', 'Reasoning requires a base, such as axioms or ontological structure, while pattern recognition can occur from nothing. The distinction between reasoning and pattern recognition is made clear, with reasoning requiring a base and pattern recognition being independent of such prerequisites.', 'The concept of understanding is differentiated from knowledge, with understanding being emphasized as different from and not synonymous with knowledge. The differentiation between understanding and knowledge is explained, highlighting that they are not synonymous concepts.']}, {'end': 9710.839, 'start': 9348.478, 'title': "Chomsky's language hypothesis", 'summary': "Discusses the limitations of gpt-3 in understanding language, highlighting chomsky's hypothesis that language understanding is a binary decision and not probabilistic, with examples showing the infinite nature and ambiguity of language.", 'duration': 362.361, 'highlights': ["The limitations of GPT-3 in understanding language are highlighted, emphasizing its inability to grasp the infinite and ambiguous nature of language, as per Chomsky's hypothesis.", 'The binary nature of language understanding is emphasized, where each sentence conveys a single thought out of many possible meanings, making it a binary decision rather than probabilistic.', 'The discussion contrasts language understanding (NLU) with text processing, suggesting a shift in terminology to differentiate the two disciplines, highlighting the need to uncover the thought behind an utterance as a binary decision in NLU.']}, {'end': 10581.235, 'start': 9711.408, 'title': 'Pattern recognition and reasoning in ai', 'summary': 'Explores the dichotomy between pattern recognition and reasoning in ai, highlighting the limitations of gpt-3 in natural language understanding and the distinction between memorizing textual patterns and true understanding, emphasizing the importance of being scientific about claims in ai.', 'duration': 869.827, 'highlights': ["GPT-3's limitations in natural language understanding and the distinction between memorizing textual patterns and true understanding GPT-3 is criticized for not demonstrating natural language understanding and is noted for memorizing tons of textual patterns, indicating it falls short of true understanding.", "The importance of being scientific about claims in AI The speaker emphasizes the need for scientific rigor in making claims about AI capabilities, particularly critiquing the overhyped expectations of GPT-3's language understanding and story-writing abilities.", 'The dichotomy between pattern recognition and reasoning in AI The discussion delves into the dichotomy between pattern recognition and reasoning in AI, highlighting the distinction between pattern recognition skills and true reasoning abilities, emphasizing the need to go beyond mere pattern recognition.']}], 'duration': 1706.307, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w8874928.jpg', 'highlights': ['GPT-3 is described as a retrieval engine more than anything, lacking NLP and NLU capabilities.', 'Understanding is binary, while pattern recognition is a skill based on what is seen and experienced, following the machine learning paradigm.', "The limitations of GPT-3 in understanding language are highlighted, emphasizing its inability to grasp the infinite and ambiguous nature of language, as per Chomsky's hypothesis.", "The importance of being scientific about claims in AI The speaker emphasizes the need for scientific rigor in making claims about AI capabilities, particularly critiquing the overhyped expectations of GPT-3's language understanding and story-writing abilities."]}, {'end': 11765.401, 'segs': [{'end': 10686.858, 'src': 'embed', 'start': 10644.679, 'weight': 0, 'content': [{'end': 10646.56, 'text': 'Is it something that we could explicitly code??', 'start': 10644.679, 'duration': 1.881}, {'end': 10649.241, 'text': "Clearly, using GPT-3 doesn't seem to be a good idea.", 'start': 10646.84, 'duration': 2.401}, {'end': 10654.585, 'text': 'Is it something that requires some degree of natural language understanding or processing or information retrieval?', 'start': 10649.622, 'duration': 4.963}, {'end': 10656.866, 'text': "These are things that we couldn't easily write with code.", 'start': 10654.945, 'duration': 1.921}, {'end': 10661.469, 'text': "So that does make me want to use GPT-3 with the caveat that I don't understand how it works.", 'start': 10657.266, 'duration': 4.203}, {'end': 10666.293, 'text': "Yeah, that's why I think the future lies in hybrid systems.", 'start': 10661.872, 'duration': 4.421}, {'end': 10676.735, 'text': "It's using programming and things like logic and mathematics and whatever in the domain where they're useful,", 'start': 10666.873, 'duration': 9.862}, {'end': 10682.097, 'text': "and using machine learning and neural nets and whatnot in the domain where they're useful.", 'start': 10676.735, 'duration': 5.362}, {'end': 10686.858, 'text': "And just like we learned that it's very difficult to solve some problems by writing code,", 'start': 10682.357, 'duration': 4.501}], 'summary': 'Hybrid systems combining programming and machine learning are the future.', 'duration': 42.179, 'max_score': 10644.679, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w10644679.jpg'}, {'end': 10786.839, 'src': 'embed', 'start': 10756.917, 'weight': 1, 'content': [{'end': 10758.158, 'text': "That's what I'm really interested in.", 'start': 10756.917, 'duration': 1.241}, {'end': 10760.561, 'text': 'Yeah, look, machine learning is great at doing these things,', 'start': 10758.218, 'duration': 2.343}, {'end': 10766.566, 'text': 'but we need to have other layers that can perform logic and in which we can encode structure.', 'start': 10760.561, 'duration': 6.005}, {'end': 10773.813, 'text': "Yeah, I think what Walid has articulated beautifully is that everything we're doing in machine learning right now is pattern recognition.", 'start': 10767.027, 'duration': 6.786}, {'end': 10777.197, 'text': 'And that is completely divorced from understanding.', 'start': 10774.474, 'duration': 2.723}, {'end': 10786.839, 'text': 'This is where I think part of the disconnect might be between you and I and Yannick and Connor is I think these terms have to be a bit better defined.', 'start': 10777.717, 'duration': 9.122}], 'summary': 'Machine learning excels at pattern recognition but lacks understanding. need for logic and structure in addition.', 'duration': 29.922, 'max_score': 10756.917, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w10756917.jpg'}, {'end': 10981.426, 'src': 'embed', 'start': 10948.211, 'weight': 2, 'content': [{'end': 10950.973, 'text': 'We need to have a better set of agreed upon definitions.', 'start': 10948.211, 'duration': 2.762}, {'end': 10956.036, 'text': 'Like you and I and Connor, we all agree that this is the definition of reasoning.', 'start': 10951.553, 'duration': 4.483}, {'end': 10959.778, 'text': 'This is the definition of pattern recognition.', 'start': 10956.536, 'duration': 3.242}, {'end': 10965.541, 'text': "Maybe go so far as to say for the time being, we're going to agree that this is our definition of intelligence.", 'start': 10959.858, 'duration': 5.683}, {'end': 10972.603, 'text': 'Can I have a go at that? So I say pattern recognition is transforming data from one substrate to another.', 'start': 10966.601, 'duration': 6.002}, {'end': 10981.426, 'text': "And that might mean transforming from gridded vision data on a planar manifold to let's say some kind of topological graph.", 'start': 10973.123, 'duration': 8.303}], 'summary': 'Agreed definitions for reasoning, pattern recognition, and intelligence are discussed, with a focus on data transformation.', 'duration': 33.215, 'max_score': 10948.211, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w10948211.jpg'}, {'end': 11128.613, 'src': 'embed', 'start': 11101.507, 'weight': 5, 'content': [{'end': 11105.431, 'text': "or the fact that you arrived at a tennis court five minutes ago and didn't leave.", 'start': 11101.507, 'duration': 3.924}, {'end': 11108.594, 'text': "So you're combining more than one data source.", 'start': 11105.931, 'duration': 2.663}, {'end': 11111.257, 'text': 'Like having extra data is not reasoning.', 'start': 11109.155, 'duration': 2.102}, {'end': 11115.902, 'text': "It's the process of taking the combined data set and doing something with it.", 'start': 11111.858, 'duration': 4.044}, {'end': 11121.288, 'text': "And so I'm saying that doing something with it, it sounds like it's first order logic.", 'start': 11116.363, 'duration': 4.925}, {'end': 11123.511, 'text': "It's like kind of old school expert system.", 'start': 11121.548, 'duration': 1.963}, {'end': 11124.891, 'text': "That's exactly what it is.", 'start': 11123.891, 'duration': 1}, {'end': 11128.613, 'text': 'Yeah, So you have a world model and what he would say.', 'start': 11125.152, 'duration': 3.461}], 'summary': 'Combining multiple data sources for processing and analysis, resembling first order logic and expert systems.', 'duration': 27.106, 'max_score': 11101.507, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11101507.jpg'}, {'end': 11172.366, 'src': 'embed', 'start': 11143.5, 'weight': 4, 'content': [{'end': 11145.161, 'text': "And then there's a whole bunch of missing information.", 'start': 11143.5, 'duration': 1.661}, {'end': 11151.784, 'text': "And now I've got all of these logical predicates that I can run to fill in the missing gaps from my world model.", 'start': 11145.441, 'duration': 6.343}, {'end': 11154.079, 'text': "Well, I wouldn't say fill in.", 'start': 11152.438, 'duration': 1.641}, {'end': 11156.54, 'text': "It's inferring or deducing.", 'start': 11154.719, 'duration': 1.821}, {'end': 11164.483, 'text': "So it's starting with a data set, which in a very general sense, all data can be represented as a relation, a mathematical relation.", 'start': 11156.56, 'duration': 7.923}, {'end': 11168.665, 'text': "And so suppose we've got all this information that's encoded in this relation.", 'start': 11164.703, 'duration': 3.962}, {'end': 11172.366, 'text': 'And then we have a sort of classic expert system.', 'start': 11169.365, 'duration': 3.001}], 'summary': 'Using logical predicates to infer missing information from a data set encoded as a relation.', 'duration': 28.866, 'max_score': 11143.5, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11143500.jpg'}, {'end': 11259.583, 'src': 'embed', 'start': 11230.14, 'weight': 3, 'content': [{'end': 11236.464, 'text': "like reasoning we consider to take place over time and it's possibly unbounded,", 'start': 11230.14, 'duration': 6.324}, {'end': 11241.108, 'text': 'like it may continue to process forever and you may never get to a transitive closure.', 'start': 11236.464, 'duration': 4.644}, {'end': 11244.29, 'text': 'It may just keep adding new assertions over and over again.', 'start': 11241.148, 'duration': 3.142}, {'end': 11249.094, 'text': 'Whereas pattern recognition is a very finite version of that.', 'start': 11244.791, 'duration': 4.303}, {'end': 11259.583, 'text': "It's like I take the relation and I apply a single function in finite time and I assert a finite number of new assertions, right?", 'start': 11249.214, 'duration': 10.369}], 'summary': 'Reasoning is unbounded, while pattern recognition is finite.', 'duration': 29.443, 'max_score': 11230.14, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11230140.jpg'}, {'end': 11381.604, 'src': 'embed', 'start': 11355.671, 'weight': 6, 'content': [{'end': 11360.136, 'text': 'But the difference is that in principle, a Turing machine is potentially infinite.', 'start': 11355.671, 'duration': 4.465}, {'end': 11361.938, 'text': 'It can sit there running forever.', 'start': 11360.516, 'duration': 1.422}, {'end': 11370.55, 'text': 'You can add more storage or whatever, whereas finite state machines are clearly just they have a finite bound in both space and time.', 'start': 11362.478, 'duration': 8.072}, {'end': 11376.799, 'text': "And so is that the distinction we're making, which is that Connor and organic may be saying yeah,", 'start': 11371.01, 'duration': 5.789}, {'end': 11381.604, 'text': 'but I can build a large enough kind of circuit and understand language.', 'start': 11376.799, 'duration': 4.805}], 'summary': 'Turing machines are potentially infinite, while finite state machines have finite bounds in space and time.', 'duration': 25.933, 'max_score': 11355.671, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11355671.jpg'}, {'end': 11574.388, 'src': 'embed', 'start': 11548.463, 'weight': 7, 'content': [{'end': 11558.473, 'text': "and so i think what we're getting at is that the essence of reasoning is that it's an unbounded computation, whereas pattern recognition, maybe,", 'start': 11548.463, 'duration': 10.01}, {'end': 11560.435, 'text': 'is bounded computation.', 'start': 11558.473, 'duration': 1.962}, {'end': 11562.317, 'text': "That's a very interesting hypothesis.", 'start': 11560.455, 'duration': 1.862}, {'end': 11566.481, 'text': 'I think, if we can nail this down a bit in a conversation with them.', 'start': 11562.677, 'duration': 3.804}, {'end': 11574.388, 'text': "do we agree with that that that's a reasonable distinction to make bounded versus unbounded, potentially infinite versus finite whatever?", 'start': 11566.481, 'duration': 7.907}], 'summary': 'Reasoning is unbounded computation, while pattern recognition is bounded.', 'duration': 25.925, 'max_score': 11548.463, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11548463.jpg'}], 'start': 10581.836, 'title': 'Machine learning and reasoning', 'summary': 'Explores limitations of machine learning, advocates for a hybrid system, emphasizes the need for logical layers and clear reasoning in pattern recognition. it also discusses the concept of reasoning, pattern recognition, and the distinction between turing and finite state machines, highlighting their implications for language understanding and gpt-3 capabilities.', 'chapters': [{'end': 10988.828, 'start': 10581.836, 'title': 'The future of machine learning', 'summary': 'Discusses the limitations of machine learning and advocates for a hybrid system combining programming and machine learning, emphasizing the need for logical layers and defining clear definitions of reasoning and pattern recognition.', 'duration': 406.992, 'highlights': ['The future lies in hybrid systems, combining programming, logic, and mathematics with machine learning and neural networks, to address various problem domains. Advocating for a hybrid system, combining programming and machine learning, to address various problem domains.', 'The need for logical layers alongside machine learning components to enable reasoning and encoding structure. Emphasizing the need for logical layers alongside machine learning components to enable reasoning and encoding structure.', 'Defining clear terms such as reasoning and pattern recognition and agreeing upon definitions for intelligence. Emphasizing the importance of defining clear terms such as reasoning and pattern recognition and agreeing upon definitions for intelligence.', 'Pattern recognition is transforming data from one substrate to another, while reasoning involves applying first order logic rules or Lambda calculus. Defining pattern recognition as the transformation of data and reasoning as the application of first order logic rules or Lambda calculus.']}, {'end': 11307.978, 'start': 10989.508, 'title': 'Reasoning and pattern recognition', 'summary': 'Discusses the concept of reasoning and pattern recognition, emphasizing the process of filling in missing information through basic reasoning and the distinction between reasoning and pattern recognition, with a brief comparison of their characteristics.', 'duration': 318.47, 'highlights': ['Basic reasoning involves filling in missing information by deducing based on structured queries and world knowledge, exemplified by the case of inferring a tennis ball on a tennis court. Basic reasoning involves deducing missing information based on structured queries and world knowledge, exemplified by inferring a tennis ball on a tennis court.', 'Reasoning combines multiple data sources and operates on the combined dataset, similar to first-order logic and expert systems. Reasoning involves combining multiple data sources and operating on the combined dataset, akin to first-order logic and expert systems.', 'Pattern recognition is described as a finite process of asserting new information from the totality of the dataset, with a distinction from reasoning in terms of symbolic vs. vector-based processing. Pattern recognition is depicted as a finite process of asserting new information from the dataset, distinguished from reasoning in terms of symbolic vs. vector-based processing.']}, {'end': 11765.401, 'start': 11307.978, 'title': 'Distinction between turing and finite state machines', 'summary': 'Discusses the distinction between turing machines and finite state machines, highlighting the potential infinity of turing machines and the bounded computation of finite state machines, and explores the implications for reasoning, language understanding, and the capabilities of gpt-3.', 'duration': 457.423, 'highlights': ['Turing machines have the potential for infinite computation, while finite state machines are bounded in both space and time. Turing machines have the capability for potential infinity in their computation, while finite state machines have a finite bound in both space and time.', 'The distinction between unbounded reasoning computation and bounded pattern recognition computation is a key hypothesis. The chapter introduces a hypothesis regarding the distinction between unbounded reasoning computation and bounded pattern recognition computation, which forms a fundamental question.', 'Experiment with GPT-3 is detailed, highlighting prompt engineering and the limitations of the model in generating accurate responses. The chapter details an experiment with GPT-3, showcasing prompt engineering and the challenges faced in generating accurate responses, demonstrating the limitations of the model.']}], 'duration': 1183.565, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w10581836.jpg', 'highlights': ['The future lies in hybrid systems, combining programming, logic, and mathematics with machine learning and neural networks, to address various problem domains.', 'Emphasizing the need for logical layers alongside machine learning components to enable reasoning and encoding structure.', 'Defining clear terms such as reasoning and pattern recognition and agreeing upon definitions for intelligence.', 'Pattern recognition is transforming data from one substrate to another, while reasoning involves applying first order logic rules or Lambda calculus.', 'Basic reasoning involves filling in missing information by deducing based on structured queries and world knowledge, exemplified by the case of inferring a tennis ball on a tennis court.', 'Reasoning combines multiple data sources and operates on the combined dataset, similar to first-order logic and expert systems.', 'Turing machines have the potential for infinite computation, while finite state machines are bounded in both space and time.', 'The distinction between unbounded reasoning computation and bounded pattern recognition computation is a key hypothesis.', 'Experiment with GPT-3 is detailed, highlighting prompt engineering and the limitations of the model in generating accurate responses.']}, {'end': 14236.065, 'segs': [{'end': 11852.775, 'src': 'embed', 'start': 11814.96, 'weight': 0, 'content': [{'end': 11816.64, 'text': 'The prompt can strain it in a way.', 'start': 11814.96, 'duration': 1.68}, {'end': 11822.433, 'text': 'This is an example of how you could potentially build software with GPT-3 using prompt engineering.', 'start': 11817.287, 'duration': 5.146}, {'end': 11824.375, 'text': 'I saw a discussion of it.', 'start': 11822.833, 'duration': 1.542}, {'end': 11827.258, 'text': 'Okay, so the database begins knowing nothing.', 'start': 11825.376, 'duration': 1.882}, {'end': 11830.401, 'text': "The database knows everything that's added to it, does not know anything else.", 'start': 11827.879, 'duration': 2.522}, {'end': 11834.005, 'text': 'When asked a question, if the answer has been added to the database, says the answer.', 'start': 11830.461, 'duration': 3.544}, {'end': 11837.189, 'text': 'When asked a question, if the answer has not been added, says it does not know.', 'start': 11834.446, 'duration': 2.743}, {'end': 11840.809, 'text': "Does the database know what is 2 plus 2? The database doesn't know.", 'start': 11837.847, 'duration': 2.962}, {'end': 11842.95, 'text': 'So this is all the prompt.', 'start': 11840.909, 'duration': 2.041}, {'end': 11844.33, 'text': "And what we're doing is we're conditioning it.", 'start': 11842.99, 'duration': 1.34}, {'end': 11847.352, 'text': 'So now if we reset and paste all of this stuff in.', 'start': 11844.911, 'duration': 2.441}, {'end': 11851.194, 'text': 'So Tom is 20 years old, has been added to the database.', 'start': 11848.072, 'duration': 3.122}, {'end': 11852.775, 'text': 'Nothing else about Tom is added to the database.', 'start': 11851.254, 'duration': 1.521}], 'summary': 'Using gpt-3 for prompt engineering to condition a database and provide answers based on its contents.', 'duration': 37.815, 'max_score': 11814.96, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11814960.jpg'}, {'end': 11973.443, 'src': 'embed', 'start': 11936.006, 'weight': 2, 'content': [{'end': 11938.327, 'text': 'how did they evaluate whether or not GPT-3 got the right answer?', 'start': 11936.006, 'duration': 2.321}, {'end': 11943.369, 'text': "But my issue is actually I don't see anything intelligent coming out.", 'start': 11939.787, 'duration': 3.582}, {'end': 11947.73, 'text': 'Look, the conditioning is just pruning the search and actually constraining.', 'start': 11943.549, 'duration': 4.181}, {'end': 11951.672, 'text': "Yeah, I have no doubt it doesn't understand semantics,", 'start': 11948.611, 'duration': 3.061}, {'end': 11967.739, 'text': "but I was trying to see if which was the challenge put forward by a few people that privately they were saying what they're trying to prove is that statistics at a large scale can approximate semantics.", 'start': 11951.672, 'duration': 16.067}, {'end': 11971.221, 'text': "So my issue is I'm convinced it doesn't do semantics.", 'start': 11968.359, 'duration': 2.862}, {'end': 11973.443, 'text': "They actually admit, we don't do any understanding.", 'start': 11971.341, 'duration': 2.102}], 'summary': "Evaluation of gpt-3's understanding: lacks intelligence and semantics, admits no understanding.", 'duration': 37.437, 'max_score': 11936.006, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11936006.jpg'}, {'end': 12013.412, 'src': 'embed', 'start': 11990.335, 'weight': 3, 'content': [{'end': 11997.92, 'text': "And you would agree, Keith, I'm sure that as soon as it's non-deterministic, it becomes almost impossible to use this in a software context.", 'start': 11990.335, 'duration': 7.585}, {'end': 12003.627, 'text': "It becomes far more difficult to scientifically evaluate this thing if it's not repeatable.", 'start': 11998.484, 'duration': 5.143}, {'end': 12010.551, 'text': "I'm getting the feeling that those that want GPT-3 to sound good are testing it with good intentions.", 'start': 12003.967, 'duration': 6.584}, {'end': 12013.412, 'text': 'And those that want it to fail are testing it to make it fail.', 'start': 12010.631, 'duration': 2.781}], 'summary': 'Non-deterministic nature of gpt-3 poses challenges for software use and scientific evaluation.', 'duration': 23.077, 'max_score': 11990.335, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11990335.jpg'}, {'end': 12171.597, 'src': 'embed', 'start': 12137.69, 'weight': 6, 'content': [{'end': 12138.951, 'text': "That's the more interesting part.", 'start': 12137.69, 'duration': 1.261}, {'end': 12142.353, 'text': 'GPT-3 is like continuing to ask itself questions.', 'start': 12139.311, 'duration': 3.042}, {'end': 12144.634, 'text': 'Yeah, and funnily enough.', 'start': 12142.733, 'duration': 1.901}, {'end': 12147.296, 'text': "so this is just being completed from something on the internet, isn't it??", 'start': 12144.634, 'duration': 2.662}, {'end': 12150.192, 'text': 'this, just in this just now, completes.', 'start': 12147.932, 'duration': 2.26}, {'end': 12159.594, 'text': "yeah, when you say age and then address, phone number, email highly correlated, so they're in the neighborhood.", 'start': 12150.192, 'duration': 9.402}, {'end': 12163.795, 'text': 'but i think that the question, though, is this basically just something on the internet?', 'start': 12159.594, 'duration': 4.201}, {'end': 12164.415, 'text': "it's gotta be.", 'start': 12163.795, 'duration': 0.62}, {'end': 12171.597, 'text': "come on, if you ask for the age, you are more likely to ask also for the phone number, and i'm guessing that's what's happening.", 'start': 12164.415, 'duration': 7.182}], 'summary': 'Discussion about gpt-3 generating content from the internet', 'duration': 33.907, 'max_score': 12137.69, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w12137690.jpg'}, {'end': 13110.974, 'src': 'embed', 'start': 13082.733, 'weight': 10, 'content': [{'end': 13084.753, 'text': "now you're giving it even more information.", 'start': 13082.733, 'duration': 2.02}, {'end': 13086.614, 'text': 'it should get it, but i wanted the.', 'start': 13084.753, 'duration': 1.861}, {'end': 13088.065, 'text': 'uh Right.', 'start': 13086.614, 'duration': 1.451}, {'end': 13089.306, 'text': "It's insane.", 'start': 13088.706, 'duration': 0.6}, {'end': 13092.867, 'text': 'With embeddings alone, this can happen, right? Yeah.', 'start': 13089.766, 'duration': 3.101}, {'end': 13098.809, 'text': 'So you have, yeah, the similarity here, but they got birds specifically, the generic type.', 'start': 13092.887, 'duration': 5.922}, {'end': 13099.71, 'text': "That's impressive.", 'start': 13099.009, 'duration': 0.701}, {'end': 13104.011, 'text': "Yeah So with patterns, I can say it's pretty good.", 'start': 13099.95, 'duration': 4.061}, {'end': 13104.712, 'text': 'Very good.', 'start': 13104.271, 'duration': 0.441}, {'end': 13107.293, 'text': 'Language competency is another issue.', 'start': 13104.952, 'duration': 2.341}, {'end': 13109.213, 'text': 'Indeed But this is impressive.', 'start': 13107.553, 'duration': 1.66}, {'end': 13110.974, 'text': 'This is not simple stuff.', 'start': 13109.233, 'duration': 1.741}], 'summary': 'Impressive language understanding with embeddings, but challenges with language competency.', 'duration': 28.241, 'max_score': 13082.733, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w13082733.jpg'}, {'end': 13369.425, 'src': 'embed', 'start': 13309.935, 'weight': 11, 'content': [{'end': 13313.737, 'text': 'Can we conclude that? Because people say you can use it to program.', 'start': 13309.935, 'duration': 3.802}, {'end': 13315.998, 'text': 'No, I mean you can, you can.', 'start': 13313.777, 'duration': 2.221}, {'end': 13321.602, 'text': 'it knows all the common functions and it knows how to find a pattern in the input output.', 'start': 13315.998, 'duration': 5.604}, {'end': 13325.364, 'text': 'but on well-known algorithms you cannot use it to do programming.', 'start': 13321.602, 'duration': 3.762}, {'end': 13326.285, 'text': "That's not true.", 'start': 13325.564, 'duration': 0.721}, {'end': 13328.606, 'text': 'Yeah I think, but.', 'start': 13326.865, 'duration': 1.741}, {'end': 13333.985, 'text': 'Okay, so it knows all the common functions and all the common built-in functions.', 'start': 13329.301, 'duration': 4.684}, {'end': 13342.634, 'text': "But I can definitely say it's very impressive in picking up a pattern from few examples,", 'start': 13335.407, 'duration': 7.227}, {'end': 13352.849, 'text': "because the ones we tried can't be especially the generalizing the type the birds, and that was quite impressive.", 'start': 13342.634, 'duration': 10.215}, {'end': 13360.853, 'text': 'Can you search for a dialogue between passive-aggressive people and just continue it? The one, two, three, four, five.', 'start': 13353.369, 'duration': 7.484}, {'end': 13367.803, 'text': 'Okay Just like the, but the bold, not just the bold ones.', 'start': 13363.979, 'duration': 3.824}, {'end': 13369.425, 'text': 'Okay Okay.', 'start': 13368.104, 'duration': 1.321}], 'summary': 'Ai can recognize patterns and common functions but not suitable for well-known algorithms.', 'duration': 59.49, 'max_score': 13309.935, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w13309935.jpg'}, {'end': 13867.907, 'src': 'embed', 'start': 13840.496, 'weight': 9, 'content': [{'end': 13844.88, 'text': "But there's pattern recognition at a very impressive level.", 'start': 13840.496, 'duration': 4.384}, {'end': 13849.684, 'text': "What's interesting, you can test it to make it look good and you can test it to make it not look good.", 'start': 13844.92, 'duration': 4.764}, {'end': 13852.087, 'text': 'It depends what your motives are.', 'start': 13849.724, 'duration': 2.363}, {'end': 13854.549, 'text': "I've seen people say, wow, this is great.", 'start': 13852.107, 'duration': 2.442}, {'end': 13863.645, 'text': "It's not clear at all to the layman how much massaging is required, how many parameters there are, what GPT-3 actually does.", 'start': 13855.49, 'duration': 8.155}, {'end': 13867.907, 'text': "It's just an autoregressive language model, just spurting out tokens.", 'start': 13864.225, 'duration': 3.682}], 'summary': 'Gpt-3 demonstrates impressive pattern recognition, but its functionality and manipulation are unclear.', 'duration': 27.411, 'max_score': 13840.496, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w13840496.jpg'}, {'end': 13941.636, 'src': 'embed', 'start': 13914.443, 'weight': 1, 'content': [{'end': 13917.264, 'text': 'When I saw the database example, I was blown away by that.', 'start': 13914.443, 'duration': 2.821}, {'end': 13921.005, 'text': 'And no one has access to GPT-3 to try this out.', 'start': 13918.163, 'duration': 2.842}, {'end': 13924.767, 'text': "And people don't really appreciate that this is just one example.", 'start': 13921.025, 'duration': 3.742}, {'end': 13930.57, 'text': 'And in that particular case, with a bit of experimentation, we realized that it was just learning some pattern.', 'start': 13925.527, 'duration': 5.043}, {'end': 13937.113, 'text': "It looked like we'd created a database application, but actually it was just learning some pattern on the internet.", 'start': 13931.11, 'duration': 6.003}, {'end': 13938.794, 'text': 'Appearances can be deceiving.', 'start': 13937.514, 'duration': 1.28}, {'end': 13939.675, 'text': 'We have to be really careful.', 'start': 13938.834, 'duration': 0.841}, {'end': 13941.636, 'text': "That's dangerous to science.", 'start': 13940.235, 'duration': 1.401}], 'summary': "Gpt-3's database example was impressive, but potentially deceptive. caution is needed in such cases.", 'duration': 27.193, 'max_score': 13914.443, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w13914443.jpg'}, {'end': 14005.166, 'src': 'embed', 'start': 13983.004, 'weight': 17, 'content': [{'end': 13993.63, 'text': 'And we, like the three of us, are too old and too set into the field and we think too narrowly to even grasp the possibilities of this.', 'start': 13983.004, 'duration': 10.626}, {'end': 13994.651, 'text': "That's what I think.", 'start': 13993.85, 'duration': 0.801}, {'end': 13996.473, 'text': 'that might be a possibility.', 'start': 13995.271, 'duration': 1.202}, {'end': 14001.38, 'text': "And yeah, I'm very much sticking to my original take on GPT-3.", 'start': 13997.174, 'duration': 4.206}, {'end': 14005.166, 'text': "But at the same time, I think we haven't even scratched the surface.", 'start': 14001.46, 'duration': 3.706}], 'summary': 'Gpt-3 holds vast potential beyond our current understanding.', 'duration': 22.162, 'max_score': 13983.004, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w13983004.jpg'}, {'end': 14123.764, 'src': 'embed', 'start': 14097.027, 'weight': 18, 'content': [{'end': 14104.692, 'text': 'but the point is, though, that if there is some kind of ontology or knowledge graph and we could reason over it using some code,', 'start': 14097.027, 'duration': 7.665}, {'end': 14111.215, 'text': 'that just implies that we need to have a two system approach, even if you read all the text.', 'start': 14104.692, 'duration': 6.523}, {'end': 14115.978, 'text': 'By the way, the trick about, I used to do it with junior students in computer science.', 'start': 14111.515, 'duration': 4.463}, {'end': 14123.764, 'text': 'I used to draw a weird symbol on the whiteboard and say, initially, look at the reaction.', 'start': 14116.699, 'duration': 7.065}], 'summary': 'Need for two-system approach in reasoning over ontology or knowledge graph.', 'duration': 26.737, 'max_score': 14097.027, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w14097027.jpg'}], 'start': 11765.765, 'title': 'Gpt-3 capabilities and limitations', 'summary': 'Delves into the use of prompt engineering with gpt-3, analyzing its database conversation, exploring language competency, pattern recognition, and ai understanding of common functions. it also discusses the limitations of gpt-3 in language understanding and the need for a balanced approach for reasoning over a knowledge graph.', 'chapters': [{'end': 12058.393, 'start': 11765.765, 'title': 'Gpt-3 prompt engineering', 'summary': "Discusses using prompt engineering with gpt-3 to improve search performance and the limitations of using statistics to approximate semantics, highlighting the challenges and potential biases in testing gpt-3's capabilities.", 'duration': 292.628, 'highlights': ["Using prompt engineering to constrain search in GPT-3, such as by asking specific questions about the number of offices in European capitals, can improve its performance (e.g., 'The BBC has five offices in Germany, how many does it have in Europe?').", "GPT-3 operates based on co-occurrence and pattern matching, and the database only knows the information added to it (e.g., 'The database knows everything that's added to it, does not know anything else.').", 'The limitations of using statistics at a large scale to approximate semantics in GPT-3 are highlighted, as it does not demonstrate understanding or intelligence beyond statistical patterns.', "Challenges in testing GPT-3's capabilities are discussed, including the biases in testing to make it succeed or fail, and the difficulty of evaluating its performance when non-deterministic and non-repeatable.", "The idea of prompt engineering and the limitations of GPT-3's understanding are further explored, emphasizing the need for unbiased and rigorous testing to truly assess its capabilities."]}, {'end': 12426.534, 'start': 12059.554, 'title': 'Gpt-3 database conversation analysis', 'summary': 'Analyzes a conversation with the gpt-3 database, revealing instances of cherry-picking and highlights its ability to generate responses based on given prompts, including its understanding of age and related attributes with temperature zero and top p to one.', 'duration': 366.98, 'highlights': ['The database can generate responses based on given prompts, showing its ability to understand age and related attributes with temperature zero and top P to one.', 'The conversation highlights instances of cherry-picking in the use of GPT-3, indicating that certain parts of the conversation were deliberately snipped and rolled with.', 'GPT-3 demonstrates its capability to continue asking itself questions and generate responses based on detected patterns, as shown in the analysis of various prompts and answers.']}, {'end': 13030.895, 'start': 12426.954, 'title': 'Prompt engineering and gpt-3', 'summary': 'Explores the challenges of prompt engineering, the limitations of gpt-3 in understanding context, and the impressive capabilities and limitations of gpt-3 in tasks such as word scrambling and text formatting.', 'duration': 603.941, 'highlights': ["GPT-3's limitations in understanding context and the challenges of prompt engineering The transcript discusses the challenges of prompt engineering and the limitations of GPT-3 in understanding context, highlighting the difficulties faced by prompt engineers and the impact of future advancements such as GPT plus one.", "Impressive capabilities and limitations of GPT-3 in word scrambling The discussion about word scrambling showcases the impressive capabilities of GPT-3 in unscrambling words and the limitations in differentiating between memorization and actual learning, offering insights into the potential and constraints of GPT-3's language processing abilities.", "GPT-3's performance in text formatting and understanding patterns The exploration of GPT-3's performance in text formatting and understanding patterns provides an understanding of its capabilities and limitations in tasks like hyphenated word cleaning, word splitting, and pattern recognition."]}, {'end': 13238.049, 'start': 13031.917, 'title': 'Language competency and pattern recognition', 'summary': 'Discusses the impressive pattern recognition and language competency demonstrated by the model, including its ability to generalize, memorize, and interpolate data, as well as its capability in programming tasks.', 'duration': 206.132, 'highlights': ["The model's impressive pattern recognition and language competency are showcased through its ability to generalize, memorize, and interpolate data.", 'It demonstrates proficiency in programming tasks, such as completing algorithms and explaining coding concepts.', "The model's capability in picking up patterns and generalizing data is highlighted as a significant strength.", "The discussion also mentions the model's retrieval ability and its proficiency in completing coding tasks, showcasing its language competency.", "The model's pattern recognition abilities extend to understanding and completing coding tasks, demonstrating its versatility."]}, {'end': 13524.058, 'start': 13238.529, 'title': 'Ai understanding of common functions', 'summary': "Explores an ai's ability to understand and utilize common programming functions, showcasing its impressive pattern recognition and memorization skills, though it also highlights limitations in detecting less common functions and effectively responding to passive-aggressive dialogues.", 'duration': 285.529, 'highlights': ['The AI showcases impressive pattern recognition and memorization skills in understanding common programming functions. The AI accurately identifies and utilizes common programming functions, demonstrating its ability to pick up patterns and memorize well-known functions.', "The AI struggles with less common functions and may give incorrect results in certain cases. The AI's limitations become evident when it fails to recognize less common functions, leading to incorrect results and showcasing a need for further development.", "The AI's response to passive-aggressive dialogues highlights its limitations in understanding and effectively engaging in nuanced conversations. The AI's interactions with passive-aggressive dialogues reveal its challenge in comprehending and appropriately responding to subtle conversational nuances, indicating areas for improvement in natural language processing."]}, {'end': 14236.065, 'start': 13524.078, 'title': 'Gpt-3 and the limits of language understanding', 'summary': 'Discusses the limitations of gpt-3 in language understanding and the potential of pattern recognition, highlighting the danger of overestimating its capabilities and the need for a two-system approach for reasoning over a knowledge graph.', 'duration': 711.987, 'highlights': ['GPT-3 exhibits impressive pattern recognition at a superficial level, showing little evidence of natural language processing or understanding.', "The chapter emphasizes the danger of overestimating GPT-3's capabilities, highlighting the need for a more critical understanding of its limitations and the potential for misleading interpretations.", 'The discussion highlights the need for a two-system approach for reasoning over a knowledge graph, suggesting the limitations of GPT-3 in discovering missing text and the importance of external reasoning mechanisms.', 'The speaker challenges the audience to think critically about the limitations of GPT-3, citing the need for deep reasoning beyond what GPT-3 can currently achieve, emphasizing the importance of common sense knowledge outside the text.', "The chapter concludes with a call for a more critical understanding of GPT-3's capabilities, cautioning against overestimating its potential and highlighting the need for a balanced approach to evaluating its performance."]}], 'duration': 2470.3, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/iccd86vOz3w/pics/iccd86vOz3w11765765.jpg', 'highlights': ['Using prompt engineering to constrain search in GPT-3 can improve its performance.', 'GPT-3 operates based on co-occurrence and pattern matching.', 'The limitations of using statistics at a large scale to approximate semantics in GPT-3 are highlighted.', "Challenges in testing GPT-3's capabilities are discussed, including biases in testing and evaluating its performance.", "The limitations of GPT-3's understanding are further explored, emphasizing the need for unbiased and rigorous testing.", 'The database can generate responses based on given prompts, showing its ability to understand age and related attributes.', 'GPT-3 demonstrates its capability to continue asking itself questions and generate responses based on detected patterns.', 'The challenges of prompt engineering and the limitations of GPT-3 in understanding context are discussed.', 'Impressive capabilities and limitations of GPT-3 in word scrambling are showcased.', "GPT-3's performance in text formatting and understanding patterns provides an understanding of its capabilities and limitations.", "The model's impressive pattern recognition and language competency are showcased through its ability to generalize, memorize, and interpolate data.", 'It demonstrates proficiency in programming tasks, such as completing algorithms and explaining coding concepts.', "The model's capability in picking up patterns and generalizing data is highlighted as a significant strength.", 'The AI showcases impressive pattern recognition and memorization skills in understanding common programming functions.', 'The AI struggles with less common functions and may give incorrect results in certain cases.', "The AI's response to passive-aggressive dialogues highlights its limitations in understanding and effectively engaging in nuanced conversations.", 'GPT-3 exhibits impressive pattern recognition at a superficial level, showing little evidence of natural language processing or understanding.', "The chapter emphasizes the danger of overestimating GPT-3's capabilities and the need for a more critical understanding of its limitations.", 'The discussion highlights the need for a two-system approach for reasoning over a knowledge graph and the importance of external reasoning mechanisms.', 'The speaker challenges the audience to think critically about the limitations of GPT-3 and the importance of common sense knowledge outside the text.', "The chapter concludes with a call for a more critical understanding of GPT-3's capabilities and the need for a balanced approach to evaluating its performance."]}], 'highlights': ["GPT-3's limitations and lack of true understanding, relying on statistical data and the illusion of reasoning.", "The skepticism towards GPT-3's capabilities, emphasizing its reliance on mass hysteria and cherry-picked examples, leading to a distorted perception of its actual performance.", 'The real problem in language is the lack of common ground and the need for background knowledge for interpretation.', 'GPT-3 faces challenges in imputing fresh information and requires retraining the entire system to accommodate new facts.', 'The need for a hybrid system incorporating machine learning and logic for artificial general intelligence, along with discussions on defining intelligence, reasoning, and pattern recognition, is stressed, suggesting a collective agreement on these definitions.', "GPT-3 controversy and viewpoints The chapter presents contrasting viewpoints from critics and advocates of GPT-3, including Gary Marcus, Walid Sabah, Connor Leahy, and Keith Duggar, highlighting the controversy and diverse perspectives surrounding GPT-3's potential for artificial general intelligence.", 'Interactive experiences with GPT-3 It showcases interactive experiences with GPT-3, demonstrating its capabilities in generating articles, text, and creative fiction, while also exploring the limitations of GPT-3 in non-interactive processes and prompt engineering.', 'Introduction of randomness through temperature scaling and top-k sampling to improve deterministic language models, addressing issues such as repeating cycles and low text quality.', 'Insights into using GPT-3 for character analogies and the impact of byte pair encoding issue, highlighting the need for stochastic sampling and the impact of temperature on model performance.', "The limitations of current AI systems are evident, as exemplified by GPT-3's inability to reason and understand beyond generating synonyms for better searches.", "GPT-3's purported advancement towards artificial general intelligence is questioned, highlighting the inability of the system to articulate complex arguments beyond producing good embeddings.", "GPT-3's hype and limitations: GPT-3's mass hysteria and media coverage are critiqued, highlighting its inability to revolutionize chatbots, fulfill natural language generation and reasoning tasks, and the press' cherry-picking of samples to amplify its capabilities.", "Illusion of understanding: GPT-3 creates an illusion of understanding complex concepts like physics and psychology, despite lacking the capacity to reason about the consequences of its actions or anticipate them. It's compared to a magician's trick, conveying an illusion without actual understanding.", "GPT-3's limitations in understanding and reasoning over new knowledge, hindering its ability to reason over new knowledge and make new inferences.", "The importance of integrating diverse technologies, such as pattern perception and symbolic systems, to address specific engineering problems, highlighting the limitations of GPT-3's dogmatic approach and advocating for a more open problem-solving perspective.", 'GPT-3 performs poorly in math and structured tasks, exhibiting errors similar to those made by humans.', 'The strengths and limitations of GPT-3 in information retrieval and reasoning are highlighted, with emphasis on its pattern matching capabilities and the absence of reasoning abilities.', 'The limitations of GPT-3 in understanding language and reasoning are highlighted, showcasing its differences from human communication and knowledge.', 'There is a 30% chance of achieving superhuman AGI by scaling GPT-3 with reinforcement learning, potentially reaching 100 trillion parameters.', 'The limitations of using statistics at a large scale to approximate semantics in GPT-3 are highlighted.', 'The challenges of prompt engineering and the limitations of GPT-3 in understanding context are discussed.', "The model's impressive pattern recognition and language competency are showcased through its ability to generalize, memorize, and interpolate data.", "The AI's response to passive-aggressive dialogues highlights its limitations in understanding and effectively engaging in nuanced conversations.", "The chapter emphasizes the danger of overestimating GPT-3's capabilities and the need for a more critical understanding of its limitations."]}