Weekly Briefing: Did ChatGPT really ace MIT's undergraduate curriculum?

Students saw red flags in the study, and poked holes in the methodology. ADVERTISEMENT [Weekly Briefing Logo]( You can also [read this newsletter on the web](. Or, if you no longer want to receive this newsletter, [unsubscribe](. Did AI really ace MIT’s undergraduate curriculum? A preprint study posted last month said that ChatGPT, the popular AI chatbot, completed the Massachusetts Institute of Technology’s undergraduate curriculum in math, computer science, and electrical engineering with 100-percent accuracy. Sounds too good to be true, right? The study still had to go through peer review. It had 15 authors, including several MIT professors. And given [other feats ChatGPT]( has performed recently, the idea that the bot could graduate from MIT didn’t seem that crazy. But soon after the study was posted, three MIT students closely evaluated the methodology and data. They said they found “glaring problems” akin to letting the chatbot cheat its way through classes. What first seemed like a landmark study is slowly turning into a cautionary tale. The three students, Neil Deshmukh, Raunak Chowdhuri, and David Koplow, collaborated after noting red flags in the paper. First, they found that some of the questions didn’t seem solvable with the information the authors gave ChatGPT; there wasn’t enough context. In other instances, the “questions” were actually assignments. The study used a technique called few-shot prompting — a tactic often used when training large language models like ChatGPT to perform a task. To do this, the chatbot is shown multiple examples so it can better understand what it’s being asked. For this study, the examples were so similar to the answers to the questions that it was, the students wrote, like being “fed the answers to a test right before taking it.” The students said they checked and double-checked their work to be fair to the paper’s authors, professors at their university. When they posted [their detailed critique,]( responses came flooding in. Some onlookers congratulated them. Authors of the paper were less thrilled. One author, Armando Solar-Lezama, a professor in the electrical-engineering and computer-science department at MIT and associate director of the university’s computer-science and artificial-intelligence laboratory, said he hadn’t realized the paper would be posted as a preprint. He added that he didn’t know about the claim that ChatGPT could ace MIT’s undergraduate curriculum, calling the idea “outrageous.” For Solar-Lezama, the paper was supposed to assess something else: which prerequisites should be mandatory for MIT students. Sometimes students discover during a class that they lack the background to completely grapple with the material. An AI analysis could help professors and the university with that problem. Solar-Lezama and other co-authors said Iddo Drori, an associate professor of the practice of computer science at Boston University, was the driving force behind the paper. Solar-Lezama gave Drori an unpaid position at MIT that allowed him to “get into the building” so they could work together. Solar-Lezama said he’d been intrigued by Drori’s ideas about training the chatbot on course materials. Solar-Lezama told our Tom Bartlett that Drori used “sloppy methodology” and that he didn’t get permission to use course materials from MIT instructors, though Drori said that he did. Solar-Lezama and two other MIT professors, also co-authors of the paper, [released a statement]( saying they hadn’t approved the posting of the preprint, and that professors hadn’t given Drori permission to use assignments and exam questions. Drori didn’t want to be interviewed by The Chronicle, and instead emailed a 500-word statement to Tom with a timeline of how and when he said the paper was prepared and posted online. Drori did acknowledge that the “perfect score” was incorrect and said he will fix issues in the second version. [Read]( full story here](. ADVERTISEMENT NEWSLETTER [Sign Up for the Teaching Newsletter]( Find insights to improve teaching and learning across your campus. Delivered on Thursdays. To read this newsletter as soon as it sends, [sign up]( to receive it in your email inbox. Lagniappe - Read. Here’s a case for [getting to know your neighbors](. (The New York Times) - Listen. The album [Jerusalem,]( by Emahoy Tsegué-Maryam Guèbrou, an Ethiopian nun turned pianist and composer, is worth your while. Especially considering that [this is her last](. (Spotify, The New Yorker) —Fernanda SUBSCRIBE TO THE CHRONICLE Enjoying the newsletter? [Subscribe today]( for unlimited access to essential news, analysis, and advice. Chronicle Top Reads THE LETTER OF THE LAW [What Counts as Discrimination on a College Campus?]( By Kelly Field [STORY IMAGE]( Mark Perry has filed hundreds of complaints with the Office for Civil Rights. His critics say he’s undoing decades of progress. SPONSOR CONTENT | Canon [A New Media Landscape Offers Both Tremendous Opportunities and Challenges]( The media industry has changed drastically over the past two decades. What has changed? What can be done to ensure students’ skillsets are competitive in this new landscape? Read more to find out. 'SOUR GRAPES' [A Florida Presidential Search Was Halted Because of ‘Anomalies.’ The Board Chair Says Nothing’s Amiss.]( By Emma Pettit [STORY IMAGE]( The state university system chancellor effectively ordered the pause, prompting Florida Atlantic University and its search firm to defend themselves. A faculty leader suggested the move was political. THE REVIEW | CONVERSATION [Did Colleges Discriminate Against Asians? The Court Didn’t Say.]( By Evan Goldstein and Len Gutkin [STORY IMAGE]( The Harvard law professor Jeannie Suk Gersen on the affirmative-action decision. ADVERTISEMENT FROM THE CHRONICLE STORE [Restructuring a University - The Chronicle Store]( [Restructuring a University]( In 2022, Henderson State University declared financial exigency after realizing it could no longer avoid hard choices. This case study of the university’s path to near-ruin highlights lessons for any college leader contemplating a restructuring to keep an institution viable. [Order your copy]( to learn about key factors to consider in a restructuring process. NEWSLETTER FEEDBACK [Please let us know what you thought of today's newsletter in this three-question survey](. This newsletter was sent to {EMAIL}. [Read this newsletter on the web](. [Manage]( your newsletter preferences, [stop receiving]( this email, or [view]( our privacy policy. © 2023 [The Chronicle of Higher Education]( 1255 23rd Street, N.W. Washington, D.C. 20037