The Edge: How to make a campus a living laboratory to study learning

Researchers at the U. of California at Irvine are mining several sources of data to better understand students’ experiences — and improve them. They say others should follow suit. ADVERTISEMENT [The Edge Logo]( Did someone forward you this newsletter? [Sign up free]( to receive your own copy. You can now read The Chronicle on [Apple News]( [Flipboard]( and [Google News](. I’m Goldie Blumenstyk, a senior writer at The Chronicle covering innovation in and around higher ed. This week I check in on a project that explores how colleges can mine an array of campus data to better understand what’s happening with students in and outside the classroom. ADVERTISEMENT SUBSCRIBE TO THE CHRONICLE Enjoying the newsletter? [Subscribe today]( for unlimited access to essential news, analysis, and advice. A research university investigates itself Five years ago, Richard Arum and a crew of researchers set out to turn their institution into a living laboratory to study learning. Now they are turning up some fascinating results. Just as significant, Arum believes this initial phase at the University of California at Irvine proves that the effort is not only replicable by other campuses but vital. Given public doubt about the value of higher education, “particularly the costly residential model,” says Arum, colleges should be measuring some of what they argue is important. “If not to improve it,” he adds, at least “to defend it.” Arum, a professor of education and sociology, is probably best-known as the co-author of [Academically Adrift]( which documented how little time students were spending on their course work. But [as he told me several years ago]( after that book came out in 2011, he realized the need for “a firmer and more empirical basis” to measure the value of college and especially a liberal-arts education. That’s when he began to create this new data-rich project on what’s happening to students inside and beyond the classroom, now known as the [UCI Measuring Undergraduate Success Trajectories Project]( (MUST Project). Last month I sat down with Arum and several of his colleagues in Irvine (and a few on Zoom) to hear what they’ve been finding. Side note: It was great to spend time on an actual campus, even if that one, despite its handsome brutalist buildings and lush landscaping, lacks my favorite higher-ed amenity, [an on-site creamery]( The scope of the research is still taking shape, but I was impressed by the breadth and focus of what I heard about, including: - Studies of how university grading policies in STEM courses may be unintentionally steering female, first-generation, and underrepresented-minority students away from majors that lead to lucrative jobs. - Analyses of more than two-million class-discussion posts over three years, drawn from the UCI learning-management system, that show underrepresented-minority students tend to post less and receive fewer peer responses than their classmates do (Arum says that could be problematic because a lot of learning occurs through peer interaction). - Inquiries into how students’ choice of roommates may affect the racial and ethnic diversity of who ends up as classmates in courses. - Plans to track whether students who engage regularly with the career-services office, and those who are active in extracurricular activities, fare better postgraduation than do others. That will draw on data from the career-services platform Handshake and from CampusGroups, UCI’s digital campus-community platform. Cool, right? The projects, some of which are led by current and former grad students, illustrate how institutions can tap into a range of existing data sources to better understand what’s happening around them. Note to the hundreds of other colleges using Handshake: What are you doing with that info? And credit where due: The research on grading policies was conducted by Oded Mcdossi, a UCI postdoc, who is also tracking the impact of student engagement; the LMS work came from Renzhe Yu, now an assistant professor at Teachers College, Columbia University; and the roommate study was done by Xunfei Li, UCI Ph.D. student. The university is also generating a heap of new data. Since 2019, the MUST Project has been regularly giving a set of five tests to about 1,000 freshmen and juniors to measure their intellectual development in five areas: critical thinking, perspective taking, understanding of confirmation bias, collaborative problem solving, and online civic reasoning. It also texts weekly surveys to a subset of that group to collect more real-time insights into emerging questions, like [how students managed their stress during Covid]( [what sources they trusted]( for pandemic information, and most recently, what their experiences have been with ChatGPT. In case you’re curious, three-quarters said they knew about the generative AI tool; 96 percent said they hadn’t used it in their course work. (Maria Calderon Leon, the MUST Project‘s lab manager, stewarded that work.) The findings I’m really interested in seeing — evaluations of the intellectual-development tests given to students over four years — aren’t available yet. The final set of assessments for the students who entered as freshmen in the fall of 2019 are being administered this spring. Like Arum, I hope the results offer deeper insights into what students are learning than did the data sets used for Academically Adrift, which [some questioned](. The MUST Project was originally funded with $1.5 million from the Andrew W. Mellon Foundation. When that ran out, the university stepped in, forming the [Postsecondary Education Research & Implementation Institute]( which now helps support the work. Michael Dennin, vice provost for teaching and learning, says having that research capacity on campus makes it easier to introduce changes. For example, when the university began to explore reasons that low-income students were leaving, he says, an analysis aided by MUST researchers helped to identify that the cause was less financial and more related to an academic policy that made it difficult for students to switch majors. In discussions about altering that policy, Dennin says, “it made more impact on the Academic Senate that the data came from UCI researchers.” The project’s four-year, $4.1-million partnership with California’s Calbright College is another source of support, [as I described in my last newsletter](. In effect, MUST is an educational R&D arm for the new online community college dedicated to serving working adults. UCI researchers, along with the University of Virginia’s Benjamin L. Castleman, an associate professor of economics and education, and colleagues from the nonprofit behavioral-design consultancy Ideas 42, are experimenting with nudges to help students stay on track, and studying different approaches to course structure and faculty-student interaction. Shannon McCarty, Calbright’s vice president for learning and instruction, hopes this work might inform other institutions that are looking to update short-term and competency-based programs. Arum is also developing a series of surveys for Calbright students that will be given when they first enroll and then periodically during their programs, including if they withdraw, and after. Arum hopes the work with Calbright, and with the John N. Gardner Institute for Excellence in Undergraduate Education, particularly on [students’ first-year experiences]( at broad-access institutions, is just the start. “We’d love for this to be taken up by other campuses,” he says. I’d love to see that, too. But after hearing what’s involved — for example, collecting and anonymizing millions of data points from an LMS or ensuring that statistically significant numbers of students receive and reply to weekly surveys — I wonder how many institutions are prepared to make that kind of commitment. Arum says he and his colleagues stand ready to help; colleges can borrow surveys and even the coding scripts Yu developed to extract information from Irvine’s LMS. And let’s not forget, Arum adds, many institutions also have “an incredible internal capacity” in the form of their own professoriate. “The biggest barrier” he says, “is institutions’ willingness.” Got a tip you’d like to share or a question you’d like me to answer? Let me know, at goldie@chronicle.com. If you have been forwarded this newsletter and would like to see past issues, [find them here](. To receive your own copy, free, register [here](. If you want to follow me on Twitter (yeah, for now at least, I’m still there), [@GoldieStandard]( is my handle. NEWSLETTER [Sign Up for the Teaching Newsletter]( Find insights to improve teaching and learning across your campus. Delivered on Thursdays. To read this newsletter as soon as it sends, [sign up]( to receive it in your email inbox. Goldie's Picks GIVING HOPE [How a Student Emergency-Aid Experiment Became a Lifeline]( By Eric Hoover [STORY IMAGE]( An initiative at Milwaukee Area Technical College that put faculty and staff members in charge of offering emergency assistance to students has spread to 28 institutions. SPONSOR CONTENT | MSU Denver [Tapping into the Limitless Potential of Supporting Neurodiverse Students]( THE NEW DEAL [The U. of Idaho Moved Fast to Acquire the U. of Phoenix. Now What?]( By Michael Vasquez [STORY IMAGE]( The $550-million purchase comes weeks after the University of Arkansas system also considered a deal with Phoenix but faced opposition from its board. INVESTIGATION [The Newest Way to Buy an Advantage in College Admissions]( By Daniel Golden and Kunal Purohit [STORY IMAGE]( Why some parents are paying to make their teen a “peer reviewed” author. THE REVIEW | OPINION [I’m a Student. You Have No Idea How Much We’re Using ChatGPT.]( By Owen Kichizo Terry [STORY IMAGE]( No professor or software could ever pick up on it. SPONSOR CONTENT | University of Miami [University of Miami’s Julio Frenk, Donna Shalala Examine Education and Health Care]( Dive into a new approach to health care that can solve current issues while preparing us for the future. ADVERTISEMENT FROM THE CHRONICLE STORE [Restructuring a University - The Chronicle Store]( [Restructuring a University]( In 2022, Henderson State University declared financial exigency after realizing it could no longer avoid hard choices. This case study of the university’s path to near-ruin highlights lessons for any college leader contemplating a restructuring to keep an institution viable. [Order your copy]( to learn about key factors to consider in a restructuring process. NEWSLETTER FEEDBACK [Please let us know what you thought of today's newsletter in this three-question survey](. This newsletter was sent to {EMAIL}. [Read this newsletter on the web](. [Manage]( your newsletter preferences, [stop receiving]( this email, or [view]( our privacy policy. © 2023 [The Chronicle of Higher Education]( 1255 23rd Street, N.W. Washington, D.C. 20037