How To Avoid the Museum of Failure

A new book develops a new science of scaling. [View this email online]( [Planet Money]( An Economist’s Guide To Scaling --------------------------------------------------------------- by Greg Rosalsky It’s been called one of the biggest product flops ever. It’s studied in business schools as a cautionary tale. It’s literally in The Museum of Failure (an actual [museum](. But back in 1996, McDonald’s executives believed that the Arch Deluxe hamburger was the solution to their problems. The company’s growth had flatlined. Competitors were eating into their sales. With its stone-ground dijon mustard, peppery bacon, and split-top potato bun, the flavorsome Arch Deluxe was supposed to help the company reverse their decline by appealing to a new, older demographic. The restaurant conducted extensive focus groups that showed adults liked the burger. And so they invested money — lots of money — in rolling it out nationwide. McDonald’s kicked off the release of the Arch Deluxe with a glitzy [event]( at Radio City Music Hall in New York City. The event featured Ronald McDonald dressed in a tuxedo, doing strut kicks with the Rockettes. The company then spent hundreds of millions of dollars blanketing the airwaves with ads that [declared]( the Arch Deluxe was “the burger with the grown-up taste.” It was [the most]( a fast-food company had ever spent on a product launch. However, almost as soon as McDonald’s took the Arch Deluxe national, sales proved to be abysmal. The burger cost more than the Big Mac and Quarter Pounder, and core McDonald’s customers rejected it. Even more important, the Arch Deluxe failed to lure more adults through McDonald’s doors, as they had hoped it would. Within a year of launching the burger, the company began phasing it out. Not long after, the McDonald’s CEO [resigned](. But if McDonald’s had read a new book, The Voltage Effect, maybe the company could have averted the Arch Deluxe catastrophe. Its author, the University of Chicago economist John List, says the focus groups that teed up McDonald’s decision to spend a fortune on launching the Arch Deluxe were flawed. “The people who participated in the focus groups weren’t a faithful reflection of McDonald’s customers as a whole,” List writes. Economists call this selection bias. The people who volunteered — or self-selected — to take part in the focus groups were McDonald’s diehards who were willing to spend their spare time testing and getting asked questions about burgers. They were not representative of customers at large. Before spending hundreds of millions of dollars rolling out the Arch Deluxe, the company should have vetted whether the burger would actually be popular with their coveted adult demographic on a smaller scale. Maybe by, for example, trying it out in select markets for a while. Kena Betancur/Getty Images The launch of the Arch Deluxe is an example of a process that The Voltage Effect seeks to systematize and improve: scaling. Scaling is the process of rolling out something on a large scale. Whether it’s a social movement seeking to improve society with a policy to reduce inequality, or a corporation trying to make a buck, scaling is central to what most major organizations are usually trying to achieve. “If an idea doesn't scale to change the world, then what is an idea really worth in the end?” List says. The New Science Of Scaling List, who headed the University of Chicago economics department for six years, and who has served as the chief economist of Uber and Lyft, is one of the leading minds in modern empirical economics. His research has focused on creating really cool experiments in the real world to answer fascinating questions, like why do people donate to [charity]( and how can you influence them to give more? How can we [nudge businesses]( to pay their taxes? And do people [actually behave]( the way classic economic theory says they do? List says he first got interested in the process of scaling about a decade ago, when he was launching an experimental preschool in Chicago Heights, Illinois. The school, which he co-founded with economists Steve Levitt and Roland Fryer, sought to figure out how to improve educational outcomes for disadvantaged youth. In 2014, List says, they got some of their first results showing effective educational curricula and techniques, and he started pitching policymakers to adopt them. “Each time they would tell me, ‘John, that's great, but we don't think your program will scale.’” Around that same time, List was also the chief economist of Uber (he’s now the chief economist of Lyft). A big part of that job was helping Uber figure out which products, services, and policies it should implement on a large scale. John List (Photo by John Boehm) As a result, List began digging into what had been written about scaling, and he was disappointed by what he found. Scaling was mostly one of those empty buzzwords that business types bloviate about. Using his vast expertise in empirical economics, List wanted to develop a science of scaling. The Five Vital Signs To Look Out For When Scaling In his new book,The Voltage Effect, List identifies five factors, or “five vital signs,” that people should consider when thinking through whether an idea, policy, or product can actually scale. The first is watching out for “false positives.” That’s when the idea an organization wants to scale may look like it holds promise, but, if they actually rigorously test the idea, they will see that the idea is bogus. This is common in social science. One study shows something works, but then when someone else tries to do a similar study, the results fail to replicate. The key to fighting false positives, List says, is basically using good social science. “I think all it takes is a few independent replications,” List says. “So at Lyft, we might find a great result in Seattle. That doesn't mean we should roll that out ubiquitously. We should replicate it in Seattle to make sure it's a true result in Seattle and then try it out in a few other markets. The idea is you should constantly be poking and prodding at your idea to make sure it's a true result.” The next vital sign, List says, is considering whether you’re “misjudging the representativeness of an initial population or situation.” This is what McDonald’s did when it erroneously used those focus groups to convince themselves of rolling out the Arch Deluxe. Sure, it was popular with a small group of loyal customers. But was that small group representative of the demographic that the company was trying to target? “What we really need to do is be more representative in our samples before we roll something out nationwide,” List says. The third vital sign, List says, is evaluating “whether your initial success depends on unscalable ingredients.” List gives the example of his experimental preschool. A mistake, he says, would have been to use their ample resources to hire 30 of the best and brightest teachers and then expect that their positive results would be replicated when their curricula was scaled nationwide. With the constrained budgets of public school systems, it would be (sadly) unrealistic to expect them to be able to hire thousands of teachers of the same caliber. And, without those high-caliber teachers, policymakers should consider whether a program’s promising results can still scale. “So the idea with this third vital sign is to try and understand the constraints that you're going to have at scale before you scale it,” List says. The fourth vital sign is watching out for “spillovers” and, more broadly, unintended consequences. An example List uses is when Uber attempted to raise the incomes of its drivers by increasing its base fares. A [recent study]( found that it worked at first, but after six weeks, Uber drivers' hourly earnings fell back down to what they were before the fare increase. Why? Basically because the opportunity of making more money on the app enticed more people to drive for Uber. That made the market more competitive and resulted in each individual driver getting fewer trips overall. At scale, the policy didn’t work as intended. Economics is complicated! The final vital sign is all about considering the business side and costs of scaling. One example, List says, is identifying whether your product has economies of scale. That is, once you scale your operation, does it get cheaper and cheaper to produce what you’re selling or providing? For instance, building a factory costs a lot of money upfront and those first few widgets the factory produces will reflect that. But, over time, those widgets will get cheaper and cheaper because the one-time fixed cost of building the factory is spread out over more and more units. This is a special sauce for scaling. There is, of course, much more in the book, so check it out. Maybe it will stop you from making an Arch Deluxe-style failure on an epic scale. Not subscribed? [Subscribe to this newsletter.]( Want to spread the love? [Share the web-version of this newsletter on social media.]( Craving more content? [Listen to our podcasts.]( --------------------------------------------------------------- Newsletter continues after sponsor message --------------------------------------------------------------- On Our Podcasts --------------------------------------------------------------- The Spider-Man Problem — Spider-Man isn't the first film franchise to be rebooted over and over again. But the infamous off-screen drama between Marvel Studios and Sony Pictures explains why it happens so frequently. [Listen here]( Two indicators: supply chain solutions — Two stories about people trying to overcome supply chain challenges. We follow a ship that is forced to get creative to bypass clogged ports, and we visit a warehouse that is running out of space. [Listen here]( Cheese Wars — A US District Court just ruled Gruyere cheese can be made outside of the Swiss French border region – including here in the US. The Indicator explores what this court battle over a cheese has to do with global trade. [Listen here]( Also on The Indicator: [Revenge of the venture capitalists]( [The opioid business plan]( and [Overdraft fees: From perk to penalty]( --------------------------------------------------------------- Stream your local NPR station. Visit NPR.org to find your local station stream. [Find a Station]( --------------------------------------------------------------- [Subscribe to Planet Money+](. Your support helps make our show possible and unlocks access to our sponsor-free episodes. What do you think of today's email? We'd love to hear your thoughts, questions and feedback: [planetmoney@npr.org](mailto:planetmoney@npr.org?subject=Newsletter%20Feedback) Enjoying this newsletter? Forward to a friend! They can [sign up here](. Looking for more great content? [Check out all of our newsletter offerings]( — including Daily News, Politics, Health and more! You received this message because you're subscribed to Planet Money emails. This email was sent by National Public Radio, Inc., 1111 North Capitol Street NE, Washington, DC 20002 [Unsubscribe]( | [Privacy Policy]( [NPR logo]