Will A.I. Ever Live Up to Its Hype?

It’s a little hard to believe that just over a year ago, a group of leading researchers asked for a six-month pause in the development of larger systems of artificial intelligence, fearing that the systems would become too powerful. “Should we risk loss of control of our civilization?” they asked.

There was no pause. But now, a year later, the question isn’t really whether A.I. is too smart and will take over the world. It’s whether A.I. is too stupid and unreliable to be useful. Consider this week’s announcement from OpenAI’s chief executive, Sam Altman, who promised he would unveil “new stuff” that “feels like magic to me.” But it was just a rather routine update that makes ChatGPT cheaper and faster.

It feels like another sign that A.I. is not even close to living up to its hype. In my eyes, it’s looking less like an all-powerful being and more like a bad intern whose work is so unreliable that it’s often easier to do the task yourself. That realization has real implications for the way we, our employers and our government should deal with Silicon Valley’s latest dazzling new, new thing. Acknowledging A.I.’s flaws could help us invest our resources more efficiently and also allow us to turn our attention toward more realistic solutions.

Others voice similar concerns. “I find my feelings about A.I. are actually pretty similar to my feelings about blockchains: They do a poor job of much of what people try to do with them, they can’t do the things their creators claim they one day might, and many of the things they are well suited to do may not be altogether that beneficial,” wrote Molly White, a cryptocurrency researcher and critic, in her newsletter last month.

Let’s look at the research.

In the past 10 years, A.I. has conquered many tasks that were previously unimaginable, such as successfully identifying images, writing complete coherent sentences and transcribing audio. A.I. enabled a singer who had lost his voice to release a new song using A.I. trained with clips from his old songs.

But some of A.I.’s greatest accomplishments seem inflated. Some of you may remember that the A.I. model ChatGPT-4 aced the uniform bar exam a year ago. Turns out that it scored in the 48th percentile, not the 90th, as claimed by OpenAI, according to a re-examination by the M.I.T. researcher Eric Martínez. Or what about Google’s claim that it used A.I. to discover more than two million new chemical compounds? A re-examination by experimental materials chemists at the University of California, Santa Barbara, found “scant evidence for compounds that fulfill the trifecta of novelty, credibility and utility.”