Gartner is ridiculous. I mean, all consultancies tend to be a little ridiculous — their job is to tell people in other industries how to do their jobs. We’re so used to them now that we rarely stop to think about what a wild idea that is.

One of Gartner’s “signature” models is the “Gartner Hype Cycle,” which looks like this:

This is the complicated version, but pay attention to the titles along the bottom. Basically, here’s the cycle:

• There’s some new technology (technology trigger)
• Everyone gets hyped about it (peak of inflated expectations)
• They realize that it can’t quite live up to the hype and get down on it (trough of disillusionment)
• Eventually they figure out what it’s actually good for (slope of enlightenment and plateau of productivity)

I find the hype cycle funny because Gartner seems blind to the role they play in it — that, in fact, the “peak of inflated expectations” might not exist without consultants hyping how much they can accomplish with this latest piece of technology. I mean, how many AI related webinars and conferences has Gartner had? The answer is just … so many.

Plus, instead of calling something “sales person feeding frenzy” they invent a term like “peak of inflated expectations.”

That’s not to say the hype cycle is worthless. Ultimately the hype cycle is just another sales tool — it’s a way for Gartner to say “Hey, you know why you’re not seeing the ROI that us and other consultants promised? It’s not because we overpromised. It’s the TROUGH OF DISILLUSIONMENT. It’s just nature, baby. We’ll get through this together. We’re probably already on the slope of enlightenment!”

So here’s an idea. Let’s try and skip the peak of inflated expectations and the trough of disillusionment for AI and figure out what it’s actually good for now, today. Let’s talk about why only 25% of AI projects deliver the expected ROI. And we’ll finish with some pretty uncomfortable truths.

The Value of Constraints

One of the things AI promises is MORE. More time. More productivity. More ads made, more code written, more … well, more stuff.

Which is more or less true. The problem is that people can already do everything AI does — it’s just that AI does it faster and cheaper.

Now, if you have a bunch of tasks that both people and AI can do, and you have been running a business for a decade or two WITHOUT AI, then odds are good you’ve figured out how to have people do the most valuable stuff. That means there are two ways to use AI:

• AI can take over stuff that people are already doing
• AI can do other stuff on the list that hasn’t been tackled yet

There’s a different problem for these two use cases.

Why AI can’t just take over

Typically, you’ll have people doing a task because it’s valuable — because it contributes to revenue in some way. If it is a valuable task, then you typically want it done well.

And AI doesn’t do anything particularly well yet. That’s not to say it does it poorly. AI does everything average. It also still has an error rate of 5-10%. And here’s the fundamental problem.

You’ve decided to pay people to do a valuable task and you’ve measured that task to make sure they’re doing a good job. Would you feel comfortable turning that task, which, again, you find valuable enough to pay people to do, over to an LLM that will do it consistently average, with a 5-10% error rate?

If you are a front line manager, you’re not happy with that tradeoff. You want the trained people to continue to do the valuable task you pay them for (NON-front line managers might feel different — we’ll talk about that in a minute). So what does that leave for AI to do?

Why AI can’t just do the stuff you’re not doing

If the stuff people already do is off the table, what about all the stuff you’ve always wanted to do, but haven’t had the people or resources for?

The problem with that is that you’ve thought to yourself “This might be useful, but it’s not worth investing money or time to find out.” You, the savvy manager you are, already decided that the ROI on that project was borderline, so you didn’t do it.

In this case, limited resources forces you to focus on the projects that are most likely to have a positive ROI, leaving aside projects that are unlikely to have a meaningful ROI. But they’re there, on your list!

What that means is, when given a shiny new tool like AI, the projects most likely to get worked on first are projects without guaranteed ROI. If they HAD guaranteed ROI, you would’ve paid a person to do them already.

THAT is why only 25% of AI projects are seeing the expected ROI. Because the ROI is shaky to begin with because it was never a sure-enough thing to spend people on it.

But, BUT, AI is in the “Peak of Inflated Expectations” or, as I prefer to call it “Salesperson/Consultant Feeding Frenzy Zone.” The world is inundated with sales people pitching AI and advertising is effective. If it doesn’t get to your CIO, it gets to your board, and you are told “we’re getting left behind if we don’t do AI! Get on it!”

And then you purchase some AI stuff, and you don’t trust it enough to do something REALLY valuable, so you go “Hey, what’s in our someday/maybe pile that we can try this AI out on?” And you do a project you’ve never felt confident enough in to dedicate real people, and, apparently, 25% of the time it pays off.

Honestly, that’s not a bad hit rate, when you think about it. And if AI is relatively cheap, and you do four projects, one of them is paying off. We’re basically hitting venture capital logic at this point, but still.

So how do you find ROI?

So, if you can’t just take over what people are doing (more on that in a bit), and you can’t just choose the next thing on your someday/maybe list, where do you find ROI?

Here’s three strategies:

1. Look for stuff people hate
2. Look for stuff where more, but average, would still be useful
3. Take a second look at that someday/maybe list

Look for stuff people hate

I can’t take credit for this one. A recent (oh no) Gartner conference had a talk from their global chief of AI who pointed out that many AI applications are just annoying — solving a problem that doesn’t exist (the one he highlighted was meeting summaries — if you’re in meetings you should be in, and skip ones you shouldn’t, why would you ever need meeting summaries?). But he highlights one deployment strategy:

He cited a use case at US healthcare company Vizient where the CTO asked employees what tasks bother them on a regular basis – the sort of thing everyone dreads having to do when they arrive at work on Monday morning. Armed with feedback from thousands of employees, the company automated the most-complained-about chores.

The result? “Instant adoption, zero change management problems,” Brethenoux said. Employees then bought in to AI and started to make good suggestions for further AI-enabled automation.

When we think of tasks people do, and the value they provide, there’s typical work tasks that provide positive value, and then there are tasks the prevent negative value. Think of expense reports, which aren’t a value-add in themselves, but prevent the company from experiencing losses due to fraud. Those types of tasks are GREAT targets for AI, because they MUST be done, but nobody really wants to do them.

Healthcare is full of these types of tasks — there’s tons of transcription, paperwork, coding, reports. And AI can help with much of that.

So find the tasks that MUST be done, but generally people hate, and see if there’s a way to reduce those. Even if you can’t completely automate them, if you can cut down the time so the person is just error-checking what an AI already did you’ve made someone’s day better.

More but average is fine

This is something where there is a task that would benefit from more hands, but for whatever reason you can’t justify the price.

Three examples spring to mind.

1. Assessments. Internal audit almost always has more to do than time, and audit falls into one of those “prevent negative value” tasks — important, but people typically don’t want to think about them too much. If AI can help perform assessments that otherwise wouldn’t be done at all, it’s a valuable use of the technology.
2. Analysis. Having more eyes on logs is never a bad thing in the cybersecurity world. At the Apres Cyber summit there were actually two great talks about using AI to comb through logs. James Pope spoke about how it was used at blackhat (super interesting and a funny story), while Rob Annand had a pretty technical talk about how Adobe uses it. In both cases, people could’ve done what the AI was doing, but it would’ve taken hundreds. Another case where “more but average” is better than “not at all.”
3. Research. In complicated topics you can research for an almost infinite amount of time, but you don’t have infinite time, nor infinite people. Again, AI can be used here to expand the capacities of the people you do have.

That’s something you’ll notice in all these cases — AI doesn’t replace a person, but it does help one person do more. And I really think this is the best case for “more but average” use cases. It’s kind of like “last mile” delivery. AI can get you 90% of the way there, but you still need an actual expert to synthesize and make sure the AI hasn’t made any mistakes.

Take a second look at the someday/maybe list

Hey, 25% isn’t a bit hit rate. You’ve just got to make sure that all four of the projects you choose justify the ROI for the other three, so that if only one hits you’re still in the black.

The common and concerning thread

Here’s what you’ll notice about these use cases. They require people, but they require a lot LESS people. In most cases, AI is augmenting what people can do so that you can have fewer people and more AI.

You’ve probably heard the phrase “AI won’t replace your job, but someone using AI will.” And my blog post here seems to highlight that.

The problem is, it’s worse than I’m making it look because I’m highlighting it from a front-line manager problem — someone who is invested in making sure that the valuable work gets done.

Unfortunately, people invested in value don’t run things. People invested in share-prices run things.

Let’s think about something like a call-center. Let’s say our imaginary call center has 100 employees. The call center manager knows the people who take calls. He hears the stories of his employee who helped a grandmother get her internet back up so her visiting grandkids could watch Bluey. The grandma asked for their address so they could send cookies! What they do is so valuable. That human connection, you know?

A regional VP of customer satisfaction doesn’t care about the cookies. They care that 100 people costs 6 million dollars a year, and a consultant just told them they could run that whole call center for two million bucks worth of AI, plus just 10 employees to double check the AI work (remember that 10% error rate), for a grand total of just $2.5 million a year.

Sure, they calculate their Net Promoter Score will fall from a 4.3 to a 2.9. But it’s already bad! And cutting costs by half will look fantastic on his next performance review.

I don’t know if you can tell I’m conflicted

AI is a useful tool. AI will lead to a huge redistribution (potentially further concentration) of labor (and wealth). AI will create enough jobs to make up for the ones it kills. According to the people who sell AI. People who use AI are dummer. Our children are using AI to cheat on all their tests.

I work in technology so I feel like I can’t ignore AI, but I also don’t want to just … do what Sam Altman says because I don’t trust him, or any of the AI sales people. It’s a powerful but dangerous tool that I don’t want to ignore, but I want to make sure I use it in a way that makes people’s lives better.

Is that possible?

In the long term I don’t know. Short term, at the very least, we can use it to help people with the tasks they hate and help do stuff that is valuable that otherwise wouldn’t get done.