The Biggest Mistake People Make With AI Is Accepting Its First Answer

0:00One of the biggest mistakes we make with AI is assuming it's first answer is the one we should act on. And I think that's becoming increasingly dangerous. So, let's discuss. Welcome to the most prolific AI marketing podcast out there in a nutshell. This is the Daily 10 Minute Podcast, what I take out. All of the techies talk around generative AI. I focus on what the everyday person, content creator, and marketer wants to hear. Now, I only got 10 minutes, I can only focus on one subject. And today is a YouTube day. Yes, this is the day to week where I pretty much will summarize what I've said on YouTube. But if you want to watch the YouTube version, which is going to be about use cases, you'll see me. You'll see my screen. Check out YouTube. Now, because of just started on YouTube, if I gave you the name of the title of the video, you probably won't find it. The best thing to do is go to my website, Andrew Miles Davis.com, Davis built the AVIS.

1:01Andrew Miles Davis.com slash YouTube. And then it will redirect you to the YouTube page. And it'll be the top videos, but obviously you can look at other videos as well. So, what did I talk about on YouTube? I'm focusing on use cases right now. So, how are people using AI? And most of us are using AI, especially the large language models, to give us answers. But I think the real skill is knowing what to do with those answers. Because imagine you've got 300 customer comments sitting in a spreadsheet. And your manager says, can you go through these and tell me what our customers are actually saying? That's a big task. And it actually used to be something we used to have to do. But now we've obviously got Gen AI, which you can just give the whole thing to. But here's where I think people make the mistake. They upload a spreadsheet, and then they might prompt something like,

2:01summarize these comments. And AI will give you a summary. Job done. Except I don't think the job is done. Because what you're going to see here, maybe yesterday, when I say yesterday, pre-AI, that would have been job done. But because of AI, I think people's expectations are going to be even higher. So, how do we go to that next level? Because summarizing something and analyzing something, are not necessarily the same thing. And I've always said, when it comes to, especially analyzing, to take things to the next level, you need to answer three simple questions. What is this? So what? What now? So let's start with what is this? So what is this? And for this example, we want to know what are these 300 customers actually telling us? Now, before I get into the detail, I just want to say as well, I'm just using this as an example of 300 comments. This can be for anything. So just be aware. Understand obviously the principles and apply it when needed.

3:08So what are these 300 customers actually telling us? Now, I've actually created a fictional data set for this because obviously I'm not going to put the confidential customers information into a YouTube video. And when I gave these 300 comments to AI, I didn't ask it to summarize them. I asked it to analyze them. And I wanted it to identify the main themes, estimate how often each one appeared, show me the evidence behind its conclusions, give me representative comments, and feel like anything it was uncertain about. And that's an important distinction because I don't just want AI to make my information shorter. I want it to help me understand it. And in this particular example, it identified four major themes. Delivery and fulfillment, product quality, customer service, and the website's ordering experience. Perfect. I've gone from 300 individual comments to four broad areas that I can actually start working with.

4:11And as some of you may know, if you've been listening to this show a while, when I break down analysis, it's part of my AI essentials, which are the 12 things that makes up 95. If not more percent of how we use AI at work. So I've always said, it is only 12 things we need to get good at 12. And then we're going to be good at 95% or more of AI. So that's all to do with what is this? But if we just stop there, there's going to be a bigger problem. Because those four themes are essentially telling me what customers talked about most. They're not necessarily telling me what matters most. And that takes us to the second question. So what? Now, this is the bit that I think a lot of people miss. And I asked AI to challenge its own analysis. Don't just tell me why your answer is right. Tell me why it might be wrong. Whatever that contradicts your main conclusion, what minority views might be hidden underneath these big themes.

5:11Is there anything here that doesn't come up very often, but could actually be incredibly important? And when I did that, AI found something interesting. There was a relatively small group of customers talking about problems, canceling their subscription and getting refunds. It was only about 5-6% of the comments. Now, if you're simply ranking everything by frequency, it's nowhere near the top. But think about this. Imagine 30% of your customers say they prefer the button on your website to be a different color. And then 5% said they can't cancel their subscription. Which one matters more? Obviously, we can't answer that simply by saying 30 is bigger than 5. Because frequency and importance are not the same thing. And I think this is really, really important lesson when we're using AI. Because AI is incredibly good at finding patterns.

6:12But the biggest pattern isn't automatically the most important pattern. So now we've moved beyond what are people saying to, so what does this actually mean? But then there's a third question, which is, as I said, what now? Imagine I've done all of this analysis, and I go back to my manager and say, right, I've analyzed 300 comments here at the four biggest themes, and I've also found this smaller cancellation issue that I think we need to pay attention to. What's the next question they're going to say? It's probably going to be something like, okay, what should we do? So now I can go back to AI and ask it to recommend free possible actions. But don't just get free nice bullet points. For each recommendation, show me the evidence supporting it. What's the likely impact? What are the risks? What are the trade-offs? What additional information would we need before actually making a decision?

7:12And look at what's happened here. So we've gone from 300 messy customer comments to breaking down ways. I find the patterns to so what IE work out what these patterns actually mean and challenge the obvious conclusions to what now, which is what can we actually do about it? Now, if you want to see me demonstrate the spreadsheet, the prompts, and all of that in AI, again, go to my YouTube channel. So the best way, as I said before, just go to angel miles davis.com slash YouTube, and it will take you there. But I think there's a bigger point, not necessarily a problem, a bigger point, because look at where the human value sits in this whole process. This is something I'm constantly asked, where's where the humans fit within this? So I'm always looking, where does the human value sit in this whole process? Before AI, somebody might have spent three hours reading those comments, highlighting them, grouping them, counting things.

8:13Today, AI can do a lot of that work instead of you, incredibly quickly. This is where I think my story lens method becomes useful. And I've mentioned this a few times this week, and I'm going to start getting more into my story lens methods in these podcasts, because the story lens is essentially a way of understanding what happens when significant change enters an established world. There are seven stages to it, this story lens method, but free a particularly relevant here. Arrival exposure, decision. The arrival is pretty obvious. AI enters this process and suddenly something that might have taken me hours can potentially be done in minutes. That's the stranger comes to town, that's that story. Everything's going okay, then this has come to town, and we have to act. That's AI. But I think exposure is much more interesting, because once AI can do that work,

9:13it exposes something about the way we were working before. And if you look at these films when I break down story, when it comes to the arrival of the stranger, it exposes what people really think. COVID, it says it's typical example. Arrival, COVID, exposes what people think of certain things, etc. But I'm not going to get into that here. So in this situation, if I was spending three hours manually grouping customer comments, was that really where my value was? Was it the best use of my time or my judgment or my experience? Or were we just doing it because someone had to do it? So then we're going to decision stage, and decision is pretty much what it says. AI can analyze the comments, but it still doesn't necessarily know which one we should choose. Because the answer to that might not exist anywhere outside of those 300 comments. So it doesn't know what my company is trying to achieve, what are the priorities,

10:17what are our budgets, and a lot more things. And I think that's one of the really interesting things that's happening to work because of AI. Human value doesn't disappear because of AI. It just moves. And if AI saves me three hours manually sourcing through customer comments, it doesn't simply do the same job faster. I would say it spends more time questioning the analysis. So when a nutshell, AI can help you analyze information faster, but finding the patterns is only the beginning. The real value comes from asking free questions. What is this? So what? What now? And AI is going to help us get better answers. But ultimately, we're still responsible for deciding what those answers mean.