SPEAKER_00 0:00

It's amazing that we live in a time of AI. And truly, I hope everyone is using AI in some capacity in their business. But this podcast episode is going to be a cautionary tale on how you should be incredibly careful using AI in your business because it can have massive effects you wouldn't dream of. My name's Dylan. I'm one of the coaches here at the ECOmerce Alley and co-host of the podcast. This is an audio exclusive episode, just as our thank you for being a listener. It's just me on this episode, no, no Josh, but he will be back on Monday, so make sure you tune in to that episode. Now, before we get into this episode, I do have a favor to ask. I would love it if you would drop a rating on whatever audio platform you're listening to this on. We are trying to become the number one e-commerce podcast in the world, and a rating goes a long way in helping us meet this goal. All right, be careful using AI. Dare I say don't use AI for one thing in particular in your business, optimizing your meta ads. Now, let me let me clarify really quick before we get into this. I'm going to be using the word optimize, optimize, optimize, but optimize and analyze to me are two different things. Optimizing is making decisions, what to kill, what to scale, and campaign frameworks. Whereas analyzing is just looking at what's wrong, thinking about the data, reasoning, iterating, you know, giving AI creative and saying, what's this creative talking about? And how can I make it better? Or how can I make more creative like it? Or that to me is analyzing. Optimizing is, okay, what should I kill? What should I scale? How should I scale? How should I structure my campaign? All of that kind of stuff. So why should you not use AI to optimize your ad campaign? Well, I'm going to tell you a story before we kind of boil this all down. I'm going to tell you a story of a client in our Max Profits mentorship program who used AI to optimize their campaign. Their revenue dropped from $268,000 last month to $196,000 this month by following AI's recommendations. That's $72,000 lost in a month. And ad spend percentage went from 30% to 40%. So that's $72,000 lost in revenue and then 10% less profit because it was being spent on inefficient ads. So this is real, real money and real profit going down the drain because of AI's recommendations. So here's a story. I jumped on a Zoom call and I was looking at the report that it spit out, and it was really impressive. I was really, really impressed with everything that it came up with. It gave like overall CPA performance, total impressions, total purchases, average CPA. It showed what was working back in February that isn't working now. It analyzed her frequency. It was pulling video completion stats. It had breakdowns for male and female audience at different age ranges. Then it ended up like spitting out the top performing ads based like based on CPA trends and the worst performing ads with the highest CPA, the best performing ads with the best CPA. Like it was impressive. I was impressed. Like I was genuinely impressed that it spit out a very detailed report that honestly probably would have taken two hours to put together if you were doing it yourself. It was pulling data that I I probably wouldn't even look at. Like if I was optimizing a campaign, I wouldn't probably dig into the breakdowns for male and female at different age ranges. It's just too granular. But when you have a I do it, it's like, oh, great, I can give it this data and it can actually digest all this data and figure out what's actually worth looking at. So I was walking through the report with her and I was giving her my thoughts and explaining why I felt it was correct and how she could improve, what ads she should create or launch, and what ad sets she should consider killing. It really turned into a great conversation about this report. But then I started I started digging deeper and I started looking past the flashy report, the charts and the graphs. And Matt Harward, I'm not sure how long you guys have been listening to the podcast. If you have not listened to any podcast episode with Matt Harvard, I would highly recommend going and finding a Matt Harvard episode. He's he's very intelligent. But he he talked about when AI first came out that you run AI, you say, write me an email, and it spits something out. And you're like, oh my gosh, that's crazy. I can't believe AI just did that. And then you like actually read the email and you're like, oh wait, this isn't actually that good. It's not that good of an email. I would never write this, I would never do this, I would never say this. It doesn't even, it's not even formatted like an email, but you were impressed initially. And that's how I felt when I was looking at this report. I was like, dang, this looks really good. But I was digging deeper, looked past everything, started actually looking at the numbers. And this is a client that we've worked with for over a year. And we know her account pretty well. So the numbers that I was I was seeing in the report, I was like, wait, this doesn't, this doesn't look right. Something doesn't feel right here. Turns out it was reporting that some of her best ads, ones that have been working for quite a while, it was reporting them as underperformers. And then I started looking deeper. And I I wanted to, and I think what what led me to this was I was looking at some of the video stuff. I was looking at the hook rates and I was looking at the hold rates and things like that. And I went over to Ads Manager to look at those things in Ads Manager. And then I started looking at the CPAs between the two because I started looking at like I pulled up the hook rate and the hold rate in the ads manager, and it looked wrong. And then I started looking at the CPAs, and I was like, wait, these are completely wrong. Claude was reporting here here's some of the some examples. Claude was reporting a $34 CPA where Meta was reporting in 29. Claude was reporting a 32 when Meta was reporting in 28. Claude was reporting a 20 when Meta was reporting in 24. Claude was reporting a 12 when Meta was reporting in 24, and Claude was reporting a 28 when Meta was reporting in 22. So 34 to 29, 32 to 28, 20 to 24, 12 to 24, 28 to 22. So sometimes it was under reporting something that was good and it was overreporting something that was, it was all over the place. So I started comparing the numbers, and turns out Claude was misreporting about 25%, plus or minus of what Meta was saying. And in some cases, up to 50%, the the Claude 12 versus Meta 24 was 50% off. It was completely wrong. If Claude told me that I had an ad that had a $12 CPA, I'd be like, scale that puppy, right? That's fantastic. But it turns out it was actually a 24. It's kind of funny. Meta's CPAs were 29, 28, 24, 24, 22. It was all kind of in the same range. Whereas Claude had anywhere from 34 to 12. What the heck happened? So I spent 20 minutes with a client breaking down this report and what she could do better, what she should change, what she should kill, just to find out it was all wrong. And thank goodness that we found this out. Now, you're probably now thinking, okay, you you started by saying she lost $72,000 in a month, but how did that happen if you caught it? Well, I took that call and kind of shifted the shifted the focus of the call away from the ads and into the into Claude and how AI works. And I explained context and tokens and usage and some technical things to try and help with for the next time. So she went back and re-ran the analysis and it told her what ads to kill inside of the campaign, and she did it. So beginning of May, she killed her top spending ad that was getting around $800 a day. Now, for those of you that are in our program, you probably hear that and you're like, well, yeah, that you you guys don't say to do that. So that makes sense. If you're not in our program, our frameworks are we don't recommend killing anything in a campaign if the performance is good at the campaign level. And if things are below target, we recommend killing the ad set level, not the ad level. So she killed ads inside of an ad set while the campaign was doing good, because AI said. And I mean it makes sense. You gave AI all this information. Why would you think that AI would not tell you the right thing? Well, I'll get into that in a second, so stay tuned. But this campaign, this one, this one tweak, this one change tanked campaign performance really bad. It went from a 2.67 ROAS, which was a $23 CPA, to a $1.8 ROAS, $34 CPA. Now, fortunately, she's working with us, so we were able to catch this and help her. She turned it back on, decreased ad spend, getting things back in check. Ad spend percentage is starting to look closer to that 30, I think, I think she's like 29% right now, so it's actually even better than it was. Her last seven-day row as is a 2.4 and CPA is 27%. So it's not quite the 2.6 and the $23 CPA, but it is getting a lot better. But here's the thing: here's here's the big takeaway. I've got one, two, three, four bullet points. I'm going to talk about four things that you should take away from this episode. If if you're using AI, here are the four things. One, don't make decisions solely with AI. Don't plug things into AI and just assume that it's right. AI hallucinates. Well, that's my next point, but it does. AI hallucinates. It'll make things up. And also, you can't just plug data into AI and it become a good media buyer. AI can only make a good media buyer better. You have to look at the things that AI spits out. I can't I keep saying Claude because that's what we use, but ChatGPT, Claude, Gemini, they're all the same. You have to look at the things that it spits out skeptical. I I always hate, you know, you do a Google search now, and the top thing is Gemini's response, right? And I, you know, I haven't been in school in a while, but you remember when teachers would say, oh, Wikipedia is the worst source for your paper. I hope teachers are saying that for the little AI thing that shows up at the top of Google because the number of times that I've searched something and it says, yeah, this is the thing, and then I like click one link and find out that's completely wrong. It's true. AI makes things up. Which brings me to my second point. AI hallucinates. So in this example, the AI was off by 25 to 50% on key metrics. Not to mention that these are the metrics that you just asked the AI to give you recommendations on what to do based on. What should I kill based on my CPA? And it had the complete wrong CPA. So not only now are you expecting AI to just be a perfect media buyer right out of the gates, but you're also giving it, or at least it's giving itself wrong information, which then just makes for a really bad situation across the board. So it was tell saying that good CPA was actually bad and bad CPA was actually good. It was flipping it, it was making a bad CPA look worse than it actually was. So it's like, oh, you should totally kill this. But in all honesty, the CPA was only like $2 higher than your average. It's not that bad, right? Here are the two big things. So those those two were just kind of general AI. Here are two big things. Number one, AI doesn't know the full story. Your ads can tank for many different reasons. You ran out of stock, you changed something on your website, your budget shifted to a new ad set. Meta just had a bad day. By the way, if you want to know if Meta has a bad day, go to e-commercealley.com/slash bad. Uh that's the bad day detector. You'll be able to see if Meta had a bad day. AI doesn't know any of this. It's looking at the raw click data, the raw impression data, the raw CPA data, the wrong, the raw add-to-cart data, right? It's just looking at the data. It doesn't know, oh, actually, you ran out of stock on Thursday, which is why Thursday and Friday were terrible. But it does see that your CPA doubled when you ran out of stock. So it's like you should kill this. You know, all of a sudden your CPA shot up, but it didn't know that you ran out of stock. And if you just trust the AI and you say, well, the AI knows, well, then you're gonna go kill something that has historically done really well for you. So that's that's the like one of the biggest things. AI doesn't know the full story. And then the next thing is AI isn't a media buyer by trade. I I guess to be fair, AI isn't really anything by trade. AI is what you mold it into. But if you give AI a bunch of data and say, optimize this campaign, it's going, it's essentially the same thing as going to AI and saying, write me an email to send to my clients about my upc to my customers about my upcoming sale. It doesn't know who your voice is, it doesn't know who your customer is, it doesn't know the benefits of your product, it doesn't know how you typically structure promos or what the promo is. The email output will be garbage, similar to what I talked about with with Matt Harward, where you know you're blown away by AI as soon as you try it, but then you look a little bit just slightly deeper. You look at it for just like two seconds longer, you're like, wait, this isn't actually that good. Just like AI doesn't know your voice or your product or how you structure promos, when you give it data, it doesn't know what your creatives actually look like. It doesn't know what your ad strategy is, it doesn't know your target or breakeven CPA. It doesn't even know how meta works. Meta changes every like six months. And AI is trained every like two years. So it only knows as of like beginning of 2025 what was actually happening with Meta. It also doesn't understand like the modern best principles that you would get from this podcast. Just like when you have Meta write you an email and you give it a ton of information about your ICP, your brand, your voice, it has to have context. You have to be able to train the AI to be a skilled media buyer through prompting. Now, you have to tell it then what to look for, what the signals are, removing the noise from the data, how the algorithm works, and then even let it know that, hey, I ran out of stock this day. I changed something on my website, I did this, I did this, I did this. Like it needs to know all of that context on top of all of the knowledge that you have. Because from the get-go, when you first log into ChatGPT or you first log into Claude or Gemini or whatever you use, it's it is not smarter than you. You may think that it is, but it isn't. You have to, that's why prompting is so important. You have to give it the baseline, foundation, understanding of the at least the things you know. Because you can't go to AI and you give it something and give it data and expect it to turn into a good media buyer. It's just not going to do it. Because you understand, hopefully, because you've been listening to this podcast, or maybe you're a client, you understand the good and the bad and how to run ads and structure and how we optimize and all of that. You have to give it that context. So then when it gives you recommendations, it's giving you recommendations based on the things that you have already told it is true. Hopefully that makes sense. Now, unfortunately, this is not an episode on how to do this. And honestly, we haven't really figured it out yet. I've been trying, I've been working on it. I've been trying to figure out a way, a pipeline of pulling the data, extracting the data, analyzing the data, taking the videos, taking the images, extracting what's happening in the videos and the images, taking the data compared to the idi vin videos and images and saying, oh, this is why it's working, this is why it isn't, here's what you should do next, like creating a whole pipeline for it. And it's like, I wouldn't even say mediocre, it's like, I don't know what whatever's lower than mediocre. It's not great. I'm working on it, but it's not great because there are so many things that go into media buying that the AI just doesn't know yet. And it takes a lot to train it, to do it well. So, all of this to say, you should be using AI. I don't want to talk you out of using AI in your business. It really will allow you to double or triple your output as a business owner, but you should be incredibly careful giving AI the reins to your ad account and doing exactly what it says. And really, you should be careful giving AI reins to anything. But this was particularly the ad account that had real revenue implications. Now, if you're one of our clients listening to this, this is not the only horror story that I've heard about people using AI to optimize their accounts. This just happens to be the worst from a big number standpoint. We hear in coaching probably one to two times a week that people did something because AI told them to. Fortunately, we are here to help those people figure out well, A, how can I use AI better? And B, well, what are the things you should have done and kind of educate and help, you know, fix what AI probably broke? But it it's it's real. The danger's real, and I hope this episode open your eyes to it. Next time you use AI, next time you give it data, next time you're trusting the AI with the recommendations it's giving you, just double check it, make sure that it's actually accurate. You do need to spend a little bit more time really making sure that the AI is telling you the right thing. And if you don't think it's right, probably don't do it. Like, don't don't just blind trust the AI. It can be, it can be dangerous. Again, if you enjoyed this episode, please leave us a rating on whatever podcast listening platform you're on. We're trying to become the number one e commerce podcast in the world, and a rating would help tremendously with that. Thanks for listening, and we'll see you in the next episode.