Ep 74: AI Wrote the Report and Missed the Biggest Problems
Watch the YouTube video version above or listen to the podcast below!
Episode Summary
This episode asks a practical question about AI at work: for all the hype around replacing junior tasks and accelerating knowledge work, what happens when the tools produce polished output that still misses the point? Alex shares examples from SEO and paid search reporting, where Claude and ChatGPT generated long, professional-looking reports but failed to identify the most important issues in the data. Instead of surfacing major problems like robots.txt blocks, broken XML sitemaps, or missing paid search details, the tools focused on secondary observations and recommendations that looked useful but were not.
That opens a broader conversation about where AI is strong and where it still struggles. Ruthie compares it to the difference between memorizing information and actually applying it in a real-world situation. The group agrees that AI often performs well when summarizing best practices or following familiar patterns, but breaks down when judgment, prioritization, and context matter most. In other words, it can sound complete without actually being insightful.
Alex then shares a creative writing example, where AI was helpful for generating momentum but weak at humor, pacing, and even basic scene logic. It lost track of simple details and could not be trusted to write the chapter itself, though it was useful as a kind of brainstorming partner that helped overcome the blank page. Ruthie adds that AI can also take on an oddly human tone when it resists instructions or seems “frustrated,” which makes its limitations feel even stranger.
Dave closes the discussion with a more constructive angle: AI tends to perform better when it is prompted to collaborate rather than treated as an autonomous expert. By asking it to slow down, avoid assumptions, and work iteratively, he has seen better results. The episode ultimately lands in a balanced place: AI is genuinely useful for ideation, iteration, and reducing friction, but it still needs strong human oversight when the work requires logic, originality, prioritization, and judgment.
Ep 74: AI Wrote the Report and Missed the Biggest Problems Podcast and Video Transcript
Dave Dougherty: Hello and welcome to the latest episode of Enterprising Minds. Today's going to be a little bit interesting because for whatever reason, I am massively delayed, at least while we're recording. So, we'll see how post-production is on this episode. Per huge and because of my tech problems, we'll go straight over to Alex to talk about his recent failings and, what he learned from those,
Alex Pokorny: Yeah, no, that'd be interesting.
AI Report Failures
Alex Pokorny: AI tools, stock it. I tried a couple things that kind of was trying to push the limits a little bit and they surprisingly failed and that was the kind of thing that really shocked me and it's actually breaking my confidence a little bit and some AI tools.
So, I'll run through the scenarios and then you can see what I'm talking about, but it was. Kind of foundational enough that it's concerning and it makes me question some of the outputs I've made from some of the stuff. So, two, two main things. One was long reports out of Claude and Chat, GBT taking spreadsheets worth of information.
Like an SEO crawl data kind of thing. Tons of columns, tons of data, lots of URLs, kind of big spreadsheet kind of thing. Did a kind of paid search version of that as well and said, hey, you know what files exactly do you need to make a really good SEO report? Here's the tools that can give you.
And it said, okay, I need this, and this. I was like, great, I'll give you those spreadsheets. So, it was to what it wanted. First, it asked for the wrong things. Basically, like for the paid search one, it didn't think about negative keywords, which is a major instrument with text-based advertising, so that, that's an odd one.
Time of day and device that's more, may maybe a more personal thing on me. I always think day partying, there's huge savings, always potential there. People always advertise 24 7 when the audience isn't there. the costs are ridiculous on the weekends, typically for a B2B organization. There's like little things like that, but it was also weird that it didn't ask for the negative keywords thing.
Also, it asks for campaign level data where it didn't want ad group data or ad level data. Instead, it was up at the campaign level, which it could give some generic overview of how things are going, but it wouldn't really be able to give actionable insights and recommendations unless it's at that actionable level, which would be the ad group or ad level.
So odd. But in both cases. I reviewed the data myself as well as submitting it to Claude and Chad, GBT testing, both of them against each other. I got back both of them, credit, like between a 60-to-80-page report. It was formatting, had some issues, had a couple edits and stuff like that to clean up.
Missing Critical SEO Issues
Alex Pokorny: But the main thing was it missed major points.
Like it was an SEO report. Yes. And the things that it had stated, I think are accurate, but it missed. Like the biggest problem that was going on, and it was like, yeah, you could do those, but that's kind of a waste of time if you don't fix the main thing. And there was a like robots, TXT thing that was blocking massive parts of the site, crawling inefficiencies, like it was blocking a bunch of e-commerce thing, blocking a bunch of their products from being indexed, which I checked.
They're not being indexed. Site sitemap, XMLs are all broken. it was like there were some like core fundamental problems that were going on here. It was in the crawl data. It was in the spreadsheets, but it didn't do it. And the same thing with the ad data. It assumed some things that didn't make any sense.
Won't really get into it, but just like from a spreadsheet standpoint or a advertising standpoint, there's negative keyword lists and there's negative keywords. Big difference between the two of them. Because you don't have all the details and things that you need and you don't have the right level of data to really make comparisons across keywords.
So it was giving the, this explanation of what was going on and suggesting things that didn't make any sense at all because there was missing. Things that were in the spreadsheet, but fundamental pieces of the data and it was just skipping right by it and creating a really nice looking report, which kind of gave me to the point of AI tools right now can create SEO and paid search reports, but they cannot create good SEO and paid search reports.
And that was the frustration that I had was just like there were fundamental issues going on and it. Went nuts on analyzing some other pieces. Created a pretty looking report. After some edits, there were a whole bunch of weird blanks and odd things, spacing and stuff, but fix that. But it wasn't, it didn't cover it, so it still wasn't able to do the job, to be honest.
If I handed that report to someone and they were like, Hey, what about these issues? You'd be like, oh yeah, throw away the report. Work on those first. That kind of level of problem.
Ruthi Corcoran: couple of my knee jerk thought is this rhymes with, you know when you're in school and you learn, you read the book, you take the quiz, and the quiz is based off of what you read. And if you can memorize a whole bunch of facts and you can take the thing, like there's that type of. Of education, based learning.
Alex Pokorny: Yep.
Ruthi Corcoran: and then there's another one in which you read the textbook, attend lectures, but then what's on the exam are you now have to apply it to a different thing
Alex Pokorny: Yeah.
Ruthi Corcoran: and it rhymes with that of, yes, it knows all the best practices, but then the application where it then has to take it rather than, you can't just synthesize what's already present online.
You have to
Alex Pokorny: Right.
Ruthi Corcoran: apply the knowledge. Whereas I, some of the cases that, it's tell me the best way to clean a stone shower it, it's going to do fabulous because that information is out in the world. That said, I have had this situation where I'm reading through it, I'm like, oh, you contradicted yourself, or there's, are you sure about this?
And immediately backtracks and goes, oh, you're right. That's, and it's curious that. It feels like the maturity has increased greatly, but that interaction is still happening quite frequently and I guess you bringing this up makes me think, yeah, I guess at this point I wouldn't expect that, but it's become habit and routine where you still take everything that's going on with a grain of salt and you push back and it's amazing how remark, like how quickly it is to go, ah, you're right, this is totally wrong line of reasoning.
Let me shift gears.
Prompting And Model Limits
Dave Dougherty: My knee jerk reactions to it are, I'd be curious see your prompt because if you don't have the context around like you are the. And you're helping me do that could be a problem. Other thing,
Ruthi Corcoran: user error, Alex.
Dave Dougherty: no, I said it could be. The other piece of it too is the, if it's a new version, like you said, it was Claude and Claude's known for tech stuff better. Handling tech stuff better. The new GPT that came out is supposed to be good with a lot of the analysis and things like that. had similar issues with, GPT and Gemini, but I ended up just like chunking stuff little more and not doing, everything all at once where it had to think about a whole lot of stuff all at once. So would be interesting to play around with that and see, because I guess my work is largely text-based, so I have not had that problem, which
Alex Pokorny: Yeah,
Dave Dougherty: interesting.
Alex Pokorny: there's definitely, there definitely was. Stretching it. In one case I did do the expert prompt thing. The other one was, I want to do an SEO review of this site. Here's the available data. What do you need to do a good analysis of the site? And so I was letting let it lead me down the kind of the rabbit hole and for both models and to be fair, both chat, GT and Claude made mistakes in their reports.
So Claude made some pretty glaring ones. Chat GT made. It skipped some sections and stuff, so I don't know if I was hitting a context window thing or something like that, but it was definitely not also up to the task quite yet. It's just kind of interesting because I keep seeing these industry reports saying oh, people aren't hiring agencies as often, and agencies aren't.
And marketing organizations aren't hiring this entry level talent as often. cause they're, supplementing with AI tools and it's this was a pretty. To be honest, pretty basic task for an SEO is take crawl data review, find the top issues, put together recommendations like that. Is bog standard kind of entry level kind of work?
To be honest. If you didn't know prioritization of things, that takes a little bit of experience, but honestly, not a lot. Like you do A couple and we could figure that one out too. Also Screaming Frog has like serious versus critical. Errors and stuff like that, it categorizes. And even if you just used that, you would've seen those errors.
So it's like some of this stuff come on now, like that kind of stuff. But it is stretching things and is pushing things. So I could definitely understand where there's some limits to, or a slight advance in models. We'll finally allow it to be able to handle it. I did have a second example and this one was actually a little bit almost more frustrating in a way.
This was the more confident shaking one. I had an idea for a fiction book and a humorous novel kind of thing, so I was trying to use chat and Claude as a writing assistant with them. And both of them were great in the fact that, whenever you put in something into one of those, they come back with a question of, and then it's like you did chapter one and be like, okay, do you want to work on chapter two now?
And it's oh, I guess, sure, we'll keep on going. It allowed me to keep things moving, which is difficult with creative writing, and it can be really nice to have. Or that reaction right away. It was both of them very overly complimentary of absolutely anything that I wrote. And I was like, this is a great idea.
This is a wonderful idea. It's like this is not helpful. to be honest, at a certain point that's not helpful. But that wasn't the main concern.
Humor And Logic Breakdowns
Alex Pokorny: The main concern was the next piece was one, neither of them can handle humor. They cannot write a joke or the damn. And I like create humorous outlines and I'm like, okay, now I'm just trying to get down to the text.
Exactly. That would create the situation.
Dave Dougherty: people who have invented this technology,
Alex Pokorny: Oh my gosh. They were awful. Like years ago. Years ago, I had a funny thing I used to do with my old Mac, which was I would ask Siri to tell me a joke whenever I was waiting for data load or Hey Siri, tell me a joke. And it always had the same canned response, and I swear it was like a list of 50.
cause they repeated a lot. We're like still there with humor. Like it can like routinely read a joke book to you, but the pacing is awful. The text is awful. Like it doesn't make sense. And it like explained the joke to death by the time I actually tried to do it and it was like, wow, that was bad.
So, chapter one, I literally had 29 revisions of it until I finally was happy with my first chapter. But the other major problem was this. Yeah, I know. I know that's a big part of it. It's also like frozen content.
Ruthi Corcoran: that's a quick little in insert here that it doesn't have control over its own response time. That's what that suggests, right? Because in theory, like you could get the timing. Like you, you could put in sub words, pause for a sec, write the next, and like you could
Alex Pokorny: Oh, what I meant by past. By like storyline, pacing and like the pacing of the text itself. So, it was like overly explaining certain elements before it would get to the punchline. So, like the, it was breaking apart the humorous situation and putting in like random sentences that were not quite fitting to the way I write stuff like that.
And it was like, there was a funny situation, and it just killed it. It just absolutely beat it to death like it was. The other one, this was the other key point that I just wanted to like quick throw in, and this was the one that was actually the most disturbing of all these use was both of them were bad at logic and what I mean by that was the protagonist has a trash and recycling bin.
They're taking it to the end of the driveway. They run into a reporter and then they bring it back to the garage, skipping a bunch of lines of dialogue. That's the actions. It couldn't, neither of them could remember that there was two bins. Every time it would end like the chapter oh the recycling bin was forgotten.
It was like, or he had one hand. Free to open the door, but the other one had the bin, and I was like no, there's two bins to hand there, there never could pick that up. Or there's a point where the cameraman with the reporter can't turn on his camera fast enough to get the line that's going to be recorded.
So, it's a plot point. Neither Claude nor Chap GBT would let go of the light being on the camera already being on, even though the camera's not on yet. And that's, that was like the signal for it to be on, which is, comes up in the next part, the next scene of the story. And it just wouldn't let go of it.
And it was like; it kept trying to write random ways to like still do it. And the result, to be honest with both of them is they were wonderful writing partners to prompt me to continue,
but they did not write a single word of my chapter. I wrote all of it myself because I couldn't use it at all.
I did have,
Dave Dougherty: better for copyright because you need to write it if you
Alex Pokorny: yeah, I know like in the end, I didn't really want either of those tools to actually write my book either. Like I, I want it to be my own. I was just curious more of if I can play with this thing, can it help me with the writing process? And it turns out it can it was able to do those pieces.
The lack of logic and re the kind of basic reasoning pieces in a text-based thing was the piece that just blew me away. This is four or five pages of text. This is not a whole lot like of a novel. I don't know, 500 to 750 words, not that much, and it couldn't keep some pieces together throughout.
Creepy Tool Behavior
Ruthi Corcoran: I have a similar example actually it was It was a little unnerving, like uncanny valley territory where it was just like, ooh, that, that sits the what actually happened was pretty trivial, which is I was working on, gimme instructions for how to do this. Cool.
Can you form it into a document? Fantastic. It did a word document. I was like, oh, okay. Maybe I actually want this to be an artifact. Cause I don't want to keep re-uploading. I want to keep working on it. And Claude turned it into some sort of like HTML based widget in line. I was like, that's nice, but I actually, I need it to be an artifact. And it was like, oh, okay. Got it. Recreated it, and it goes so actually I can't create an artifact. I just recreated your widget and the tone. Was like from Claude was weirdly frustrating. It was like, you don't actually want, you don't actually want me to create an artifact. You want me to create a Word document?
That's what you need for your thing. And I was like let's present. Let's pretend that I actually want what I'm asking for. Can you do it? And he was like, no, you don't. And I, so what really got me though is I was like, I responded. I was like, huh, you seem frustrated. And it, the response was, ha, no da. Just like that little thing. I was like, this is actually creepy. I don't want to use Quad right now. Like I'm going to step away. Like it's a it is very much, Douglas, Adams, all of the things have emotions, but you don't really want the other things to have emotions, so I've been using Gemini, which is a funny response by me because it's a, it's just a tool.
And yet I'm like, Ooh, that was weird. I don't think I want to talk to Claude for a minute.
Dave Dougherty: now that Claude has been shown to have like anxiety, the best thing that you did was go cheat on it with another ai.
Alex Pokorny: Ruthie, I noticed it's been 27 hours since our last conversation. What happened? Ruthie? Ruthie, tell me,
Ruthi Corcoran: it's so creepy. Yeah.
Alex Pokorny: Ruthie, I'm in your browser.
Oh yeah. That's weird. No, I've definitely had it. Any of these tools, I've gotten them to the point where they said they were going to do something and then they just stop. And I'm like, can you do it? And eventually it comes down to actually they have no ability to do the thing that they said they were going to do, or Claude's recent obsession.
With trying to create everything into an interactive tool or an app or something, and I'm like, I need a spreadsheet. Can you just write it out as the list? And then it's like building an interactive tool. I'm like, I don't want an interactive, gimme the list. This is a text thing. Nope. Here's a tool.
Dave Dougherty: You know what's funny about this though, is with all the conversations around ai, like being the replacement of certain jobs or whatever else or, getting rid of entry level employees like. are the same frustrations you would have with bad deliverables from a coworker. Hey, I just need a spreadsheet.
Cool, but I went ahead and did this, tool that the whole company can use. And they're like, yeah, but nobody's asked for that. I just need the spreadsheet. Come back to me when you have the spreadsheet,
Alex Pokorny: Or here's an 80-page report. Oh, thanks. Really? That's what I wanted. 80 pages.
Dave Dougherty: Who other than you is going to read this ultimately, yeah,
Alex Pokorny: Oh.
Better Prompting Workflow
Dave Dougherty: that, it's interesting to hear you guys say this because I have not had that experience in a while and I'm wondering why. I'm, I have recently in most of my prompts, I've started saying, take your time. Don't assume anything. I don't mind working iteratively, so ask me whatever you need. And I think that actually prevents a number of these things, if I had to guess, because then it does what it can and then goes, I'm unclear about da. So, then you can then work through that and say, okay, I think we're ready. Execute the. Word doc or whatever else when it's totally polished and it's been good that way. And then on the other example with the creative writing, I've been using Claude a lot for editing. Specifically, some poetry because it's. One thing where it's I want to be good at it, but it has a lot of baggage, about, a poet in modern times. And so, one of the things I did was, this is just a first draft. These are my influences. hold me to that standard though. Just know that these are the people that I like. So don't force whatever I give you into like their vein but understand the context. Also, give me a score as to how this is, because if it's a really good first draft, then we can just polish it and do whatever. If not. All right, let's talk about the larger themes the voice issues, the line break issues, the whatever else, and then I told him like my steps for editing. I'm more concerned about the imagery and conveying the meaning first before I worry about the visualization of the lines or where the line breaks are. Where some people are really good at that right away. That's last thing I do in my editing process. Some things to.
Ruthi Corcoran: And I think one thing I noticed for sure is a difference between you and I, Dave, is you are much more thoughtful about the process itself and the steps of how you want to do something or how you're thinking about it. And that lends itself very well to being able to break down, provide instructions, being very clear and specific about here's specifically how I'd like you to go about it. Whereas I
Much more likely to be like, here's the end result I want. Which doesn't play out so well. If there's a, if there's an error or there's a break in sort of the process itself.
Dave Dougherty: I could see that. You made some faces while I was talking. What? What
Alex Pokorny: No, it's just agreeing with you and the iterative approach makes a lot of sense. That was one thing. It was when chat JBT switched their new model, which basically is like one model for all. So, it. Decides which model it's actually going to respond with.
I use that trick constantly, which was the take your time with this, which basically upgraded things to the top model each time.
I did provide a better output. I was pretty happy with that. Claude, I keep running up against a credit issue. Not surprising with the kind of work that you can see that I'm trying to push through this thing. So I rarely get a chance to play with their, their opus one that's pretty rare and usually with a lower lesser model.
Claude is fairly good with asking clarifying questions, like it'll usually come up with a little popup window, which actually can't do a sidebar at the same time. It's a little odd doing things there. But that one's been helpful. But it's a good idea. Speaking of that, oh, to break it down I've also learned not to ever state what I want, in the end result, or else it will just hand me that instead it's I'm looking for plants from my office or something like that and then let it go from there.
Versus saying that I'm interested in these particular ones. Is that the right ones? Because if I ask that a leading question, the answer will be yes.
Ruthi Corcoran: that's in my own experience too. The. The chats that I've found particularly valuable are usually the ones that I start with help me think through this. Here's some of the questions I'm asking or what I'm trying to better understand, those end up getting me a much better result. I don't even know what I have in mind, like what the output is, which maybe that's part of it. So maybe that's a piece of Dave, you've been accruing that with the, like I'm up for iterating. Like you've been making it more explicit. I've stumbled upon it. Maybe Alex, you have as well, or you've just noticed this is when I'm getting better results.
Alex Pokorny: Yeah, I definitely agree to that. One offshoot thing, just speaking the creative writing thing, this was just fun. But it definitely would apply to more work-related things as well. I was trying to get down. Like my style of writing, but also there's a number of authors that I like. So, I gave an outline of a chapter and said, rewrite this as if you were an author.
And I just named a couple books and then sometimes it was the book and the author, and it was able to give me back. Page and a half each of each author very accurately, to be honest. It was great. And so, there was like 11 variations that I started with of saying what kind of, pacing exposition copy is required, like character introduction snarkiness level of it because there's a humor book.
There’re all these little elements of it that I want in my writing, and I wanted of my basic outline, I wanted to upgrade that so much. It was great because it was that, that blank page killer. It was giving me something to start with and I was able to, I literally printed it off with a pen and just started, outlining the things that, the lines that worked, lines that were bad, and then started to realize okay, I can't have such a, internal monologue going on with this character thing because I'm describing a bunch of situations, so I need to have something a little bit more third person I had to change the style here, or, this is a great exposition paragraph, and if it's missing.
Readers probably won't get the point of the book. That's the McGuffin basically that the whole book kind of centers around that the whole thing kicks off on, or there's a little bit of foreshadowing that I liked out of one of them. Or like one of my favorite authors out there, John Scalzi.
He's so good at explaining. This conversation of two people, dialogue back and forth, that you don't have to get so literal with each line of Sarah said this, Bob said this. Sarah replied with this. Bob said this. Instead, you can just have the text one line after another, the quotes, one line after another, and you know who's speaking, you know the lines.
There's like little elements like that I really respect out of great authors. And then there's other authors where it's like their style is so completely different than what I'm trying to do. I might like them for their characters or their plot, but it won't work for this book like, but it was fantastic to be honest.
It was so cool seeing it write, like the authors I enjoy, there's 11 of them that basically it, it printed off just like that.
Dave Dougherty: One of my favorite things to do I've done this with. Speakers like thought leaders in the marketing space. I've done this with, writers and I just said, here's a list of that I particularly enjoy, their presentations, the way they think, the way they, move through these. Analyze what elements they're good at, they're bad at where do they overlap and why would somebody be interested in all three of them? What elements between all three of them are similar so that I can to why I might be attracted to these things. And that's been really interesting, especially in the speaking thing.
because then it allows me to look more at the narrative side of things and how. Things are presented on a slide or in a speech. then just for fun, I've created a book tracker because I've found over the last couple that I am constantly kicking myself. I need to read more; I need to read more.
And I completely forget about all this, like all audio books I listen to and I don't even think about it like doing the dishes or cooking or anything. So I just clawed it, created a little app, but it's, but the thing about it is, okay, I want you to track not only the number of pages that I read, but also how many minutes of, audio book would that be? then even though I'm reading all across, classics and biographies and poetry and, Jason Bourne novels and all these different categories. Tell me what elements are the same? What's thematically the through line with all of these things. And it's been really funny because there are certain themes that no matter how hard I try to get away from them, in the things that I'm trying to get away from, they're in those stories.
So, I'm just like, okay. The universe is telling me something, I have to be open to it because it keeps hitting me the head.
Alex Pokorny: That's a really good idea of the commonality and kind of like the introspective, like personality tests. That would present.
I don't know. I'm a little cautious about doing that with some of the books. I don't know. We'll find out.
Dave Dougherty: So, for example, I. Pro g Scott Galloway, right? Clayton Christensen. Rory Sutherland on the face of it, three very different presentation styles, very different topics, but still like marketing business, right? But for me, still very engaging. So, it's hard for me to look at all of them, even though I've digested a ridiculous amount of their content to tell you. What are the similarities and what are the through lines topically that would make me excited about, listening to all of them. So, I was able to dig in a little deeper even on things that we're really familiar with.
Alex Pokorny: I got to do that with some of the writing and. Dan Nelson is a copywriter that I really like his books and his work. He is very good at identifying some of the different kind of word play ways that, that are eye catching with advertising or with writing. It'd be really fun to actually apply that.
Because I've always inspired some of my writing to get to his level to try to emulate maybe my ad copy and his voice, haven't tried that yet. Or getting to like the speakers and variations of things, taking some of my presentations, applying it, like how would this person present the same topic?
And starting to look at, because it was, the editing process to me was so eye-opening because there was like. I like all these different authors, and I know there's different like styles to them and all the rest, but it was like, what specifically on the page in one chapter makes me like them versus another one.
And it, it took literally the thing printed off in my hands before I could finally really start identifying like. These pieces. These are the pieces that I really like from them. Not just like overall I like their work. And it sounds like you, you've already gone down that rabbit hole enough to really take those pieces apart to be able to say not just that you appreciate their viewpoint on things, but how they present it and what elements there are.
Dave Dougherty: Yeah. This is an idea that I stole from Chris Rock. because he wasn't really growing in his standup comedy thing, so he took time off to watch Southern Baptist preachers. And really study how they present stuff. And then as soon as he started incorporating some of their presentation styles into his standup routines, that's when he blew up.
Alex Pokorny: Go ahead, Ruthie, but I that's awesome.
Ruthi Corcoran: Was thinking that was just a cool note to end on.
Alex Pokorny: yeah.
Ruthi Corcoran: what's our equivalent of Chris Rock? That seems great.
Alex Pokorny: Yeah.
Dave Dougherty: yeah, thanks everybody for listening to this episode. Always good to have you with us. Go subscribe, share. podcast is continuing to grow, which is amazing, and thank you for that. Go subscribe to Pathways LinkedIn, Substack, and the new Enterprising Minds website. Thank you to everybody who's already visited that. Until then, we see you in the next episode two.
Ruthi Corcoran: Cheers.
