I gave a generative AI tool, a content farm, and a writer the same prompt. Here’s what happened.

Great, another article about generative AI. 

I’m sorry or you’re welcome. 

Even before ChatGPT made big waves, SaaS-ified versions of AI writing tools were coming to market with alarming alacrity. 

One phrase consistently stood out: “It’s an assistant, not a replacement.” 

Thought leaders and wonky wonks sang this chorus but the SaaS tools didn’t seem to hear it. Their homepages make promises like:

• “Generate high-quality blog posts in minutes” 
• “Write 10X faster”
• “Go from 0-1,000 words in four minutes flat”

It seemed like generative AI’s biggest proponents and its loudest vendors were at odds around how, exactly, we should be using this thing. 

Yes, I get that there are some fantastic marketing use cases for ChatGPT and other AI tools. (Our Head of Ops uses it to speed up her spreadsheet work.) But that’s not what this is about, for me. 

It’s about the overpromise of productized content services and AI content tools.

Use generative AI to create captivating blog posts and sales collateral in minutes!

I wasn’t convinced. 

You get what you pay for—we’ve all heard the phrase. If it’s applicable anywhere, it’s applicable in the marketing world. I wondered, “Can I really get quality output while spending next to nothing?"

But I’m biased; unfortunately or fortunately, I make my living from the written word. 

So I put it to the test. 

‍

The experiment 

What better way to compare quality than to provide a writer (like a real human writer) and a generative AI tool with the same input to see what they each come up with? 

For good measure, I brought in a productized content marketing service—colloquially known as a “content farm”—so that we’d have three points of comparison. 

In short, these were the three authors I leaned on for this experiment:

• A paid generative AI SaaS tool (I’m not naming names, but it wasn’t ChatGPT.)
• A productized content service (Again, I’m not naming names.)
• A human writer on the Beam writer’s bench

And this was the prompt: 

‍

‍

That’s it. I kept the assignment purposefully broad for two reasons:

First, the Beam Team is convinced that good content starts with experts, not writers. Since this was a topic about writing, I knew that our human writer was an expert in what they’d be writing about.

Second, I’m looking at ROI—not just on my money, but on my time. Generative AI proponents like to point out that the output is only as good as the input. I’d like to point out that you should be able to send a good writer in the right direction and get your money’s worth. 

Here’s what I paid for each of the three 1,000 word drafts: 

• The paid generative AI SaaS tool draft: Free as part of a trial
• The productized content service: $150
• The human writer on theBeam bench: $400

I left each draft completely unedited. The only finagling I did was with the AI draft; I had to pull multiple elements of the AI version together because the tool I used needed some additional prompts and human input. 

‍

The drafts

We have an internal editing checklist at Beam that I considered as I read each draft. We ask questions like:

• Is it specific?
• Is it conversational?
• Is it organized? 

I was looking for specificity, a conversational tone throughout, and organization that flowed from one point to another. I invite you to do the same.

Take a look at the introductions, body copy, and conclusions of each draft. I won’t reveal the authors until the end. Here we go. 

The introductions

Sample 1:

As content marketing continues to remain crucial to brands' growth, companies must find new ways to create stellar marketing content. This is especially true for the SaaS industry more than any other. 

Particular content marketing strategies work specifically for SaaS brands. The customers in this industry are knowledgeable and specific about what they're looking for. Therefore, you must have the answers to the questions even before they're asked. Additionally, you have to know how to present these answers strategically and in an engaging manner.

Sample 2:

Content marketing is all about engaging with customers and prospects in a way that builds trust and drives sales. The best way to do this is by creating content that's relevant and valuable to your audience, but you can't do it alone. You need subject matter experts (SMEs) who can share their knowledge on topics related to your business. In this guide, we'll explore how SMEs can help with content marketing and give tips for finding the right people and getting them involved in your project.

Sample 3: 

Imagine honing a craft or skill for 10,000 hours—the equivalent of five years of full-time work. According to the popular book Outliers by Malcolm Gladwell, that’s approximately how long it takes to become an expert.

Expertise is hard-won: that’s why it’s critical to tap into your team’s internal experts when it comes to content marketing. 

When you capture and convey the insights, wisdom, and passion of experts in their own words, you build trust in your company—and, as an extension, create trust in your products or services. 

All pretty different, no? 

Let’s take a look at the next section.

The first bit of body copy

Sample 1: 

The first step in content creation is curating ideas. How you select and organize the results of brainstorming sessions with your subject matter experts will determine the success or failure of your project.

One of the approaches to this is to create a knowledge bank. This is a repository of all the insights provided by the experts that can be used to create content. Even if the insights are not polished, you can use them as a springboard for future content ideas. A simple, spontaneous thought can be turned into an influential article with a little research.

Sample 2:

The first step to finding an internal expert for your content marketing is to find someone who can communicate the subject matter. The person needs to be knowledgeable about their field and also have the ability to express themselves in a way that will resonate with your target audience.

Good candidates include:

• Someone who has a passion for the topic, which will make it easier for them to produce articles on their own time without compensation;

• An expert in his or her field—someone who has been practicing their profession long enough that they have developed unique insights into how things work; and

• Someone with strong writing skills, preferably with experience writing for an audience outside of his or her company.

Sample 3:

First, the obvious: in all likelihood, your experts are not on the content marketing team, and they are probably not writers. In fact, they may range from overworked C-level execs to camera-shy software developers. 

The good news is none of that matters. You simply need a strong writer to get in the same room (or Zoom) with your experts.

Start by identifying what topics your company wants to cover. The goal of content marketing is to attract, engage, and retain your audience. So for example a B2B software company might want to:

• Write about the pain points of their target audience 

• Tell the story of how X type of software guides their audience to solve problems

• Show expert-led deep dive technical applications and real-life use cases

• Share thought leadership on where the industry is headed 

Are you starting to see which is which?

Let’s look at one more section:

Toward the end

Sample 1:

Prepare for interviews by coming up with questions that are focused on the audience's needs and concerns, rather than on what you want them to say. If you're writing a blog post about how artificial intelligence has changed the world of marketing strategy, then ask questions about how AI can help marketers be more creative and efficient without feeling like they need an advanced degree in computer science — or worse yet, if there are any hidden downsides that could make things much harder down the road.

Most importantly, create a consistent process for conducting these interviews so that everyone involved knows what's expected of them.

Sample 2:

Structuring the content creation process as an interview minimizes the work for the interviewee. Rather than dumping several questions in their lap or giving them a topic and expecting an article, you make the job easier by helping guide them through the process.

This way, the bulk of the work lies with the marketing team. The expert answers the questions to the best of their ability, and the marketing team gets to work editing the responses into a cohesive article. The questions, in this case, should tap into the subject matter's expertise and be within the scope of their knowledge. It’s advisable to send the questions to the interviewee before the interview so they can be fully prepared.

For maximum effect, research hot topics and use them for mapping your questions. Your target audience should always be at the back of your mind, so you can create meaningful and relevant content that will gain online traction.

‍Sample 3:

All new initiatives require buy-in, and expert-led content requires securing the cooperation and time of team members across your organization.

To convince relevant stakeholders that expert-led content itself is worthwhile, you can point to examples of how expert-led content has been successful elsewhere, share your detailed plan of topics and the team members you would engage, and calculate the time and budgetary requirements involved.

Once you have top-down support, it’s time to secure buy-in from your experts themselves. One option is to make the process collaborative: consider building a shared resource such as a Miro board where you can lay out your initial topics and crowdsource granular ideas and angles from experts company-wide.

You can also dive right into one-on-one conversations. In that case, you want to make your process and expectations crystal clear to respect your experts’ time and set them at ease. Let them know succinctly:

• Why you value their expertise

• What you hope to cover in the conversation 

• When, where, and how you will conduct the interview

I’ll spoil it for you now:

Sample 1 is the productized service. 

Sample 2 is AI. 

Sample 3 is the Beam writer.

What surprised me most is that I liked the AI draft better than the draft I spent $0.15/word on. It’s (slightly) more cohesive. It has (somewhat) better ideas.

That said, both the content farm and the AI drafts suffer from the same drawbacks: they’re not all that conversational, and they’re not at all specific. 

Again, I’m biased, but our writer came back with the opposite: a conversational tone, candid recommendations of books to read and software to try, and a ground-level view of getting started with SMEs. I’m not sure what all seven deadly sins of content marketing are, but one of them is certainly “making generalized or obvious statements.”

Comparing these three drafts, I was struck by how “good” content is so clearly more than the sum of its parts: the hook, the intro, flow, word choice, examples, actionability, and so on. After all, writing isn’t just stringing a bunch of words together. 

So, again, I was looking for:

• Specificity
• Organization
• A conversational tone

Which of the three drafts do you think hit all three?

‍

‍

The takeaways

Bear with me for a couple more minutes while I share the three things I’m walking away with from this experiment.

One: AI will replace some writers, but not the best ones.

When I first posted the three introductions on LinkedIn, everyone knew immediately which one was from our writer. But folks had a harder time distinguishing between the content farm draft and the AI draft. So did I.

Two: Keep your time and effort in mind.

I had the AI draft back in seconds, but it took me a good hour or two to piece everything together. The content farm draft was, in my opinion, unusable. I would have to start over from scratch. In contrast, our writer’s draft would require maybe 30 minutes of my time to get it publish-ready.

Three: You’re not just paying for the words.

I said it before, and I’ll say it again: you get what you pay for. You can’t replace the critical thinking and first-hand knowledge of a good writer.

When you hire one, you’re paying for expectations—you expect that what you’ll get back will be nearly ready to publish. You’re paying for a partner—you can bounce ideas back and forth and get feedback on your direction. You’re paying for an expert—someone who knows their stuff and levels up your marketing efforts.

So. That’s what happened, and that’s what I learned when I put AI and cheap content to the test.

I’m still not convinced.

But I’d love to hear your take. Send me a DM, and we’ll keep the conversation going.