AI Crawlers, Meet the Firewall: Why I’m Backing Cloudflare’s Bold Move
Something pretty major just happened in the internet world, and I’ve got thoughts.
On July 1st, Cloudflare announced they’re now blocking all AI crawlers by default—unless a site explicitly says, “Come on in.” As someone who uses Cloudflare across both my personal projects and our work at Augusto, I’m giving this move a solid, two-thumbs-up, slow clap.
What’s the Big Deal?
Let’s break it down: right now, AI bots roam the internet like overly curious raccoons—scraping up content, training models, and sometimes spitting back that same info without ever pointing back to the source.
- No credit.
- No link.
- No traffic
- No compensation.
Meanwhile, the people and teams who actually wrote the blog posts, created the tools, or built the guides? Left wondering why their traffic dropped while ChatGPT suddenly sounds suspiciously like their site.
That’s the problem Cloudflare just decided to confront head-on.
What Did Cloudflare Do?
Cloudflare, which powers roughly 1 in 5 websites on the internet (yes, really), is now blocking AI bots like GPTBot, ClaudeBot, and others by default on their Free, Pro, and Business plans.
Want to train your next LLM? Great—go ask for permission first. Or maybe even talk about compensation. (This will be interesting)
This flips the script. Instead of publishers needing to jump through hoops to opt out, AI companies now have to work to opt in.
Why I’m Into This
I’ve been a Cloudflare fan for years. I use it for everything from DNS to WAFs to performance tuning. But this move feels bigger than a feature toggle—it’s a stand.
As someone who builds digital products and works closely with creators, devs, and businesses trying to grow, this is huge. It sends a clear message: your content has value.
I’m all for AI. (You’ve probably heard me talk way too much about automation, agents, and LLM workflows.) But I’m also for fairness. If AI is built on the backs of real people’s writing, research, or creativity—let’s at least give those people a seat at the table.
Or hey, maybe even a royalty check.
Here is an image of Cloudflare, an AI crawler on this site
What This Means for All of Us
This changes how we think about:
- Content ownership – Who gets to decide how your work is used?
- Attribution and traffic – Should AI models send users back to the source?
- Value exchange – If AI tools profit off your work, should you be cut in?
We’re entering a phase where content licensing for AI might be a real, scalable business model. And that’s exciting—especially for creators, educators, and product companies who’ve been burned by the “take now, ask never” approach.
Part of me realizes that AI is now probably now the new search engine and SEO is changing. The question is where do you want your content or your information.
As a small business, I would like AI to analyze my content. If I were a larger syndicate for news or content, I would want to discuss its value.
Final Thought
I think Cloudflare just lit a match. This is a pivotal moment in how we balance innovation and respect on the internet.
The AI tools we’re building need great content to learn from. But we also need to protect the people who make that content.
So yeah—I am interested to see how this progresses.