Your Website Might Be Invisible to AI — Here's Why That Matters
More and more people don't search anymore. They ask.
Instead of typing "yoga classes near me" into Google and scrolling through a page of blue links, they open ChatGPT, Claude or Perplexity and ask a question in plain English: "Who runs good yoga retreats in Surrey?" or "Can you recommend a reliable electrician in Caterham?" The AI thinks for a moment and hands back a short, confident answer — usually naming two or three specific businesses.
Here's the uncomfortable question every business owner should be asking: when an AI recommends businesses like yours, is it recommending you?
For a surprising number of small businesses, the answer is no — and not because they're not good enough. It's because their website has, often without anyone realising, told the AI to go away.
The one line of code that makes you disappear
Most websites have a small, invisible file called robots.txt. It sits quietly in the background and tells automated visitors — "crawlers" — which parts of the site they're allowed to read. It's ordinary, sensible housekeeping, and most owners never think about it.
The problem is that the AI assistants people now use to find businesses have their own crawlers. OpenAI's is called GPTBot. Anthropic's Claude uses ClaudeBot. Amazon's is Amazonbot. These are the programs that go out and read your website so the AI can learn what you do, where you are, and whether you're worth recommending.
And a great many websites — particularly those built on older platforms or with a "block the bots" template switched on — are quietly blocking exactly these crawlers. The robots.txt file says, in effect: ChatGPT, you are not allowed to read this site. Claude, you're not welcome either.
The effect is total. It doesn't matter how good your service is, how many happy customers you have, or how beautifully your site is written. If the crawler can't read the page, the AI has nothing to recommend. You are simply not in the conversation — and you'll never know, because there's no error message, no warning, no dip in a report. You just quietly don't come up.
I see this constantly. In a recent assessment I ran for a yoga teacher with more than twenty years' experience, glowing testimonials and a genuinely excellent local reputation, the single biggest issue wasn't her content or her design. It was that her site was blocking every major AI assistant outright. Someone asking Claude or ChatGPT for a good yoga teacher in her area literally could not be shown her business — however perfect an answer she'd have been.
Why this matters more every month
It's tempting to file this under "something to worry about later." I'd push back on that.
The shift from searching to asking is happening now, and it's happening fastest among exactly the people who used to skim Google results — busy people who want one good answer rather than ten options. When someone asks an AI for a recommendation, they typically get a shortlist of two or three names. There is no page two. Being left off that shortlist isn't like ranking tenth on Google, where a determined searcher might still scroll to you. It's like not existing.
And unlike a lot of marketing problems, this one compounds. AI systems increasingly favour businesses they can read easily, understand clearly, and see corroborated across several places online. The businesses getting themselves recommended today are building a head start that gets harder to close the longer you leave it.
The good news: it's fixable, often in an afternoon
If there's a silver lining, it's this: unblocking the AI crawlers is usually one of the quickest, cheapest fixes in the whole of digital marketing. It's often a one-line change to that robots.txt file. There's no redesign, no rebuild, no big invoice. It's the rare marketing problem where a few minutes of the right work can genuinely reopen a channel that was completely closed.
But — and this is the part worth understanding — unblocking the crawlers only gets you back into the room. Being recommended takes a bit more, and it's the same handful of things that also help you rank on ordinary Google:
- A clear, single statement of who you are and what you do. AI systems, like people, trust a business most when its expertise is stated plainly in one place, not scattered in fragments across a homepage and a few testimonials.
- Answering the real questions people ask. Content written as direct questions and answers — "Do I need to be flexible to start yoga?", "How much does rewiring a house cost?" — is exactly what both Google's snippets and AI assistants pull from first.
- Being visible somewhere other than your own website. Reviews, a claimed Google Business Profile, a mention or two elsewhere online. AI tends to trust what it can confirm from more than one source.
None of that requires starting from scratch. For most small businesses it's a series of small, sensible improvements to what's already there.
How to check where you stand
You can get a rough sense yourself. Ask ChatGPT or Claude the question a customer would ask — "recommend a [what you do] in [your town]" — and see whether you appear. Then ask a friend to look at your site's robots.txt file (you can usually see it by adding /robots.txt to the end of your web address) and check whether GPTBot, ClaudeBot or Amazonbot are listed as disallowed.
That'll tell you whether you have a problem. It won't tell you the full picture — how you compare to competitors, what's holding back your ordinary search visibility, or which fixes will actually move the needle for your particular business.
That's what my website, SEO and AI-visibility assessment is for. I go through your site the way both Google and the AI assistants do, find what's helping and what's quietly working against you, and give you a plain-English, prioritised list of what to fix first — from the five-minute wins to the bigger structural pieces. No jargon, no upsell to a rebuild you don't need.
If you've built a genuinely good business, it's worth making sure the tools people now use to find businesses can actually see it.
Curious whether AI can see your site? Get in touch and I'll take a look.