Life sciences growth · Video
Is Your Site Blocking AI Search?
VayoMed Team ·
About this video
Episode 07 separates crawler policies for model training from access for search answers. It uses the website audit to explain practical robots.txt and page-fetch checks.
Full transcript
Is your website blocking AI search? Probably not. But training bots and search bots are different, and your rules should be too.
We read 121 robots files from life-science websites. Only six blocked any tracked AI bot at the root. Three of those blocked GPTBot. None blocked the search bots that fetch pages for ChatGPT, Claude or Perplexity answers.
So decide training and search separately. GPTBot collects data for training. OAI-SearchBot and Claude-SearchBot fetch pages for answers.
And there's no secret AI file. Only 17 of 157 home pages served an llms.txt, and Google says its AI features need no special file.
Three steps. One: write separate rules for training and search bots. Two: fetch your key product pages yourself, and check they load. Three: make sure nothing blocks the pages you want cited.
Decide who reads your site. On purpose.