WEBVTT

00:00:00.800 --> 00:00:03.270
Is your website blocking AI search?

00:00:03.380 --> 00:00:04.520
Probably not.

00:00:04.520 --> 00:00:07.344
But training bots and
search bots are different,

00:00:07.344 --> 00:00:09.070
and your rules should be too.

00:00:09.790 --> 00:00:13.910
We read 121 robots files
from life-science websites.

00:00:13.910 --> 00:00:17.590
Only six blocked any
tracked AI bot at the root.

00:00:17.590 --> 00:00:19.355
Three of those blocked GPTBot.

00:00:20.030 --> 00:00:23.442
None blocked the search bots
that fetch pages for ChatGPT,

00:00:23.770 --> 00:00:25.820
Claude or Perplexity answers.

00:00:26.790 --> 00:00:29.680
So decide training and search separately.

00:00:29.990 --> 00:00:32.460
GPTBot collects data for training.

00:00:33.830 --> 00:00:37.301
OAI-SearchBot and Claude-SearchBot
fetch pages for answers.

00:00:37.540 --> 00:00:39.700
And there's no secret AI file.

00:00:39.700 --> 00:00:43.990
Only 17 of 157 home pages
served an llms.txt,

00:00:45.460 --> 00:00:49.070
and Google says its AI
features need no special file.

00:00:49.730 --> 00:00:50.780
Three steps.

00:00:50.930 --> 00:00:54.980
One: write separate rules for
training and search bots.

00:00:55.070 --> 00:00:58.130
Two: fetch your key
product pages yourself,

00:00:58.130 --> 00:00:59.520
and check they load.

00:00:59.550 --> 00:01:03.440
Three: make sure nothing blocks
the pages you want cited.

00:01:04.160 --> 00:01:06.000
Decide who reads your site.

00:01:06.000 --> 00:01:07.330
On purpose.