5 min
AI Crawlers Explained: GPTBot, ClaudeBot and How to Control Them
Which AI crawlers visit your site, what each one does with your content, and how to allow, block or limit them with robots.txt, verification and edge rules.
read →$ ls ./blog/bot-detection
Bot detection
Spotting scrapers, scanners and spoofed crawlers: the signs to look for, verifying search engines, and controlling AI crawlers.
3 min
Is Your Site Being Scraped? Signs to Look For and What to Do
The symptoms of scraping and automated abuse in your analytics, logs and bills, how to confirm what's going on, and the practical steps to take next.
read →4 min
Spoofed Googlebot: How to Verify Search Engine Crawlers
Many requests claiming to be Googlebot aren't. Verify crawlers with reverse and forward DNS or published IP ranges, and learn why user-agent checks aren't enough.
read →other topics