User-agent: GPTBot Disallow: / # Block Common Crawl (CCBot) User-agent: CCBot Disallow: / User-agent: Twitterbot Disallow: / User-agent:* Disallow: /*100923808.html Disallow: /*?$ Disallow: /?* Disallow: /*?NOJP Disallow: /*?*=*