Stop OpenAI’s ChatGPT User agent from scanning and using your website’s content

Background:

As Wikipedia Describes ChatGPT: AI chatbot developed by OpenAI. ChatGPT is an artificial intelligence (AI) chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI’s GPT-3.5 and GPT-4 families of large language models (LLMs) and has been fine-tuned (an approach to transfer learning) using both supervised and reinforcement learning techniques.

ChatGPT can access real-time info from the web using plug-ins.

If you don’t want ChatGPT User-agent to use your website’s data, you can block it using your robots.txt file.

A ‘robots.txt’ file is generally located at the root of the web server and can be accessed using SFTP or can be viewed using this URL:

https://yourwebsite.com/robots.txt

(replace ‘yourwebsite.com’ with your domain name)

You need to add this rule to your website’s ‘robot.txt’ file

To allow ChatGPT plugins to access your website, you can explicitly add the ChatGPT-User to your site’s robots.txt:

User-agent: ChatGPT-User
Disallow:

To allow plugins to access only parts of your site, you can add the ChatGPT-User to your site’s robots.txt like this:

User-agent: ChatGPT-User
Disallow:
Allow: /directory-1/
Allow: /directory-2/

To disallow plugins to access your site, you can add the ChatGPT-User to your site’s robots.txt:

User-agent: ChatGPT-User
Disallow: /

Reference ChatGPT User-agent: https://platform.openai.com/docs/plugins/bot

I hope this information helps you to take the right decision.

Thanks & Regards
Mandar Apte