11 lines
332 B
Markdown
11 lines
332 B
Markdown
https://github.com/ai-robots-txt/ai.robots.txt
|
|
|
|
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
|
|
|
|
### Using .htaccess
|
|
``` .htaccess
|
|
# Block bot traffic from certain user agents
|
|
RewriteEngine On
|
|
RewriteCond %{HTTP_USER_AGENT} (Amazonbot|SemrushBot|meta-externalagent|Bytespider) [NC]
|
|
RewriteRule .* - [R=503,L]
|
|
``` |