Crawler Rules
Generated robots.txt
Upload the generated file to your site's root (
https://yourdomain.com/robots.txt) — it must live there exactly, subfolders don't count. Everything runs entirely in your browser.
