Create a custom robots.txt file for your website.
Create a valid robots.txt file for your website by filling in a simple form. The robots.txt file tells search engine crawlers which pages or directories to crawl or skip. A well-configured robots.txt prevents private pages from being indexed and improves crawl efficiency.
# Click Generate to see output