Robots Txt Generator
Create a draft robots.txt from a user-agent name and allowed or disallowed paths. The output expresses crawler access preferences, rather than protecting private material from visitors.
- Runs in your browser
- Nothing is uploaded
- No signup; device limits apply
Loading the tool. If it does not appear, reload this page.
How to use this tool
Enter crawler names and one path per line, with a sitemap address if appropriate. Review the result against the real root-level robots.txt and merge intentionally with existing rules.
Results and examples
Disallow: /private/ requests that a crawler not fetch that path. It does not create login protection or guarantee that a URL cannot be discovered through links elsewhere.
Limitations
- robots.txt is not an access-control boundary or an index-removal command.
- An unintended rule for / can affect crawling of the entire site; inspect scope before deployment.
Fields
| Field | Guidance and constraints |
|---|---|
| User-agent | Check the unit and format displayed beside this field. |
| Allow paths (one per line) | Check the unit and format displayed beside this field. |
| Disallow paths (one per line) | Check the unit and format displayed beside this field. |
| Sitemap URL (optional) | Check the unit and format displayed beside this field. |
| Extra block: User-agent: AdsBot-Google → Disallow: / | Check the unit and format displayed beside this field. |
| robots.txt | Check the unit and format displayed beside this field. |
Data handling
Processing stays in the current browser tab; the site does not upload tool input for the calculation itself.
Sources
Frequently asked questions
Does Disallow make a page private?
Disallow: /private/ requests that a crawler not fetch that path. It does not create login protection or guarantee that a URL cannot be discovered through links elsewhere.
Why is blocking / significant?
robots.txt is not an access-control boundary or an index-removal command. An unintended rule for / can affect crawling of the entire site; inspect scope before deployment.
