robots.txt Builder — Free, Local | How to use and interpret results
Draft a robots.txt group from User-agent and Allow/Disallow paths. The result is crawler guidance text, not authentication or an automatic removal from search results.
- Runs in your browser
- Nothing is uploaded
- No signup; device limits apply
Numbers stay on your device. Nothing is sent to a server.
How to use this tool
Enter the target agent and paths, plus an absolute sitemap URL if needed. Review grouping and existing rules before merging this draft into a live file.
Results and examples
User-agent * with Disallow /admin asks matching crawlers not to request that path prefix. Generation does not install the file or make every bot obey it immediately.
Worked example
User-agent * with Disallow /admin asks matching crawlers not to request that path prefix. Generation does not install the file or make every bot obey it immediately.
Limitations
- Robots rules are not authentication or access control for sensitive files.
- Crawler behaviour, indexing and existing-rule conflicts are not simulated.
Fields
| Field | Guidance and constraints |
|---|---|
| User-agent | Check the unit and format displayed beside this field. |
| Disallow | Check the unit and format displayed beside this field. |
| Allow (optional) | Check the unit and format displayed beside this field. |
| Sitemap URL (optional) | Check the unit and format displayed beside this field. |
Data handling
Processing stays in the current browser tab; the site does not upload tool input for the calculation itself.
Sources
Frequently asked questions
What does the /admin rule request?
User-agent * with Disallow /admin asks matching crawlers not to request that path prefix. Generation does not install the file or make every bot obey it immediately.
Can robots.txt protect private files?
Robots rules are not authentication or access control for sensitive files. Crawler behaviour, indexing and existing-rule conflicts are not simulated.
Related tools
You do not need to start over. Continue with the tool that fits your next step.
