Robots.txt Generator

Robots.txt is a plain-text file at the root of a site telling crawlers which paths they may fetch. It is built from rule groups: a User-agent line naming one bot or the catch-all star, Allow and Disallow path lines, an optional Crawl-delay, and Sitemap pointers. Hand-writing looks trivial until an over-broad Disallow shadows an Allow, or a missing file leaves staging wide open. This generator assembles the file from editable rule rows with a live output pane, ready to copy or download as robots.txt for the domain root.

Rule groups the file is built from

• 🤖 One group per crawler, from a named bot to the star matching everyone, with add and remove controls
• 🚫 Multiple Disallow paths plus Allow exceptions in the same group for fine control
• ⏱️ Optional Crawl-delay for bots that hammer the server, blank when unneeded
• 🗺️ One or more Sitemap lines appended after the groups for discovery
• 🧩 Three presets: allow-all for open sites, block-all for staging, WordPress defaults shielding admin paths while freeing admin-ajax

Occasions that call for a fresh robots file

• 🚧 Sealing a staging host so it never leaks into the index
• 🛡️ Slowing one aggressive scraper without blocking well-behaved search bots

Advisory protocol, not a padlock

• 📨 Robots.txt is a request polite crawlers honor; it cannot force a rogue bot to stop
• 📍 The file only works at the domain root and takes effect on the next crawl, not instantly

Nothing you type travels anywhere during editing, since the whole file is composed inside your browser and only the file you save ever touches your server.