Sitemap and robots.txt explained: the two files that make your site legible

A sitemap and a robots file do not make a site rank: they make it legible, and legibility decides whether the pages get read at all.

Two small files sit at the root of most websites, and most owners never open either. The sitemap is the list of pages you want found; the robots file is the standing instruction for what crawlers may enter. Together they are the site's first conversation with a search engine, and like many first conversations, they are held once and remembered wrong.

The honest framing: these files do not make a site rank. Ranking comes from the pages themselves, from what they say and how fast and how well they are built. What the two files decide is legibility: whether the pages get read at all, whether the crawler is invited into the rooms that matter, and whether the list it is handed matches the building that actually exists.

The failure versions are quiet. The sitemap still listing pages deleted a year ago, describing to every crawler a wing of the building that was demolished. The staging environment left open to crawlers, so its test pages, placeholder text and half-built forms and all, get indexed for the world while the real site waits. The robots file configured once during a rushed relaunch, now blocking the very pages the business lives on.

The practice is to let the sitemap reflect what actually exists. Generated from the build, or checked against it before the gap costs anything, never hand-typed in a burst of launch enthusiasm and forgotten. A page added should appear in it; a page removed should disappear from it; and nobody should have to remember to do either by hand.

The robots file gets re-read on every relaunch and every environment change, because it is the one file whose mistakes stay invisible until they have already cost months. Then a search-console-style check closes the loop: what was submitted, what was actually indexed, and what is sitting there that nobody intended to be public. The gap between those lists is the whole story.

We ship both files with every site we build and re-check them at each relaunch, because legibility is part of the handover, not an extra. The broader diagnosis of a site that earns no traffic is its own topic; this is the ground floor of it. Guidance, not guarantees: the files make your site readable, and the pages still have to deserve the reading.