Is a private folder or robots.txt enough to protect a development website?

Simply placing a development website in a folder named “private” or blocking crawling with robots.txt is not enough to protect private content. The key is whether “private” means actual access restrictions, rather than just a folder name or the absence of public links. Google’s official documentation explicitly states that robots.txt cannot force all crawlers to comply; protecting private files on a server requires other measures, such as password protection. If the goal is merely to stay out of search results, it is also important to distinguish between blocking crawling and preventing indexing.

A folder name is not an access restriction

Putting test pages in a private folder and preventing unauthorized people from reading them are two different things. A name can communicate a purpose, but it cannot replace a protection mechanism. What matters is not whether the URL looks obscure, but whether other people can access its content.

The supplied forum material offers differing views on this point: one post says that Google may still find a folder labeled “private”; another says that Google cannot access it, but adds two conditions: no links to it are posted online, and other people are prevented from browsing it. The latter cannot be reduced to “it is safe as long as you do not share the URL,” nor does it demonstrate that a folder name alone can protect a website.

robots.txt manages crawling; it does not lock down a website

Google’s official guidance explains that Googlebot and some trustworthy crawlers follow robots.txt, but not all crawlers do, and different crawlers may interpret its syntax differently. It is therefore suitable for specifying crawling restrictions, but cannot serve as an access barrier protecting confidential content.

• Blocking crawling: Asking crawlers that follow the rules not to fetch specified content.
• Restricting access: Using measures such as passwords to limit others’ access to private files.
• Controlling indexing: Managing whether URLs or pages appear in search results, which is distinct from the first two.

For example, even if a development page is blocked by robots.txt, Google may still index its URL if other websites link to it, and display the URL or link text in search results. This does not mean that Google has crawled the blocked content, but it does show that “not crawled” does not mean “not discoverable in search.”

First decide whether the goal is confidentiality or simply avoiding indexing

If a development website contains material that should not be public, prioritize the password protection recommended in the official guidance rather than relying solely on hidden paths or robots.txt. If a page can be accessible but should not appear in Google search results, the officially listed methods include a noindex meta tag or response header; removing the page entirely is another option.

These measures address different problems and should not be treated as interchangeable switches. Google also cautions that when multiple crawling and indexing rules are used together, some may counteract one another. Protecting a development site should begin with establishing who can read its content, followed by managing crawling and search presentation—not using a robots.txt file as a substitute for all protection.