Does robots.txt keep a page out of Google?

No. It stops the page being crawled, not listed. A blocked URL can still appear in results, without a description, if something links to it. Use a noindex directive on a page crawlers are allowed to fetch.

Is robots.txt a security measure?

The opposite. It is a public file listing the paths you would rather were not visited, readable by anyone. Never rely on it to protect anything.

Should I block CSS and JavaScript?

No. Search engines render pages to understand them, and blocking the assets means rendering a broken version of your own site for the crawler that is judging it.

Where should I declare my sitemap?

In robots.txt, with an absolute URL. It is the one place every crawler looks without being told, and it works regardless of whether you have also submitted it anywhere.

A robots.txt tester that works on sites you do not own

For your own verified property, the report Google publishes is the authority and nothing can beat it - it shows the exact file Google fetched, when it fetched it, and how it parsed it. It has one hard limit: you must have proved you own the site.

Open the Robots & Sitemap Checker →

Search Console and Softland, side by side

	Search Console	Softland
Sites you have verified	Authoritative - the file as Google fetched it	A current fetch
Sites you have not verified	Not available	Works - robots.txt is public
Per-crawler rules	Google crawlers	Every user agent block in the file
Declared sitemaps	Shown, with indexing status	Listed
Account	Required, plus verified ownership	None

The cases where you do not own the site

An agency taking on a client and working out why nothing ranks, before any access has been granted. A developer inheriting a site with no documentation. Someone evaluating a competitor to see which sections they keep out of search. A buyer doing diligence on a site before acquiring it. An engineer checking whether a supplier has accidentally blocked the endpoint an integration depends on.

None of those can use a tool that requires verified ownership, and all of them are asking a question about a file that is deliberately public. Robots.txt is served to every crawler on the internet by design, so reading it is not an intrusion.

The mistake that removes a site from search

A staging site is built with a robots.txt that disallows everything, which is correct. Then it is promoted to production, and the file comes with it. The site is live, complete, and invisible - and the symptom is simply an absence of traffic, with nothing anywhere announcing the cause.

It is worth checking immediately after any launch, migration or replatform, and worth checking on any site that lost its traffic overnight. Two characters in one file, and it happens to organisations with experienced teams several times a year.

Robots.txt does not do what most people think

It controls crawling, not indexing. A page blocked in robots.txt can still be listed in search results if other pages link to it - the crawler respects the block, does not fetch the page, and lists the URL anyway with no description because it was never allowed to read one. To keep a page out of the index you need a noindex directive, and that only works on a page the crawler is permitted to fetch.

Blocking a URL in robots.txt and putting noindex on it is therefore self-defeating: the block prevents the crawler from ever seeing the noindex. It is also not a security control. The file is a public list of the paths you would rather nobody visited, which is the opposite of concealment for anything genuinely sensitive.

When Search Console is the better choice

The site is yours and verified - then use the authoritative report, which shows the exact file Google fetched and how it parsed it.
You need to know whether a specific URL is indexed rather than whether it is crawlable, which is a different question entirely.
You want the history of fetch attempts and any errors Google encountered retrieving the file.

Frequently asked questions

Does robots.txt keep a page out of Google?: No. It stops the page being crawled, not listed. A blocked URL can still appear in results, without a description, if something links to it. Use a noindex directive on a page crawlers are allowed to fetch.
Is robots.txt a security measure?: The opposite. It is a public file listing the paths you would rather were not visited, readable by anyone. Never rely on it to protect anything.
Should I block CSS and JavaScript?: No. Search engines render pages to understand them, and blocking the assets means rendering a broken version of your own site for the crawler that is judging it.
Where should I declare my sitemap?: In robots.txt, with an absolute URL. It is the one place every crawler looks without being told, and it works regardless of whether you have also submitted it anywhere.

Try it yourself

Read a site robots.txt and find its sitemaps.

Open the Robots & Sitemap Checker

Search Console is a trademark of its respective owner. Softland is not affiliated with, endorsed by or sponsored by Search Console. This comparison reflects how each product works rather than what either costs, because pricing and plan limits change; check Search Console’s own site for its current terms.