Robots.txt explained in one line: it tells search engine bots which pages they can visit. A noindex tag is a different tool that tells search engines to keep a page out of search results.
You open yourdomain.com/robots.txt, see a few lines that look like code, and close the tab. Now a worry lingers: could one wrong line hide your whole site from Google?
You already know your field, so this guide skips the jargon and gives you the part nobody taught you.
What Is a Robots.txt File? (Robots.txt Explained Simply)
| Term | Meaning |
| Robots.txt | A plain text file on your website that tells search engine bots which pages they can visit and which pages they should skip. |
A robots.txt file is a set of visiting rules for search engine bots. It lives at one fixed address: yourdomain.com/robots.txt.
People call these bots crawlers. They travel from link to link, read each page, and report back to the search engine.
The file sits inside a wider area called technical SEO. That area covers how easily search engines reach and read your site.
Think of a consultant with a public website. The site has a services page for everyone and a private client login page. A robots.txt file can ask bots to visit the first page and skip the second.
Anyone can read your robots.txt file by typing its address into a browser. Bots also follow it by choice, so search engines obey it and bad bots often ignore it.
How Does Robots.txt Work?
| Term | Meaning |
| Crawling | When a search engine bot visits a page and reads what’s on it. |
| Directive | One rule line in your robots.txt file that gives a bot an instruction. |
A bot checks your robots.txt file first, then follows the rules that match its name. That check happens before it reads any page on your site.
Google’s bot, called Googlebot, follows this same routine. It’s the first step in how Google finds and reads pages.
Here’s the process in four steps:
- A bot arrives at your site.
- It asks for yourdomain.com/robots.txt before anything else.
- It looks for rules that match its name.
- It skips the areas you blocked and visits the rest.
Rule lines are the heart of the file, and most files use only four kinds.
This table shows the four directives you’ll see most often.
| Directive | What it does | Example |
| User-agent | Names the bot the rules apply to | User-agent: * (every bot) |
| Disallow | Asks bots to skip a path | Disallow: /private/ |
| Allow | Makes an exception inside a blocked area | Allow: /private/info-page/ |
| Sitemap | Points bots to your list of pages | Sitemap: https://example.com/sitemap.xml |
The sitemap is a list of your pages that helps bots find them. You can also submit your sitemap directly to Google.
Here’s a simple example file a small WordPress site might use:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
The star means every bot. The file blocks the admin area, keeps one exception open, and shares the sitemap address.
What Is Noindex, and How Is It Different From Robots.txt?
| Term | Meaning |
| Noindex | A tag that tells search engines to keep a page out of their search results. |
Noindex tells search engines to keep a page out of search results. Robots.txt only controls whether bots visit the page.
You can add noindex in two ways. The first is a meta robots tag, a short line of code in the hidden top part of a page.
The second is an X-Robots-Tag, a message your website’s server sends along with the page. Most WordPress users skip the code and tick a box in their SEO plugin instead.
This table puts the two tools side by side.
| What to compare | Robots.txt | Noindex |
| What it controls | Whether bots visit a page | Whether a page appears in search results |
| Where it lives | One file for the whole site | On each page, or in its server message |
| Best for | Saving bot visits on low-value areas | Keeping a page out of search results |
| Main limit | A blocked page can still appear if other sites link to it | Bots must visit the page to read the tag |
The key difference is simple. Robots.txt controls visits, and noindex controls what shows up in results.
That difference has a surprising side effect. A page you block in robots.txt can still appear in results.
Say a coach blocks a private client page in robots.txt. A friend’s blog links to it, and Google may still list the bare address with no description.
Should You Use Robots.txt, Noindex, or Both?
| Term | Meaning |
| Indexing | When a search engine saves a page in its big library so the page can appear in search results. |
Use robots.txt to guide where bots spend their visits. Use noindex to keep a page out of search results.
When should you use noindex?
Use noindex for pages that visitors need but searchers don’t. Thank-you pages, booking confirmations, and login pages fit this group.
A trainer’s “thanks for booking” page is a good example. Clients see it once, and nobody needs to find it on Google.
Noindex works well here because bots can still open the page. They read the tag and follow it.
When should you use robots.txt?
Use robots.txt for areas bots have no reason to visit. Admin areas and internal search result pages fit this group.
Blocking them saves bot visits for the pages that matter. On a small site, this rarely changes much.
Why should you never use both on the same page?
If robots.txt blocks a page, bots never open it. They never see the noindex tag either, so the page can still appear in results.
Google’s own documentation confirms that a blocked page hides its noindex tag.
Robots.txt is also not a security tool. Anyone can read your file, so never list private pages in it.
Real private pages need a password.
What Should You Block, and What Should You Never Block?
| Term | Meaning |
| Crawl budget | The number of pages a search engine bot will visit on your site in a set time. |
Block pages that add nothing in search, and never block pages you want people to find. For most small sites, blocking very little is the safest choice.
Crawl budget mostly matters for very large sites. A professional blog with a few dozen posts rarely runs out of it.
This table shows what usually belongs on each side.
| Page or file | Block it? | Why |
| Admin area | Yes | Visitors and searchers never need it |
| Internal search result pages | Usually yes | They create endless thin pages |
| Blog posts and service pages | Never | You want people to find these |
| CSS and JavaScript files | Never | These code files control how pages look and work, and Google needs them |
| Your whole site | Never | One line can hide everything |
One short line can hide your whole site. The line Disallow: / tells every bot to skip everything.
Picture a consultant who builds a new site behind a full block. Launch day arrives, and nobody removes the line.
Weeks later, no page shows up in search.
How Do You Create or Edit Robots.txt and Add Noindex in WordPress?
| Term | Meaning |
| Root directory | The main top-level folder of your website, where the robots.txt file has to sit. |
Most WordPress sites already have a basic robots.txt file, and your SEO plugin can edit it. You add noindex to a single page with one plugin setting, and you never touch code.
The file has to sit in the root directory. Bots only look for it at your site’s top level, so a copy in a subfolder does nothing.
How do you check and edit your robots.txt file?
- Type your site address followed by /robots.txt into a browser.
- Read every line. Look for Disallow: / because that line blocks everything.
- Open your SEO plugin and find its robots.txt editor.
- Change one thing at a time, then save.
- Reload yourdomain.com/robots.txt to confirm the change.
How do you add noindex to one page?
- Open the page in your editor.
- Find your SEO plugin’s settings for that page.
- Switch on the option that keeps the page out of search results.
- Update the page, then test it using the steps in the next section.
Which WordPress setting can hide your whole site?
Under Settings, then Reading, WordPress has a box that asks search engines to stay away from your entire site. Tick it while you build, and untick it on launch day.
A coach who builds a booking site in private often forgets this box. The site goes live, and search engines never come.
How Do You Test Robots.txt and Noindex (and Avoid Common Mistakes)?
| Term | Meaning |
| URL Inspection | A free tool in Google Search Console that shows how Google sees one specific page. |
Use Google Search Console to test both. This free Google tool shows whether Google can visit a page and whether the page can appear in results.
That matters for your credibility. If your best pages never reach search, potential clients never find you.
How do you run the test?
- Open Search Console and choose your site.
- Open the robots.txt report to confirm Google can fetch your file.
- Paste a page address into the URL Inspection tool at the top.
- Read the result to see whether robots.txt blocks the page or a noindex tag applies.
- Open the page indexing report to spot pages you did not expect.
What do the two confusing status messages mean?
“Indexed, though blocked by robots.txt” means Google listed the page but could not read it. Remove the block and add noindex if you want the page out of results.
“Excluded by ‘noindex’ tag” means Google read your noindex tag and left the page out. That’s good if you planned it, and a problem if you didn’t.
Which mistakes should you avoid?
Five mistakes show up again and again:
- Blocking the whole site with Disallow: /
- Blocking CSS and JavaScript files that Google needs
- Using robots.txt and noindex on the same page
- Leaving noindex on a page after launch
- Listing private pages in a public robots.txt file
Run these checks during your next basic SEO audit.
Important FAQs
Does robots.txt stop a page from appearing in Google?
Not always. Robots.txt stops bots from visiting a page, but the web address can still appear if other sites link to it. Use noindex to keep a page out of results.
Can I use robots.txt and noindex on the same page?
You can, but you shouldn’t. Robots.txt blocks bots from opening the page, so they never see the noindex tag. Pick one tool for each page.
Is robots.txt a safe way to hide private pages?
No. Anyone can read your robots.txt file, and it only asks bots to stay away. Protect private pages with a password instead.
Does every website need a robots.txt file?
No. A site without one still gets visited by search engines. A file helps when you want to block certain areas, such as an admin area.
How long does Google take to notice changes to robots.txt or noindex?
It varies. Google often rechecks your robots.txt file within a day, but pages can take days or weeks to update in results. Use the URL Inspection tool to ask for a fresh look at a key page.
What Should You Do Next?
| Term | Meaning |
| Robots.txt check | A quick look at your robots.txt file to confirm it does not block pages you want people to find. |
Do one small thing today. Type yourdomain.com/robots.txt into your browser and look for a line that says Disallow: /.
Then open Search Console and confirm your best pages appear. Those pages build your credibility and bring clients to you, and income follows from them.
Ready for the next step? The SEO for Beginners guide walks you through the rest of the basics in order.