How To Access Robots.txt In WordPress

Last updated on July 31, 2021.

In this short guide, I’m showing you how to create, access, and modify robots.txt in WordPress in 4 simple ways. 

If you are serious about your website (and especially if you have a bigger website), you should definitely control its robots.txt file. With WordPress websites, fortunately, it is quite easy to do. 

Let me show you 4 simple ways to access the robots.txt file in WordPress. 

SEO NEWSLETTER

Subscribe to receive free weekly updates with SEO news, SEO tips, SEO tools, and more!
Join 3000+ ambitious SEOs.

Basic information about the robots.txt file

Robots.txt is the file that informs search engine bots about the pages or files that should or should not be crawled. 

  • The robots.txt file is supposed to protect a website from overloading it with requests from crawlers (check my full guide on the crawl budget optimization). 
  • The robots.txt file is not a way to block web pages or files from getting indexed and appearing in search results. 
  • Search engine crawlers always start crawling your website by checking the directives in robots.txt.
  • Search engines should but do not always respect the directives provided in robots.txt (Google does).

In case you need a small reminder, here is some basic technical information about robots.txt:

  • The only correct robots.txt location is the root (the main directory) of the website. This applies to any website whether or not it is a WordPress website. 
  • One website can only have one robots.txt file
  • The only acceptable name for the file is robots.txt
  • Robots.txt needs to be a UTF-8 encoded text file. 

With that in mind, the WordPress robots.txt location is the same as for any other website. It is the root of the website. In the case of my website, this is https://seosly.com/robots.txt. 

Example robots.txt file
This is how a default WordPress robots.txt file looks like.

A good thing about WordPress is that it lets you access your robots.txt in a few different ways even if you don’t have FTP access. 

☝️ If you want to learn more about the robots.txt file, how it works, what it is, make sure to check the introduction to robots.txt in Google Search Central. 

4 ways to access the robots.txt in WordPress

And here are the four ways you can access and modify the robots.txt file of your WordPress site

#1: Use an SEO plugin 

There are many WordPress SEO plugins but there are practically only two that really count: 

There is also All in One SEO but I am not a big fan of this plugin. All of those SEO plugins let you access and modify robots.txt easily.

Access robots.txt with Rank Math 

If you are using Rank Math, here is what you need to do:

  • Log in to your WordPress Dashboard as admin. Note that only admins can modify plugins. 
  • Navigate to Rank Math SEO
Rank Math SEO settings
  • In the left sidebar under Rank Math, you will see different Rank Math settings. Click General Settings. 
Robots.txt location in WordPress
  • You will now see a list of general SEO settings. Click on Edit robots.txt.
How to access robots.txt in RankMath
  • If you don’t do anything, Rank Math will automatically handle robots.txt for you as you can see in the screenshot below.
Robots.txt in Rank Math
  • If you want to modify robots.txt, simply start typing in the text field and click Save Changes.
Robots.txt in Rank Math in WordPress
  • If you are not sure about what to put in robots.txt or you do something wrong, just click Reset Options
Resetting robots.txt settings in Rank Math

That’s it for Rank Math and robots.txt. 

Access robots.txt with Yoast SEO

Here is what to do if you are using Yoast SEO:

  • Log in to your WordPress Dashboard as an admin. Remember that only admins can modify plugin settings. 
  • Navigate to SEO
Robots.txt in Yoast SEO
  • In the left sidebar, you will see a couple of settings under SEO. Click on Tools.
Robots.txt in Yoast SEO
  • Click on File editor.
How to access robots.txt in WordPress in Yoast SEO
  • Now you can modify the robots.txt file. 
Modifying robots.txt in Yoast SEO in WordPress
  • When you are done, simply click Save changes to robots.txt. 

In most WordPress websites, these two methods using these two SEO plugins will do. But there are other ways too. 

#2: Use a dedicated robots.txt plugin

There are also lots of different WordPress plugins made specifically to let you edit robots.txt. 

Here are the most popular robots.txt WordPress plugins you might want to check: 

Each one lets you easily access and modify robots.txt. Below are the quick steps to take to access the robots.txt file using these 3 WordPress plugins. 

❗ Note that some plugins do create a robots.txt file that is placed at the root of your website while others create it dynamically. If robots.txt is created dynamically, then you will not find it in the root directory of your website (e.g. via FTP). 

❗ Remember not to use multiple robots.txt plugins at the same time! 

Virtual Robots.txt

Here is how to use the Virtual Robots.txt plugin:

Virtual Robots.txt WordPress Plugin
  • When in WordPress dashboard, navigate to Settings and then Virtual Robots.txt.
Virtual Robots.txt WordPress Plugin
  • You can now see your robots.txt file and modify it. When you are done, click Save Changes
Virtual Robots.txt WordPress Plugin settings

This is it!

Better Robots.txt

This robots.txt plugin is a bit different than the plugins I showed you above. With Better Robots.txt, you don’t modify a text file but simply adjust settings in a user-friendly panel inside WordPress. 

Here is what to: 

  • Install and activate Better Robots.txt. The plugin is also named WordPress Robots.txt optimization.
Better Robots.txt WordPress Plugin
  • Grant the necessary permissions to the plugin.
Better Robots.txt WordPress Plugin configuration
  • Navigate to Better Robots.txt. The plugin will be available in the left sidebar.
Better Robots.txt WordPress Plugin configuration
  • You will now see a lot of useful and user-friendly settings. 
Better Robots.txt WordPress Plugin settings
  • The plugin also lets you manually allow or disallow specific crawlers.
Better Robots.txt WordPress Plugin crawler settings
  • You can also adjust some settings to protect your data or optimize loading performance.
Better Robots.txt WordPress Plugin advanced settings
  • There are also some other interesting settings the plugin offers. I recommend checking those. 

Better Robots.txt essentially lets you create a fully customized robots.txt file without you having to touch a line of code. 

Robots.txt Editor 

And here is how to use Robots.txt Editor:

Robots.txt Editor WordPress Plugin
  • Navigate to Plugins > Installed Plugins in your WordPress dashboard. You can also access robots.txt by going to Settings > Reading.
Robots.txt Editor WordPress Plugin
  • Click Settings under Robots.txt Editor.
Robots.txt Editor WordPress Plugin
  • You can now access and edit your robots.txt file. When you are done, click Save Changes
Robots.txt Editor WordPress Plugin

❗ Be careful and avoid using multiple robots.txt editors at one time. This can really mess things up. If you use an SEO plugin, you should stick with its robots.txt editor only. 

#3: Access robots.txt via cPanel in your hosting 

Here I will show you how to upload robots.txt in WordPress without FTP access. 

You can also use the cPanel from your hosting to create or modify robots.txt. 

Note that some of the plugins shown above dynamically create a robots.txt file. It means that you won’t be able to find the file in the root directory of your website. 

❗ If you use a plugin to create and manage robots.txt, then you should not try to manually add a new robots.txt using cPanel or FTP (below). 

However, if you don’t rely on those plugins, you are free to manually create and update a robots.txt file to your WordPress website. 

Here is how to do it using Bluehost:

  • Log in to the cPanel in your hosting. 
  • Unfold section Advanced.
Bluehost cPanel
  • Click on File Manager
Bluehost File Manager
  • Navigate to the root directory of your website. 
  • Unless you use a plugin that dynamically generates robots.txt, you should see a robots.txt file at the root.
Robots.txt in the root directory
  • Click on the robots.txt file and choose Edit.
Editing robots.txt in cPanel in Bluehost
  • You can now edit the contents of robots.txt. When done, click Save Changes.
Saving changes to robots.txt in cPanel

If your website does not have a robots.txt file, then you need to create and upload one. 

Here is what to do:

  • Open a text file editor like Notepad or Notepad++. 
  • Create a robots.txt file.
Creating a robots.txt file in Notepad
  • Upload the file to the main directory of your website.  In the case of Bluehost, you simply click Upload when in File manager. Just make sure you are at the root.
Upload robots.txt via cPanel in Bluehost

This method will work both for WordPress and non-WordPress websites. 

#4: Use FTP to access robots.txt 

Here I’m showing you how to access robots.txt via FTP. 

The quickest and the most “pro” way to access robots.txt is to use FTP. This method also works for any website type, WordPress sites including. 

❗ I don’t recommend this method if you use a plugin to automatically generate robots.txt for you. 

This method is similar to what you do using the cPanel in your hosting except that this time you are using an FTP client to access your website. 

To use this method you need an FTP username and password and an FTP client like Filezilla

Here is what you need to do:

  • Connect to your website via an FTP client.
  • Navigate to the root directory of your website if you aren’t there already.
Modify robots.txt via FTP
  • Download the robots.txt file if it’s there.
  • Open the robots.txt file with a text editor like Notepad++.
  • Modify the file as needed. Save changes.
  • Upload the file to the root directory. This will overwrite the existing robots.txt. 
Modify robots.txt via FTP

If your website does not have a robots.txt file, this is also the way to create and upload one. 

How to test the robots.txt file on your website

If you have implemented some changes to robots.txt, I strongly suggest you test your robots.txt to make sure the file is correct and really does what it is supposed to do.

Here is what to do:

Robots.txt tester in Google Search Console

Here are other guides you might be interested in:

GET ACCESS TO PREMIUM SEO CONTENT

Members of SEOSLY Pro get access to Olga’s SEO audit video guide where she documents her 500+ step SEO audit process, Olga’s mini video SEO audits, SEO audit templates, pro SEO auditing tips, and much more!

Join SEOSLY Pro to become a technical SEO expert in no time.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *