Guidelines To Create & Optimize robots.txt File In WordPress

So, here you are about to learn how to create robots.txt file for WordPress and how to optimize it properly!

In my previous post, I have discussed all the essential things about robots.txt WordPress as it is an important SEO factor.

If you don’t use this text file or doesn’t optimize the robots.txt properly, you would face several issues like 

  1. Search engines would get confused
  2. Website hackers would get invited
  3. Sensitive data could be seen
  4. raise duplicate content issues 

One of the biggest misconceptions about this file is that people think that if they add any file or folder in this file then it won’t get indexed in the Google.

Well, they should know that crawling and indexing are different things. This file guides the Google bots what to crawl and what not to crawl from your website. It doesn’t tell them what to index.

So don’t get mismatched thoughts about these two concepts. The working of the robots.txt file is related to crawling and not indexing.

How To Create robots.txt File?

To create robots.txt file for WordPress, let me provide you a tour of the robots.txt and its syntax, so that you will understand properly.

create-robots-txt

The syntax contains mainly three words, “User-agent”, “Disallow” and “Allow” and it can be written according to the need of the user.

User-agent – Search engine crawler (Like Google, Bing)

Disallow – The File or directory or URL that you want to block. 

Allow – To give crawl access to bots

1. To stop all the bots from your server, then you should write:-

User-agent: *
Disallow: /

After adding this syntax, no bot will crawl your website. The one important thing, you should keep in your mind is that when bots crawl your website for spam and any malware then they don’t read the robots.txt file.

2. If you want to allow the robots to crawl your whole website, then you should write:-

User-agent: *
Disallow:

The same thing can be done if you create an empty robots.txt file or no text like this which can guide the crawling bots about your website.

3. If you don’t want any particular folder (Say cgi-bin) of the directory of your website to get crawled, then you should write:-

User-agent: *
Disallow: /cgi-bin/

4. What would you do if you don’t want the bots to crawl more than one folder? Let me show you how can you do that:-

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-plugins/
Disallow: /wp-admin/
Disallow: /wp-includes/

You can see that for every folder, I have written a provided a separate syntax. It states that you can’t put two or more folders in the same line.

If you write like this “Disallow: /contents/jolly/work/” then it won’t be considered because here you are showing that a folder named “work” is there inside the folder “jolly” which is also inside the folder “content” So basically, the syntax is wrong and won’t fulfill your action.

5. With the help of the robots.txt file for WordPress, you can disallow any bad bot to crawl your website like :-

User-agent: BadBot
Disallow: /

6. You may be thinking as there is no option for allowing the file in the robots.txt file for WordPress. Well, there is a way to do it. The file you want to allow for the bots to crawl, you should write:-

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

The above code will block all the search engines except Google Images to crawl your image uploads.

Also, you can allow one specific file in the directory and block all other like

User-agent: *
Allow: /my-folder/my-file.php
Disallow: /my-folder/

It will block the entire folder (my-folder), but the specific file (my-file.php) can be accessed by the spiders. However, make sure to add the “Allow” code before the “Disallow”.

7. The other thing you can do with this file is to disallow the page or post you don’t want to get crawled, then you should write :- 

User-agent: *
Disallow: /page-permalink
Disallow: /post-permalink

Ex: Disallow: /about-me/

8. If you don’t want to allow the search engines to crawl the CSS, PHP and Javascript files, then you should write :-

User-agent: *
Disallow: /*.php$
Disallow: /*.css$
Disallow: /*.js$

$ – to match the patterns

9. If you use Google Adsense on your site, you may write the following to serve the better ads:-

User-agent: Mediapartners-Google
Disallow:

10. If you want to allow the Google spiders only, then you should write :-

User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /

User-agent: * – It is for all the search engine spiders to follow the rules specified in the robots.txt file.

How To Optimize robots.txt For WordPress?

It would be good to create robots.txt file to stop the search engines crawl some core files and directories. Also, it is a very good practice to add the sitemap in robots.txt. Let’s see how to optimize robots.txt WordPress file.

If your WordPress site is new and you didn’t monetize with the Google Adsense, you may use the robots txt sample like below

Sitemap: http://www.yoursite.com/sitemap_index.xml

User-agent: *
Disallow: /cgi-bin/
Disallow: /page/
Disallow: /wp-admin/
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: /feed/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php

If you use Adsense ads and your WordPress site is not new, it would be better to use the robots txt example given below

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Sitemap: http://www.yoursite.com/sitemap.xml

Conclusion

Robots.txt file for WordPress is an important part of SEO. If it is misused or used incorrectly, it would affect your search engine rankings. 

This file is located in the root of the web directory and the guidelines listed in the file will be followed by search engine bots. 

You should understand that there isn’t any universal rule to create and optimize robots.txt file; it may vary according to the content of the website and your crawl preferences for search engines. 

I hope that I have helped to create robots.txt file, how to optimize it for WordPress SEO and I have provided the example robots.txt file as well. Don’t forget to add sitemap in robots txt file. 

Now, discuss with me about this post! Do you have WordPress robots.txt to optimize your website? Would you like to add any other essential commands in this text file for better performance? Share your insights in the comment section. 

11 Comments

    • Nirmala
  1. Nirmala
    • Nirmala
    • Nirmala
  2. shajeer
    • Nirmala

Leave a Reply

0 Shares
Share
Tweet
+1
Pin
Stumble