Guidelines To Create & Optimize robots.txt File In WordPress

So, here you are about to learn how to create robots.txt file for WordPress and how to optimize it properly!

In my previous post, I have discussed all the essential things about robots.txt WordPress as it is an important SEO factor.

If you don’t use this text file or doesn’t optimize the robots.txt properly, you would face several issues like 

  1. Search engines would get confused
  2. Website hackers would get invited
  3. Sensitive data could be seen
  4. raise duplicate content issues 

One of the biggest misconceptions about this file is that people think that if they add any file or folder in this file then it won’t get indexed in the Google.

Well, they should know that crawling and indexing are different things. This file guides the Google bots what to crawl and what not to crawl from your website. It doesn’t tell them what to index.

So don’t get mismatched thoughts about these two concepts. The working of the robots.txt file is related to crawling and not indexing.

How To Create robots.txt File?

To create robots.txt file for WordPress, let me provide you a tour of the robots.txt and its syntax, so that you will understand properly.

create-robots-txt

The syntax contains mainly three words, “User-agent”, “Disallow” and “Allow” and it can be written according to the need of the user.

User-agent – Search engine crawler (Like Google, Bing)

Disallow – The File or directory or URL that you want to block. 

Allow – To give crawl access to bots

1. To stop all the bots from your server, then you should write:-

User-agent: *
Disallow: /

After adding this syntax, no bot will crawl your website. The one important thing, you should keep in your mind is that when bots crawl your website for spam and any malware then they don’t read the robots.txt file.

2. If you want to allow the robots to crawl your whole website, then you should write:-

User-agent: *
Disallow:

The same thing can be done if you create an empty robots.txt file or no text like this which can guide the crawling bots about your website.

3. If you don’t want any particular folder (Say cgi-bin) of the directory of your website to get crawled, then you should write:-

User-agent: *
Disallow: /cgi-bin/

4. What would you do if you don’t want the bots to crawl more than one folder? Let me show you how can you do that:-

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-plugins/
Disallow: /wp-admin/
Disallow: /wp-includes/

You can see that for every folder, I have written a provided a separate syntax. It states that you can’t put two or more folders in the same line.

If you write like this “Disallow: /contents/jolly/work/” then it won’t be considered because here you are showing that a folder named “work” is there inside the folder “jolly” which is also inside the folder “content” So basically, the syntax is wrong and won’t fulfill your action.

5. With the help of the robots.txt file for WordPress, you can disallow any bad bot to crawl your website like :-

User-agent: BadBot
Disallow: /

6. You may be thinking as there is no option for allowing the file in the robots.txt file for WordPress. Well, there is a way to do it. The file you want to allow for the bots to crawl, you should write:-

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

The above code will block all the search engines except Google Images to crawl your image uploads.

Also, you can allow one specific file in the directory and block all other like

User-agent: *
Allow: /my-folder/my-file.php
Disallow: /my-folder/

It will block the entire folder (my-folder), but the specific file (my-file.php) can be accessed by the spiders. However, make sure to add the “Allow” code before the “Disallow”.

7. The other thing you can do with this file is to disallow the page or post you don’t want to get crawled, then you should write :- 

User-agent: *
Disallow: /page-permalink
Disallow: /post-permalink

Ex: Disallow: /about-me/

8. If you don’t want to allow the search engines to crawl the CSS, PHP and Javascript files, then you should write :-

User-agent: *
Disallow: /*.php$
Disallow: /*.css$
Disallow: /*.js$

$ – to match the patterns

9. If you use Google Adsense on your site, you may write the following to serve the better ads:-

User-agent: Mediapartners-Google
Disallow:

10. If you want to allow the Google spiders only, then you should write :-

User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /

User-agent: * – It is for all the search engine spiders to follow the rules specified in the robots.txt file.

How To Optimize robots.txt For WordPress?

It would be good to create robots.txt file to stop the search engines crawl some core files and directories. Also, it is a very good practice to add the sitemap in robots.txt. Let’s see how to optimize robots.txt WordPress file.

If your WordPress site is new and you didn’t monetize with the Google Adsense, you may use the robots txt sample like below

Sitemap: http://www.yoursite.com/sitemap_index.xml

User-agent: *
Disallow: /cgi-bin/
Disallow: /page/
Disallow: /wp-admin/
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: /feed/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php

If you use Adsense ads and your WordPress site is not new, it would be better to use the robots txt example given below

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Sitemap: http://www.yoursite.com/sitemap.xml

Conclusion

Robots.txt file for WordPress is an important part of SEO. If it is misused or used incorrectly, it would affect your search engine rankings. 

This file is located in the root of the web directory and the guidelines listed in the file will be followed by search engine bots. 

You should understand that there isn’t any universal rule to create and optimize robots.txt file; it may vary according to the content of the website and your crawl preferences for search engines. 

I hope that I have helped to create robots.txt file, how to optimize it for WordPress SEO and I have provided the example robots.txt file as well. Don’t forget to add sitemap in robots txt file. 

Now, discuss with me about this post! Do you have WordPress robots.txt to optimize your website? Would you like to add any other essential commands in this text file for better performance? Share your insights in the comment section. 

by Nirmala
Nirmala is an avid blogger, WordPress enthusiast who has been blogging since 2010. She loves to write useful WP tips & tricks on this active blog.

18 thoughts on “Guidelines To Create & Optimize robots.txt File In WordPress”

    • Yes Prakash! We should create the proper file to stay away from the duplicate content issues. Thanks for reading the post. Keep coming!

      Reply
  1. Thanks, Nirmala.

    It’s a very nice article, and I found useful over here.

    I don’t this line that why Robots.txt blog my image files.

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    I’ll text the code and back to tell your the result.

    Oh. By the way, I hear from Yoast. We shouldn’t blog resources files such CSS and JS. Hence the code such should change a little bit.

    User-agent: *
    Allow: /wp-admin/admin-ajax.php

    Just the above code block instead of

    User-agent: *
    Disallow: /wp-plugins/
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Reply
  2. Hello Kimsea,

    You are welcome and thanks for getting time to read my post.

    It is good to allow the Google bots to crawl the images to show in the image search results. So, I have specifically listed the codes for my readers to include in their robots.txt file.

    Yes, I agree with you that the one line code that you have shared will block the files of wp-admin that we would like to hide.

    Thanks again for coming by, stay tuned!

    Reply
    • Thanks for reading the post and leaving your kind words. Glad I could help you understand the essential WordPress stuff, keep coming!

      Reply
    • Hey Imran,

      Nice to see you here and great that I have written the useful content for you. It would be better to create a custom robots.txt file to perform well on search engines.

      Thanks for coming by! Have a good day, everyday!

      Reply
  3. That is a great tip particularly to those new to the blogosphere.
    Short but very precise information… Many thanks for sharing this one.

    A must read article!

    Reply
  4. Very good information you have provided on this website. I also have a website, on which I share similar information with people.

    Reply
  5. Hey,
    I am Ashish malik an internet marketer and trainer. I have been looking for a detailed description of robots.txt and it was really amazing to get this. Thank you so much for sharing. Will be reading more your articles now.

    Reply
  6. Very informative guide! Creating and optimizing a robots.txt file in WordPress can be daunting, but your post simplifies it perfectly. Thanks for the step-by-step instructions! 👍😊

    Reply

Leave a Comment