Today, I will teach you how to upload Robots.txt in WordPress which plays and important role in giving Google Bot command they need to follow and how they should crawl your website.
Have you ever heard the word Robots.txt and how it affects your website? Not everyone understands the function of robots.txt especially in WordPress which makes it a little bit hard to add robots.txt on WordPress.
This article will give you an in-depth understanding about creating, editing and uploading robots.txt in WordPress, I was once asked where to put robots.txt in WordPress which gave me the ability to write this article in order to teach others who don’t have robots.txt on their WordPress.
It’s quite hard for a novice to have the knowledge of putting robots.txt in WordPress without reading this article, most WordPress users think having a WordPress gives them automatic robot file generator but that’s a lie.
WordPress robots.txt doesn’t come as a feature on WordPress settings but with the help of yoast SEO plugin its possible.
Putting wrong information on your robots.txt might affect your SEO and also give crawling bot the wrong details to crawl.
Now I will give you the best WordPress robots.txt guide to enable you make bot understand your website easily.
Before explaining how to edit and upload robots.txt on WordPress I will give a simple explanation on what robots.txt means.
What is WordPress Robots.txt?
Robots in layman language are any type of “bot” that visits websites on the Internet. The most common type of bot is search engine crawlers. These bots “crawl” around the web to help search engines like Google index and rank the billions of pages on the Internet.
How to Create, Edit and Upload WordPress Robots.txt File
WordPress gives you a default robots.txt file generated automatically without you enabling any settings but in order to tell search engine bot what to do you need to tell them now to crawl your website, when to crawl your website and how to crawl it and which page they are allowed to crawl.
WordPress robots.txt by default allows search engine crawl all your pages including, categories, pages.
This guide will put you through on how to create, edit and upload WordPress robots.txt file easily.
Before proceeding you should already have a sitemap enabled for website, if you have sitemap on your website you can follow this step now:
How Create, Edit and Upload WordPress Robots.txt File on Yoast SEO Plugin
- Install Yoast SEO plugin if you don’t have one
- Go to Yoast SEO plugin option on your sidebar
- Click on the dropdown menu indicating tools
- Click on File editor
- Click on create robotsx.txt then your website HTML version should be displayed next on your screen
- Once you click the create button, you will be able to edit your robots.txt file directly from same interface
- Now it’s time to upload WordPress robots.txt file now
Keep reading to see how to write your robots.txt command…
How Create, Edit and Upload WordPress Robots.txt File on All in One SEO Plugin
- Install All in one SEO plugin if you don’t have one
- Navigate to All in one SEO on your sidebar
- Click Feature Manager and activate Robots.txt
- After that you can manage your Robots.txt from all in one SEO robots.txt
You can also create your robots.txt using SFTP but I won’t be explaining that.
Below is how to the robots.txt code to upload on your wordpress robots.txt file looks like:
You can check out how my robots.txt looks like by visit https://www.deegeek.com.ng/robots.txt.
After copying the above robots.txt file command paste in your robots.txt and save changes that’s all then you can check your robot file by visit https://yoursiteurl.com/robots.txt.
After updating your robots.txt don’t forget to update your robots.txt on google search console also…
Robots.txt blocks crawling, but not necessarily indexing. You can use it to add specific rules to shape how search engines and other bots interact with your site, but it will not explicitly control whether your content is indexed or not.
Don’t misuse the Disallow command this might block Google from mis-crawling your website and might destroy your SEO.
Don’t forget to Leave a comment if you enjoyed this guide and need more understanding on any robots.txt query.