Wondering why you’re not visible in Google search results? If this is the case, it is possible that search engines do not have the faintest idea of the existence of your pages.
You own a WordPress website, you are an entrepreneur, head of a small business or a simple online shop or you are self-employed and you live on your website (or want to live it). You’ve heard talking about SEO, your agency or developer gave you advice, you read a lot on the subject but still, you can’t stand out on Google. There may be a reason for that!
Did you know that according to a study carried out in 2017, on a very large number of websites, nearly 95% of the WordPress sites audited all had a recurring “problem” that prevented them from fetching the maximum performance and visibility of their content? This “error”, which is more akin to a problem of configuration, has two consequences:
- A difficulty for crawlers (Google, Bing,…) in finding your content
- An obstacle to identifying new content on your site for indexing robots.
Results? Your recent articles, products or services do not appear on Google.
What is it about?
Each website presents at its root (on hosting server) a file called “robots.txt”. This text file, of an extreme simplicity, which most of the time is either misconfigured or totally forgotten, turns out to be one of the most important part of your website. And that’s an understatement because it’s the only file of your website that actually “talks” to search engines!
When your site is “visited” by a search engine, in a systematic way and before any other operation, it searches in the root of your website the robots.txt file in order to read its content. So this is the first file read and loaded related to your site. And the reason why search engines read this file is that it contains information, rather instructions, strictly to their intent.
These instructions are simple but essential:
- What can I read and index
- What should I not read and index
In principle, if you have a website, you want your content to be taken into consideration by search engines in order to be found in search results by your visitors. However, for this to be possible, it is necessary to tell crawlers what they can do and what they can’t do. And how important it is to be seen on Google, it’s just as important that some part of your site is not accessible (to protect some data, insignificant internal pages, etc.)
Also, given that robots.txt is the first scanned data from your website, it is crucial to be able to inform search engines of new developments recently put in place on the website (new product or addition of articles, etc.) And it is once again the robots.txt file, via the “Sitemap”, which allows it.
In short, you might as well say that this file is of paramount importance. Yet while the strict reason why you make a website is to be found on the web, this simple file that serves just that, happens to be neglected or forgot more than 9 times out of 10! And you are part of that statistic, for sure!
Better Robots.txt plugin
That’s why we designed Better Robots. txt, a simple plugin that allows:
- To embed in your robots.txt indexing instructions for more than fifteen indexing robots, among the most popular (informing them of what they can index or not);
- To detect and inject the URL of your Sitemap (index) in the robots.txt file (100% compatible with YOAST SEO plugin);
- To define a crawl delay (allowing to set the number of seconds that robots must wait between each successive query) avoiding overloading your server with too many indexing requests;
- To block unwanted indexing robots (Bad bots) – scrapers,
- To directly edit the content of your Robots.txt to customize more advanced instructions;
Tested on several hundred websites, no matter the area of activity, the simple configuration of this file allowed, in 100% of cases, to boost the organic ranking of the websites in question (until doubling it in some cases). What for? Because, in a few clicks, you have brought crawlers to discover or rediscover the content of your website by unblocking it!
It is not an “obscure hack” but a simply unknown SEO technique (see this article by Neil Patel).
So now that you know more, what does your robots.txt looks like?
To identify the content of your robots.txt (and also see if it exists), you need to copy and paste the URL of your Website (with http://or https://) in the fields shown below:
By way of comparison, here is ours: https://www.better-robots.com/robots.txt. Do you see the difference?
Wondering if your agency knows about this? Test their website with this tool. If you see that this is not the case, our plugin will allow you to solve this problem!