Better Robots.txt The simplest and easiest way to fully benefit from your website.

100% compatible with 

By optimizing your robots.txt the right way, you can tell search engine bots to spend their crawl budgets wisely. And that’s what makes the robots.txt file so useful in a SEO context.

Better Robots.txt

UNIQUE Better Robots.txt was conceived to solve a specific lack of optimization which concerns most WordPress websites. This plugin is a source of SEO juice just waiting to be unlocked.

Better Robots.txt

VALUABLE Better-Robots.txt ensures that your website, no matter the content (and its quality) will always be crawled by search engines so that you can get the best of it.

Better Robots.txt

USER CENTRIC We conceived this plugin to be efficient and easy to configure. You don’t need to have any technical experience to use it. A few clicks and you are on business.

FIRST STEP: let's see if you have a robots.txt and what's in it. You can enter a website below, click submit and it will detect if the site has a robots.txt file and display what the file says (it shows results here on this page).

You don't have any Robots.txt file (404 error)?

You are in a real trouble ! And Better Robots.txt can create one for your website.

You have these 2 lines ?

Better Robots.txt

Great ! But this is the basic skeleton of a robots.txt file. And Better Robots.txt plugin can optimize it.

Did you know that... The robots.txt file is a simple text file placed on your web server which tells webcrawlers (like Googlebot) if they should access a file or not.

  • The robots.txt file controls how search engine spiders see and interact with your webpages;
  • This file, and the bots they interact with, are fundamental parts of how search engines work;
  • The first thing a search engine crawler looks at when it is visiting a page is the robots.txt file;
  • The robots.txt is a source of SEO juice just waiting to be unlocked.

HOW TO OPTIMIZE YOUR ROBOTS.TXT?

Entrepreneurs, bloggers, ... Whatever you do on WordPress, this plugin is a ``MUST HAVE``.

Better Robots.txt

Google traffic from a website (in June 2018) optimized with Better Robots.txt.

1. Boost your content with your sitemap ! Make sure your pages, articles and products, even the latest, are taken into consideration by search engines !

Better Robots.txt plugin was made to work with Yoast SEO plugin (probably the best SEO Plugin for WordPress websites). It will detect if you are currently using Yoast SEO and if the sitemap feature is activated.

If yes, then it will add automatically an instruction in the Robots.txt file asking bots/crawlers to read the sitemap and check if you have made recent changes in your website (so that search engines can index the new content).

If you want to add your own sitemap (or if you are using another SEO plugin), then you just have to copy-paste the Sitemap URL and Better Robots.txt will add to your Robots.txt file.

From Better Robots.txt PRO plugin

Better Robots.txt

From Better Robots.txt PRO plugin

2. Boost your Website indexing Get your website crawled by most popular search engines !

We have identified a list of most popular search engines (which will be updated frequently). Each time you select a bot, Better Robots.txt plugin add an instruction in your robots.txt file. Like this (if you add selected “GoogleBot”):

Better Robots.txt

This is a simple instruction asking Google to read and index your website.

We wanted it simple and we did it !

3. Protect your data and content Block bad bots from scraping your website and commercializing your information.

Better Robots.txt plugin helps you block most popular bad bots from crawling and scraping your data.

When it comes to things crawling your site, there are good bots and bad bots. Good bots, like Google bot, crawl your site to index it for search engines. Others crawl your site for more nefarious reasons such as stripping out your content (text, price, …) for republishing, downloading whole archives of your site or extracting your images. Some bots were even reported to pull down entire websites as a result of a heavy broadband…

Better Robots.txt plugin protects your website against spiders/scrapers identified as bad bots by Distil Networks. We will provide with time (and upgrades) a more complete and detailed list of bad bots blocked by Better Robots.com.

From Better Robots.txt PRO plugin

better Robots.txt

From Better Robots.txt PRO plugin

4. Hide & protect your backlinks Avoid competitors from identifying your profitable backlinks

Backlinks, also called “inbound links” or “incoming links” are created when one website links to another. The link to an external website is called a backlink.Backlinks are especially valuable for SEO because they represent a “vote of confidence” from one site to another. In essence, backlinks to your website are a signal to search engines that others vouch for your content.

If many sites link to the same webpage or website, search engines can infer that content is worth linking to, and therefore also worth surfacing on a SERP. So, earning these backlinks generates a positive effect on a site’s ranking position or search visibility. In the SEM industry, it is very common for specialists to identify where these backlinks come from (competitors) in order to sort out the best of them and generate high quality backlinks for their customers.

Considering that the creation of very profitable backlinks, for a company, takes time (time + energy + budget), allowing your competitors to identify and duplicate them so easily is a pure loss of efficiency.

Better Robots.txt helps you block all SEO crawlers (aHref, Majestic, Semrush) to keep your backlinks undetectable.

Test it or get it, but do something ! Don't take the chance of having useless content on your website.

FREEMIUM

$ 0

Available for download freely on WordPress
Allow to add indexing instructions to 16 crawlers
Easy setup of crawl-delay

PREMIUM

$ 45

Only available on Better-Robots.com
Allow to add indexing instructions to 16 crawlers
Easy setup of crawl-delay
Verify your sitemap index from Yoast plugin
Add the sitemap URL to your Robots.txt
Block scrapers (Bad bots)

Best SEO practices with Better Robots.txt plugin Learn more about what to do with your robots.txt & your sitemap.

Better Robots.txt

Keep up with your Performance Stay in the loop for the latest Better Robots.txt news and features.

Great support to optimize your Robots.txt We will help you fix any WordPress indexing issues with our plugin.

Better Robots.txt