Better Robots.txt (WP) plugin The simplest and easiest way to fully benefit from your website.

Better Robots.txt

UNIQUE Better Robots.txt was conceived to solve a specific lack of optimization which concerns most WordPress websites. This plugin is a source of SEO juice just waiting to be unlocked.

Better Robots.txt

VALUABLE Better-Robots.txt ensures that your website, no matter the content (and its quality) will always be crawled by search engines so that you can get the best of it.

Better Robots.txt

USER CENTRIC We conceived this plugin to be efficient and easy to configure. You don’t need to have any technical experience to use it. A few clicks and you are on business.

Entrepreneurs, bloggers, ... Whatever you do on WordPress, this plugin is a ``MUST HAVE``.

Did you know that... The robots.txt file is a simple text file placed on your web server which tells webcrawlers (like Googlebot) if they should access a file or not.

  • The robots.txt file controls how search engine spiders see and interact with your webpages;
  • This file, and the bots they interact with, are fundamental parts of how search engines work;
  • The first thing a search engine crawler looks at when it is visiting a page is the robots.txt file;
  • The robots.txt is a source of SEO juice just waiting to be unlocked.
Better Robots.txt

Did you know that in 2018 more than 20% of search queries are made on Google Images ? More info

Boost your Alt Texts for Google Images Auto-add Alt texts to all your image for pages, posts & products with BIALTY WP plugin

1. Boost your content with your sitemap ! Make sure your pages, articles and products, even the latest, are taken into consideration by search engines !

Better Robots.txt plugin was made to work with Yoast SEO plugin (probably the best SEO Plugin for WordPress websites) and ANY other sitemap generator. It will detect if you are currently using Yoast SEO (if the sitemap feature is activated) or if any other sitemap is available on your website.

If yes, then it will add automatically instructions in the Robots.txt file asking bots/crawlers to read it and check if you have made recent changes in your website (so that search engines can index the new content).

If you want to add your own sitemap (or if you are using another SEO plugin), then you just have to copy-paste the Sitemap URL (mywebsite.com/sitemap.xml) and Better Robots.txt will add it to your Robots.txt file.

Better Robots.txt

From Better Robots.txt PRO plugin

Better Robots.txt

From Better Robots.txt PRO plugin

2. Boost your Website indexing Get your website crawled by most popular search engines !

We have identified a list of most popular search engines (which will be updated frequently). Each time you select a bot, Better Robots.txt plugin add an instruction in your robots.txt file. Like this (if you add selected “GoogleBot”):

Better Robots.txt

This is a simple instruction asking Google to read and index your website.

We wanted it simple and we did it !

3. Protect your data and content Block bad bots from scraping your website and commercializing your information.

Better Robots.txt plugin helps you block most popular bad bots from crawling and scraping your data.

When it comes to things crawling your site, there are good bots and bad bots. Good bots, like Google bot, crawl your site to index it for search engines. Others crawl your site for more nefarious reasons such as stripping out your content (text, price, …) for republishing, downloading whole archives of your site or extracting your images. Some bots were even reported to pull down entire websites as a result of a heavy broadband…

Better Robots.txt plugin protects your website against spiders/scrapers identified as bad bots by Distil Networks. We will provide with time (and upgrades) a more complete and detailed list of bad bots blocked by Better Robots.txt.

Better Robots.txt

From Better Robots.txt PRO plugin

Better Robots.txt

Will your website support and look good on new dual screen phones?

Boost your responsive design Get Instant & dynamic mobile previews for 15 phone formats with MOBILOOK WP plugin

Other features Social Medias, store's crawlability, multisite (directory based network sites), crawler traps, ...

Test it or get it, but do something ! Don't take the chance of having useless content on your website.

FREE

$ 0

Crawling instructions for 16 search engines
Easy setup of crawl-delay
Spam backlink blocker
Custom rules management

PREMIUM

$ 64.99

Crawling instructions of 16 crawlers
Include popular chinese Search engines
Easy setup of crawl-delay
Sitemap detection
Bad bot blocker
Spam Backlink blocker
Backlink protector
Store’s crawlability optimization
Custom rules management
Crawler traps blocker
Crawling instructions for social medias
Growth hacking tools (150+)
Plugin recommendations
+ FREE A1 BACKLINK

TRIAL (14 days)

$ 0

Crawling instructions of 16 crawlers
Include popular chinese Search engines
Easy setup of crawl-delay
Sitemap detection
Bad bot blocker
Spam Backlink blocker
Backlink protector
Store’s crawlability optimization
Custom rules management
Crawler traps blocker
Crawling instructions for social medias
Growth hacking tools (150+)
Plugin recommendations

Better Robots.txt

NEW

Boost your Title Attribute for Google Images Auto-add Image Title attributes to all your image for pages, posts & products with BIGTA WP plugin

Best SEO practices with Better Robots.txt plugin Learn more about what to do with your robots.txt & your sitemap.

Looking for high quality backlinks ? Better Robots.txt and Partners provide A1 Bakclinks

Thousands of People and
Companies Love Using Better Robots.txt PRO

Other WordPress plugins (SEO) ? Alt Text, Responsive design, Title attribute ...

Better Robots.txt

BIALTY Bulk Image Alt Text (Alt tag, Alt attribute) for Google Images

Better Robots.txt

BETTER ROBOTS.TXT Boost your ranking on search engines with an optimized robots.txt

Better Robots.txt

MOBILOOK Get Dynamic/Instant Mobile Previews of your Websites (+Facebook debugger)

Better Robots.txt

APP-ADS.TXT App-ads.txt & Ads.txt Manager for WordPress & advertising

Better Robots.txt

VIDSEO Video SEO embedder – player (Youtube, Vimeo) with video transcription

Better Robots.txt

BIGTA Bulk Image title attribute (Title text, Title tag) for Google Images

Click here to see Video transcripts and all the content that could be added to your website from Automatic Youtube Video... ▼

Click here to see Video transcripts and all the content that could be added to your website from Automatic Youtube Video transcription

 

Robots.txt explained PRO

Welcome to this week's web design video blog.

Today Nick and I are going to explain the purpose of the robots.txt file and also share the common rules that you might want to use to communicate with search engine robots like Googlebot.

Better Robots.txt

So the primary purpose of the robots.txt file is to restrict access to your website by search engine robots or BOTS. The file is quite literally a simple dot txt text file that can be opened and created in almost any notepad HTML editor or word processor. To make a start, name your file robots.txt and add it to the root layer of your website. This is quite important as all of the main reputable search engine spiders will automatically look for this file to take instruction before crawling your website. So here's how the file should look to start with. On the very first line: add "User-agent". This first command essentially addresses the instructions to all search BOTS. Once you've adjust a specific or in our case with the asterisks all search BOTS you come on to the allow and disallow commands that you can use to specify your restrictions. To simply ban the bots from the entire website directory including the home page, you'll add the following code: disallow with a capital of D : space and a forward slash. This first four slash represents the root layer of your website.

In most cases you won't want to stripped your entire website just specific folders or files. To do this you specify each restriction on its own line proceed with the disallow command. In example here you can see the necessary code to strict access to a folder called admin along with everything inside it.

If you're looking to restrict individual pages or files the format is very similar online for we're not restricting their entire secure folder just one HTML file within it. You should bear in mind that these directives are case sensitive. You need to ensure that you specify on your robots.txt file the exact match against the file or folder name of your website. So those are the basics and next we come onto slightly more advanced pattern matching techniques.

These can be quite handy if you're looking to block files or directories in bulk without having lines of commands on your robots file. These bulk commands are known as pattern matching and the most common one that you might want to use would be to restrict the access to all dynamically generated URLs that contain the question mark for example. So if you check out line 5 all you need to do to catch all of these type is the forward slash Asterix and then a question mark symbol. you can also use pattern matching to block access to all directories that begin with admin for example: if you check out line 6 you'll see how we've again use an Asterix to match all directories let's start with admin.

So again if for example you had the following folders on your root directory admin - panel admin - files and admin - secure then this one line on your robots file would block access to all three of these folders as they all start with admin. The follow pattern matching command is the ability to identify and restrict access to all files that end with a specific extension. On line 7 you'll see how the single short command will instruct the search engine BOTS and spiders not to crawl and ideally cache any pages that contain the dot PHP extension. So after your initial forward slash, use the asterisks followed by a full stop and then the extension. To signify an extension instead of a string you can clued the command with the dollar sign. The dollar tells the bots that the file to be restricted must end with this extension.

So that's it using those six different techniques you should be able to put together a well optimized robots.txt file that Flags content to search engines that you don't wish to be crawled or cached.

It is important that we point out however that website hackers often frequent robots.txt files as they can indicate where security vulnerability may lie that they might want to throw themselves at. Always be sure to password protect and test the security of your dynamic pages particularly if you're advertising their location on a robots.txt file.

Thank you for watching this week's web design video blog

VidSEO plugin (NEW)

Skyrocket your SEO with Transcripts Get access to FREE & non-indexed content from Youtube with video transcripts (see example)

Great support to optimize your Robots.txt We will help you fix any WordPress indexing issues with our plugin. Let us know how we can help you!

Better Robots.txt

What people think about Better Robots.txt plugin