Skip to main content

SEO Tips about Robot.txt | Statement meaning of Robot.txt

This is a SEO tips. It is a must requirement for an SEO to prevent indexing of search pages. Google webmaster tools do this automatically. If it is not done then use robot.txt to prevent custom search pages on your blog or website. Its code is like this. This can be tested in google webmaster tools whether the google bot is indexing or not.

To see the websites' robot.txt file write this in the address bar of the browser. E.g.

Read these instruction carefully to understand the robot.txt statements. (Resources google and other branded websites)

What do the robot instructions mean?

Here is an explanation of what the different words mean in a robot.txt file

The "User-agent" part is there to specify directions to a specific robot if needed.

There are two ways to use this in your file.

If you want to tell all robots the same thing you put a " * " after the "User-agent" It would look like this...

User-agent: * (This line is saying "these directions apply to all robots")

If you want to tell a specific robot something (in this example Googlebot) it would look like this...

User-agent: Googlebot (this line is saying "these directions apply to just Googlebot")


The "Disallow" part is there to tell the robots what folders they should not look at.

This means that if, for example you do not want search engines to index the photos on your site then you can place those photos into one folder and exclude it.

Lets say that you have put all these photos into a folder called "photos".
Now you want to tell search engines not to index that folder.

Here is what your robot.txt file should look like:

User-agent: *
Disallow: /photos

The above two lines of text in your robots.txt file would keep robots from visiting your photos folder. The "User-agent *" part is saying "this applies to all robots". The "Disallow: /photos" part is saying "don't visit or index my photos folder".

Googlebot specific instructions

The robot that Google uses to index their search engine is called Googlebot It understands a few more instructions than other robots. The instructions it follows are well defined in the Google help pages (see resources below).

In addition to the "User-name" and "Disallow"

Googlebot also uses the...

The "Allow:" instructions lets you tell a robot that it is okay to see a file in a folder that has been "Disallowed" by other instructions.

To illustrate this, let's take the above example of telling the robot not to visit or index your photos. We put all the photos into one folder called "photos" and we made a robot.txt file that looked like this...

User-agent: *
Disallow: /photos

Now let's say there was a photo called mycar.jpg in that folder that you want Googlebot to index. With the Allow: instruction, we can tell Googlebot to do so, it would look like this...

User-agent: *
Disallow: /photos
Allow: /photos/mycar.jpg

This would tell Googlebot that it can visit "mycar.jpg" in the photo folder, even though

Nokia E6
<a href="">Free Essay</a>


Popular posts from this blog

How to pay SAIL Mediclaim Insurance coverage Premium

It is that time of the year when ex-employees of Steel Authority of India (SAIL) as well as its steel plants like Bhilai (BSP), Rourkela (RSP), Bokaro, Durgapur, Salem, etc; run around to restore their medical insurance coverage premiums under the SAIL Mediclaim Scheme. Ex-employees are generally under stress to get the mediclaim renewed for themselves and also their partners as they rarely get the renewal types on time, by message. Over these years, we have seen that Insurance Companies and also Third Party Administers (TPA) have not been extremely customer pleasant however, for the last few years, the renewal type is available online on the SAIL site and also there is additionally a stipulation to make the premium payment online through State Bank of India (SBI). The online premium payment is a extremely simple and practical choice yet considering that a lot of the ex-employees might not be really Internet savvy, I think the ideal option would certainly be for their youngs

How to Change Adsense Payment Method | How to fill Adsense Payment Detail | Wire Transfer to Receive Money to your Account

Google Adsense made it very easy to receive Adsense payment directly to your account. It is important to fill the account details carefully. A fear is always carried in our mind whenever we perform a financial transaction and there remains a risk of money loss. But to overcome this psychological fear, we need to do a research. First go to your bank and get the SWIFT details of the bank. IFCI code is already written on your bank passbook and also available on Chaque leaflet. Therefore the Straight rule for this is to do a good homework for filling SWIFT details for WIRE TRANSFER. Before this first see the form of Adsense. You can reach to this form in these steps: Go to setting > Payment > click on Payment Method > Click Add payment method.Your will see the form as in this image at left. A sample data has been filled up in the given form. You need to change the details as per your bank details. After filling your details of a primary account, a secondary acc

Use Mobile Marketing To Your Benefit With Home Based Business

Mobile Marketing-- Just What Is It?. Mobile marketing is all over. We can see it on our portable devices, on our tablets, pads and even in Cyberspace. Little do we know that we are currently engaged in mobile marketing, nevertheless we don't actually know what it is that we are getting involved in. On the side of the online marketers and company owner, they often disregard the aspect of mobile marketing, which causes a loss of opportunity and higher reach. In order to set the record directly, various questions shall be answered connecting to what mobile marketing is and what are the typical strategies are behind it. Mobile marketing is a marketing method that involves mobile phones. This does not just restrict to cellphone, but also covers handheld devices like tablets, pads, and other cordless devices. Another known meaning of mobile marketing is the use of the mobile medium as a method to market. This includes circulation of any advertising messages or advertisemen

How to auto fit Excel cell to text width while copy paste