Category — search engines

1) Google Trends

You can use this tool to compare daily unique visitors to your site compared to multiple competitors. When you’re signed in with a google account, it will display volume numbers. You can segment by region or time period and also see what sites or search terms visitors used. As an example, I searched twitter.com.

Here are the results:

link: Here

2) Google Insights for Search

You can view trending of interest in your company versus that of your competitor’s company versus that of keywords of products or services in your category. It also allows you to filter by region. As an example, you could track the interest for pepsi in Brazil in the last 12 months and compare it to the interest in Canada.

Link Here

3) Google ad planner

This service is still in beta, but it can give you some valuable demographic information. You can: search for websites relevant to your audience, access aggregated statistics on the number of unique visitors, create lists of websites where you’d like to advertise and store them in a media plan, and generate aggregated website statistics for your media plan.

Link Here

4) Google alternative view for search results

This is an experimental service from google. You need to go to the link below and click on: “join this experiment”. You can search for keywords and specify a month/year filter.

Link Here

5) twist.flaptor.com

This site will allow you to search for daily trends on twitter. This includes a graph of the search term and individual tweets that contained it.

Link Here

What is robots.txt?

A robots.txt file on your website tells a search engine to ignore specified files or directories. You might want to create one to exclude certain content because it is private (such as an administrative interface) or that it is misleading or irrelevant to the categorization to your website as a whole.

You can create the robots.txt file manually, using any text editor. It should be an ascii-encoded text file and the filename should all be in lowercase. Once you’ve created your robots.txt file, save it to the root of your domain with the name robots.txt. This is where robots will check for your file. If it’s saved anywhere else, they will not find it.

Setting it up

• User-agent: the search engine robot
• Disallow: the URL that will be blocked

You can include as many entries as you want.

Blocking search engine bots

A comprehensive list of search engine bots can be found here

Blocking a page:
Disallow: /my_page.html

Blocking an entire directory (which includes all pages in it):
Disallow: /blocked_dir/

Block a site:
Disallow: /

To block files of a specific file type (.exe) from the google bot:
User-agent: Googlebot
Disallow: /*.exe$

More advanced settings (only works with the goolgebot)

Block access to all URLs that include a question mark:
User-agent: Googlebot
Disallow: /*?

If you have created a robots.txt file for your site and would like to check the validity of it, there is a free service that will do this for you here