PowerMapper Software Blog

• W3C recommends a maximum of 64 characters for page titles.
• Bing recommends a title between 5 and 65 characters long.
• Yahoo recommends a maximum title length of 67 characters (although this advice is obsolete since Bing now supplies Yahoo's search results) 
• Google don't have any guidance for content publishers, but recommend a maximum title length of approximately 60 characters for Google's own pages.

• Bing - titles up to 70 characters are displayed (previously longer titles were truncated to whole words around 53 characters, but this no longer happens)
• Yahoo - results are now provided by Bing (since mid-2010)
• Google - titles up to 71 characters are displayed, longer titles are truncated to whole words, with the following exceptions:
  
  • Results included from Google patent search - patent title is limited to 71 characters but "Google Patent Search" is added to the end of the 71 character title.
  • The title limit used to be 66 characters, and some old documents remain in the index with titles still truncated at 66 characters

Most computers will open PDF documents automatically, but you may need to download Adobe Acrobat Reader.

Update: Orginally posted April 2010, updated April 2011

In Part 1 we covered how a search engine crawler visits web pages. In this part we're going to investigate how words on web pages are indexed.

You'll recall the three phases of search engines:

• Crawling (or spidering) the web, finding pages people want to search
• Indexing words on web pages
• Searching the index (i.e. the bit that happens when you type a search into Google)

A search engine index works very like the way the index in book works: in a book each word in the index lists page numbers the word appears on; in a search index each word has a list of pages the word appears on.

Here's what happens when you search for "blue widgets":

1. Get the list of pages containing the word "blue"
2. Get the list of pages containing the word "widgets"
3. Return the pages that appear in both lists

The really clever stuff on search engines happens when deciding which pages are most relevant and get listed first, which we'll cover in Part 3.

Having some background on how search engines work is very useful when you're trying to optimize your site. We hope we've a bit of perspective on this, having spent the best part of a decade implementing search engines and web crawlers.

How do words on a web page end up searchable? This happens in three phases:

• Crawling (or spidering) the web, finding pages people want to search
• Indexing words on web pages
• Searching the index (i.e. the bit that happens when you type a search into Google)

The two important components here are the web crawler (which we'll cover in this post) and the index (which works just like an index in a book, and we'll cover in the next post).

The Web Crawler

The web crawler has a very simple job to do: it visits each page on the web and does two things:

• Adds words on the page to the index
• Adds links on the page to the list of pages to visit

Once it's finished with a page it moves to the next page on the list to visit and repeats the process.

It all sounds simple, so what can go wrong?

Note: In reality, it's a bit more complex since large search engines like Google speed things up by running multiple web crawlers, and store multiple copies of the index, but the basic process is the same.

Extracting Text and Links: HTML

When a search engine scans a page for text, it's looking for words in the between your HTML tags. It also extracts links from tags like <a href="somelink.htm">

As an example, consider the tag above. If you miss out the closing quote from the link name you end up with broken HTML, but some browsers (e.g. Internet Explorer) have error handling code that detects this and guess that the closing angle bracket ends the link. Most search engines are less forgiving and include the angle bracket and all the text that follows it as part of the link. This means somelink.htm won't get indexed.

Extracting Text: JavaScript

Content created by JavaScript is mostly invisible to search engines. In particular, they can't "run" the JavaScript to produce HTML. The best they can do is try extracting links and words from strings embedded in the JavaScript.

Note: There are lots of reasons for this but security is one of them. How many recent browser security holes were down to JavaScript problems? 80%, 90%, more?

Extracting Text: Images

This presents a problem if all your text is in images - the search engine can't see any text in your images - though some search engines index ALT text and the image file name (notably Google Image Search).

Extracting Text: Flash

Flash is problematic, although some (but not all) search engines can extract text and links from Flash movies. Don't depend on this though - the search engines use a tool provided by Adobe which does a good job most of the time, but it can't read all Flash movies and crashes in some cases. You won't see the crash of course, but the text of movies triggering the crash will never appear in any search indexes.

Preventing Indexing Problems

So what can you do to stop this happening? Finding all these problems manually is very difficult, but by amazing co-incidence we produce a tool called SortSite that checks every page on web site for these sorts of problems.

Google wants sites in its index, but it doesn't want sites that use sneaky techniques to increase their rankings. If Google detects a site using these techniques, they penalize the site's ranking (or remove it altogether).

Google has a set of freely available guidelines on the sort of techniques they frown on, but it's often difficult to know when you're breaking them - especially if you're not the one writing the site code.

Matt Cutts (a Google engineer) made a posting 18 months ago showing a site with a ranking penalty due to keywords stuffed into 1 pixel high text.

By a strange quirk of fate, the example he chose was www.villamagdala.co.uk - the hotel where my wife and I stayed the weekend I proposed to her. At the bottom of their home page they have a bunch of keywords in 1 pixel high white text. They still don't know they've been penalized - the hidden keywords were still there a year and half after Matt's posting.

Now if the hotel owners or site developers had used our excellent site optimization tool, SortSite, they'd see they were breaking Google's guidelines on hidden text (and would know about a bunch of other stuff like broken links and usability problems).