If we look at how one would create a Search Engine we might find out how they must operate, or even how they could be improved. As much as it might appear that Search Engine results are mysterious, they can be broken down to their component parts and understood. Once you add simple on top of simple you tend to get complex, and the whole system in operation can appear to be daunting. If we break it down to its components, processing power, storage capacity and specialty algorithms these all just become parts of a larger understandable system.
From the point of view of a Search Engine one of the easiest ways to evaluate a general array of web related documents for various factors would be to use a baseline to measure all others against.
For instance if we take keyword density. Look at some well known documents in literature, or newspaper articles, or magazine pieces. Wouldn't that standard, perhaps something written before the web era illustrate a natural level of keyword density for all other documents?
Using this technique you could easily eliminate as spam many documents that would fall outside the range of these general documents. Another possibility would be to take an article written without Optimization. A sample Wikipedia article could provide such a baseline.
The Wikipedia article about “Tennis” provides some indication of a keyword density for an encyclopedic entry on the web.
This document is 9141 words long. This includes all links and anything else on the page, not just the main article, since in evaluating a web document the source includes everything being displayed. | Term | Keyword Density | Times Occurring
| | Tennis | 1.8% | 165 | | Court | 0.6%
| 54
| U.S. Open
| < 0.1%
| 9
| Racquet
| < 0.1% | 5
| Racket
| < 0.1% | 2
|
In looking at the Wikipedia "Tennis" article we determined the keywords from the title and general topics covered. Over all such an article is very broad, covering the whole subject in a lot of text. The second "article" we looked at was Henry David Thoreau's "On the Duty of Civil Disobedience", or also called "Resistance to Civil Government". This essay is 9408 words long, and because it is pre-web writing (1849) there is no navigation or links included in the word count. | Term | Keyword Density | Times Occurring
| | State | 0.6% | 61
| | Government | 0.62%
| 58
| Slavery
| 0.12%
| 11 | American
| < 0.1% | 8
| Civil
| < 0.1% | 6 | | Constitution | < 0.1% | 6
| Revolution
| < 0.1% | 6
| | Disobedience | < 0.1% | 3
|
When you evaluate an older piece, like something from Thoreau you begin to realize the way modern documents are organized to reveal much smaller specific pieces of information. Essays like Civil Disobedience are relatively long and were not designed for web usage and these essays were frequently read aloud as persuasion pieces. Essays, like speeches in this sense are not search friendly, they have many ideas, but very few facts. This type of writing does not answer questions, it incites discussion. Even blogs which are generally not search friendly are very specific in nature.
The next piece we decided to look at in the pre-web era was something more akin to modern writing formats. This is the book review, written by Lance Morrow in 1976 for Time magazine analyzing two new books about the Soviet Union. At 1217 words it is more in line with a modern web piece than something from Thoreau. The Inscrutable Soviets | Term | Keyword Density | Times Occurring
| | Russia | 1.4% | 17
| | Soviet | 1%
| 13
| America
| 0.2%
| 6
| | Moscow | 0.4% | 3 |
It is easy to see the keywords, as derived from the title or the article, as well as the titles of the books reviewed are used very infrequently relative to modern web specific promotion and short web pages.
Conclusion: The web has forced a world in which documents must become smaller, more highly defined in order to be found by Search Engines and in turn by web users. Most people perform searches that require a specific answer. - The score of the latest game.
- A date in history that refers to a specific event.
- Product information about a small range of specific desires.
Basically, highly specific pages need to exist for a Search Engine to be able to properly answer the above queries. Keyword Density is a possible factor but as we can see from the above array of articles and web pages this is only one factor. And to determine how much of a factor depends on the situation, but over all it would be a very small one and by definition would not define, or help to define longer more conceptual documents such as Thoreau's "Resistance to Civil Government". A Search Engine could certainly use Keyword Density as a factor, even a large factor if they wish but you might see that whatever density level you set for keyword phases they are rather arbitrary. Unless you could successfully segment content into Articles or Newspapers, Historic Literature or Product Literature you would not want to utilize the concept of Keyword Density, or it may be to the detriment of other types of information.
|