This is the second in a series of articles attempting to look at how programs are built and how one would begin to think about building their own search engine. This article is about understanding the nature of taking shortcuts and how the major search engines need these to perform up to current expectations.

Part I: Creating a New Program

Part III: How Search Engines Look at Information

The first thing you want to do is determine what you want to accomplish. If I were starting my own search engine then I first want to think about what exists and why. What don't I like about Google or Yahoo! search or MSN search.

In regards to Google and the other major search engines, everyone dislikes bad results. But you have to determine what is a bad result. If you are searching for contemporary information then for the most part the results are easy to find. This is especially true with Google's search engine, since they frequently source news collections sites, such as Digg. And it is a point like this that needs to be highlighted. This is one of many shortcuts that Google takes to have better results. If there were no public relation sites, such as openpr.com or news aggregators, such as digg.com then Google would have a tougher time getting those results to the public. They would have to work much harder at understanding what is todays news and what is yesterday's news. And yesterday's news is just history, locked in documents called web pages.

So news sites, public relation sites and so on are shortcuts that Google has capitalized on to make sure their results are better, then if they were simply relying on their own algorithms. Another of these shortcuts are separate programs to interpret certain types of search queries. Let's use the example:

“grams in a pound”

The first thing you will notice is that they get the right answer. The next thing you should realize is that they had to write a separate program to deal with that type of language. They provide their traditional results, but also the answer to your question.

But if you try a little different search, say “1 pound is x grams”, they do not translate this into their program. You just receive the traditional results - and this is a matter of differentiating language, something which at this point needs to be very hard-coded.

So lets say that:

1.We want to produce good results
2.We don't want to write separate programs specific to certain topics

But we must understand the nature of shortcuts, and since the web changes (i.e., all sorts of people out there are adding all sorts of new structures all the time), the search engine's role must adapt. I suppose the ideal situation is some all-encompassing artificial intelligence that would adapt on its own, understanding that the web changes and that people are changing it. But for now lets further understand the use of shortcuts and how to make them work better.

This is the first in a series of articles attempting to look at how programs are built and how one would begin to think about building their own search engine.

If you have the opportunity to build a program, you will notice that there are few resources to help you think about the overall structure. There are plenty of web resources for specific issues, but not too many when you want to think about the overall idea of how things should fit.

For web program ideas there are many open-source programs you can open up and look at – oscommerce, pligg, wordpress...etc but in some cases you are starting from scratch and these open-source programs are rather all over the place. A lot of companies use these open-source programs and end up hiring professionals to do any modifications, even just color changes. Anyway, open-source always has the downside of being a little too understandable to just about anyone with a little knowledge, and that may not be a good thing for some applications. So you could modify any of these open-source programs for your own purpose, but that may just drive you in the wrong direction.

Some projects require SEO, and some programs do not. SEO may also steer you away from some open-source applications, since you might want to be innovative. Some programs are needed because they are for a job which may have strict requirements. But there are some projects that you just do for yourself.

Perhaps, if you are like me you want to make something that you need for yourself, or maybe you are just interested in creating something new and interesting. If you are the type that likes to get into new projects, which may or may not result in a business, then you are generally starting from scratch.

There are many programs that need creating, or improving, whether they are just something small or if they are something large. Many people have been thinking about, or are currently in the midst of creating their own search engine, simply because they do not like the way the current crop of companies deal with information. But when you think about creating your own program you may come with all sorts of opinions, but soon you will face a reality.

When you are thinking about the overall structure of a project you should realize that the thinking part will consume a great deal of time. Why are you doing it? What exactly do you want to accomplish? So the philosophy is a major part, and that can take months, then a little programming, then some more thinking.