Waiting for the Ultimate and Best Search Engine

Welcome reader! Like all other weeks, today I will have another round of personal conversation with you. Strictly non personal finance. Today I wish to talk about search engines and our, bloggers, struggle to appear at the top of search results.

Readers who are not yet familiar with the concept of SEO (Search Engine Optimization), SEO is a technique which puts a particular site ahead of other sites in a search result. SEO technique essentially means cracking the algorithms that search engines like Google, Bing or yahoo use to rank search results.

There are serious flaws in today’s search algorithms, better article often get placed after not-so-better articles. Throughout this article I would give reference to a search term “Best credit cards”. I would mostly mention Google, but it implies all search engines, not just Google.

Searching Google for “Best credit card” brought me a 2008 and a 2009 article on the front page. They were from reputed financial newspapers, but their coverage is now outdated and the searchers will not get any benefit reading them in 2012. There are millions of other search terms for which search engines pull up totally irrelevant result. There is only one possible reason I could think of.

Improper/inaccurate search algorithm.

Most of the Google policies/algorithm is unknown to the public. I see the main reason being the nature of the business. The same way car manufacturers do not disclose their technology or latest innovation, Google keeps their algorithms close to their chest. The other reason could be to prevent unwanted exploitation (hacking) to break the ranking algorithm.

If you know what factors exactly determines the search ranking, you would start devising ways to improve upon those factors. For years Google thought that incoming links to an article determines the popularity or quality of the article. publishers/advertisers started gaming them with buying links, creating network of publishers to link to an article or even creating various dummy sites just to create in-bound links to articles they wanted to rank higher in searches.

Example, I can write an article on “Best credit cards” and ask all my blogger friends to link to it from their sites. I can even buy links from other finance sites for my article. Within a few months and a few thousand dollars later I may get into first 20 Google search results for “best credit cards”.

The beauty of this arrangement is, once I start ranking high with a powerful search term like “best credit cards”, top card issuers would start approaching me to put banner advertisement on my article. That would in turn earn me more money than I had originally spent to get a higher search rank.

Now imagine I had written a superfluous and vague article that do not exactly give any guide to readers as to what could the best credit card be, or, what could the criterion to select better cards be. Readers would have no value reading my article.

If similar practice becomes rampant, searchers would lose interest in Google and invariably Google, as a company, would suffer. I am sure these are the same vocal tonic that are given to Google employees by their management “Improve our algorithm so that Google continue to give best search result, compared to our competitors”.

So what would Google do?

They know very well what to do. No matter what improvement they do people will find ways to game them. In 2012, online ad business is expected to touch $40B. I am sure serious money is being invested to analyze, diagnose, tear-apart and reverse engineer Google search algorithm each time they do an update to the existing ones. Be Panda or penguin, people will find ways to score higher ranks.

Google actually chose to do policing rather than trying to improve its algorithm. Its putting enormous resources on to stop some of the SEO tactics which exploit Google’s algorithmic weaknesses. Google would still be working towards making search algorithm stronger, but nothing would be complete till it determines an article’s effectiveness like a human being. They would probably rely on policing till they find out “Moksha” (the ultimate state).

When we read a blog post, we do bookmark occasional ones when we find it exceptionally good and useful. Google bots should be that capable. With various upgrades, they are moving towards that or at-least trying to reach there.

I heard they have discarded some ridiculous factors like “keyword density” in their search algo. In past, somehow they found out that when you are writing about “Best credit cards”, the phrase “best credit cards”, should appear between ‘X’ and ‘Y’ times within the article body to be considered as a better article.

Well, I could have written a perfect post on “best credit cards” without mentioning that phrase anywhere, and guess what, I would have sat in a ‘ghost town’ of Google search result, probably on 100th page, where nobody goes to.

Agreed, it’s not easy to replicate human intelligence in machines. We may lose to computers in the game of chess but, we are more powerful than machines when it come to intuition, judgement and inference.

When we read something we process millions of information in our subconscious mind. To determine the effectiveness of a write-up we go back years in the past and compare with everything we know since that point. It’s not easy for a computer to do the same thing in fraction of second.

Yes Google bots spend only that much time on a particular webpage when they go out for acquiring updates to all the indexed pages Google has. Number of links to a domain or an article is a good measure compared to any other method to determine an article quality, but it is only good in absence of other better robust alternatives.

If I start a site today and write a splendid article on “best credit cards” Google is not going to recognize me at all because my domain and article is just a new kid with no incoming links to it. A human on other hand, would readily bookmark my post and base most future credit card application decisions on my article. Who is the loser? Google, the searcher and I – all.

As the ending thought, I feel sorry for million dollar start-ups who do nothing but analyzing Google algorithms and developing custom ranking systems based on interpolation/extrapolation of sample search results. So far they are accurate in predicting authority sites. A high domain authority does reflect higher search traffic, but for how long SEO MOZ? or for that matter every other SEO firms?

Once Google (or any other search engine)  acquires human intelligence in their algorithm, we wouldn’t need those parasytic companies.

Sunday chit-chat is my only opportunity to break free of real-life financial facts I share on this blog. In reality I am a thinker and often I fly to faraway lands to re-establish world order. On one such flight I was thinking about a new technology start-up developing this human like algorithm, this start-up would take away all traffic from Google, Yahoo, Bing etc.

That day we, the bloggers, would stop thinking about links, SEO and other metrices. We would only concentrate on writing better content. Still, things would not be smooth, a portion of that $40B online ad revenue actually belongs to SEO optimization, a part of which come to us in terms of additional monthly income from the blog. That opportunity would cease to exist.

That is acceptable to me as long as I am capable of writing quality articles to attract banner ads from advertisers. I welcome that tomorrow.

Readers, I want to leave you today with a question to think on, if possible write a comment if you find an answer. Why do you think Warren buffet never invested in tech companies like Microsoft, Google or Apple?