Recently in Hyperix Category

A lot has been written about cloud computing in the last year and each day seems to bring news of a new player in the cloud arena. So what does the cloud have to offer search engine companies like Hyperix? Well that depends on how deep our pockets are. After all, we need a lot of bandwidth, processing power and data storage to run any real search engine. And as we don't have deep pockets, nor an angel or venture firm backing us we've had to be find creative solutions and innovate where possible.

Up to this point we've been focusing solely on the technology we're using that will differentiate ourselves from any other vertical search platform entities out there. We've got our own small web crawling cluster setup which we've used for some time to test different web crawlers, collect and parse data and measure a variety web crawler values which determine how many CPU cycles, RAM, bandwidth, and storage is necessary to create the vertical search indexes we want. We've also been focusing on the quality of the data we're crawling, the algorithm which ranks the pages crawled, the parsing engines, and the results pages.

Since we're getting closer to the public unveiling of the first vertical search engine produced from the Hyperix search platform I think it's time to release a couple of tidbits about the platform. Hints have been out there for a while and some people within the search community are already aware of this, but one of the primary components of Hyperix is the use of Hadoop.

Here's the quick intro of what Hadoop does from the open source site:

For my first post in 2008 I've decided to write a little note about opportunities for vertical search platforms such as the one we're developing at Hyperix.

I've said for years now that while the big search engines do a great job for some things they do poorly at others and that many opportunities exist for those who go the vertical search route as a business model. There's a lot of data out there that can be made into a business product. Today Tim O'Reilly on the O'Reilly Radar wrote about one such opportunity for vertical search in the finance sector in his piece "Predicting Financial Market Performance with Real Time Web Data". To quote Tim;