On Friday at the Open Source conference, Jimmy Wales, founder of Wikipedia and Wikia, the open source search engine project, announced the release of an open-source Web crawling site called Grub. Grub crawls the web indexing pages for the Wikia search engine.
It's a clever idea building upon other distributed projects such as SETI @ Home. Crawling the web is costly so if you have thousands of clients doing it for you that will save you money and could make crawling cost effective. However I have to wonder what percentage of an actual crawl will be performed by Grub distributed clients. Also if my computer is contributing to this project, which although is open source, is still a for profit venture, shouldn't I profit from it as well?
Grub aims to compete with Google. If they can get enough computing power behind them they might be able to get an index as large as Google's and maybe even bigger but the key to getting and keeping market share will be the results returned. And that is all in the algorithms of the search engine.
A distributed web crawling client for Project Phoenix is something to consider and providing people a portion of the revenue stream could make it attractive to users.
Leave a comment - Sign in with SpaceRef, Google, Yahoo or OpenID accounts