Hadoop Scales Really Well - Yahoo! Launches World’s Largest Hadoop Production Application

By Marc Boucher on February 19, 2008 12:41 PM | 0 Comments

Hadoop
Some exciting news today from Eric Baldeschwieler, Senior Director, Grid Computing on the Yahoo Developer Network, Yahoo! Launches World's Largest Hadoop Production Application. I'll note that my company Hyperix is using Hadoop for our vertical search platform.

Here's some of the stats:

Some Webmap size data:

* Number of links between pages in the index: roughly 1 trillion links
* Size of output: over 300 TB, compressed!
* Number of cores used to run a single Map-Reduce job: over 10,000
* Raw disk used in the production cluster: over 5 Petabytes




Categories:

Tags:

Leave a comment - Sign in with SpaceRef, Google, Yahoo or OpenID accounts

Recent Blog Entries

Thoughts on Apple's iPad - Why it Will Succeed
I haven't used one and I can't buy one yet, as I'm Canada, but I do have some thoughts on…
By Marc Boucher | Comments (0)
Bigelow Space Station 1/30th Scale Model
I received two Bigelow Space Station models today. They are 1/30 scale model and include one B.A. Standard Module, two…
By Marc Boucher | Comments (1)
What if Twitter was Down for Several Days? Perhaps it's Time for a new Internet Protocol
Anil Dash has an opinion piece today on CNN which basically says don't let a service like Twitter or Facebook…
By Marc Boucher | Comments (0)
Using Social Media Tools Like Twitter to add Value to Advertisers Campaigns
SpaceRef has recently started using Twitter as an additional marketing tool as part of our advertisers campaigns. We don't spam…
By Marc Boucher | Comments (0)
Apple 12″ PowerBook G4 Meet Yellow Dog Linux
I hate it when a perfectly good computer just sits around doing nothing. In this case it's my old Apple…
By Marc Boucher | Comments (0)
New Media Hearings - CRTC Should Once Again Do Nothing
Ten years ago I testified at the Canadian Radio-television Telecommunications Commission's (CRTC) New Media hearings in Ottawa and argued that…
By Marc Boucher | Comments (0)