Over a million developers have joined DZone.

How Can a Mac Mini Outperform a 1,636-Node Hadoop Cluster?

DZone's Guide to

How Can a Mac Mini Outperform a 1,636-Node Hadoop Cluster?

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

This recent article from Big Data Republic describes a hands-on performance comparison between GraphChi - a graph computation tool from GraphLab - and a 1,636-node Hadoop cluster. The task set for both was to process a Twitter graph with 1.5 billion edges, and the result, surprisingly enough, was a significantly quicker processing time for GraphChi.

The article goes into detail on the techniques behind the optimization of GraphChi, and seems not so much to point to the superiority of GraphChi (though it does lean that way), but to highlight the importance of planning and optimization when it comes to massive datasets. It's an interesting look at the work that still needs to go into big data tools to make them as efficient as they can be.

Take a look at the full article for a look at how much difference optimization can make. Also, to be fair, this comparison originally took place a year ago. Hadoop 2.0 rematch, anyone?

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.


Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}