Presentation: Scalability Challenges in Big Data Science
Presentation: Scalability Challenges in Big Data Science
Join the DZone community and get the full member experience.
Join For FreeThe open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.
Scalability Challenges in Big Data Science
Yesterday I gave a talk on scalability and machine learning at the BerlinBuzzword conference. I give an overview of different ways to scale data analysis and machine learning methods. I cover MapReduce (of course), large scale training of SVMs via stochastic gradient descent, but also stream mining, and real-time (as you know, “you don’t just scale into real-time”).
The conference continues today, follow the conference on Twitter on the #bbuzz hashtag.
Update: On scribd, the hyperlinks are somehow lost, so here is the list:
Scalable Databases
Multithreadding and Messaging Frameworks
MapReduce
- hadoop
- disco
- The MapReduce paper: Map-Reduce for Machine Leanring on Multicore
Large Scale Classifier Training
Other frameworks
- Pregel paper: Pregel: A System for Large-Scale Graph Processing
- Storm
- Esper
Stream processing
- Alex Smola’s lecture
- Heavy hitter’s paper Efficient computation of Frequent and Top-k Elements in Data Streams
- Hashing paper Feature Hashing for Large Scale Multitask Learning
- Count-min sketches website
- Stream Mining Clustering paper: A Framework for Clustering Massive-Domain Data Streams
- My post One does not simply scale into real-time
TWIMPACT:
Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.
Like This Article? Read More From DZone
Published at DZone with permission of Mikio Braun , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}