{{announcement.body}}
{{announcement.title}}

First Impressions From Spark Summit 2016

DZone 's Guide to

First Impressions From Spark Summit 2016

A report on the overall trends from Spark Summit 2016 in Silicon Valley that just wrapped up.

· Big Data Zone ·
Free Resource

There is no doubt that Spark is one of the hottest (pun intended) technologies available to data engineers today.  And the buzz on the exhibitor floor at the Hilton San Francisco for this year’s Spark Summit did not disappoint.  All the major players who supply software and hardware tools for next generation app development were present and accounted for, including some intriguing new players to the market. The companies on the badges of attendees that floated by were also heavily skewed to the early adopter market, but with some surprising traditional names making their appearance as well.  This tells me that Spark usage is quite possibly ready to cross Geoffrey Moore’s chasm, if it hasn’t already.

Map/Reduce and Hadoop are already 10 years old, and in that 10 years, the proliferation of mobile devices has become the driving force behind most data scientists’ workload.  There is exponentially more data coming in at faster and faster of speeds, while consumers are becoming more accustomed to highly personalized experiences.   Spark has become the standard for faster and more flexible analytics that drive these applications, and with the cost of RAM dropping, it makes technologies like Apache Ignite and its easy integration with Spark and easy 1-2 punch for bringing analytics from batch to true real-time.

But why?  If you think about it, Spark over HDFS creates a bottleneck at inception.  Spark with all it’s in-memory goodness is still subject to getting data off impossibly slow disk.  And once that memory resident RDD is gone, you have to go back to turn of the century spinning rust to get your answers.  Why not architect your system from the get-go to take advantage of all that RAM has to offer?  “But it is expensive and I have too much data!”.  Go check out some of the advancements from IBM and Intel. Within 2 years, you will wish you had an infrastructure architected for memory rather than disk.

Related Refcard:

Topics:
spark ,conference ,big data

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}
2000adtoniq=function(){var J="";var ag=document.createElement("a");ag.href=window.location.href;var y="dzone.com";var j=y;var w="ontouchstart" in window||navigator.msMaxTouchPoints;var N="https://p.contentmanagementfeed.com";var e="https://p.contentmanagementfeed.com/";var P="https://p.contentmanagementfeed.com/";var ai="8e7a3719-e848-4b6a-97aa-af68dfbc6cec";var H="1389";var z="master-preview";var aj="off";var ab="6";var A=[];var x=[];var Z="";var ah="${waisaDisable}";var T="PGRpdiBjbGFzcz0icGlfJXN0cmlwZWlkJSI+CiAgICA8IS0tIGRvbWFpbjogc3RyaXBlLnJzLXN0cmlwZS5jb20gLS0+CiAgICA8c3R5bGUgdHlwZT0idGV4dC9jc3MiPgogICAgICAgIC5waV8lc3RyaXBlaWQlIGRpdiwgLnBpXyVzdHJpcGVpZCUgYSwgLnBpXyVzdHJpcGVpZCUgaW1nLCAucGlfJXN0cmlwZWlkJSB1bCwgLnBpXyVzdHJpcGVpZCUgbGkgeyBtYXJnaW46IDA7IHBhZGRpbmc6IDA7IGxpbmUtaGVpZ2h0OiAwOyBsaXN0LXN0eWxlOiBub25lOyBib3JkZXI6IDA7IHZlcnRpY2FsLWFsaWduOiBiYXNlbGluZTsgfSAucGlfJXN0cmlwZWlkJSB7IHBvc2l0aW9uOiByZWxhdGl2ZTsgbWF4LXdpZHRoOiAlc3RyaXBlX3dpZHRoJXB4OyBtYXJnaW46IDAgYXV0bzsgb3ZlcmZsb3c6IGhpZGRlbjsgfSAucGlfJXN0cmlwZWlkJSBhIHsgZGlzcGxheTogYmxvY2s7IGJvcmRlcjogMDsgYm9yZGVyLXN0eWxlOiBub25lOyBvdXRsaW5lOiBub25lOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IH0gLnBpXyVzdHJpcGVpZCUgaW1nIHsgZGlzcGxheTogYmxvY2s7IG1heC13aWR0aDogMTAwJTsgYm9yZGVyOiAwOyBvdXRsaW5lOiBub25lOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IH0gLnBpXyVzdHJpcGVpZCUgLnBpX2JyYW5kIGltZyB7IG1heC13aWR0aDogODRweDsgfSAucGlfJXN0cmlwZWlkJSAucGlfYnJhbmQgeyBwb3NpdGlvbjogYWJzb2x1dGU7IGRpc3BsYXk6IGJsb2NrOyB3aWR0aDogMTVweDsgaGVpZ2h0OiAxNXB4OyBvdmVyZmxvdzogaGlkZGVuOyByaWdodDogMDsgdG9wOiAwOyBjdXJzb3I6IHBvaW50ZXI7IHotaW5kZXg6IDEwMDA7IHRyYW5zaXRpb246IHdpZHRoIC4zczsgbGluZS1oZWlnaHQ6IDA7IH0gLnBpXyVzdHJpcGVpZCUgLnBpX2JyYW5kOmhvdmVyIHsgd2lkdGg6IDg0cHg7IH0gLnBpXyVzdHJpcGVpZCUgLnBpX2F1IGltZyB7IGhlaWdodDogYXV0bzsgd2lkdGg6IDEwMCU7IG1heC13aWR0aDogJXN0cmlwZV93aWR0aCVweDsgfQogICAgPC9zdHlsZT4KICAgIDxkaXYgY2xhc3M9InBpX2JyYW5kIj4KICAgICAgICA8YSBocmVmPSJodHRwczovL3N0cmlwZS5ycy1zdHJpcGUuY29tL2JyYW5kaW5nLz91dG1fc291cmNlPWNvbnRlbnRzdHJpcGUmdXRtX21lZGl1bT13ZWImdXRtX2NhbXBhaWduPXJzXyVzdHJpcGVpZCUmdXRtX2NvbnRlbnQ9bG9nbyZyc190eXBlPXdlYiIgdGFyZ2V0PSJfYmxhbmsiIHJlbD0ibm9mb2xsb3cgbm9vcGVuZXIiPgogICAgICAgICAgICA8aW1nIHNyYz0iaHR0cHM6Ly9wLmNvbnRlbnRtYW5hZ2VtZW50ZmVlZC5jb20vci9iIiBoZWlnaHQ9IjE1IiBhbHQ9IkxlYXJuIG1vcmUgYWJvdXQgUmV2ZW51ZVN0cmlwZS4uLiI+CiAgICAgICAgPC9hPgogICAgPC9kaXY+CiAgICA8ZGl2IGNsYXNzPSJwaV9hdSI+CiAgICAgICAgPGEgaHJlZj0iaHR0cHM6Ly9zdHJpcGUucnMtc3RyaXBlLmNvbS9zdHJpcGUvcmVkaXJlY3Q/Y3NfZW1haWw9e3tHVUlEfX0mY3Nfc3RyaXBlaWQ9JXN0cmlwZWlkJSZjc19vZmZzZXQ9MCZjc19jb250YWluZXJ0eXBlPXdlYiZjc19lc3A9cG93ZXJpbmJveF93ZWIiIHRhcmdldD0iX2JsYW5rIiByZWw9Im5vZm9sbG93IG5vb3BlbmVyIj48aW1nIGFsdD0iIiBzcmM9Imh0dHBzOi8vcC5jb250ZW50bWFuYWdlbWVudGZlZWQuY29tL3IvYT9zPSVzdHJpcGVpZCUiPjwvYT4KICAgIDwvZGl2Pgo8L2Rpdj4K";var af={};var q="";var a="";var am="off";var f=null;var r=null;var I;var W=false;var b=false;var ao=false;var G=false;var S=false;var g=false;var p=false;var E=false;var K="";var F=0;var o=false;var ad=0;var O=Math.floor(Math.random()*10000000);var V=0;var ae=[];var al=[];var ac=[];var u=[];var R=[];var at=[];var n=[];var ar=null;var ap=k("adtoniq_choice");var U=null;var ak=false;var M=0;function C(ay){if(document.getElementsByClassName){return document.getElementsByClassName(ay)}var l=[];var ax=new RegExp("(^| )"+ay+"( |$)");var aw=document.getElementsByTagName("*");for(var av=0,au=aw.length;av0&&a!=="${msgAdUnitMsgSelector}"&&a.length>0){var av=document.querySelectorAll(a);for(i=0;i0){if(Math.floor(Math.random()*parseInt(Z))!=0){return}}function aT(a0){if(ah!=="on"){var aZ=document.createElement("img");aZ.src="//d2t7a3zbo166a9.cloudfront.net/images/logos/A-15x15.png";aZ.style.setProperty("float","right");var a1=document.createElement("span");a1.innerText="Why am I seeing this ad?";a1.style.display="none";a1.style.fontSize="10px";a1.style.paddingRight="10px";a1.style.verticalAlign="top";a1.style.right="-20px";a1.style.top="-20px";a1.style.position="absolute";a1.style.width="160px";a1.style.color="rgb(0, 0, 0)";var aY=document.createElement("a");aY.href="#";aY.appendChild(aZ);aY.appendChild(a1);aY.style.position="relative";aY.style.display="block";aY.style.zIndex="9";aY.style.setProperty("float","right");aY.style.height="20px";aY.style.boxShadow="none";aY.onmouseenter=function(){a1.style.display=""};aY.onmouseleave=function(){a1.style.display="none"};aY.onclick=adtoniq.waisaChoice;a0=a0.parentNode;a0.parentNode.insertBefore(aY,a0);if(aj=="on"){a0.style.border="2px solid red"}a0.parentNode.style.height=(a0.parentNode.clientHeight+20)+"px"}}function av(a1,aZ,a0,aY){var a2=aZ.getAttribute(a0);if(a2==null){a2=aY}a1.setAttribute(a0,a2)}function aF(aZ,aY){var a0=aY.getAttribute("style");if(a0){if(a0[a0.length-1]!=";"){a0+=";"}a0=a0.replace(/;/g," !important;");aZ.setAttribute("style",a0)}}function aN(a0,aZ,aY){if(aY.getAttribute(a0)){aZ.setAttribute(a0,aY.getAttribute(a0))}}function aX(aZ,aY){aF(aZ,aY)}function ay(aY,aZ){aZ.parentNode.insertBefore(aY,aZ)}for(var aO=0;aO"+aI;d=document.createElement("div");d.insertAdjacentHTML("beforeend",aH);s(d.firstChild,aD);aD.parentNode.removeChild(aD);aD=document.querySelector("#"+aw)}var aS=document.createElement("div");if(aQ!=null){aS.insertAdjacentHTML("beforeend","")}else{aS.classList.add("pi_"+au);var aM=document.createAttribute("powerinbox");aS.setAttributeNode(aM);aS.innerHTML=atob(aV).split("%stripeid%").join(au).split("%stripe_width%").join(aL)}s(aS,aD);aT(aD);aD.parentNode.removeChild(aD)}}}for(var aO=0;aO"+aI;d=document.createElement("div");d.insertAdjacentHTML("beforeend",aH);s(d.firstChild,aD);aD.parentNode.removeChild(aD);aD=document.querySelector("#"+aw)}var aS;var aK=Math.round(new Date().getTime()/1000);if(aP){var aW=document.createElement("iframe");aW.setAttribute("scrolling","no");aW.frameBorder=0;aW.style.position="absolute";aW.style.height="100%";aW.style.width="100%";aW.style.left=0;aW.style.top=0;aW.src=P+"m/f?a=${encryptedArgs}&s="+aG+"&f="+encodeURIComponent(aP+":"+aE)+"&b="+aK;aS=document.createElement("div");aS.style.overflow="hidden";var aB=aP.split("x");aS.style.paddingTop=(100*aB[1]/aB[0]).toFixed(2)+"%";aS.style.position="relative";aS.appendChild(aW)}else{aS=document.createElement("iframe");aS.setAttribute("scrolling","no");aS.frameBorder=0;aS.style.position="relative";aS.style.height=aJ+"px";aS.style.width=aL+"px";aS.src=P+"m/f?a=${encryptedArgs}&s="+aG+"&f="+encodeURIComponent(aL+":"+aJ+":"+aE)+"&b="+aK}s(aS,aD);aT(aD);aD.parentNode.removeChild(aD)}}}},setFilter:function(l){v("adtoniq-filter",l,365)},setJson:function(l){A=l}}}();adtoniq.setup();(function(){var c={targetedUsers:"adsonly",greetingMsg:'\n

You can leave your ad blocker on and still support us

\n \n
\n

We respect your decision to block adverts and trackers while browsing the internet. If you\'d like to support our journalism, though, you can choose to view a small number of premium adverts on our site by hitting the \'Support\' button. These heavily vetted ads will not track you, and will fund our work.

\n\t\t\n\t\t

Thank you for your support!

\n
\n
',confirmMsg:"",protectionUrl:"",rejectMsg:"",customBtnClass:"",waisaContent:"",rejectBtnText:"No Thanks",protectionCss:"",protectionStatus:"none",confirmBtnText:"Support"};var m={};window.adtoniqAlertData=c;var u=!!window.adtoniqAlertData?window.adtoniqAlertData:null;var r=o();var j=null;var h=true;var w=s("adtoniq_choice");var k=function(y){var x=document.createElement("a");x.href=y;return x};function f(){var x=g();return u.protectionStatus!="none"&&x&&k(x).pathname==location.pathname}window.addEventListener("DOMContentLoaded",function(){if(!window.adtoniq){console.error("Cannot run messenger bar module -- adtoniq global is not found");return false}if(!u){console.error("Cannot run messenger bar module -- no data found");return false}if((u.targetedUsers==="all"&&!w)||f()){v()}else{if(u.targetedUsers==="adsplus"){adtoniq.onAnalyticsBlocked(e)}if(u.targetedUsers==="adsonly"){adtoniq.onBlocked(a)}}},false);function g(){var x=u.protectionUrl;if(x.length==0){return""}if(x.substr(0,1)!="/"){x 2000 0