The new release of Spring introduces features to make server-sent events simple.

What’s good about online schedulers is that you can get an access to your data from any of your devices. Your schedule will be safe if something happens to your laptop or smartphone. But the overall quantity of available schedulers can confuse you. There are JavaScript and .NET scheduling tools. There are third-party plug-ins for big frameworks and built-in parts of these frameworks. And finally, there are stand-alone tools which can be integrated with other components. This article’s aim is to take a look at the four most popular .NET scheduling components to make clear which of them is worthy of attention and good for beginners. Here are the scheduling tools: DayPilot ASP.NET Event Calendar DHTMLX Scheduler .NET Devexpress ASP.NET Scheduler Telerik .NET Scheduler We’ll use Microsoft Visual Studio 2013 to create demo applications. We will check how the basic calendar looks like and behaves. The other important thing is documentation. Good structured documentation pages with understandable step-by-step guides will be a plus. DayPilot ASP.NET Event Calendar DayPilot provides you with different kinds of time management tools: ASP.NET Event Calendar is an event calendar that allows you to plan your day hour by hour: ASP.NET Monthly Calendar will help you create a month schedule: ASP.NET Scheduler is useful if you want to group you events into different categories: ASP.NET Gantt Chart for those who want to build a Gantt Chart: Since Scheduler and Gantt Chart are more specialized tools we’ll take a look at something more commonly used. Let’s say, the Event Calendar. Basic features: Integrated message bar Drag-and-drop Export. Available file formats are BMP, PNG, GIF and JPG Customizable appearance The full list of features is available on this page. If you want to check what you can get from it, check this demo page. The tutorial page is a little bit messy at first sight. There are pages for different kinds of features for all of the DayPilot components. However, it’s not too hard to find the page you want. Which is, in our case, this basic calendar tutorial page. It’s a step-by-step guide which shows how to create a project in Visual Studio, create a database and attach it to your project, etc. It’s a little bit hard to get started with it if you haven’t used Visual Studio before. If you’re lazy enough, you can simply download the source code for this example here. After the job is done, you can check the results: You can’t do much with this simple event calendar. This testing event can be dragged through the calendar grid to change its beginning date. It was created during the calendar creation and by default, there’s no possibility to create a new one. To add this functionality you should spend some more time with tutorials. And you can’t change the scale of the calendar. Therefore, if you want to get a possibility to create long term events as well, you should use both Event Calendar and Monthly Calendar in your project. But here’s some good news. There is an open-source version of Event Calendar available. The cost of DayPilot calendar may vary from $499 to $4999 depending on the license and product version. Well, let’s move along. DHTMLX Scheduler .NET DHTMLX Scheduler .NET is another stand-alone tool that was made to help you create your own online scheduler. Check this demo page to see how it looks like. Basic features: Different views: Decade view, Grid view, Year view Recurring events Drag-and-drop Touchscreen support Easy customizable Integration with Google Maps and Google Calendar You can check all its features on this page. There is more than one possible way to get this scheduler. You can download the package with required files or install them via NuGet or Package Manager Console. This install guide can help you choose the proper way. Let’s talk about documentation. Since DHTMLX Scheduler .NET is basically a .NET wrapper for this JavaScript scheduler, the documentation consists of two parts: server side and client side articles. There are basic calendar creation guides and a bunch of configuration guides. Moreover, there is a video tutorial available. But what we’re interested in right now is how we can create a basic calendar. The developer’s blog contains this tutorial explaining how to build a basic calendar in ASP.NET MVC5. It’s quite detailed and contains a bunch of screenshots. And here’s what we got: This time we can change the scale. There are “Day”, “Week” and “Month” buttons available. The current day is highlighted and there is a “Today” button that will help you in case you lost yourself within the calendar grid. There are several ways of how you can create a new task. You can use drag-and-drop with this calendar. So, select the time period you want and the event property window will appear on your screen: You can also use right click to get access to this window, but in this case you’ll have to set the time period by yourself. Scale changing feature is useful if you want to create a long term event: You can drag the event to change its start date. This is all that you can get by default. The important thing is that, unlike the DayPilot tool, this scheduler can be used to show days, weeks and months. This basic app, as you can see, gives you a possibility to add and change the tasks. Documentation structure is easy to understand, so it’s not too hard to use it to improve your application. The cost of DHTMLX Scheduler .NET is $299 for Commercial License and $979 for Enterprise license. Devexpress ASP.NET Scheduler DevExpress provides you with huge amount of different tools. There are WinForms, iOS, ASP.NET components for desktop and online applications. Very detailed subscription of Devexpress scheduler is available on this page. You can get the installation file on this download page. There’s no possibility to install the scheduler separately from other .NET and MVC components. There’s a lot of documentation. Tons of it. For every single component. The scheduler documentation page is separated into several parts: “Start Here”, Learn More”, “How To” and “Reference”. Each of them consists of chapters that contain a bunch of articles on a specific topic. Everything that you need to know about the every single tool. But if you want to create a basic scheduler fast, you can check two video tutorials available on YouTube: The Project Wizard which will tell you how to create a Devexpress project and Getting Started with the ASPxScheduler which is about the scheduler creation process. This 14 minutes is all that you need to create your first online application. The result is shown below: As you can see, there are a couple of scales available: “Day”, “Work Week”, “Full Week” and “Month”. Moreover, there is a Timeline mode, which can help you trace the current week events. A navigation panel on the right side will help you switch between month and years. Drag-and-drop behavior is available, so you can select the proper area of the calendar grid to create a new event. After the right-click you can choose what you want to do: The properties window will appear. It allows you to set the basic properties (Start time, End time, Subject, etc.) and a couple of additional ones (Label, for example): According to these properties, different events will look differently, which is pretty handy. Here’s the Timeline mode as an example: The cost of Devexpress scheduer is $899.99 for the .NET bundle. Telerik .NET Scheduler Just like the previous developer, Telerik provides you with a wide variety of tools: a mobile application platform for iOS, Android and Windows Mobile; HTML5 and JavaScript widgets for web applications; .NET components for desktop, mobile and online applications and much more. But we’re here for one reason. RadScheduler page contains a detailed list of features with a demo page for each of them. And, like in case of Devexpress, this time we can’t download the scheduler separately. We should get the whole .NET bundle. The documentation page contains comprehensive information about available components. There are tutorials, API documentation, videos, infographics and other useful stuff. It may be a little bit complicated to define where to start from. But there is a YouTube video tutorial that allows you to dive straight into the action instead of reading tons of docs. First 13 minutes are enough to get required basic information about how the things work. As a result you will get something like this: There is something familiar with this one. We can choose between “Day”, “Week”, “Month” and “Timeline” modes. The last one works pretty much the same as the Devexpress Timeline option. This time we have a mini calendar which is useful for navigation. There’s also a “Show full/business hours” toggle button that allows you to separate your work events from the private ones: New event creation is intuitive and similar to other calendars. This time there is a wide variety of properties that allows you to configure the recurrent events. And, as usual, drag-and-drop is available. The cost of Telerik scheduler is $999 for the .NET bundle. Conclusion Described schedulers can be divided into two groups. The first two of them are stand-alone tools. In case of DayPilot there’s no possibility to get the calendar you like. The only possible way to get it is to buy the whole bundle. DHTMLX Scheduler .NET is the only component that can be bought separately from any other tool, which is great if you’re not interested in additional components. The remaining two components are the part of big .NET bundles. And maybe it’s not a good idea to buy them if you want to create a simple online calendar. DayPilot and DHTMLX Scheduler .NET DayPilot components look great. The documentation contains detailed tutorials. The bad thing is that the basic scheduler which was built according to the beginner’s guide is kinda low functional. To make something really useful you should spend some more time with it. There’s also no possibility to change the scale of a calendar, so you have to chose either Event Calendar or Monthly Calendar to plan your events properly. The good news is that, besides the paid components, there is an open-source version available. You can get it here. DHTMLX Scheduler .NET is more attractive, especially for beginners. There are step-by-step guides of different levels of complexity and a video tutorial. The basic app looks pretty useful. It allows you to change the calendar scale, add and configure the tasks. There are Commercial and Enterprise versions available on this page. You can also get the 30-days trial version. As an alternative, you can try the JavaScript version of this scheduler. You have two options with this one: commercial Pro Edition and open-source Standard edition. Devexpress ASP.NET Scheduler and Telerik .NET Scheduler Those two are the components of big and advanced sets of tools. They’re integrated with Visual Studio. As a result, there’s a project wizard for both of them that allows you to configure your application properties. The documentation contains dozens of pages and it may confuse you at the first glance. But you can find an article about the every single property if you spend some time with it. There are video tutorials available as well, so everyone can find the most convenient way of learning. The Devexpress documentation structure looks more comfortable. Every big theme is separated into several parts which are grouped according to the difficulty level. The basic scheduler is pretty functional. For example, there is a possibility to set a proper label for a task. There are a lot of possible combinations of tools that you can buy. But it’s always good to try some before to make a decision and trial versions will help you with that. Check the details on this page. Telerik documentation is not so well grouped which can confuse the beginners. Scheduler creation is intuitive and the basic app is pretty usable. Just like in the previous case, Telerik provides you with a wide variety of possible bundles of tools. And, once again, you can try it before buy it. Trial versions are waiting for you.

Learn how to use the Bean Validation outside of your Java EE container.

Learn unit testing concepts, JUnit constructs, how to write good assertions, JUnit 4 annotations, and JUnit test suites.

How to use Spring Integration to keep domain and integration concerns separate and simple for many Spring-based applications.

How to put Spring, Spring Boot, and Spring Cloud together to create a microservice.

Dialog is like any other window that pops up in front of current window, used to show some short message, taking user input or to ask user decisions.

Take your first steps in using JUnit to unit test your Java code with the help of tools like Maven, a build tool, and IntelliJ IDEA, a popular IDE.

H2 as a memory database for Spring-based applications is lightweight, easy to use, and emulates other RDBMS with the help of JPA and Hibernate.

In this article I will tell you how Apache Camel can turn a full-stack Linux microcomputer (like Raspberry Pi) into a device collecting the GPS coordinates.

I’ve built a small example of running a standalone Java application that both serves static HTML, JavaScript, CSS content, and also publishes a REST web service.

A step-by-step guide for migrating a web application from Struts to Spring MVC, covering essential changes in libraries, configurations, and code structure.

This is a really simple way to provide high-availability with failover and load balancing to any Java backend using JDBC and Spring Boot .

Learn about how to integrate Apache Camel and Fabric8 into an existing Kubernetes CDI service.

[This article was written by Juergen Hoeller.] Spring is well-known to actively support the latest versions of common open source projects out there, e.g. Hibernate and Jackson but also common server engines such as Tomcat and Jetty. We usually do this in a backwards-compatible fashion, supporting older versions at the same time - either through reflective adaptation or through separate support packages. This allows for applications to selectively decide about upgrades, e.g. upgrading to the latest Spring and Jackson versions while preserving an existing Hibernate 3 investment. With the upcoming Spring Framework 4.2, we are taking the opportunity to support quite a list of new open source project versions, including some rather major ones: Hibernate ORM 5.0 Hibernate Validator 5.2 Undertow 1.2 / WildFly 9 Jackson 2.6 Jetty 9.3 Reactor 2.0 SockJS 1.0 final Moneta 1.0 (the JSR-354 Money & Currency reference implementation) While early support for the above is shipping in the Spring Framework 4.2 RCs already, the ultimate point that we’re working towards is of course 4.2 GA - scheduled for July 15th. At this point, we’re eagerly waiting for Hibernate ORM 5.0 and Hibernate Validator 5.2 to GA (both of them are currently at RC1), as well as WildFly 9.0 (currently at RC2) and Jackson 2.6 (currently at RC3). Tight timing… By our own 4.2 GA on July 15th, we’ll keep supporting the latest release candidates, rolling any remaining GA support into our 4.2.1 if necessary. If you’d like to give some of those current release candidates a try with Spring, let us know how it goes. Now is a perfect time for such feedback towards Spring Framework 4.2 GA! P.S.: Note that you may of course keep using e.g. Hibernate ORM 3.6+ and Hibernate Validator 4.3+ even with Spring Framework 4.2. A migration to Hibernate ORM 5.0 in particular is likely to affect quite a bit of your setup, so we only recommend it in a major revision of your application, whereas Spring Framework 4.2 itself is designed as a straightforward upgrade path with no impact on existing code and therefore immediately recommended to all users.

Continuing my Spring-Cloud learning journey, earlier I had covered how to write the infrastructure components of a typical Spring-Cloud and Netflix OSS based micro-services environment - in this specific instance two critical components, Eureka to register and discover services and Spring Cloud Configuration to maintain a centralized repository of configuration for a service. Here I will be showing how I developed two dummy micro-services, one a simple "pong" service and a "ping" service which uses the "pong" service. Sample-Pong microservice The endpoint handling the "ping" requests is a typical Spring MVC based endpoint: @RestController public class PongController { @Value("${reply.message}") private String message; @RequestMapping(value = "/message", method = RequestMethod.POST) public Resource pongMessage(@RequestBody Message input) { return new Resource<>( new MessageAcknowledgement(input.getId(), input.getPayload(), message)); } } It gets a message and responds with an acknowledgement. Here the service utilizes the Configuration server in sourcing the "reply.message" property. So how does the "pong" service find the configuration server, there are potentially two ways - directly by specifying the location of the configuration server, or by finding the Configuration server via Eureka. I am used to an approach where Eureka is considered a source of truth, so in this spirit I am using Eureka to find the Configuration server. Spring Cloud makes this entire flow very simple, all it requires is a "bootstrap.yml" property file with entries along these lines: --- spring: application: name: sample-pong cloud: config: discovery: enabled: true serviceId: SAMPLE-CONFIG eureka: instance: nonSecurePort: ${server.port:8082} client: serviceUrl: defaultZone: http://${eureka.host:localhost}:${eureka.port:8761}/eureka/ The location of Eureka is specified through the "eureka.client.serviceUrl" property and the "spring.cloud.config.discovery.enabled" is set to "true" to specify that the configuration server is discovered via the specified Eureka server. Just a note, this means that the Eureka and the Configuration server have to be completely up before trying to bring up the actual services, they are the pre-requisites and the underlying assumption is that the Infrastructure components are available at the application boot time. The Configuration server has the properties for the "sample-pong" service, this can be validated by using the Config-servers endpoint - http://localhost:8888/sample-pong/default, 8888 is the port where I had specified for the server endpoint, and should respond with a content along these lines: "name": "sample-pong", "profiles": [ "default" ], "label": "master", "propertySources": [ { "name": "classpath:/config/sample-pong.yml", "source": { "reply.message": "Pong" } } ] } As can be seen the "reply.message" property from this central configuration server will be used by the pong service as the acknowledgement message Now to set up this endpoint as a service, all that is required is a Spring-boot based entry point along these lines: @SpringBootApplication @EnableDiscoveryClient public class PongApplication { public static void main(String[] args) { SpringApplication.run(PongApplication.class, args); } } and that completes the code for the "pong" service. Sample-ping micro-service So now onto a consumer of the "pong" micro-service, very imaginatively named the "ping" micro-service. Spring-Cloud and Netflix OSS offer a lot of options to invoke endpoints on Eureka registered services, to summarize the options that I had: 1. Use raw Eureka DiscoveryClient to find the instances hosting a service and make calls using Spring's RestTemplate. 2. Use Ribbon, a client side load balancing solution which can use Eureka to find service instances 3. Use Feign, which provides a declarative way to invoke a service call. It internally uses Ribbon. I went with Feign. All that is required is an interface which shows the contract to invoke the service: package org.bk.consumer.feign; import org.bk.consumer.domain.Message; import org.bk.consumer.domain.MessageAcknowledgement; import org.springframework.cloud.netflix.feign.FeignClient; import org.springframework.http.MediaType; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.ResponseBody; @FeignClient("samplepong") public interface PongClient { @RequestMapping(method = RequestMethod.POST, value = "/message", produces = MediaType.APPLICATION_JSON_VALUE, consumes = MediaType.APPLICATION_JSON_VALUE) @ResponseBody MessageAcknowledgement sendMessage(@RequestBody Message message); } The annotation @FeignClient("samplepong") internally points to a Ribbon "named" client called "samplepong". This means that there has to be an entry in the property files for this named client, in my case I have these entries in my application.yml file: samplepong: ribbon: DeploymentContextBasedVipAddresses: sample-pong NIWSServerListClassName: com.netflix.niws.loadbalancer.DiscoveryEnabledNIWSServerList ReadTimeout: 5000 MaxAutoRetries: 2 The most important entry here is the "samplepong.ribbon.DeploymentContextBasedVipAddresses" which points to the "pong" services Eureka registration address using which the service instance will be discovered by Ribbon. The rest of the application is a routine Spring Boot application. I have exposed this service call behind Hystrix which guards against service call failures and essentially wraps around this FeignClient: package org.bk.consumer.service; import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand; import org.bk.consumer.domain.Message; import org.bk.consumer.domain.MessageAcknowledgement; import org.bk.consumer.feign.PongClient; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.stereotype.Service; @Service("hystrixPongClient") public class HystrixWrappedPongClient implements PongClient { @Autowired @Qualifier("pongClient") private PongClient feignPongClient; @Override @HystrixCommand(fallbackMethod = "fallBackCall") public MessageAcknowledgement sendMessage(Message message) { return this.feignPongClient.sendMessage(message); } public MessageAcknowledgement fallBackCall(Message message) { MessageAcknowledgement fallback = new MessageAcknowledgement(message.getId(), message.getPayload(), "FAILED SERVICE CALL! - FALLING BACK"); return fallback; } } Boot"ing up I have dockerized my entire set-up, so the simplest way to start up the set of applications is to first build the docker images for all of the artifacts this way: mvn clean package docker:build -DskipTests and bring all of them up using the following command, the assumption being that both docker and docker-compose are available locally: docker-compose up Assuming everything comes up cleanly, Eureka should show all the registered services, at http://dockerhost:8761 url - The UI of the ping application should be available at http://dockerhost:8080 url - Additionally a Hystrix dashboard should be available to monitor the requests to the "pong" app at this url http://dockerhost:8989/hystrix/monitor?stream=http%3A%2F%2Fsampleping%3A8080%2Fhystrix.stream: References 1. The code is available at my github location - https://github.com/bijukunjummen/spring-cloud-ping-pong-sample 2. Most of the code is heavily borrowed from the spring-cloud-samples repository - https://github.com/spring-cloud-samples

When creating a camel route using http, the destination might require a ssl connection with a self signed certificate. Therefore on our http client we should register a TrustManager that suports the certificate. In our case we will use the https4 component of Apache Camel Therefore we should configure the routes and add them to the camel context RouteBuilder routeBuilder = new RouteBuilder() { @Override public void configure() throws Exception { from("http://localhost") .to("https4://securepage"); } }; routeBuilder.addRoutesToCamelContext(camelContext); But before we proceed on starting the camel context we should register the trust store on the component we are going to use. Therefore we should implement a function for creating an ssl context with the trustore. Supposed the jks file that has the certificate imported is located on the root of our classpath. private void registerTrustStore(CamelContext camelContext) { try { KeyStore truststore = KeyStore.getInstance("JKS"); truststore.load(getClass().getClassLoader().getResourceAsStream("example.jks"), "changeit".toCharArray()); TrustManagerFactory trustFactory = TrustManagerFactory.getInstance("SunX509"); trustFactory.init(truststore); SSLContext sslcontext = SSLContext.getInstance("TLS"); sslcontext.init(null, trustFactory.getTrustManagers(), null); SSLSocketFactory factory = new SSLSocketFactory(sslcontext, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER); SchemeRegistry registry = new SchemeRegistry(); final Scheme scheme = new Scheme("https4", 443, factory); registry.register(scheme); HttpComponent http4 = camelContext.getComponent("https4", HttpComponent.class); http4.setHttpClientConfigurer(new HttpClientConfigurer() { @Override public void configureHttpClient(HttpClientBuilder builder) { builder.setSSLSocketFactory(factory); Registry registry = RegistryBuilder.create() .register("https", factory) .build(); HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(registry); builder.setConnectionManager(ccm); } }); } catch (IOException e) { e.printStackTrace(); } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (CertificateException e) { e.printStackTrace(); } catch (KeyStoreException e) { e.printStackTrace(); } catch (KeyManagementException e) { e.printStackTrace(); } } After that our route would be able to access the destination securely.

I have just recorded a 5 minute video that demonstrates running an out of stock example from Apache Camel release, the camel-example-servlet packaged as a docker container and running on a kubernetes platform, such as openshift 3. camel-servlet-example scaled up to 3 running containers (pods) which is easy with kubernetes and fabric8 In this video I have already deployed the example and then demonstrates how we can use the fabric8 web console to manage our application. And also connect to the running container and see inside, such as the Camel routes visually as shown above. Then I run a simple bash script from my laptop that sends a HTTP GET to the Camel example and prints the response. The script runs in a endless loop and demonstrates how kubernetes can easily scale up and down multiple Camel containers and load balance across the running containers. And at the end its even self healing when I force killing docker containers. So I suggest to grab a fresh cup of tea or coffee and sit back and play the 5 minutes video. The video is hosted on vimeo and can be seen from this link.

As I get older, I get increasingly to a functionalist position on the world around me. That the social structures and constructs that we see around us are there through a process of evolution and serve some sort of positive benefit because otherwise they would have fallen foul of natural selection. Sometimes this can seem counter-intuitive; crime, for example, has a positive benefit in allowing the vast majority of us to know clearly the boundaries of things that are wrong – a world without law-breaking would probably be a world without laws, not one without crime. When I look at organisations, I seem to see an inexorable move away from structures based on professional expertise. The boundaries by which we define organisations seem to be caught in a previous age, and their utility is becoming less and less. I had the privilege yesterday to spend my time working with a group of Market Insight professionals from a very broad set of organisations. The challenges that they are facing seem in many ways to be linked to a sense of purpose: what the heck is “insight”, and why should a single operational part of an organisation have sole responsibility for it? Organising an organisation by specialism – sales, marketing, operations, production, HR, Finance, IT, legal and so on – is decreasing in function. If that sounds outlandish, remember that most manufacturing organisation moved from specialism-focus to product-focus at the beginning of the last century with the advent of the production line. And today, with matrix management and project-centric management the norm rather than the exception, we’ve implicitly acknowledged the passing of professional specialism. The creation of new divisions like “digital” or “innovation” – at their core, multi-disciplinary activities – again implicitly acknowledges that modern organisations need to have a design based on the things that people are doing, not the skills that they collectively have in common. With the ways in which we can communicate and collaborate (and the breaking down of 9-5 “in the office” through flexible working patterns) the big benefit of specialist operating divisions, communities of practice, becomes lessened dramatically too. Do I gain more professionally (and does my organisation gain more) by surrounding myself with colleagues who have skills rather than necessarily activity in common? This isn’t going to happen overnight. Our professional structures are so ingrained, that it will be at an evolutionary rather than revolutionary pace that change will permeate; some of the memes (like matrix management) are already in place; there are some mutations that will probably wither (looking at you, Holocracy). How to prepare? I’ve heard a lot in recent years about “T-shaped” workers: people with a strong expertise (the down stroke of the T) but with a breadth of knowledge and experience (the horizontal stroke). I personally am hedging more on being “comb-shaped” – many more areas of domain knowledge, and enough understanding to be able to bring in an expert where necessary. The irony that because of follicle challenge I haven’t used a comb in many years isn’t lost…

Written by Craig Wentworth. To understand the furor that’s greeted recent vendor announcements around open source analytics computing engine Spark, and some commentary seemingly setting up a Spark versus Hadoop battle, it’s worth taking a moment to recap on what each actually is (and is not). As I covered in last year’s MWD report on Hadoop and its family of tools, when people talk about Apache Hadoop they’re often referring to a whole framework of tools designed to facilitate distributed parallel processing of large datasets. That processing was traditionally confined to MapReduce batch jobs in Hadoop’s early days, though Hadoop 2 brought the YARN resource scheduler and opened up Hadoop to streaming, real-time querying and a wider array of analytical programming applications (beyond MapReduce). Spark has been designed to run on top of Hadoop’s Distributed File System (amongst other data platforms) as an alternative to MapReduce – tuned for real-time streaming data processing and fast interactive queries, and with multi-genre analytics applicability (machine learning, time series, graph, SQL, streaming out-of-the-box). It gets that speed advantage by caching in-memory (rather than writing interim results to disk, as MapReduce does), but with that approach comes a need for higher-spec physical machines (compared with MapReduce’s tolerance for commodity hardware). So, Spark isn’t about to replace Hadoop -- but it may well supplant MapReduce (especially in growing real-time use cases). Those “Spark vs Hadoop” headlines are about as meaningful as one proclaiming “mushrooms vs pizza." Yes, mushroom might be a more suitable topping than, say, pepperoni (especially in a vegetarian use case), but it’ll still be deployed on the same dough and tomato sauce pizza platform. Nobody’s about to suggest the mushroom should go it alone! But what’s behind the headlines and the hype is a story of enterprise adoption – or at least vendors anticipating that adoption and investing in ‘the weaponization of Spark’ as it faces the more exacting standards of security, scaling performance, consistency, etc. which come with mainstream enterprise deployment. Big names like IBM, Databricks (the company formed by the originators of Spark), and MapR made commitments in and around the Spark Summit earlier this month. MapR has announced three new Quick Start Solutions for its Hadoop distribution to help customers get started with Spark in real-time security log analytics, genome sequencing, and time series analytics; Databricks’ cloud-hosted Spark platform (formerly known as Databricks Cloud) has become generally available; and IBM announced a raft of measures designed to give Spark a significant shot in the arm – it’s open sourcing its SystemML technology to bolster Spark’s machine learning capabilities, integrating Spark into its own analytics platforms, investing in Spark training and education, committing 3,500 of its researchers and developers to work on Spark-related projects, and offering Spark as a service on its Bluemix developer cloud. Given the overlap with Databricks’ business model (of offering development, certification, and support for Spark), IBM’s intentions are likely to tread on some toes before long – but for now, at least, both companies are content to focus on the combined push benefiting the Spark community and its enterprise aspirations overall (though clearly IBM’s betting on all this investment buying it some influence over where Spark goes next). It’s worth bearing in mind that not all its supporters champion Spark wholesale and all the interested parties tend to be interested in particular bits of Spark (as wide-ranging as it is) because of overlaps with their own preferred toolsets. For instance, although Spark supports many analytics genres, Cloudera focuses on its machine learning capabilities (as it has its own SQL-on-Hadoop tool in Impala), and MapR and Hortonworks also promote Drill and Hive as their favoured source of SQL-on-Hadoop. IBM’s support is focused on Spark’s machine learning and in-memory capabilities (hence the SystemML open sourcing news). In the face of such strong vendor preferences, how long before some of Spark’s current features fall away (or at least start to show the effects of being starved of as much care and feeding as is bestowed upon vendors’ favourite Spark components)? The Spark community is at much the same place the Hadoop one was at a while back – it’s showing great promise and suitability in key growth workloads (in Spark’s case, such as real-time IoT applications). However, the product as it stands is too immature for many enterprise tastes. Cue enterprise software vendors stepping up to help grow Spark up fast. Their challenge though is to smooth out the edges without smothering what made it so interesting in the first place.