It’s fairly easy. You need to add the following uses-permission in the Android manifest to get call history programmatically. interface in your activity. It has three methods. abstract Loader onCreateLoader(int id, Bundle args) //Instantiate and return a new Loader for the given ID. abstract void onLoadFinished(Loader loader, D data) //Called when a previously created loader has finished its load. abstract void onLoaderReset(Loader loader) //Called when a previously created loader is being reset, and thus making its data unavailable. To initialize a query, we need to call LoaderManager.initLoader() at the very first place. We are going to add a button and call this in that button events here and after this background framework will be initialized. As soon as the background framework is initialized, it calls your implementation of onCreateLoader(). To start the query, we have to return a CursorLoader from this method. @Override public Loader onCreateLoader(int loaderID, Bundle args) { Log.d(TAG, "onCreateLoader() >> loaderID : " + loaderID); switch (loaderID) { case URL_LOADER: // Returns a new CursorLoader return new CursorLoader( this, // Parent activity context CallLog.Calls.CONTENT_URI, // Table to query null, // Projection to return null, // No selection clause null, // No selection arguments null // Default sort order ); default: return null; } } We are going access our expected data from a Cursor. And we will get this in theonLoadFinished() method. @Override public void onLoadFinished(Loader loader, Cursor managedCursor) { Log.d(TAG, "onLoadFinished()"); StringBuilder sb = new StringBuilder(); int number = managedCursor.getColumnIndex(CallLog.Calls.NUMBER); int type = managedCursor.getColumnIndex(CallLog.Calls.TYPE); int date = managedCursor.getColumnIndex(CallLog.Calls.DATE); int duration = managedCursor.getColumnIndex(CallLog.Calls.DURATION); sb.append("Call Log Details "); sb.append("\n"); sb.append("\n"); sb.append(""); while (managedCursor.moveToNext()) { String phNumber = managedCursor.getString(number); String callType = managedCursor.getString(type); String callDate = managedCursor.getString(date); Date callDayTime = new Date(Long.valueOf(callDate)); String callDuration = managedCursor.getString(duration); String dir = null; int callTypeCode = Integer.parseInt(callType); switch (callTypeCode) { case CallLog.Calls.OUTGOING_TYPE: dir = "Outgoing"; break; case CallLog.Calls.INCOMING_TYPE: dir = "Incoming"; break; case CallLog.Calls.MISSED_TYPE: dir = "Missed"; break; } sb.append("") .append("Phone Number: ") .append("") .append(phNumber) .append(""); sb.append(""); sb.append(""); sb.append("") .append("Call Type:") .append("") .append(dir) .append(""); sb.append(""); sb.append(""); sb.append("") .append("Date & Time:") .append("") .append(callDayTime) .append(""); sb.append(""); sb.append(""); sb.append("") .append("Call Duration (Seconds):") .append("") .append(callDuration) .append(""); sb.append(""); sb.append(""); sb.append(""); } sb.append(""); managedCursor.close(); callLogsTextView.setText(Html.fromHtml(sb.toString())); } Output: Full Source code: https://github.com/rokon12/call-log

Written by Josh Long on the Spring blog. This blog was inspired by a talk that Laurent Doguin, a developer advocate over at Couchbase, and I gave at Couchbase Connect last year. Merci Laurent! This is a demo of the Spring Data Couchbase integration. From the project page, Spring Data Couchbase is: The Spring Data Couchbase project provides integration with the Couchbase Server database. Key functional areas of Spring Data Couchbase are a POJO centric model for interacting with Couchbase Buckets and easily writing a Repository style data access layer. What is Couchbase? Couchbase is a distributed data-store that enjoys true horizontal scaling. I like to think of it as a mix of Redis and MongoDB: you work with documents that are accessed through their keys. There are numerous client APIs for all languages. If you’re using Couchbase for your backend and using the JVM, you’ll love Spring Data Couchbase. The bullets on the project home page best enumerate its many features: Spring configuration support using Java based @Configuration classes or an XML namespace for the Couchbase driver. CouchbaseTemplate helper class that increases productivity performing common Couchbase operations. Includes integrated object mapping between documents and POJOs. Exception translation into Spring’s portable Data Access Exception hierarchy. Feature Rich Object Mapping integrated with Spring’s Conversion Service. Annotation based mapping metadata but extensible to support other metadata formats. Automatic implementation of Repository interfaces including support for custom finder methods (backed by Couchbase Views). JMX administration and monitoring Transparent @Cacheable support to cache any objects you need for high performance access. Running Couchbase Use Vagrant to Run Couchbase Locally You will need to have Couchbase installed if you don’t already (naturally). Michael Nitschinger (@daschl, also lead of the Spring Data Couchbase project), blogged about how to get a simple4-node Vagrant cluster up and running here. I’ve reproduced his example here in the vagrantdirectory. To use it, you’ll need to install Virtual Box and Vagrant, of course, but then simply run vagrant up in the vagrant directory. To get the most up-to-date version of this configuration script, I went to Michael’s GitHub vagrants project and found that, beyond this example, there are numerous other Vagrant scripts available. I have a submodule in this code’s project directory that points to that, but be sure to consult that for the latest-and-greatest. To get everything running on my machine, I chose the Ubuntu 12 installation of Couchbase 3.0.2. You can change how many nodes are started by configuring the VAGRANT_NODES environment variable before startup: VAGRANT_NODES=2 vagrant up You’ll need to administer and configure Couchbase on initial setup. Point your browser to the right IP for each node. The rules for determining that IP are well described in the README. The admin interface, in my case, was available at 192.168.105.101:8091 and192.168.105.102:8091. For more on this process, I recommend that you follow theguidelines here for the details. Here’s how I did it. I hit the admin interface on the first node and created a new cluster. I usedadmin for the username and password for the password. On all subsequent management pages, I simply joined the existing cluster by pointing the nodes to 192.168.105.101 and using the aforementioned admin credential. Once you’ve joined all nodes, look for theRebalance button in the Server Nodes panel and trigger a cluster rebalance. If you are done with your Vagrant cluster, you can use the vagrant halt command to shut it down cleanly. Very handy is also vagrant suspend, which will save the state of the nodes instead of shutting them down completely. If you want to administer the Couchbase cluster from the command line there is the handycouchbase-cli. You can simply use the vagrant ssh command to get into each of the nodes (by their node-names: node1, node2, etc..). Once there, you can run cluster configuration commands. For example the server-list command will enumerate cluster nodes. /opt/couchbase/bin/couchbase-cli server-list -c 192.168.56.101-u admin -p password It’s easy to trigger a rebalance using: /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.56.101-u admin -p password Couchbase In the Cloud and on Cloud Foundry Couchbase lends itself to use in the cloud. It’s horizontally scalable (like Gemfire or Cassandra) in that there’s no single point of failure. It does not employ a master-slave or active/passive system. There are a few ways to get it up and running where your applications are running. If you’re running a Cloud Foundry installation, then you can install the the Cumulogic Service Broker which then lets your Cloud Foundry installation talk to the Cumulogic platform which itself can manage Couchbase instances. Service brokers are the bit of integration code that teach Cloud Foundry how to provision, destroy and generally interact with a managed service, like Couchbase, in this case. Using Spring Data Couchbase to Store Facebook Places Let’s look at a simple example that reads data (in this case from the Facebook Places API using Spring Social Facebook’s FacebookTemplate API) and then loads it into the Couchbase server. Get a Facebook Access Token You’ll also need a Facebook access token. The easiest way to do this is to go to the Facebook Developer Portal and create a new application and then get an application ID and an application secret. Take these two values and concatenate them with a pike character (|). Thus, you’ll have something of the form: appID|appSecret. The sample application uses Spring’s Environment mechanism to resolve the facebook.accessToken key. You can provide a value for it in the src/main/resources/application.properties file or using any of the other supported Spring Boot property resolution mechanisms. You could even provide the value as a -D argument: -Dfacebook.accessToken=...|... Telling Spring Data Couchbase About our Cluster Data in Couchbase is stored in buckets. It’s logically the same as a database in a SQL RDBMS. It is typically replicated across nodes and has its own configuration. We’ll be using the defaultbucket, but it’s a snap to create more buckets. Let’s look at the basic configuration required to use Spring Data Couchbase (in this case, in terms of a Spring Boot application): @SpringBootApplication @EnableScheduling @EnableCaching public class Application { @EnableCouchbaseRepositories @Configuration static class CouchbaseConfiguration extends AbstractCouchbaseConfiguration { @Value("${couchbase.cluster.bucket}") private String bucketName; @Value("${couchbase.cluster.password}") private String password; @Value("${couchbase.cluster.ip}") private String ip; @Override protected List bootstrapHosts() { return Arrays.asList(this.ip); } @Override protected String getBucketName() { return this.bucketName; } @Override protected String getBucketPassword() { return this.password; } } // more beans } A Spring Data Couchbase Repository Spring Data provides the notion of repositories - objects that handle typical data-access logic and provide convention-based queries. They can be used to map POJOs to data in the backing data store. Our example simply stores the information on businesses it reads from Facebook’s Places API. To acheive this we’ve created a simple Place entity that Spring Data Couchbase repositories will know how to persist: @Document(expiry = 0) class Place { @Id private String id; @Field private Location location; @Field @NotNull private String name; @Field private String affilitation, category, description, about; @Field private Date insertionDate; // .. getters, constructors, toString, etc } The Place entity references another entity, Location, which is basically the same. In the case of Spring Data Couchbase, repository finder methods map to views - queries written in JavaScript - in a Couchbase server. You’ll need to setup views on the Couchbase servers. Go to any Couchbase server’s admin console and visit the Views screen, then clickCreate Development View and name it place, as our entity will be demo.Place (the development view name is adapted from the entity’s class name by default). We’ll create two views, the generic all, which is required for any Spring Data Couchbase POJO, and the byName view, which will be used to drive the repository’s findByName finder method. This mapping is by convention, though you can override which view is employed with the @View annotation on the finder method’s declaration. First, all: Now, byName: When you’re done, be sure to Publish each view! Now you can use Spring Data repositories as you’d expect. The only thing that’s a bit different about these repositories is that we’re declaring a Spring Data Couchbase Query type for the argument to the findByName finder method, not a String. Using the @Query is straightforward: Query query = new Query(); query.setKey("Philz Coffee"); Collection places = placeRepository.findByName(query); places.forEach(System.out::println); Where to go from Here We’ve only covered some of the basics here. Spring Data Couchbase supports the Java bean validation API, and can be configured to honor validation constraints on its entities. Spring Data Couchbase also provides lower-level access to the CouchbaseClient API, if you want it. Spring Data Couchbase also implements the Spring CacheManager abstraction - you can use@Cacheable and friends with data on service methods and it’ll be transparently persisted to Couchbase for you. The code for this example is in my Github repository, co-developed with my pal Laurent Doguin (@ldoguin) over at Couchbase.

This post is a guest post by Activiti co-founder and community member Joram Barrez (@jbarrez) who works for Alfresco. Thanks Joram! I’d like to see more of these community guest posts, so - as usual - don’t hesitate to ping me (@starbuxman) with ideas and contributions! -Josh Introduction Activiti is an Apache-licensed business process management (BPM) engine. Such an engine has as core goal to take a process definition comprised of human tasks and service calls and execute those in a certain order, while exposing various API’s to start, manage and query data about process instances for that definition. Contrary to many of its competitors, Activiti is lightweight and integrates easily with any Java technology or project. All that, and it works at any scale - from just a few dozen to many thousands or even millions of process executions. The source code of Activiti can be found on Github. The project was founded and is sponsored by Alfresco, but enjoys contributions from all across the globe and industries. A process definition is typically visualized as a flow-chart-like diagram. In recent years, the BPMN 2.0 standard (an OMG standard, like UML) has become the de-facto ‘language’ of these diagrams. This standard defines how a certain shape on the diagram should be interpreted, both technically and business-wise and how it is stored, as a not-so-hipster XML file.. but luckily most of the tooling hides this for you. This is a standard, and you can use any number of compliant tools to design (and even run) your BPMN processes. That said, if you’re asking me, there is no better choice than Activiti! Spring Boot integration Activiti and Spring play nicely together. The convention-over-configuration approach in Spring Boot works nicely with Activiti’s process engine is setup and use. Out of the box, you only need a database, as process executions can span anywhere from a few seconds to a couple of years. Obviously, as an intrinsic part of a process definition is calling and consuming data to and from various systems with all kinds of technologies. The simplicity of adding the needed dependencies and integrating various pieces of (boiler-plate) logic with Spring Boot really makes this child’s play. Using Spring Boot and Activiti in a microservice approach also makes a lot of sense. Spring Boot makes it easy to get a production-ready service up and running in no time and - in a distributed microservice architecture - Activiti processes can glue together various microservices while also weaving in human workflow (tasks and forms) to achieve a certain goal. The Spring Boot integration in Activiti was created by Spring expert Josh Long. Josh and I did a webinar a couple of months ago that should give you a good insight into the basics of the Activiti integration for Spring Boot. The Activiti user guide section on Spring Boot is also a great starting place to get more information. Getting Started The code for this example can be found in my Github repository. The process we’ll implement here is a hiring process for a developer. It’s simplified of course (as it needs to fit on this web page), but you should get the core concepts. Here’s the diagram: As said in the introduction, all shapes here have a very specific interpretation thanks to the BPMN 2.0 standard. But even without knowledge of BPMN, the process is pretty easy to understand: When the process starts, the resume of the job applicant is stored in an external system. The process then waits until a telephone interview has been conducted. This is done by a user (see the little icon of a person in the corner). If the telephone interview wasn’t all that, a polite rejection email is sent. Otherwise, both a tech interview and financial negotiation should happen. Note that at any point, the applicant can cancel. That’s shown in the diagram as the event on the boundary of the big rectangle. When the event happens, everything inside will be killed and the process halts. If all goes well, a welcome email is sent. This is the BPMN for this process Let’s create a new Maven project, and add the dependencies needed to get Spring Boot, Activiti and a database. We’ll use an in memory database to keep things simple. org.activiti spring-boot-starter-basic ${activiti.version} com.h2database h2 1.4.185 So only two dependencies is what is needed to create a very first Spring Boot + Activiti application: @SpringBootApplication public class MyApp { public static void main(String[] args) { SpringApplication.run(MyApp.class, args); } } You could already run this application, it won’t do anything functionally but behind the scenes it already creates an in-memory H2 database creates an Activiti process engine using that database exposes all Activiti services as Spring Beans configures tidbits here and there such as the Activiti async job executor, mail server, etc. Let’s get something running. Drop the BPMN 2.0 process definition into thesrc/main/resources/processes folder. All processes placed here will automatically be deployed (ie. parsed and made to be executable) to the Activiti engine. Let’s keep things simple to start, and create a CommandLineRunner that will be executed when the app boots up: @Bean CommandLineRunner init( final RepositoryService repositoryService, final RuntimeService runtimeService, final TaskService taskService) { return new CommandLineRunner() { public void run(String... strings) throws Exception { Map variables = new HashMap(); variables.put("applicantName", "John Doe"); variables.put("email", "[email protected]"); variables.put("phoneNumber", "123456789"); runtimeService.startProcessInstanceByKey("hireProcess", variables); } }; } So what’s happening here is that we create a map of all the variables needed to run the process and pass it when starting process. If you’d check the process definition you’ll see we reference those variables using ${variableName} in many places (such as the task description). The first step of the process is an automatic step (see the little cogwheel icon), implemented using an expression that uses a Spring Bean: which is implemented with activiti:expression="${resumeService.storeResume()}" Of course, we need that bean or the process would not start. So let’s create it: @Component public class ResumeService { public void storeResume() { System.out.println("Storing resume ..."); } } When running the application now, you’ll see that the bean is called: . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v1.2.0.RELEASE) 2015-02-16 11:55:11.129 INFO 304 --- [ main] MyApp : Starting MyApp on The-Activiti-Machine.local with PID 304 ... Storing resume ... 2015-02-16 11:55:13.662 INFO 304 --- [ main] MyApp : Started MyApp in 2.788 seconds (JVM running for 3.067) And that’s it! Congrats with running your first process instance using Activiti in Spring Boot! Let’s spice things up a bit, and add following dependency to our pom.xml: org.activiti spring-boot-starter-rest-api ${activiti.version} Having this on the classpath does a nifty thing: it takes the Activiti REST API (which is written in Spring MVC) and exposes this fully in your application. The REST API of Activiti is fully documented in the Activiti User Guide. The REST API is secured by basic auth, and won’t have any users by default. Let’s add an admin user to the system as shown below (add this to the MyApp class). Don’t do this in a production system of course, there you’ll want to hook in the authentication to LDAP or something else. @Bean InitializingBean usersAndGroupsInitializer(final IdentityService identityService) { return new InitializingBean() { public void afterPropertiesSet() throws Exception { Group group = identityService.newGroup("user"); group.setName("users"); group.setType("security-role"); identityService.saveGroup(group); User admin = identityService.newUser("admin"); admin.setPassword("admin"); identityService.saveUser(admin); } }; } Start the application. We can now start a process instance as we did in the CommandLineRunner, but now using REST: curl -u admin:admin -H "Content-Type: application/json" -d '{"processDefinitionKey":"hireProcess", "variables": [ {"name":"applicantName", "value":"John Doe"}, {"name":"email", "value":"[email protected]"}, {"name":"phoneNumber", "value":"1234567"} ]}' http://localhost:8080/runtime/process-instances Which returns us the json representation of the process instance: { "tenantId": "", "url": "http://localhost:8080/runtime/process-instances/5", "activityId": "sid-42BAE58A-8FFB-4B02-AAED-E0D8EA5A7E39", "id": "5", "processDefinitionUrl": "http://localhost:8080/repository/process-definitions/hireProcess:1:4", "suspended": false, "completed": false, "ended": false, "businessKey": null, "variables": [], "processDefinitionId": "hireProcess:1:4" } I just want to stand still for a moment how cool this is. Just by adding one dependency, you’re getting the whole Activiti REST API embedded in your application! Let’s make it even cooler, and add following dependency org.activiti spring-boot-starter-actuator ${activiti.version} This adds a Spring Boot actuator endpoint for Activiti. If we restart the application, and hithttp://localhost:8080/activiti/, we get some basic stats about our processes. With some imagination that in a live system you’ve got many more process definitions deployed and executing, you can see how this is useful. The same actuator is also registered as a JMX bean exposing similar information. { completedTaskCountToday: 0, deployedProcessDefinitions: [ "hireProcess (v1)" ], processDefinitionCount: 1, cachedProcessDefinitionCount: 1, runningProcessInstanceCount: { hireProcess (v1): 0 }, completedTaskCount: 0, completedActivities: 0, completedProcessInstanceCount: { hireProcess (v1): 0 }, openTaskCount: 0 } To finish our coding, let’s create a dedicated REST endpoint for our hire process, that could be consumed by for example a javascript web application (out of scope for this article). So most likely, we’ll have a form for the applicant to fill in the details we’ve been passing programmatically above. And while we’re at it, let’s store the applicant information as a JPA entity. In that case, the data won’t be stored in Activiti anymore, but in a separate table and referenced by Activiti when needed. You probably guessed it by now, JPA support is enabled by adding a dependency: org.activiti spring-boot-starter-jpa ${activiti.version} and add the entity to the MyApp class: @Entity class Applicant { @Id @GeneratedValue private Long id; private String name; private String email; private String phoneNumber; // Getters and setters We’ll also need a Repository for this Entity (put this in a separate file or also in MyApp). No need for any methods, the Repository magic from Spring will generate the methods we need for us. public interface ApplicantRepository extends JpaRepository { // .. } And now we can create the dedicated REST endpoint: @RestController public class MyRestController { @Autowired private RuntimeService runtimeService; @Autowired private ApplicantRepository applicantRepository; @RequestMapping(value="/start-hire-process", method= RequestMethod.POST, produces= MediaType.APPLICATION_JSON_VALUE) public void startHireProcess(@RequestBody Map data) { Applicant applicant = new Applicant(data.get("name"), data.get("email"), data.get("phoneNumber")); applicantRepository.save(applicant); Map variables = new HashMap(); variables.put("applicant", applicant); runtimeService.startProcessInstanceByKey("hireProcessWithJpa", variables); } } Note we’re now using a slightly different process called ‘hireProcessWithJpa’, which has a few tweaks in it to cope with the fact the data is now in a JPA entity. So for example, we can’t use ${applicantName} anymore, but we now have to use ${applicant.name}. Let’s restart the application and start a new process instance: curl -u admin:admin -H "Content-Type: application/json" -d '{"name":"John Doe", "email": "[email protected]", "phoneNumber":"123456789"}' http://localhost:8080/start-hire-process We can now go through our process. You could create a custom endpoints for this too, exposing different task queries with different forms … but I’ll leave this to your imagination and use the default Activiti REST end points to walk through the process. Let’s see which task the process instance currently is at (you could pass in more detailed parameters here, for example the ‘processInstanceId’ for better filtering): curl -u admin:admin -H "Content-Type: application/json" http://localhost:8080/runtime/tasks which returns { "order": "asc", "size": 1, "sort": "id", "total": 1, "data": [{ "id": "14", "processInstanceId": "8", "createTime": "2015-02-16T13:11:26.078+01:00", "description": "Conduct a telephone interview with John Doe. Phone number = 123456789", "name": "Telephone interview" ... }], "start": 0 } So, our process is now at the Telephone interview. In a realistic application, there would be a task list and a form that could be filled in to complete this task. Let’s complete this task (we have to set the telephoneInterviewOutcome variable as the exclusive gateway uses it to route the execution): curl -u admin:admin -H "Content-Type: application/json" -d '{"action" : "complete", "variables": [ {"name":"telephoneInterviewOutcome", "value":true} ]}' http://localhost:8080/runtime/tasks/14 When we get the tasks again now, the process instance will have moved on to the two tasks in parallel in the subprocess (big rectangle): { "order": "asc", "size": 2, "sort": "id", "total": 2, "data": [ { ... "name": "Tech interview" }, { ... "name": "Financial negotiation" } ], "start": 0 } We can now continue the rest of the process in a similar fashion, but I’ll leave that to you to play around with. Testing One of the strengths of using Activiti for creating business processes is that everything is simply Java. As a consequence, processes can be tested as regular Java code with unit tests. Spring Boot makes writing such test a breeze. Here’s how the unit test for the “happy path” looks like (while omitting @Autowired fields and test e-mail server setup). The code also shows the use of the Activiti API’s for querying tasks for a given group and process instance. @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = {MyApp.class}) @WebAppConfiguration @IntegrationTest public class HireProcessTest { @Test public void testHappyPath() { // Create test applicant Applicant applicant = new Applicant("John Doe", "[email protected]", "12344"); applicantRepository.save(applicant); // Start process instance Map variables = new HashMap(); variables.put("applicant", applicant); ProcessInstance processInstance = runtimeService.startProcessInstanceByKey("hireProcessWithJpa", variables); // First, the 'phone interview' should be active Task task = taskService.createTaskQuery() .processInstanceId(processInstance.getId()) .taskCandidateGroup("dev-managers") .singleResult(); Assert.assertEquals("Telephone interview", task.getName()); // Completing the phone interview with success should trigger two new tasks Map taskVariables = new HashMap(); taskVariables.put("telephoneInterviewOutcome", true); taskService.complete(task.getId(), taskVariables); List tasks = taskService.createTaskQuery() .processInstanceId(processInstance.getId()) .orderByTaskName().asc() .list(); Assert.assertEquals(2, tasks.size()); Assert.assertEquals("Financial negotiation", tasks.get(0).getName()); Assert.assertEquals("Tech interview", tasks.get(1).getName()); // Completing both should wrap up the subprocess, send out the 'welcome mail' and end the process instance taskVariables = new HashMap(); taskVariables.put("techOk", true); taskService.complete(tasks.get(0).getId(), taskVariables); taskVariables = new HashMap(); taskVariables.put("financialOk", true); taskService.complete(tasks.get(1).getId(), taskVariables); // Verify email Assert.assertEquals(1, wiser.getMessages().size()); // Verify process completed Assert.assertEquals(1, historyService.createHistoricProcessInstanceQuery().finished().count()); } Next steps We haven’t touched any of the tooling around Activiti. There is a bunch more than just the engine, like the Eclipse plugin to design processes, a free web editor in the cloud (also included in the .zip download you can get from Activiti's site, a web application that showcases many of the features of the engine, … The current release of Activiti (version 5.17.0) has integration with Spring Boot 1.1.6. However, the current master version is compatible with 1.2.1. Using Spring Boot 1.2.0 brings us sweet stuff like support for XA transactions with JTA. This means you can hook up your processes easily with JMS, JPA and Activiti logic all in the same transaction! ..Which brings us to the next point … In this example, we’ve focussed heavily on human interactions (and barely touched it). But there’s many things you can do around orchestrating systems too. The Spring Boot integration also has Spring Integration support you could leverage to do just that in a very neat way! And of course there is much much more about the BPMN 2.0 standard. Read more about itin the Activiti docs.

The Streams API is a real gem in Java 8, and I keep finding more or less unexpected uses for them. I recently wrote about using them as ForkJoinPool facade. Here’s another interesting example: Walking recursive data structures. Without much ado, have a look at the code: class Tree { private int value; private List children = new LinkedList<>(); public Tree(int value, List children) { super(); this.value = value; this.children.addAll(children); } public Tree(int value, Tree... children) { this(value, asList(children)); } public int getValue() { return value; } public List getChildren() { return Collections.unmodifiableList(children); } public Stream flattened() { return Stream.concat( Stream.of(this), children.stream().flatMap(Tree::flattened)); } } It’s pretty boring, except for the few highlighted lines. Let’s say we want to be able to find elements matching some criteria in the tree or find particular element. One typical way to do it is a recursive function – but that has some complexity and is likely to need a mutable argument (e.g. a set where you can append matching elements). Another approach is iteration with a stack or a queue. They work fine, but take a few lines of code and aren’t so easy to generalize. Here’s what we can do with this flattened function: // Get all values in the tree: t.flattened().map(Tree::getValue).collect(toList()); // Get even values: t.flattened().map(Tree::getValue).filter(v -> v % 2 == 0).collect(toList()); // Sum of even values: t.flattened().map(Tree::getValue).filter(v -> v % 2 == 0).reduce((a, b) -> a + b); // Does it contain 13? t.flattened().anyMatch(t -> t.getValue() == 13); I think this solution is pretty slick and versatile. One line of code (here split to 3 for readability on blog) is enough to flatten the tree to a straightforward stream that can be searched, filtered and whatnot. It’s not perfect though: It is not lazy and flattened is called for each and every node in the tree every time. It probably could be improved using a Supplier. Anyway, it doesn’t matter for typical, reasonably small trees, especially in a business application on a very tall stack of libraries. But for very large trees, very frequent execution and tight time constraints the overhead might cause some trouble.

This article by Baron Schwartz comes to you from the DZone Guide to Database and Persistence Management.

Introduction JPA translates entity state transitions to database DML statements. Because it’s common to operate on entity graphs, JPA allows us to propagate entity state changes from Parents to Child entities. This behavior is configured through the CascadeType mappings. JPA vs Hibernate Cascade Types Hibernate supports all JPA Cascade Types and some additional legacy cascading styles. The following table draws an association between JPA Cascade Types and their Hibernate native API equivalent: JPA EntityManager action JPA CascadeType Hibernate native Session action Hibernate native CascadeType Event Listener detach(entity) DETACH evict(entity) DETACH or EVICT Default Evict Event Listener merge(entity) MERGE merge(entity) MERGE Default Merge Event Listener persist(entity) PERSIST persist(entity) PERSIST Default Persist Event Listener refresh(entity) REFRESH refresh(entity) REFRESH Default Refresh Event Listener remove(entity) REMOVE delete(entity) REMOVE orDELETE Default Delete Event Listener saveOrUpdate(entity) SAVE_UPDATE Default Save Or Update Event Listener replicate(entity, replicationMode) REPLICATE Default Replicate Event Listener lock(entity, lockModeType) buildLockRequest(entity, lockOptions) LOCK Default Lock Event Listener All the above EntityManager methods ALL All the above Hibernate Session methods ALL From this table we can conclude that: There’s no difference between calling persist, merge or refresh on the JPAEntityManager or the Hibernate Session. The JPA remove and detach calls are delegated to Hibernate delete and evict native operations. Only Hibernate supports replicate and saveOrUpdate. While replicate is useful for some very specific scenarios (when the exact entity state needs to be mirrored between two distinct DataSources), the persist and merge combo is always a better alternative than the native saveOrUpdate operation. As a rule of thumb, you should always use persist for TRANSIENT entities and merge for DETACHED ones.The saveOrUpdate shortcomings (when passing a detached entity snapshot to aSession already managing this entity) had lead to the merge operation predecessor: the now extinct saveOrUpdateCopy operation. The JPA lock method shares the same behavior with Hibernate lock request method. The JPA CascadeType.ALL doesn’t only apply to EntityManager state change operations, but to all Hibernate CascadeTypes as well. So if you mapped your associations with CascadeType.ALL, you can still cascade Hibernate specific events. For example, you can cascade the JPA lock operation (although it behaves as reattaching, instead of an actual lock request propagation), even if JPA doesn’t define a LOCK CascadeType. Cascading best practices Cascading only makes sense only for Parent – Child associations (the Parent entity state transition being cascaded to its Child entities). Cascading from Child to Parent is not very useful and usually, it’s a mapping code smell. Next, I’m going to take analyse the cascading behaviour of all JPA Parent – Childassociations. One-To-One The most common One-To-One bidirectional association looks like this: @Entity public class Post { @Id @GeneratedValue(strategy = GenerationType.AUTO) private Long id; private String name; @OneToOne(mappedBy = "post", cascade = CascadeType.ALL, orphanRemoval = true) private PostDetails details; public Long getId() { return id; } public PostDetails getDetails() { return details; } public String getName() { return name; } public void setName(String name) { this.name = name; } public void addDetails(PostDetails details) { this.details = details; details.setPost(this); } public void removeDetails() { if (details != null) { details.setPost(null); } this.details = null; } } @Entity public class PostDetails { @Id @GeneratedValue(strategy = GenerationType.AUTO) private Long id; @Column(name = "created_on") @Temporal(TemporalType.TIMESTAMP) private Date createdOn = new Date(); private boolean visible; @OneToOne @PrimaryKeyJoinColumn private Post post; public Long getId() { return id; } public void setVisible(boolean visible) { this.visible = visible; } public void setPost(Post post) { this.post = post; } } The Post entity plays the Parent role and the PostDetails is the Child. The bidirectional associations should always be updated on both sides, therefore the Parent side should contain the addChild andremoveChild combo. These methods ensure we always synchronize both sides of the association, to avoid Object or Relational data corruption issues. In this particular case, the CascadeType.ALL and orphan removal make sense because the PostDetails life-cycle is bound to that of its Post Parent entity. Cascading the one-to-one persist operation The CascadeType.PERSIST comes along with the CascadeType.ALL configuration, so we only have to persist the Post entity, and the associated PostDetails entity is persisted as well: Post post = new Post(); post.setName("Hibernate Master Class"); PostDetails details = new PostDetails(); post.addDetails(details); session.persist(post); Generating the following output: INSERT INTO post(id, NAME) VALUES (DEFAULT, Hibernate Master Class'') insert into PostDetails (id, created_on, visible) values (default, '2015-03-03 10:17:19.14', false) Cascading the one-to-one merge operation The CascadeType.MERGE is inherited from the CascadeType.ALL setting, so we only have to merge the Post entity and the associated PostDetails is merged as well: Post post = newPost(); post.setName("Hibernate Master Class Training Material"); post.getDetails().setVisible(true); doInTransaction(session -> { session.merge(post); }); The merge operation generates the following output: SELECT onetooneca0_.id AS id1_3_1_, onetooneca0_.NAME AS name2_3_1_, onetooneca1_.id AS id1_4_0_, onetooneca1_.created_on AS created_2_4_0_, onetooneca1_.visible AS visible3_4_0_ FROM post onetooneca0_ LEFT OUTER JOIN postdetails onetooneca1_ ON onetooneca0_.id = onetooneca1_.id WHERE onetooneca0_.id = 1 UPDATE postdetails SET created_on = '2015-03-03 10:20:53.874', visible = true WHERE id = 1 UPDATE post SET NAME = 'Hibernate Master Class Training Material' WHERE id = 1 Cascading the one-to-one delete operation The CascadeType.REMOVE is also inherited from the CascadeType.ALL configuration, so the Post entity deletion triggers a PostDetails entity removal too: Post post = newPost(); doInTransaction(session -> { session.delete(post); }); Generating the following output: delete from PostDetails where id = 1 delete from Post where id = 1 The one-to-one delete orphan cascading operation If a Child entity is dissociated from its Parent, the Child Foreign Key is set to NULL. If we want to have the Child row deleted as well, we have to use the orphan removalsupport. doInTransaction(session -> { Post post = (Post) session.get(Post.class, 1L); post.removeDetails(); }); The orphan removal generates this output: SELECT onetooneca0_.id AS id1_3_0_, onetooneca0_.NAME AS name2_3_0_, onetooneca1_.id AS id1_4_1_, onetooneca1_.created_on AS created_2_4_1_, onetooneca1_.visible AS visible3_4_1_ FROM post onetooneca0_ LEFT OUTER JOIN postdetails onetooneca1_ ON onetooneca0_.id = onetooneca1_.id WHERE onetooneca0_.id = 1 delete from PostDetails where id = 1 Unidirectional one-to-one association Most often, the Parent entity is the inverse side (e.g. mappedBy), the Child controling the association through its Foreign Key. But the cascade is not limited to bidirectional associations, we can also use it for unidirectional relationships: @Entity public class Commit { @Id @GeneratedValue(strategy = GenerationType.AUTO) private Long id; private String comment; @OneToOne(cascade = CascadeType.ALL) @JoinTable( name = "Branch_Merge_Commit", joinColumns = @JoinColumn( name = "commit_id", referencedColumnName = "id"), inverseJoinColumns = @JoinColumn( name = "branch_merge_id", referencedColumnName = "id") ) private BranchMerge branchMerge; public Commit() { } public Commit(String comment) { this.comment = comment; } public Long getId() { return id; } public void addBranchMerge( String fromBranch, String toBranch) { this.branchMerge = new BranchMerge( fromBranch, toBranch); } public void removeBranchMerge() { this.branchMerge = null; } } @Entity public class BranchMerge { @Id @GeneratedValue(strategy = GenerationType.AUTO) private Long id; private String fromBranch; private String toBranch; public BranchMerge() { } public BranchMerge( String fromBranch, String toBranch) { this.fromBranch = fromBranch; this.toBranch = toBranch; } public Long getId() { return id; } } Cascading consists in propagating the Parent entity state transition to one or more Child entities, and it can be used for both unidirectional and bidirectional associations. One-To-Many The most common Parent – Child association consists of a one-to-many and a many-to-one relationship, where the cascade being useful for the one-to-many side only: @Entity public class Post { @Id @GeneratedValue(strategy = GenerationType.AUTO) private Long id; private String name; @OneToMany(cascade = CascadeType.ALL, mappedBy = "post", orphanRemoval = true) private List comments = new ArrayList<>(); public void setName(String name) { this.name = name; } public List getComments() { return comments; } public void addComment(Comment comment) { comments.add(comment); comment.setPost(this); } public void removeComment(Comment comment) { comment.setPost(null); this.comments.remove(comment); } } @Entity public class Comment { @Id @GeneratedValue(strategy = GenerationType.AUTO) private Long id; @ManyToOne private Post post; private String review; public void setPost(Post post) { this.post = post; } public String getReview() { return review; } public void setReview(String review) { this.review = review; } } Like in the one-to-one example, the CascadeType.ALL and orphan removal are suitable because the Comment life-cycle is bound to that of its Post Parent entity. Cascading the one-to-many persist operation We only have to persist the Post entity and all the associated Comment entities are persisted as well: Post post = new Post(); post.setName("Hibernate Master Class"); Comment comment1 = new Comment(); comment1.setReview("Good post!"); Comment comment2 = new Comment(); comment2.setReview("Nice post!"); post.addComment(comment1); post.addComment(comment2); session.persist(post); The persist operation generates the following output: insert into Post (id, name) values (default, 'Hibernate Master Class') insert into Comment (id, post_id, review) values (default, 1, 'Good post!') insert into Comment (id, post_id, review) values (default, 1, 'Nice post!') Cascading the one-to-many merge operation Merging the Post entity is going to merge all Comment entities as well: Post post = newPost(); post.setName("Hibernate Master Class Training Material"); post.getComments() .stream() .filter(comment -> comment.getReview().toLowerCase() .contains("nice")) .findAny() .ifPresent(comment -> comment.setReview("Keep up the good work!") ); doInTransaction(session -> { session.merge(post); }); Generating the following output: SELECT onetomanyc0_.id AS id1_1_1_, onetomanyc0_.NAME AS name2_1_1_, comments1_.post_id AS post_id3_1_3_, comments1_.id AS id1_0_3_, comments1_.id AS id1_0_0_, comments1_.post_id AS post_id3_0_0_, comments1_.review AS review2_0_0_ FROM post onetomanyc0_ LEFT OUTER JOIN comment comments1_ ON onetomanyc0_.id = comments1_.post_id WHERE onetomanyc0_.id = 1 update Post set name = 'Hibernate Master Class Training Material' where id = 1 update Comment set post_id = 1, review='Keep up the good work!' where id = 2 Cascading the one-to-many delete operation When the Post entity is deleted, the associated Comment entities are deleted as well: Post post = newPost(); doInTransaction(session -> { session.delete(post); }); Generating the following output: delete from Comment where id = 1 delete from Comment where id = 2 delete from Post where id = 1 The one-to-many delete orphan cascading operation The orphan-removal allows us to remove the Child entity whenever it’s no longer referenced by its Parent: newPost(); doInTransaction(session -> { Post post = (Post) session.createQuery( "select p " + "from Post p " + "join fetch p.comments " + "where p.id = :id") .setParameter("id", 1L) .uniqueResult(); post.removeComment(post.getComments().get(0)); }); The Comment is deleted, as we can see in the following output: SELECT onetomanyc0_.id AS id1_1_0_, comments1_.id AS id1_0_1_, onetomanyc0_.NAME AS name2_1_0_, comments1_.post_id AS post_id3_0_1_, comments1_.review AS review2_0_1_, comments1_.post_id AS post_id3_1_0__, comments1_.id AS id1_0_0__ FROM post onetomanyc0_ INNER JOIN comment comments1_ ON onetomanyc0_.id = comments1_.post_id WHERE onetomanyc0_.id = 1 delete from Comment where id = 1 If you enjoy reading this article, you might want to subscribe to my newsletter and get a discount for my book as well. Many-To-Many The many-to-many relationship is tricky because each side of this association plays both the Parent and the Child role. Still, we can identify one side from where we’d like to propagate the entity state changes. We shouldn’t default to CascadeType.ALL, because the CascadeTpe.REMOVE might end-up deleting more than we’re expecting (as you’ll soon find out): @Entity public class Author { @Id @GeneratedValue(strategy=GenerationType.AUTO) private Long id; @Column(name = "full_name", nullable = false) private String fullName; @ManyToMany(mappedBy = "authors", cascade = {CascadeType.PERSIST, CascadeType.MERGE}) private List books = new ArrayList<>(); private Author() {} public Author(String fullName) { this.fullName = fullName; } public Long getId() { return id; } public void addBook(Book book) { books.add(book); book.authors.add(this); } public void removeBook(Book book) { books.remove(book); book.authors.remove(this); } public void remove() { for(Book book : new ArrayList<>(books)) { removeBook(book); } } } @Entity public class Book { @Id @GeneratedValue(strategy=GenerationType.AUTO) private Long id; @Column(name = "title", nullable = false) private String title; @ManyToMany(cascade = {CascadeType.PERSIST, CascadeType.MERGE}) @JoinTable(name = "Book_Author", joinColumns = { @JoinColumn( name = "book_id", referencedColumnName = "id" ) }, inverseJoinColumns = { @JoinColumn( name = "author_id", referencedColumnName = "id" ) } ) private List authors = new ArrayList<>(); private Book() {} public Book(String title) { this.title = title; } } Cascading the many-to-many persist operation Persisting the Author entities will persist the Books as well: Author _John_Smith = new Author("John Smith"); Author _Michelle_Diangello = new Author("Michelle Diangello"); Author _Mark_Armstrong = new Author("Mark Armstrong"); Book _Day_Dreaming = new Book("Day Dreaming"); Book _Day_Dreaming_2nd = new Book("Day Dreaming, Second Edition"); _John_Smith.addBook(_Day_Dreaming); _Michelle_Diangello.addBook(_Day_Dreaming); _John_Smith.addBook(_Day_Dreaming_2nd); _Michelle_Diangello.addBook(_Day_Dreaming_2nd); _Mark_Armstrong.addBook(_Day_Dreaming_2nd); session.persist(_John_Smith); session.persist(_Michelle_Diangello); session.persist(_Mark_Armstrong); The Book and the Book_Author rows are inserted along with the Authors: insert into Author (id, full_name) values (default, 'John Smith') insert into Book (id, title) values (default, 'Day Dreaming') insert into Author (id, full_name) values (default, 'Michelle Diangello') insert into Book (id, title) values (default, 'Day Dreaming, Second Edition') insert into Author (id, full_name) values (default, 'Mark Armstrong') insert into Book_Author (book_id, author_id) values (1, 1) insert into Book_Author (book_id, author_id) values (1, 2) insert into Book_Author (book_id, author_id) values (2, 1) insert into Book_Author (book_id, author_id) values (2, 2) insert into Book_Author (book_id, author_id) values (3, 1) Dissociating one side of the many-to-many association To delete an Author, we need to dissociate all Book_Author relations belonging to the removable entity: doInTransaction(session -> { Author _Mark_Armstrong = getByName(session, "Mark Armstrong"); _Mark_Armstrong.remove(); session.delete(_Mark_Armstrong); }); This use case generates the following output: SELECT manytomany0_.id AS id1_0_0_, manytomany2_.id AS id1_1_1_, manytomany0_.full_name AS full_nam2_0_0_, manytomany2_.title AS title2_1_1_, books1_.author_id AS author_i2_0_0__, books1_.book_id AS book_id1_2_0__ FROM author manytomany0_ INNER JOIN book_author books1_ ON manytomany0_.id = books1_.author_id INNER JOIN book manytomany2_ ON books1_.book_id = manytomany2_.id WHERE manytomany0_.full_name = 'Mark Armstrong' SELECT books0_.author_id AS author_i2_0_0_, books0_.book_id AS book_id1_2_0_, manytomany1_.id AS id1_1_1_, manytomany1_.title AS title2_1_1_ FROM book_author books0_ INNER JOIN book manytomany1_ ON books0_.book_id = manytomany1_.id WHERE books0_.author_id = 2 delete from Book_Author where book_id = 2 insert into Book_Author (book_id, author_id) values (2, 1) insert into Book_Author (book_id, author_id) values (2, 2) delete from Author where id = 3 The many-to-many association generates way too many redundant SQL statements and often, they are very difficult to tune. Next, I’m going to demonstrate the many-to-many CascadeType.REMOVE hidden dangers. The many-to-many CascadeType.REMOVE gotchas The many-to-many CascadeType.ALL is another code smell, I often bump into while reviewing code. The CascadeType.REMOVE is automatically inherited when usingCascadeType.ALL, but the entity removal is not only applied to the link table, but to the other side of the association as well. Let’s change the Author entity books many-to-many association to use theCascadeType.ALL instead: @ManyToMany(mappedBy = "authors", cascade = CascadeType.ALL) private List books = new ArrayList<>(); When deleting one Author: doInTransaction(session -> { Author _Mark_Armstrong = getByName(session, "Mark Armstrong"); session.delete(_Mark_Armstrong); Author _John_Smith = getByName(session, "John Smith"); assertEquals(1, _John_Smith.books.size()); }); All books belonging to the deleted Author are getting deleted, even if other Authorswe’re still associated to the deleted Books: SELECT manytomany0_.id AS id1_0_, manytomany0_.full_name AS full_nam2_0_ FROM author manytomany0_ WHERE manytomany0_.full_name = 'Mark Armstrong' SELECT books0_.author_id AS author_i2_0_0_, books0_.book_id AS book_id1_2_0_, manytomany1_.id AS id1_1_1_, manytomany1_.title AS title2_1_1_ FROM book_author books0_ INNER JOIN book manytomany1_ ON books0_.book_id = manytomany1_.id WHERE books0_.author_id = 3 delete from Book_Author where book_id=2 delete from Book where id=2 delete from Author where id=3 Most often, this behavior doesn’t match the business logic expectations, only being discovered upon the first entity removal. We can push this issue even further, if we set the CascadeType.ALL to the Book entity side as well: @ManyToMany(cascade = CascadeType.ALL) @JoinTable(name = "Book_Author", joinColumns = { @JoinColumn( name = "book_id", referencedColumnName = "id" ) }, inverseJoinColumns = { @JoinColumn( name = "author_id", referencedColumnName = "id" ) } ) This time, not only the Books are being deleted, but Authors are deleted as well: doInTransaction(session -> { Author _Mark_Armstrong = getByName(session, "Mark Armstrong"); session.delete(_Mark_Armstrong); Author _John_Smith = getByName(session, "John Smith"); assertNull(_John_Smith); }); The Author removal triggers the deletion of all associated Books, which further triggers the removal of all associated Authors. This is a very dangerous operation, resulting in a massive entity deletion that’s rarely the expected behavior. If you enjoyed this article, I bet you are going to love my book as well. SELECT manytomany0_.id AS id1_0_, manytomany0_.full_name AS full_nam2_0_ FROM author manytomany0_ WHERE manytomany0_.full_name = 'Mark Armstrong' SELECT books0_.author_id AS author_i2_0_0_, books0_.book_id AS book_id1_2_0_, manytomany1_.id AS id1_1_1_, manytomany1_.title AS title2_1_1_ FROM book_author books0_ INNER JOIN book manytomany1_ ON books0_.book_id = manytomany1_.id WHERE books0_.author_id = 3 SELECT authors0_.book_id AS book_id1_1_0_, authors0_.author_id AS author_i2_2_0_, manytomany1_.id AS id1_0_1_, manytomany1_.full_name AS full_nam2_0_1_ FROM book_author authors0_ INNER JOIN author manytomany1_ ON authors0_.author_id = manytomany1_.id WHERE authors0_.book_id = 2 SELECT books0_.author_id AS author_i2_0_0_, books0_.book_id AS book_id1_2_0_, manytomany1_.id AS id1_1_1_, manytomany1_.title AS title2_1_1_ FROM book_author books0_ INNER JOIN book manytomany1_ ON books0_.book_id = manytomany1_.id WHERE books0_.author_id = 1 SELECT authors0_.book_id AS book_id1_1_0_, authors0_.author_id AS author_i2_2_0_, manytomany1_.id AS id1_0_1_, manytomany1_.full_name AS full_nam2_0_1_ FROM book_author authors0_ INNER JOIN author manytomany1_ ON authors0_.author_id = manytomany1_.id WHERE authors0_.book_id = 1 SELECT books0_.author_id AS author_i2_0_0_, books0_.book_id AS book_id1_2_0_, manytomany1_.id AS id1_1_1_, manytomany1_.title AS title2_1_1_ FROM book_author books0_ INNER JOIN book manytomany1_ ON books0_.book_id = manytomany1_.id WHERE books0_.author_id = 2 delete from Book_Author where book_id=2 delete from Book_Author where book_id=1 delete from Author where id=2 delete from Book where id=1 delete from Author where id=1 delete from Book where id=2 delete from Author where id=3 This use case is wrong in so many ways. There are a plethora of unnecessary SELECT statements and eventually we end up deleting all Authors and all their Books. That’s why CascadeType.ALL should raise your eyebrow, whenever you spot it on a many-to-many association. When it comes to Hibernate mappings, you should always strive for simplicity. TheHibernate documentation confirms this assumption as well: Practical test cases for real many-to-many associations are rare. Most of the time you need additional information stored in the “link table”. In this case, it is much better to use two one-to-many associations to an intermediate link class. In fact, most associations are one-to-many and many-to-one. For this reason, you should proceed cautiously when using any other association style. Conclusion Cascading is a handy ORM feature, but it’s not free of issues. You should only cascade from Parent entities to Children and not the other way around. You should always use only the casacde operations that are demanded by your business logic requirements, and not turn the CascadeType.ALL into a default Parent-Child association entity state propagation configuration. Code available on GitHub.

In the traditional 3-tier architecture, data processing is performed by the application server where the data itself is stored in the database server. Application server and database server are typically two different machine. Therefore, the processing cycle proceeds as follows Application server send a query to the database server to retrieve the necessary data Application server perform processing on the received data Application server will save the changed data to the database server In the traditional data processing paradigm, we move data to the code. It can be depicted as follows ... Then big data phenomenon arrives. Because the data volume is huge, it cannot be hold by a single database server. Big data is typically partitioned and stored across many physical DB server machines. On the other hand, application servers need to be added to increase the processing power of big data. However, as we increase the number of App servers and DB servers for storing and processing the big data, more data need to be transfer back and forth across the network during the processing cycle, up to a point where the network becomes a major bottleneck. Moving Code to Data To overcome the network bottleneck, we need a new computing paradigm. Instead of moving data to the code, we move the code to the data and perform the processing at where the data is stored. Notice the change of the program structure The program execution starts at a driver, which orchestrate the execution happening remotely across many worker servers within a cluster. Data is no longer transferred to the driver program, the driver program holds a data reference in its variable rather than the data itself. The data reference is basically an id to locate the corresponding data residing in the database server Code is shipped from the program to the database server, where the execution is happening, and data is modified at the database server without leaving the server machine. Finally the program request a save of the modified data. Since the modified data resides in the database server, no data transfer happens over the network. By moving the code to the data, the volume of data transfer over network is significantly reduced. This is an important paradigm shift for big data processing. In the following session, I will use Apache Spark to illustrate how this big data processing paradigm is implemented. RDD Resilient Distributed Dataset (RDD) is how Spark implements the data reference concept. RDD is a logical reference of a dataset which is partitioned across many server machines in the cluster. To make a clear distinction between data reference and data itself, a Spark program is organized as a sequence of execution steps, which can either be a "transformation" or an "action". Programming Model A typical program is organized as follows From an environment variable "context", create some initial data reference RDD objects Transform initial RDD objects to create more RDD objects. Transformation is expressed in terms of functional programming where a code block is shipped from the driver program to multiple remote worker server, which hold a partition of the RDD. Variable appears inside the code block can either be an item of the RDD or a local variable inside the driver program which get serialized over to the worker machine. After the code (and the copy of the serialized variables) is received by the remote worker server, it will be executed there by feeding the items of RDD residing in that partition. Notice that the result of a transformation is a brand new RDD (the original RDD is not mutated) Finally, the RDD object (the data reference) will need to be materialized. This is achieved through an "action", which will dump the RDD into a storage, or return its value data to the driver program. Here is a word count example # Get initial RDD from the context file = spark.textFile("hdfs://...") # Three consecutive transformation of the RDD counts = file.flatMap(lambda line: line.split(" ")) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a + b) # Materialize the RDD using an action counts.saveAsTextFile("hdfs://...") When the driver program starts its execution, it builds up a graph where nodes are RDD and edges are transformation steps. However, no execution is happening at the cluster until an action is encountered. At that point, the driver program will ship the execution graph as well as the code block to the cluster, where every worker server will get a copy. The execution graph is a DAG. Each DAG is a atomic unit of execution. Each source node (no incoming edge) is an external data source or driver memory Each intermediate node is a RDD Each sink node (no outgoing edge) is an external data source or driver memory Green edge connecting to RDD represents a transformation. Red edge connecting to a sink node represents an action Data Shuffling Although we ship the code to worker server where the data processing happens, data movement cannot be completely eliminated. For example, if the processing requires data residing in different partitions to be grouped first, then we need to shuffle data among worker server. Spark carefully distinguish "transformation" operation in two types. "Narrow transformation" refers to the processing where the processing logic depends only on data that is already residing in the partition and data shuffling is unnecessary. Examples of narrow transformation includes filter(), sample(), map(), flatMap() .... etc. "Wide transformation" refers to the processing where the processing logic depends on data residing in multiple partitions and therefore data shuffling is needed to bring them together in one place. Example of wide transformation includes groupByKey(), reduceByKey() ... etc. Joining two RDD can also affect the amount of data being shuffled. Spark provides two ways to join data. In a shuffle join implementation, data of two RDD with the same key will be redistributed to the same partition. In other words, each of the items in each RDD will be shuffled across worker servers. Beside shuffle join, Spark provides another alternative call broadcast join. In this case, one of the RDD will be broadcasted and copied over to every partition. Imagine the situation when one of the RDD is significantly smaller relative to the other, then broadcast join will reduce the network traffic because only the small RDD need to be copied to all worker servers while the large RDD doesn't need to be shuffled at all. In some cases, transformation can be re-ordered to reduce the amount of data shuffling. Below is an example of a JOIN between two huge RDDs followed by a filtering. Plan1 is a naive implementation which follows the given order. It first join the two huge RDD and then apply the filter on the join result. This ends up causing a big data shuffling because the two RDD is huge, even though the result after filtering is small. Plan2 offers a smarter way by using the "push-down-predicate" technique where we first apply the filtering in both RDDs before joining them. Since the filtering will reduce the number of items of each RDD significantly, the join processing will be much cheaper. Execution Planning As explain above, data shuffling incur the most significant cost in the overall data processing flow. Spark provides a mechanism that generate an execute plan from the DAG that minimize the amount of data shuffling. Analyze the DAG to determine the order of transformation. Notice that we starts from the action (terminal node) and trace back to all dependent RDDs. To minimize data shuffling, we group the narrow transformation together in a "stage" where all transformation tasks can be performed within the partition and no data shuffling is needed. The transformations becomes tasks that are chained together within a stage Wide transformation sits at the boundary of two stages, which requires data to be shuffled to a different worker server. When a stage finishes its execution, it persist the data into different files (one per partition) of the local disks. Worker nodes of the subsequent stage will come to pickup these files and this is where data shuffling happens Below is an example how the execution planning turns the DAG into an execution plan involving stages and tasks. Reliability and Fault Resiliency Since the DAG defines a deterministic transformation steps between different partitions of data within each RDD RDD, fault recovery is very straightforward. Whenever a worker server crashes during the execution of a stage, another worker server can simply re-execute the stage from the beginning by pulling the input data from its parent stage that has the output data stored in local files. In case the result of the parent stage is not accessible (e.g. the worker server lost the file), the parent stage need to be re-executed as well. Imagine this is a lineage of transformation steps, and any failure of a step will trigger a restart of execution from its last step. Since the DAG itself is an atomic unit of execution, all the RDD values will be forgotten after the DAG finishes its execution. Therefore, after the driver program finishes an action (which execute a DAG to its completion), all the RDD value will be forgotten and if the program access the RDD again in subsequent statement, the RDD needs to be recomputed again from its dependents. To reduce this repetitive processing, Spark provide a caching mechanism to remember RDDs in worker server memory (or local disk). Once the execution planner finds the RDD is already cache in memory, it will use the RDD right away without tracing back to its parent RDDs. This way, we prune the DAG once we reach an RDD that is in the cache. Overall speaking, Apache Spark provides a powerful framework for big data processing. By the caching mechanism that holds previous computation result in memory, Spark out-performs Hadoop significantly because it doesn't need to persist all the data into disk for each round of parallel processing. Although it is still very new, I think Spark will take off as the main stream approach to process big data.

What does implementing microservices mean for a software developer? Especially, for the rookies, greenhorns, and newbs out there? I’m not talking about microservice software architecture here; this is about microservices software development. And not just that, the ultimate implementation goal should be “microservices done right”. For this post, I’ll go with Java. Yes, it’s wordy. Yes, it’s resource intensive (especially when used for the sole purpose of returning a single string). However the concept of classes and objects goes well with my intention of explaining how to do microservices correctly. Plus, it makes sense to use microservices in environments that are heavily biased towards Java. Anyway, please feel free to add your own “Hello, World!” microservice in your favorite language in the comments section below. Hello, monolith! As a prerequisite, you should be familiar with the following piece of code, what it does, and why it has to look the way it does (read this tutorial if you don’t): class Starter { public static void main(String[] args) { System.out.println(“Hello, World!”); } } This is a simple console application that yields the string “Hello, World!” This is not written in the microservice way. This is an example of when not to use the microservices approach: if all you need on your console is a single string, this is all you need. Hello, code duplication! In addition to this console application, I want this string to be available on the web by calling http://localhost:80/helloWorld.servlet from a browser. Here is the required code, implemented as plain HTTP servlet (yes, it’s wordy. Get over it.) class HelloWorldServlet extends HttpServlet { public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.getWriter().println(“Hello, World!”); } } The string “Hello, World!” has to be “implemented” again. Sure, this is no big deal. But this simple string could be so much more. It could be the result of a complex calculation or it could be the result of a time consuming search query. So, just imagine that the string “Hello, World!” is the result of a week’s worth of hard work (If you’re new to programming, it may very well be...). How should you go about making it available to apps and services that you create? Step 1: HelloWorldService.java To save yourself from duplicating a week’s worth of coding, allow me to introduce the HelloWorldService class: class HelloWorldService { public String greet() { return “Hello, World!”; } } You can re-use this fine piece of software craftmanship in all your apps and classes without re-implementing or duplicating code. Here’s our console application again: class Starter { HelloWorldService helloWorldService = new HelloWorldService(); public static void main(String[] args) { String message = helloWorldService.greet(); System.out.println(message); } } The same goes for servlets: class HelloWorldServlet extends HttpServlet { HelloWorldService helloWorldService = new HelloWorldService(); public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String message = helloWorldService.greet(); response.getWriter().println(message); } } It also works great for Spring MVC controllers: @Controller class HelloWorldController { HelloWorldService helloWorldService = new HelloWorldService(); @RequestMapping("/helloWorld") public String greet() { String message = helloWorldService.greet(); return message; } } I could go on and show more examples, but I think you get the point (spoiler: it’s the bold lines that matter). Those of you who are familiar with microservices could point out that this may be fine for getting rid of code duplication, but this is no microservice. You’re right, but to get to “microservices done right,” you have to be able to separate you app’s concerns, which is what I did here in the most possible basic way: I separated the app’s frontend concerns from its backend concerns. The frontend is either a console app or a servlet, the backend is HelloWorldService. Serviceward, ho! To go down microservice lane from here, all we have to do is wrap HelloWorldService into some kind of web component that makes it accessible via HTTP, right? Let’s see… First, we could just use our servlet code from above, as it conveniently returns the string as a response to any HTTP request. But we won’t. Why? Because there’s something missing: fault tolerance. What could possibly fail when returning a simple string? That’s not the point. What matters is that the client side (the code that calls HelloWorldService) should be given enough information to effectively react to failures. We face two possible problems: The service as a whole may be unavailable The service may be unable to return a proper response The service is unavailable If a service is unavailable, it’s the client that is responsible for dealing with the situation. Frameworks like unirest.io save you the effort of writing many lines of code when dealing with HTTP requests. Future> future = Unirest.post("HTTP://helloworld.myservices.local/greet") .header("accept", "application/json") .asJsonAsync(new Callback() { public void failed(UnirestException e) { //tell them UI folks that the request went south } public void completed(HttpResponse response) { //extract data from response and fulfill it’s destiny } public void cancelled() { //shot a note to UI dept that the request got cancelled } } ); With this code, the client now knows when the service is not available or has timed out following no response. Wee can easily have an error message displayed in place of the string we expected to receive. Try/catch is probably the right solution here. Invalid responses however pose more of a challenge. The service fails If the service fails, we can just return a string with an appropriate error message. But how can you know if a message is an error message or a correct response? Yes, you can start every error message with [ERROR] or invent another “smart” (read: not-so-smart) workaround, but this won’t be a solution you’ll be proud of. And, there’s always the possibility that even valid responses may begin with ERROR because it’s simply part of the message. I’d go with JSON or XML for wrapping the answer. I prefer JSON because it’s a little less wordy than XML. And I really like using the JSON-HTML tool over at json.bloople.net for visualizing results during development. Of course, you might go for any of the numerous alternatives, like protobuf or a proprietary solution of your own. The main point is that you need to be able to apply structure to responses: { “status”:”ok”, ”message”:”Hello, World!” } By checking the status attribute, you can easily decide whether to handle an error or to display an appropriate message. { “status”:”error”, ”message”:”Invalid input parameter” } The possibilities are endless here. You can add an error code or additional properties. This all boils down to a single important point: apply structure to your responses. Structure, why? Because structure not only helps you keep your code maintainable, it also serves as the foundation of the API of your service. An API definition consists of more than a URL like this: GET HTTP://helloworld.myservices.local/greet API definitions also consist of the response structures that can be expected as a response (you know this already from a few lines back): { “status”:”ok”, ”message”:”Hello, World!” } Most important takeaway Keeping the API specifications of a service’s request and response stable is a key requirement for succeeding with microservices. Conclusion Are you (and your project) ready for microservices? If you read this and kept asking yourself, what good is all the overhead of microservices, then either your project won’t benefit from microservices or you’re just not there yet (for mindset perspective see my previous post about the value of microservices ). If you can’t stop thinking about microservices, then you probably are ready.

RESTEasy (and Jersey as well) contain a minimal web server within their libraries which enables their users to start up a tiny web server.

Many organizations in which I've coached are concerned when the people on their teams aren't being "utilized" at 100% of their capacity. If someone is at work for 8 hours per day, minus an hour for lunch and breaks, that's 7 hours of potential capacity. Some organizations are progressive enough to see that the organization's overhead of administrative activities lowers that value to 6 hours per day. By extension, a team's capacity is simply a multiple of the number of team members and the number of hours available to be utilized, i.e. a team of 5 has 30 person-hours per day. So, anything less than 30 person-hours per day spent on tasks means that a team isn't being utilized 100%. Which is bad, apparently. A Thought Exercise If you're reading this, you have a computer and it's connected to a network. Your computer might be a small one that makes phone calls or a large one that can also handle the complex scenery of games at 60 frames per second without breaking a sweat. Regardless, it's a computer. Thanks to John von Neumann, almost all computers today have the same basic architecture, the heart of which is the Central Processing Unit or CPU. When the CPU on your computer, regardless of its size, reaches 100% utilization, what happens? Your computer slows down. Significantly. The CPU is saturated with instructions and does its best to process them, but it has a finite limit on how quickly it can do that. There's a finite limit to the speed at which the instructions and the data they use can be passed around among the internal components of the CPU. That processor just can't go any faster. To a person using the computer, it appears to be locked up. To see this in action, try running a modern game like Faster Than Light on a 1st generation iPad. The older processor can't keep up with the computing demand from a game targeted towards more a powerful CPU. The communications network to which your computer is connected might be wireless or have a "hard" connection, but behind the scenes the public internet is dominated by TCP/IP and Ethernet. What happens when the network to which your computer is connected is running at 100% capacity? The network slows down. Significantly. The equipment moving the packets of data about will experience numerous collisions and will have to send requests back to your computer to resend data. The equipment will also begin to simply "drop" packets because it can't process them quickly enough. As far as you're concerned the network has become either very slow or has simply stopped. To see this in real life, look at what happens to a web site when it's the subject of a Denial of Service (DoS) attack, which effectively saturates that site's network with data requests. The site either crashes completely or appears to be unresponsive because it can't handle the traffic. Knowledge work such as software development requires a huge amount of thinking (processing) and a huge amount of collaboration (communication). The human brain is a processor, and like the von Neumann architecture has subcomponents (lobes), working memory, etc. What happens when our processor hits 100% utilization? We simply can't process all the inputs fast enough and our ability to process information slows down dramatically. We even reach a state where the decision-making centre in our brain shuts itself down. Coupled with a decreasing ability to process information and make decisions, we experience increasing levels of anxiety. Now, let's look at teams. They represent a communications network, with the people being the nodes in the network. When the team reaches 100% utilization, the individual processors (the brains of the people!) begin to slow as the ability to process information diminishes and anxiety increases. This has the effect of slowing the team's ability to communicate and operate effectively. When one or multiple people reach the point of shutting down, the network collapses. Why does anyone think that 100% utilization is a good thing? In manufacturing, 100% utilization of the capacity of your workers is actually desirable. The thinking aspects have already been done in the design and engineering of the item being created, and the assembly process is infinitely repeatable until a new item comes along. This approach to the process was taken from the manufacturing world and applied to the development of software, with the thinking that assembly was done by the programmers after all the design and engineering had taken place. Well, that simply isn't true. The "assembly" of software isn't programming, it's the compilation and packing into something deployable. Writing software is a confluence of design and engineering that has creative and technical aspects. It's not an assembly line, and probably won't ever be within my lifetime. As long ago as the late 1950's, management guru Peter Drucker realized that there were people who "think for a living". He coined the term knowledge worker to describe that type of work. Developing software is a classic knowledge worker role and therefore has different rules for productivity. As Drucker said in his 1999 book Management Challenges of the 21st Century, Productivity of the knowledge worker is not — at least not primarily — a matter of the quantity of output. Quality is at least as important. If we're pushing individuals and teams to 100% capacity, the quality of work and therefore the productivity of the team will be reduced. In 2001, Tom deMarco released Slack: Getting Past Burnout, Busywork and the Myth of Total Efficiency, an entire book describing the problems created by trying to achieve 100% utilization. I highly recommend the book, and also recommend you give your manager a copy. :) Here are some selected quotes that are pertinent to reducing the load on our processor and communication network: Very successful companies have never struck me as particularly busy; in fact, they are, as a group, rather laid back. Energy is evident in the workplace, but it's not the energy tinged with fear that comes from being slightly behind on everything. The principal resource needed for invention is slack. When companies can't invent, it's usually because their people are too damn busy. People under time pressure don't think faster. - Tim Lister How Can We Deal With This? First and foremost, stop trying to do so damned much! This is truly a slow down to go faster type of solution. For a Team If you're an agile team using iterations or sprints, pull less work into a sprint even if your velocity says you should be able to pull more. The extra time can be used to increase the quality of the work that you do complete, which is in itself a motivating factor for knowledge workers. Increased quality means decreased rework, which means the team as a whole delivers faster over the long term. Similarly, the reduced anxiety and stress on the team members increases their ability to think about what they're doing, meaning better and more innovative solutions to problems. Also, ensure that teams are focusing on activities that contribute directly to their work. Meetings tend to be the worst offenders of this, including the meetings defined within agile processes. Some suggestions: No meeting can be scheduled unless there is a specific question to be answered or specific information to be passed along. Reduce the default meeting length from 1 hour to 30 minutes in whatever calendar tool you use. Even better, reduce it to 20 minutes. Choose blocks of time that are deemed meeting free zones. No one, regardless of their position in the org chart can schedule a meeting with the team or the team members during that time. Minimize meetings where people are on a phone. My observation is that people who call into meetings tend to speak longer than when they are in the room with everyone else. I know that I do this, and I see it often in others. For an Individual Learn to say, "No." At the very least, learn to say, "Not yet." It's really difficult to do this, and I speak from personal experience. We want to help others, we want to be team players. However, if we don't say no, we take on too much, our processing and communication ability suffers and we end up disappointing those we were trying to help! Also, learn to take breaks. Yes, take breaks. The same concepts from Slack apply here, with brief diversions clearing our mind and allowing us to work better afterwards. At the extreme, if you feel like you can't keep your eyes open, take a nap! I highly recommend the Pomodoro Technique as at least a starting point for giving yourself some slack. Step away from your work for just a few minutes and return recharged. Conclusion We don't want the CPU in our computer to be working at 100% and we don't want the communications network to which it's connected to be at 100% capacity either. Our brains are processors and teams are a communications network, so why would we want those to be 100% busy all the time? In fact, ensuring that teams and the people on them are always busy is a provably wrong approach for software development. It has it's roots in a manufacturing mindset and developing software is knowledge work. Software development requires substantially different ways to make teams and the individual people on them as productive as possible. Finally, there is no magic number for what percentage of utilization is best. People are variable and 90% utilization for a person might be fine on a given day while %30 is all that the same person can handle on the next. Don't aim for a number, aim for an environment where people know that they don't have to appear busy. That will leave plenty of capacity for each person's processor and their team's communications network to run smoothly and deliver real value more effectively.

I’ve been playing around with dplyr over the weekend and wanted to extract the values from a data frame column to use in a later filtering step. I had a data frame: library(dplyr) df = data.frame(userId = c(1,2,3,4,5), score = c(2,3,4,5,5)) And wanted to extract the userIds of those people who have a score greater than 3. I started with: highScoringPeople = df %>% filter(score > 3) %>% select(userId) > highScoringPeople userId 1 3 2 4 3 5 And then filtered the data frame expecting to get back those 3 people: > df %>% filter(userId %in% highScoringPeople) [1] userId score <0 rows> (or 0-length row.names) No rows! I created vector with the numbers 3-5 to make sure that worked: > df %>% filter(userId %in% c(3,4,5)) userId score 1 3 4 2 4 5 3 5 5 That works as expected so highScoringPeople obviously isn’t in the right format to facilitate an ‘in lookup’. Let’s explore: > str(c(3,4,5)) num [1:3] 3 4 5 > str(highScoringPeople) 'data.frame': 3 obs. of 1 variable: $ userId: num 3 4 5 Now it’s even more obvious why it doesn’t work – highScoringPeople is still a data frame when we need it to be a vector/list. One way to fix this is to extract the userIds using the $ syntax instead of the select function: highScoringPeople = (df %>% filter(score > 3))$userId > str(highScoringPeople) num [1:3] 3 4 5 > df %>% filter(userId %in% highScoringPeople) userId score 1 3 4 2 4 5 3 5 5 Or if we want to do the column selection using dplyr we can extract the values for the column like this: highScoringPeople = (df %>% filter(score > 3) %>% select(userId))[[1]] > str(highScoringPeople) num [1:3] 3 4 5 Not so difficult after all.

Like many other Java developers, when i start a new Java development project that requires a database, i have hopes and dreams of what my database looks like: Java API (of course) Embeddable Pure Java Simple jar file for inclusion in my project Database stored in a directory on disk Faster than a rocket First I’m going to review these points, and then i’m going to talk about the database i chose for my latest project, which is in production now with hundreds of users accessing the web application each month. What I Want from My Database Here’s what i’m looking for in my database. These are the things that literally make me happy and joyous when writing code. Java API I code in Java. It’s natural for me to want to use a modern Java API for my database work. Embeddable My development productivity and programming enjoyment skyrocket when my database is embedded. The database starts and stops with my application. It’s easy to destroy my database and restart from scratch. I can upgrade my database by updating my database jar file. It’s easy to deploy my application into testing and production, because there’s no separate database server to startup and manage. (I know about the issue with clustering and an embedded database, but i’ll get to that.) Pure Java Back when i developed software that would be deployed on all manner of hardware, i was a stickler that all my code be pure Java, so that i could be confident that my code would run wherever customers and users deployed it. In this day of SaaS, i’m less picky. I develop on the Mac. I test and run in production on Linux. Those are the systems i care about, so if my database has some platform-specific code in it to make it run fast and well, i’m fine with that. Just as long as that platform-specific configuration is not exposed to me as the developer. Simple Jar File for Inclusion in My Project I really just want one database jar file to add to my project. And i don’t want that jar file messing with my code or the dependencies i include in my project. If the database uses Guava 1.2, and i’m using Guava 0.8, that can mess me up. I want my database to not interfere with jars that i use by introducing newer or older versions of class files that i already reference in my project’s jars. Database Stored in a Directory on Disk I like to destroy my database by deleting a directory. I like to run multiple, simultaneous databases by configuring each database to use a separate directory. That makes me super productive during development, and it makes it more fun for me to program to a database. Faster Than a Rocket I think that’s just a given. My Latest Project That Needs a Database My latest project is Floify.com. Floify is a Mortgage Borrower Portal, automating the process of collecting mortgage loan documents from borrowers and emailing milestone loan status updates to real estate agents and borrowers. Mortgage loan originators use Floify to automate the labor-intensive parts of their loan processes. The web application receives about 500 unique visitors per month. Floify experienced 28% growth in january 2015. Floify’s vital statistics are: 38,301 loan documents under management 3,619 registered users 3,113 loan packages under management The Database I Chose for My Latest Project When i started Floify, i looked for a database that met all the criteria i’ve described above. I decided against databases that were server-based (Postgres, etc). I decided against databases that weren’t Java-based (MongoDB, etc). I decided against databases that didn’t support ACID transactions. I narrowed my choices to OrientDB and Neo4j. It’s been a couple years since that decision process occurred, but i distinctly remember a few reasons why i ultimately chose OrientDB over Neo4j: Performance benchmarks for OrientDB were very impressive. The OrientDB development team was very active. Cost. OrientDB is free. Neo4j cost more than what i was willing to pay or what i could afford. I forget which it was. My Favourite OrientDB Features Here are some of my favourite features in OrientDB. These are not competitive advantages to OrientDB. It’s just some of the things that make me happy when coding against an embeddable database. I can create the database in code. I don’t have to use SQL for querying, but most of the time, i do. I already know SQL, and it’s just easy for me. I use the document database, and it’s very pleasant inserting new documents in Java. I can store multi-megabyte binary objects directly in the database. My database is stored in a directory on disk. When scalability demands it, i can upgrade to a two-server distributed database. I haven’t been there yet. Speed. For me, OrientDB is very fast, and in the few years i’ve been using it, it’s become faster. OrientDB doesn’t come in a single jar file, as would be my ideal. I have to include a few different jars, but that’s an easy tradeoff for me. Future In the future, as Floify’s performance and scalability needs demand it, i’ll investigate a multi-server database configuration on OrientDB. In the meantime, i’m preparing to upgrade to OrientDB 2.0, which was recently released and promises even more speed. Go speed. :-)

Originally Written by Matt Kalan Welcome to part two of our three-part series on MongoDB and Hadoop. In part one, we introduced Hadoop and how to set it up. In this post, we'll look at a Hive example. Introduction & Setup of Hadoop and MongoDB Hive Example Spark Example & Key Takeaways For more detail on the use case, see the first paragraph of part 1. Summary Use case: aggregating 1 minute intervals of stock prices into 5 minute intervals Input:: 1 minute stock prices intervals in a MongoDB database Simple Analysis: performed in: - Hive - Spark Output: 5 minute stock prices intervals in Hadoop Hive Example I ran the following example from the Hive command line (simply typing the command “hive” with no parameters), not Cloudera’s Hue editor, as that would have needed additional installation steps. I immediately noticed the criticism people have with Hive, that everything is compiled into MapReduce which takes considerable time. I ran most things with just 20 records to make the queries run quickly. This creates the definition of the table in Hive that matches the structure of the data in MongoDB. MongoDB has a dynamic schema for variable data shapes but Hive and SQL need a schema definition. CREATE EXTERNAL TABLE minute_bars ( id STRUCT, Symbol STRING, Timestamp STRING, Day INT, Open DOUBLE, High DOUBLE, Low DOUBLE, Close DOUBLE, Volume INT ) STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id", "Symbol":"Symbol", "Timestamp":"Timestamp", "Day":"Day", "Open":"Open", "High":"High", "Low":"Low", "Close":"Close", "Volume":"Volume"}') TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/marketdata.minbars'); Recent changes in the Apache Hive repo make the mappings necessary even if you are keeping the field names the same. This should be changed in the MongoDB Hadoop Connector soon if not already by the time you read this. Then I ran the following command to create a Hive table for the 5 minute bars: CREATE TABLE five_minute_bars ( id STRUCT, Symbol STRING, Timestamp STRING, Open DOUBLE, High DOUBLE, Low DOUBLE, Close DOUBLE ); This insert statement uses the SQL windowing functions to group 5 1-minute periods and determine the OHLC for the 5 minutes. There are definitely other ways to do this but here is one I figured out. Grouping in SQL is a little different from grouping in the MongoDB aggregation framework (in which you can pull the first and last of a group easily), so it took me a little while to remember how to do it with a subquery. The subquery takes each group of 5 1-minute records/documents, sorts them by time, and takes the open, high, low, and close price up to that record in each 5-minute period. Then the outside WHERE clause selects the last 1-minute bar in that period (because that row in the subquery has the correct OHLC information for its 5-minute period). I definitely welcome easier queries to understand but you can run the subquery by itself to see what it’s doing too. INSERT INTO TABLE five_minute_bars SELECT m.id, m.Symbol, m.OpenTime as Timestamp, m.Open, m.High, m.Low, m.Close FROM (SELECT id, Symbol, FIRST_VALUE(Timestamp) OVER ( PARTITION BY floor(unix_timestamp(Timestamp, 'yyyy-MM-dd HH:mm')/(5*60)) ORDER BY Timestamp) as OpenTime, LAST_VALUE(Timestamp) OVER ( PARTITION BY floor(unix_timestamp(Timestamp, 'yyyy-MM-dd HH:mm')/(5*60)) ORDER BY Timestamp) as CloseTime, FIRST_VALUE(Open) OVER ( PARTITION BY floor(unix_timestamp(Timestamp, 'yyyy-MM-dd HH:mm')/(5*60)) ORDER BY Timestamp) as Open, MAX(High) OVER ( PARTITION BY floor(unix_timestamp(Timestamp, 'yyyy-MM-dd HH:mm')/(5*60)) ORDER BY Timestamp) as High, MIN(Low) OVER ( PARTITION BY floor(unix_timestamp(Timestamp, 'yyyy-MM-dd HH:mm')/(5*60)) ORDER BY Timestamp) as Low, LAST_VALUE(Close) OVER ( PARTITION BY floor(unix_timestamp(Timestamp, 'yyyy-MM-dd HH:mm')/(5*60)) ORDER BY Timestamp) as Close FROM minute_bars) as m WHERE unix_timestamp(m.CloseTime, 'yyyy-MM-dd HH:mm') - unix_timestamp(m.OpenTime, 'yyyy-MM-dd HH:mm') = 60*4; I can definitely see the benefit of being able to use SQL to access data in MongoDB and optionally in other databases and file formats, all with the same commands, while the mapping differences are handled in the table declarations. The downside is that the latency is quite high, but that could be made up some with the ability to scale horizontally across many nodes. I think this is the appeal of Hive for most people - they can scale to very large data volumes using traditional SQL, and latency is not a primary concern. Post #3 in this blog series shows similar examples using Spark. Introduction & Setup of Hadoop and MongoDB Hive Example Spark Example & Key Takeaways To learn more, watch our video on MongoDB and Hadoop. We will take a deep dive into the MongoDB Connector for Hadoop and how it can be applied to enable new business insights. WATCH MONGODB & HADOOP << Read Part 1

There are a number of distributed computation systems that can process Big Data in real time or near-real time. This article will start with a short description of three Apache frameworks, and attempt to provide a quick, high-level overview of some of their similarities and differences. Apache Storm In Storm, you design a graph of real-time computation called a topology, and feed it to the cluster where the master node will distribute the code among worker nodes to execute it. In a topology, data is passed around between spouts that emit data streams as immutable sets of key-value pairs called tuples, and bolts that transform those streams (count, filter etc.). Bolts themselves can optionally emit data to other bolts down the processing pipeline. Apache Spark Spark Streaming (an extension of the core Spark API) doesn’t process streams one at a time like Storm. Instead, it slices them in small batches of time intervals before processing them. The Spark abstraction for a continuous stream of data is called a DStream (for Discretized Stream). A DStream is a micro-batch of RDDs (Resilient Distributed Datasets). RDDs are distributed collections that can be operated in parallel by arbitrary functions and by transformations over a sliding window of data (windowed computations). Apache Samza Samza ’s approach to streaming is to process messages as they are received, one at a time. Samza’s stream primitive is not a tuple or a Dstream, but a message. Streams are divided into partitions and each partition is an ordered sequence of read-only messages with each message having a unique ID (offset). The system also supports batching, i.e. consuming several messages from the same stream partition in sequence. Samza`s Execution & Streaming modules are both pluggable, although Samza typically relies on Hadoop’s YARN (Yet Another Resource Negotiator) and Apache Kafka. Common Ground All three real-time computation systems are open-source, low-latency, distributed, scalable and fault-tolerant. They all allow you to run your stream processing code through parallel tasks distributed across a cluster of computing machines with fail-over capabilities. They also provide simple APIs to abstract the complexity of the underlying implementations. The three frameworks use different vocabularies for similar concepts: Comparison Matrix A few of the differences are summarized in the table below: There are three general categories of delivery patterns: At-most-once: messages may be lost. This is usually the least desirable outcome. At-least-once: messages may be redelivered (no loss, but duplicates). This is good enough for many use cases. Exactly-once: each message is delivered once and only once (no loss, no duplicates). This is a desirable feature although difficult to guarantee in all cases. Another aspect is state management. There are different strategies to store state. Spark Streaming writes data into the distributed file system (e.g. HDFS). Samza uses an embedded key-value store. With Storm, you’ll have to either roll your own state management at your application layer, or use a higher-level abstraction called Trident. Use Cases All three frameworks are particularly well-suited to efficiently process continuous, massive amounts of real-time data. So which one to use? There are no hard rules, at most a few general guidelines. If you want a high-speed event processing system that allows for incremental computations, Storm would be fine for that. If you further need to run distributed computations on demand, while the client is waiting synchronously for the results, you’ll have Distributed RPC (DRPC) out-of-the-box. Last but not least, because Storm uses Apache Thrift, you can write topologies in any programming language. If you need state persistence and/or exactly-once delivery though, you should look at the higher-level Trident API, which also offers micro-batching. A few companies using Storm: Twitter, Yahoo!, Spotify, The Weather Channel... Speaking of micro-batching, if you must have stateful computations, exactly-once delivery and don’t mind a higher latency, you could consider Spark Streaming…specially if you also plan for graph operations, machine learning or SQL access. The Apache Spark stack lets you combine several libraries with streaming (Spark SQL, MLlib, GraphX) and provides a convenient unifying programming model. In particular, streaming algorithms (e.g. streaming k-means) allow Spark to facilitate decisions in real-time. A few companies using Spark: Amazon, Yahoo!, NASA JPL, eBay Inc., Baidu… If you have a large amount of state to work with (e.g. many gigabytes per partition), Samza co-locates storage and processing on the same machines, allowing to work efficiently with state that won’t fit in memory. The framework also offers flexibility with its pluggable API: its default execution, messaging and storage engines can each be replaced with your choice of alternatives. Moreover, if you have a number of data processing stages from different teams with different codebases, Samza ‘s fine-grained jobs would be particularly well-suited, since they can be added/removed with minimal ripple effects. A few companies using Samza: LinkedIn, Intuit, Metamarkets, Quantiply, Fortscale… Conclusion We only scratched the surface of The Three Apaches. We didn’t cover a number of other features and more subtle differences between these frameworks. Also, it’s important to keep in mind the limits of the above comparisons, as these systems are constantly evolving.

Here I will consider two different ways of standing up a local instance of Netflix Eureka. If you are not familiar with Eureka, it provides a central registry where (micro)services can register themselves and client applications can use this registry to look up specific instances hosting a service and to make the service calls. Approach 1: Native Eureka Library The first way is to simply use the archive file generated by the Netflix Eureka build process: 1. Clone the Eureka source repository here: https://github.com/Netflix/eureka 2. Run "./gradlew build" at the root of the repository, this should build cleanly generating a war file in eureka-server/build/libs folder 3. Grab this file, rename it to "eureka.war" and place it in the webapps folder of either tomcat or jetty. For this exercise I have used jetty. 4. Start jetty, by default jetty will boot up at port 8080, however I wanted to instead bring it up at port 8761, so you can start it up this way, "java -jar start.jar -Djetty.port=8761" The server should start up cleanly and can be verified at this endpoint - "http://localhost:8761/eureka/v2/apps" Approach 2: Spring-Cloud-Netflix Spring-Cloud-Netflix provides a very neat way to bootstrap Eureka. To bring up Eureka server using Spring-Cloud-Netflix the approach that I followed was to clone the sample Eureka server application available here: https://github.com/spring-cloud-samples/eureka 1. Clone this repository 2. From the root of the repository run "mvn spring-boot:run", and that is it!. The server should boot up cleanly and the REST endpoint should come up here: "http://localhost:8761/eureka/apps". As a bonus, Spring-Cloud-Netflix provides a neat UI showing the various applications who have registered with Eureka at the root of the webapp at "http://localhost:8761/". Just a few small issues to be aware of, note that the context url's are a little different in the two cases "eureka/v2/apps" vs "eureka/apps", this can be adjusted on the configurations of the services which register with Eureka. Conclusion Your mileage with these approaches may vary. I have found Spring-Cloud-Netflix a little unstable at times but it has mostly worked out well for me. The documentation at the Spring-Cloud site is also far more exhaustive than the one provided at the Netflix Eureka site.

Although Java Architecture for XML Binding (JAXB) is fairly easy to use in nominal cases (especially since Java SE 6), it also presents numerous nuances. Some of the common nuances are due to the inability to exactlymatch (bind) XML Schema Definition (XSD) types to Java types. This post looks at one specific example of this that also demonstrates how different XSD constructs that enforce the same XML structure can lead to different Java types when the JAXB compiler generates the Java classes. The next code listing, for Food.xsd, defines a schema for food types. The XSD mandates that valid XML will have a root element called "Food" with three nested elements "Vegetable", "Fruit", and "Dessert". Although the approach used to specify the "Vegetable" and "Dessert" elements is different than the approach used to specify the "Fruit" element, both approaches result in similar "valid XML." The "Vegetable" and "Dessert" elements are declared directly as elements of the prescribed simpleTypes defined later in the XSD. The "Fruit" element is defined via reference (ref=) to another defined element that consists of a simpleType. Food.xsd Although Vegetable and Dessert elements are defined in the schema differently than Fruit, the resulting valid XML is the same. A valid XML file is shown next in the code listing for food1.xml. food1.xml Spinach Watermelon Pie At this point, I'll use a simple Groovy script to validate the above XML against the above XSD. The code for this Groovy XML validation script (validateXmlAgainstXsd.groovy) is shown next. validateXmlAgainstXsd.groovy #!/usr/bin/env groovy // validateXmlAgainstXsd.groovy // // Accepts paths/names of two files. The first is the XML file to be validated // and the second is the XSD against which to validate that XML. if (args.length < 2) { println "USAGE: groovy validateXmlAgainstXsd.groovy " System.exit(-1) } String xml = args[0] String xsd = args[1] import javax.xml.validation.Schema import javax.xml.validation.SchemaFactory import javax.xml.validation.Validator try { SchemaFactory schemaFactory = SchemaFactory.newInstance(javax.xml.XMLConstants.W3C_XML_SCHEMA_NS_URI) Schema schema = schemaFactory.newSchema(new File(xsd)) Validator validator = schema.newValidator() validator.validate(new javax.xml.transform.stream.StreamSource(xml)) } catch (Exception exception) { println "\nERROR: Unable to validate ${xml} against ${xsd} due to '${exception}'\n" System.exit(-1) } println "\nXML file ${xml} validated successfully against ${xsd}.\n" The next screen snapshot demonstrates running the above Groovy XML validation script against food1.xmland Food.xsd. The objective of this post so far has been to show how different approaches in an XSD can lead to the same XML being valid. Although these different XSD approaches prescribe the same valid XML, they lead to different Java class behavior when JAXB is used to generate classes based on the XSD. The next screen snapshot demonstrates running the JDK-provided JAXB xjc compiler against the Food.xsd to generate the Java classes. The output from the JAXB generation shown above indicates that Java classes were created for the "Vegetable" and "Dessert" elements but not for the "Fruit" element. This is because "Vegetable" and "Dessert" were defined differently than "Fruit" in the XSD. The next code listing is for the Food.java class generated by the xjc compiler. From this we can see that the generated Food.java class references specific generated Java types for Vegetable and Dessert, but references simply a generic Java String for Fruit. Food.java (generated by JAXB jxc compiler) // // This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802 // See http://java.sun.com/xml/jaxb // Any modifications to this file will be lost upon recompilation of the source schema. // Generated on: 2015.02.11 at 10:17:32 PM MST // package com.blogspot.marxsoftware.foodxml; import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; import javax.xml.bind.annotation.XmlElement; import javax.xml.bind.annotation.XmlRootElement; import javax.xml.bind.annotation.XmlSchemaType; import javax.xml.bind.annotation.XmlType; /** * Java class for anonymous complex type. * * The following schema fragment specifies the expected content contained within this class. * * * * * * * * * * * * * * * * */ @XmlAccessorType(XmlAccessType.FIELD) @XmlType(name = "", propOrder = { "vegetable", "fruit", "dessert" }) @XmlRootElement(name = "Food") public class Food { @XmlElement(name = "Vegetable", required = true) @XmlSchemaType(name = "string") protected Vegetable vegetable; @XmlElement(name = "Fruit", required = true) protected String fruit; @XmlElement(name = "Dessert", required = true) @XmlSchemaType(name = "string") protected Dessert dessert; /** * Gets the value of the vegetable property. * * @return * possible object is * {@link Vegetable } * */ public Vegetable getVegetable() { return vegetable; } /** * Sets the value of the vegetable property. * * @param value * allowed object is * {@link Vegetable } * */ public void setVegetable(Vegetable value) { this.vegetable = value; } /** * Gets the value of the fruit property. * * @return * possible object is * {@link String } * */ public String getFruit() { return fruit; } /** * Sets the value of the fruit property. * * @param value * allowed object is * {@link String } * */ public void setFruit(String value) { this.fruit = value; } /** * Gets the value of the dessert property. * * @return * possible object is * {@link Dessert } * */ public Dessert getDessert() { return dessert; } /** * Sets the value of the dessert property. * * @param value * allowed object is * {@link Dessert } * */ public void setDessert(Dessert value) { this.dessert = value; } } The advantage of having specific Vegetable and Dessert classes is the additional type safety they bring as compared to a general Java String. Both Vegetable.java and Dessert.java are actually enums because they come from enumerated values in the XSD. The two generated enums are shown in the next two code listings. Vegetable.java (generated with JAXB xjc compiler) // // This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802 // See http://java.sun.com/xml/jaxb // Any modifications to this file will be lost upon recompilation of the source schema. // Generated on: 2015.02.11 at 10:17:32 PM MST // package com.blogspot.marxsoftware.foodxml; import javax.xml.bind.annotation.XmlEnum; import javax.xml.bind.annotation.XmlEnumValue; import javax.xml.bind.annotation.XmlType; /** * Java class for Vegetable. * * The following schema fragment specifies the expected content contained within this class. * * * * * * * * * * * * */ @XmlType(name = "Vegetable") @XmlEnum public enum Vegetable { @XmlEnumValue("Carrot") CARROT("Carrot"), @XmlEnumValue("Squash") SQUASH("Squash"), @XmlEnumValue("Spinach") SPINACH("Spinach"), @XmlEnumValue("Celery") CELERY("Celery"); private final String value; Vegetable(String v) { value = v; } public String value() { return value; } public static Vegetable fromValue(String v) { for (Vegetable c: Vegetable.values()) { if (c.value.equals(v)) { return c; } } throw new IllegalArgumentException(v); } } Dessert.java (generated with JAXB xjc compiler) // // This file was generated by the JavaTM Architecture for XML Binding(JAXB) Reference Implementation, v2.2.8-b130911.1802 // See http://java.sun.com/xml/jaxb // Any modifications to this file will be lost upon recompilation of the source schema. // Generated on: 2015.02.11 at 10:17:32 PM MST // package com.blogspot.marxsoftware.foodxml; import javax.xml.bind.annotation.XmlEnum; import javax.xml.bind.annotation.XmlEnumValue; import javax.xml.bind.annotation.XmlType; /** * Java class for Dessert. * * The following schema fragment specifies the expected content contained within this class. * * * * * * * * * * * */ @XmlType(name = "Dessert") @XmlEnum public enum Dessert { @XmlEnumValue("Pie") PIE("Pie"), @XmlEnumValue("Cake") CAKE("Cake"), @XmlEnumValue("Ice Cream") ICE_CREAM("Ice Cream"); private final String value; Dessert(String v) { value = v; } public String value() { return value; } public static Dessert fromValue(String v) { for (Dessert c: Dessert.values()) { if (c.value.equals(v)) { return c; } } throw new IllegalArgumentException(v); } } Having enums generated for the XML elements ensures that only valid values for those elements can be represented in Java. Conclusion JAXB makes it relatively easy to map Java to XML, but because there is not a one-to-one mapping between Java and XML types, there can be some cases where the generated Java type for a particular XSD prescribed element is not obvious. This post has shown how two different approaches to building an XSD to enforce the same basic XML structure can lead to very different results in the Java classes generated with the JAXB xjccompiler. In the example shown in this post, declaring elements in the XSD directly on simpleTypes restricting XSD's string to a specific set of enumerated values is preferable to declaring elements as references to other elements wrapping a simpleType of restricted string enumerated values because of the type safety that is achieved when enums are generated rather than use of general Java Strings.

A common task in Python (especially while testing or debugging) is to redirect sys.stdout to a stream or a file while executing some piece of code. However, simply "redirecting stdout" is sometimes not as easy as one would expect; hence the slightly strange title of this post. In particular, things become interesting when you want C code running within your Python process (including, but not limited to, Python modules implemented as C extensions) to also have its stdout redirected according to your wish. This turns out to be tricky and leads us into the interesting world of file descriptors, buffers and system calls. But let's start with the basics. Pure Python The simplest case arises when the underlying Python code writes to stdout, whether by calling print, sys.stdout.write or some equivalent method. If the code you have does all its printing from Python, redirection is very easy. With Python 3.4 we even have a built-in tool in the standard library for this purpose - contextlib.redirect_stdout. Here's how to use it: from contextlib import redirect_stdout f = io.StringIO() with redirect_stdout(f): print('foobar') print(12) print('Got stdout: "{0}"'.format(f.getvalue())) When this code runs, the actual print calls within the with block don't emit anything to the screen, and you'll see their output captured by in the stream f. Incidentally, note how perfect the with statement is for this goal - everything within the block gets redirected; once the block is done, things are cleaned up for you and redirection stops. If you're stuck on an older and uncool Python, prior to 3.4 [1], what then? Well, redirect_stdout is really easy to implement on your own. I'll change its name slightly to avoid confusion: from contextlib import contextmanager @contextmanager def stdout_redirector(stream): old_stdout = sys.stdout sys.stdout = stream try: yield finally: sys.stdout = old_stdout So we're back in the game: f = io.StringIO() with stdout_redirector(f): print('foobar') print(12) print('Got stdout: "{0}"'.format(f.getvalue())) Redirecting C-level streams Now, let's take our shiny redirector for a more challenging ride: import ctypes libc = ctypes.CDLL(None) f = io.StringIO() with stdout_redirector(f): print('foobar') print(12) libc.puts(b'this comes from C') os.system('echo and this is from echo') print('Got stdout: "{0}"'.format(f.getvalue())) I'm using ctypes to directly invoke the C library's puts function [2]. This simulates what happens when C code called from within our Python code prints to stdout - the same would apply to a Python module using a C extension. Another addition is the os.system call to invoke a subprocess that also prints to stdout. What we get from this is: this comes from C and this is from echo Got stdout: "foobar 12 " Err... no good. The prints got redirected as expected, but the output from puts and echo flew right past our redirector and ended up in the terminal without being caught. What gives? To grasp why this didn't work, we have to first understand what sys.stdout actually is in Python. Detour - on file descriptors and streams This section dives into some internals of the operating system, the C library, and Python [3]. If you just want to know how to properly redirect printouts from C in Python, you can safely skip to the next section (though understanding how the redirection works will be difficult). Files are opened by the OS, which keeps a system-wide table of open files, some of which may point to the same underlying disk data (two processes can have the same file open at the same time, each reading from a different place, etc.) File descriptors are another abstraction, which is managed per-process. Each process has its own table of open file descriptors that point into the system-wide table. Here's a schematic, taken from The Linux Programming Interface: File descriptors allow sharing open files between processes (for example when creating child processes with fork). They're also useful for redirecting from one entry to another, which is relevant to this post. Suppose that we make file descriptor 5 a copy of file descriptor 4. Then all writes to 5 will behave in the same way as writes to 4. Coupled with the fact that the standard output is just another file descriptor on Unix (usually index 1), you can see where this is going. The full code is given in the next section. File descriptors are not the end of the story, however. You can read and write to them with the read and write system calls, but this is not the way things are typically done. The C runtime library provides a convenient abstraction around file descriptors - streams. These are exposed to the programmer as the opaque FILE structure with a set of functions that act on it (for example fprintf and fgets). FILE is a fairly complex structure, but the most important things to know about it is that it holds a file descriptor to which the actual system calls are directed, and it provides buffering, to ensure that the system call (which is expensive) is not called too often. Suppose you emit stuff to a binary file, a byte or two at a time. Unbuffered writes to the file descriptor with write would be quite expensive because each write invokes a system call. On the other hand, using fwrite is much cheaper because the typicall call to this function just copies your data into its internal buffer and advances a pointer. Only occasionally (depending on the buffer size and flags) will an actual write system call be issued. With this information in hand, it should be easy to understand what stdout actually is for a C program. stdout is a global FILE object kept for us by the C library, and it buffers output to file descriptor number 1. Calls to functions like printf and puts add data into this buffer. fflush forces its flushing to the file descriptor, and so on. But we're talking about Python here, not C. So how does Python translate calls to sys.stdout.write to actual output? Python uses its own abstraction over the underlying file descriptor - a file object. Moreover, in Python 3 this file object is further wrapper in an io.TextIOWrapper, because what we pass to print is a Unicode string, but the underlying write system calls accept binary data, so encoding has to happen en route. The important take-away from this is: Python and a C extension loaded by it (this is similarly relevant to C code invoked via ctypes) run in the same process, and share the underlying file descriptor for standard output. However, while Python has its own high-level wrapper around it - sys.stdout, the C code uses its own FILE object. Therefore, simply replacing sys.stdout cannot, in principle, affect output from C code. To make the replacement deeper, we have to touch something shared by the Python and C runtimes - the file descriptor. Redirecting with file descriptor duplication Without further ado, here is an improved stdout_redirector that also redirects output from C code [4]: from contextlib import contextmanager import ctypes import io import os, sys import tempfile libc = ctypes.CDLL(None) c_stdout = ctypes.c_void_p.in_dll(libc, 'stdout') @contextmanager def stdout_redirector(stream): # The original fd stdout points to. Usually 1 on POSIX systems. original_stdout_fd = sys.stdout.fileno() def _redirect_stdout(to_fd): """Redirect stdout to the given file descriptor.""" # Flush the C-level buffer stdout libc.fflush(c_stdout) # Flush and close sys.stdout - also closes the file descriptor (fd) sys.stdout.close() # Make original_stdout_fd point to the same file as to_fd os.dup2(to_fd, original_stdout_fd) # Create a new sys.stdout that points to the redirected fd sys.stdout = io.TextIOWrapper(os.fdopen(original_stdout_fd, 'wb')) # Save a copy of the original stdout fd in saved_stdout_fd saved_stdout_fd = os.dup(original_stdout_fd) try: # Create a temporary file and redirect stdout to it tfile = tempfile.TemporaryFile(mode='w+b') _redirect_stdout(tfile.fileno()) # Yield to caller, then redirect stdout back to the saved fd yield _redirect_stdout(saved_stdout_fd) # Copy contents of temporary file to the given stream tfile.flush() tfile.seek(0, io.SEEK_SET) stream.write(tfile.read()) finally: tfile.close() os.close(saved_stdout_fd) There are a lot of details here (such as managing the temporary file into which output is redirected) that may obscure the key approach: using dup and dup2 to manipulate file descriptors. These functions let us duplicate file descriptors and make any descriptor point at any file. I won't spend more time on them - go ahead and read their documentation, if you're interested. The detour section should provide enough background to understand it. Let's try this: f = io.BytesIO() with stdout_redirector(f): print('foobar') print(12) libc.puts(b'this comes from C') os.system('echo and this is from echo') print('Got stdout: "{0}"'.format(f.getvalue().decode('utf-8'))) Gives us: Got stdout: "and this is from echo this comes from C foobar 12 " Success! A few things to note: The output order may not be what we expected. This is due to buffering. If it's important to preserve order between different kinds of output (i.e. between C and Python), further work is required to disable buffering on all relevant streams. You may wonder why the output of echo was redirected at all? The answer is that file descriptors are inherited by subprocesses. Since we rigged fd 1 to point to our file instead of the standard output prior to forking to echo, this is where its output went. We use a BytesIO here. This is because on the lowest level, the file descriptors are binary. It may be possible to do the decoding when copying from the temporary file into the given stream, but that can hide problems. Python has its in-memory understanding of Unicode, but who knows what is the right encoding for data printed out from underlying C code? This is why this particular redirection approach leaves the decoding to the caller. The above also makes this code specific to Python 3. There's no magic involved, and porting to Python 2 is trivial, but some assumptions made here don't hold (such as sys.stdout being a io.TextIOWrapper). Redirecting the stdout of a child process We've just seen that the file descriptor duplication approach lets us grab the output from child processes as well. But it may not always be the most convenient way to achieve this task. In the general case, you typically use the subprocess module to launch child processes, and you may launch several such processes either in a pipe or separately. Some programs will even juggle multiple subprocesses launched this way in different threads. Moreover, while these subprocesses are running you may want to emit something to stdout and you don't want this output to be captured. So, managing the stdout file descriptor in the general case can be messy; it is also unnecessary, because there's a much simpler way. The subprocess module's swiss knife Popen class (which serve as the basis for much of the rest of the module) accepts a stdout parameter, which we can use to ask it to get access to the child's stdout: import subprocess echo_cmd = ['echo', 'this', 'comes', 'from', 'echo'] proc = subprocess.Popen(echo_cmd, stdout=subprocess.PIPE) output = proc.communicate()[0] print('Got stdout:', output) The subprocess.PIPE argument can be used to set up actual child process pipes (a la the shell), but in its simplest incarnation it captures the process's output. If you only launch a single child process at a time and are interested in its output, there's an even simpler way: output = subprocess.check_output(echo_cmd) print('Got stdout:', output) check_output will capture and return the child's standard output to you; it will also raise an exception if the child exist with a non-zero return code. Conclusion I hope I covered most of the common cases where "stdout redirection" is needed in Python. Naturally, all of the same applies to the other standard output stream - stderr. Also, I hope the background on file descriptors was sufficiently clear to explain the redirection code; squeezing this topic in such a short space is challenging. Let me know if any questions remain or if there's something I could have explained better. Finally, while it is conceptually simple, the code for the redirector is quite long; I'll be happy to hear if you find a shorter way to achieve the same effect. [1] Do not despair. As of February 2015, a sizable chunk of the worldwide Python programmers are in the same boat. [2] Note that bytes passed to puts. This being Python 3, we have to be careful since libc doesn't understand Python's unicode strings. [3] The following description focuses on Unix/POSIX systems; also, it's necessarily partial. Large book chapters have been written on this topic - I'm just trying to present some key concepts relevant to stream redirection. [4] The approach taken here is inspired by this Stack Overflow answer.

This post covers the JCache API at a high level and provides a teaser – just enough for you to (hopefully) start itching about it ;-) In this post …. JCache overview JCache API, implementations Supported (Java) platforms for JCache API Quick look at Oracle Coherence Fun stuff – Project Headlands (RESTified JCache by Adam Bien) , JCache related talks at Java One 2014, links to resources for learning more about JCache What is JCache? JCache (JSR 107) is a standard caching API for Java. It provides an API for applications to be able to create and work with in-memory cache of objects. Benefits are obvious – one does not need to concentrate on the finer details of implementing the Caching and time is better spent on the core business logic of the application. JCache components The specification itself is very compact and surprisingly intuitive. The API defines high level components (interfaces) some of which are listed below Caching Provider – used to control Caching Managers and can deal with several of them, Cache Manager – deals with create, read, destroy operations on a Cache Cache – stores entries (the actual data) and exposes CRUD interfaces to deal with the entries Entry – abstraction on top of a key-value pair akin to a java.util.Map Hierarchy of JCache API components JCache Implementations JCache defines the interfaces which of course are implemented by different vendors a.k.a Providers. Oracle Coherence Hazelcast Infinispan ehcache Reference Implementation – this is more for reference purpose rather than a production quality implementation. It is per the specification though and you can be rest assured of the fact that it does in fact pass the TCK as well From the application point of view, all that’s required is the implementation to be present in the classpath. The API also provides a way to further fine tune the properties specific to your provider via standard mechanisms. You should be able to track the list of JCache reference implementations from the JCP website link public class JCacheUsage{ public static void main(String[] args){ //bootstrap the JCache Provider CachingProvider jcacheProvider = Caching.getCachingProvider(); CacheManager jcacheManager = jcacheProvider.getCacheManager(); //configure cache MutableConfiguration jcacheConfig = new MutableConfiguration<>(); jcacheConfig.setTypes(String.class, MyPreciousObject.class); //create cache Cache cache = jcacheManager.createCache("PreciousObjectCache", jcacheConfig); //play around String key = UUID.randomUUID().toString(); cache.put(key, new MyPreciousObject()); MyPreciousObject inserted = cache.get(key); cache.remove(key); cache.get(key); //will throw javax.cache.CacheException since the key does not exist } } JCache provider detection JCache provider detection happens automatically when you only have a single JCache provider on the class path You can choose from the below options as well //set JMV level system property -Djavax.cache.spi.cachingprovider=org.ehcache.jcache.JCacheCachingProvider //code level config System.setProperty("javax.cache.spi.cachingprovider","org.ehcache.jcache.JCacheCachingProvider //you want to choose from multiple JCache providers at runtime CachingProvider ehcacheJCacheProvider = Caching.getCachingProvider("org.ehcache.jcache.JCacheCachingProvider"); //which JCache providers do I have on the classpath? Iterable jcacheProviders = Caching.getCachingProviders(); Java Platform support Compliant with Java SE 6 and above Does not define any details in terms of Java EE integration. This does not mean that it cannot be used in a Java EE environment – it’s just not standardized yet. Could not be plugged into Java EE 7 as a tried and tested standard Candidate for Java EE 8 Project Headlands: Java EE and JCache in tandem By none other than Adam Bien himself ! Java EE 7, Java SE 8 and JCache in action Exposes the JCache API via JAX-RS (REST) Uses Hazelcast as the JCache provider Highly recommended ! Oracle Coherence This post deals with high level stuff w.r.t JCache in general. However, a few lines about Oracle Coherence in general would help put things in perspective Oracle Coherence is a part of Oracle’s Cloud Application Foundation stack It is primarily an in-memory data grid solution Geared towards making applications more scalable in general What’s important to know is that from version 12.1.3 onwards, Oracle Coherence includes a reference implementation for JCache (more in the next section) JCache support in Oracle Coherence Support for JCache implies that applications can now use a standard API to access the capabilities of Oracle Coherence This is made possible by Coherence by simply providing an abstraction over its existing interfaces (NamedCache etc). Application deals with a standard interface (JCache API) and the calls to the API are delegated to the existing Coherence core library implementation Support for JCache API also means that one does not need to use Coherence specific APIs in the application resulting in vendor neutral code which equals portability How ironic – supporting a standard API and always keeping your competitors in the hunt ;-) But hey! That’s what healthy competition and quality software is all about ! Talking of healthy competition – Oracle Coherence does support a host of other features in addition to the standard JCache related capabilities. The Oracle Coherence distribution contains all the libraries for working with the JCache implementation The service definition file in the coherence-jcache.jar qualifies it as a valid JCache provider implementation Curious about Oracle Coherence ? Quick Starter page Documentation Installation Further reading about Coherence and JCache combo – Oracle Coherence documentation JCache at Java One 2014 Couple of great talks revolving around JCache at Java One 2014 Come, Code, Cache, Compute! by Steve Millidge Using the New JCache by Brian Oliver and Greg Luck Hope this was fun :-) Cheers !

Retry-After is a lesser known HTTP response header.

Basic Authentication is the simplest way to secure access to a resource.