Hello, how are you? Today we will talk about situations in which the use of the JPA/Hibernate is not recommended. Which alternatives do we have outside the JPA world? What we will talk about: JPA/Hibernate problems Solutions to some of the JPA/Hibernate problems Criteria for choosing the frameworks described here Spring JDBC Template MyBatis Sormula sql2o Take a look at: jOOQ and Avaje Is a raw JDBC approach worth it? How can I choose the right framework? Final thoughts I have created 4 CRUDs in my github using the frameworks mentioned in this post, you will find the URL at the beginning of each page. I am not a radical that thinks that JPA is worthless, but I do believe that we need to choose the right framework for the each situation. If you do not know I wrote a JPA book (in Portuguese only) and I do not think that JPA is the silver bullet that will solve all the problems. I hope you like the post. [= JPA/Hibernate Problems There are times that JPA can do more harm than good. Below you will see the JPA/Hibernate problems and in the next page you will see some solutions to these problems: Composite Key: This, in my opinion, is the biggest headache of the JPA developers. When we map a composite key we are adding a huge complexity to the project when we need to persist or find a object in the database. When you use composite key several problems will might happen, and some of these problems could be implementation bugs. Legacy Database: A project that has a lot of business rules in the database can be a problem when wee need to invoke StoredProcedures or Functions. Artifact size: The artifact size will increase a lot if you are using the Hibernate implementation. The Hibernate uses a lot of dependencies that will increase the size of the generated jar/war/ear. The artifact size can be a problem if the developer needs to do a deploy in several remote servers with a low Internet band (or a slow upload). Imagine a project that in each new release it is necessary to update 10 customers servers across the country. Problems with slow upload, corrupted file and loss of Internet can happen making the dev/ops team to lose more time. Generated SQL: One of the JPA advantages is the database portability, but to use this portability advantage you need to use the JPQL/HQL language. This advantage can became a disadvantage when the generated query has a poor performance and it does not use the table index that was created to optimize the queries. Complex Query: That are projects that has several queries with a high level of complexity using database resources like: SUM, MAX, MIN, COUNT, HAVING, etc. If you combine those resources the JPA performance might drop and not use the table indexes, or you will not be able to use a specific database resource that could solve this problem. Framework complexity: To create a CRUD with JPA is very simples, but problems will appear when we start to use entities relationships, inheritance, cache, PersistenceUnit manipulation, PersistenceContext with several entities, etc. A development team without a developer with a good JPA experience will lose a lot of time with JPA ‘rules‘. Slow processing and a lot of RAM memory occupied: There are moments that JPA will lose performance at report processing, inserting a lot of entities or problems with a transaction that is opened for a long time. After reading all the problems above you might be thinking: “Is JPA good in doing anything?”. JPA has a lot of advantages that will not be detailed here because this is not the post theme, JPA is a tool that is indicated for a lot of situations. Some of the JPA advantages are: database portability, save a lot of the development time, make easier to create queries, cache optimization, a huge community support, etc. In the next page we will see some solutions for the problems detailed above, the solutions could help you to avoid a huge persistence framework refactoring. We will see some tips to fix or to workaround the problems described here. Solutions to Some of the JPA/Hibernate problems We need to be careful if we are thinking about removing the JPA of our projects. I am not of the developer type that thinks that we should remove a entire framework before trying to find a solution to the problems. Some times it is better to choose a less intrusive approach. Composite Key Unfortunately there is not a good solution to this problem. If possible, avoid the creation of tables with composite key if it is not required by the business rules. I have seen developers using composite keys when a simple key could be applied, the composite key complexity was added to the project unnecessarily. Legacy Databases The newest JPA version (2.1) has support to StoredProcedures and Functions, with this new resource will be easier to communicate with the database. If a JPA version upgrade is not possible I think that JPA is not the best solution to you. You could use some of the vendor resources, e.g. Hibernate, but you will lose database and implementations portability. Artifact Size An easy solution to this problem would be to change the JPA implementation. Instead of using the Hibernate implementation you could use the Eclipsellink, OpenJPA or the Batoo. A problem might appear if the project is using Hibernate annotation/resources; the implementation change will require some code refactoring. Generated SQL and Complexes Query The solution to these problems would be a resource named NativeQuery. With this resource you could have a simplified query or optimized SQL, but you will sacrifice the database portability. You could put your queries in a file, something like SEARCH_STUDENTS_ORACLE or SEARCH_STUDENTS_MYSQL, and in production environment the correct file would be accessed. The problem of this approach is that the same query must be written for every database. If we need to edit the SEARCH_STUDENTS query, it would be required to edit the oracle and mysql files. If your project is has only one database vendor the NativeQuery resource will not be a problem. The advantage of this hybrid approach (JPQL and NativeQuery in the same project) is the possibility of using the others JPA advantages. Slow Processing and Huge Memory Size This problem can be solved with optimized queries (with NativeQuery), query pagination and small transactions. Avoid using EJB with PersistenceContext Extended, this kind of context will consume more memory and processing of the server. There is also the possibility of getting an entity from database as a “read only” entity, e.g.: entity that will only be used in a report. To recover an entity in a “read only” state is not needed to open a transaction, take a look at the code below: String query = "select uai from Student uai"; EntityManager entityManager = entityManagerFactory.createEntityManager(); TypedQuery typedQuery = entityManager.createQuery(query, Student.class); List resultList = typedQuery.getResultList(); Notice that in the code above there is no opened transaction, all the returned entities will be detached (non monitored by the JPA). If you are using EJB mark your transaction as NOT_SUPPORTED or you could use @Transactional(readOnly=true). Complexity I would say that there is only one solution to this problem: to study. It will be necessary to read books, blogs, magazines or any other trustful source of JPA material. More study is equals to less doubts in JPA. I am not a developer that believes that JPA it is the only and the best solution to every problem, but there are moments that JPA is not the best to tool to use. You must be careful when deciding about a persistence framework change, usually a lot of classes are affected and a huge refactoring is needed. Several bugs may be caused by this refactoring. It is needed to talk with the project mangers about this refactoring and list all the positive and negative effects. In the next four pages we will see 4 persistence frameworks that can be used in our projects, but before we see the frameworks I will show how that I choose each framework. Criteria for Choosing the frameworks Described Here Maybe you will think: “why the framework X is not here?”. Below I will list the criteria applied for choosing the framework displayed here: Found in more than one source of research: we can find in forums people talking about a framework, but it is harder to find the same framework appearing in more than one forum. The most quoted frameworks were chosen. Quoted by different sources: Some frameworks that we found in the forums are indicated only by its committers. Some forums does not allow “self merchandise”, but some frameworks owners still doing it. Last update 01/05/2013: I have searched for frameworks that have been updated in this past year. Quick Hello World: Some frameworks I could not do a Hello World with less than 15~20min, and with some errors. To the tutorials found in this post I have worked 7 minutes in each framework: starting counting in its download until the first database insert. The frameworks that will be displayed in here has good methods and are easy to use. To make a real CRUD scenario we have a persistence model like below: A attribute with a name different of the column name: socialSecurityNumber —-> social_security_number A date attribute a ENUM attribute With this characteristics in a class we will see some problems and how the framework solve it. Spring JDBC Template One of the most famous frameworks that we can find to access the database data is the Spring JDBC Template. The code of this project can be found in here: https://github.com/uaihebert/SpringJdbcTemplateCrud The Sprint JDBC Template uses natives queries like below: As it is possible to see in the image above the query has a database syntax (I will be using MySQL). When we use a native SQL query it is possible to use all the database resources in an easy way. We need an instance of the object JDBC Template (used to execute the queries), and to create the JDBC Template object we need to set up a datasource: We can get the datasource now (thanks to the Spring injection) and create our JDBCTemplate: PS.: All the XML code above and the JDBCTemplate instantiation could be replace by Spring injection and with a code bootstrap, just do a little research about the Spring features. One thing that I did not liked is the INSERT statement with ID recover, it is very verbose: With the KeyHolder class we can recover the generated ID in the database, unfortunately we need a huge code to do it. The other CRUD functions are easier to use, like below: Notice that to execute a SQL query it is very simple and results in a populated object, thanks to the RowMapper. The RowMapper is the engine that the JDBC Template uses to make easier to populate a class with data from the database. Take a look at the RowMapper code below: The best news about the RowMapper is that it can be used in any query of the project. The developer that is responsible to write the logic that will populate the class data. To finish this page, take a look below in the database DELETE and the database UPDATE statement: About the Spring JDBC Template we can say: Has a good support: Any search in the Internet will result in several pages with tips and bug fixes. A lot of companies use it: several projects across the world use it Be careful with different databases for the same project: The native SQL can became a problem with your project run with different databases. Several queries will need to be rewritten to adapt all the project databases. Framework Knowledge: It is good to know the Spring basics, how it can be configured and used. To those that does not know the Spring has several modules and in your project it is possible to use only the JDBC Template module. You could keep all the other modules/frameworks of your project and add only the necessary to run the JDBC Template. MyBatis MyBatis (created with the name iBatis) is a very good framework that is used by a lot of developers. Has a lot of functionalities, but we will only see a few in this post. The code of this page can be found in here: https://github.com/uaihebert/MyBatisCrud To run your project with MyBatis you will need to instantiate a Session Factory. It is very easy and the documentation says that this factory can be static: When you run a project with MyBatis you just need to instantiate the Factory one time, that is why it is in a static code. The configuration XML (mybatis.xml) it is very simple and its code can be found below: The Mapper (an attribute inside the XML above) will hold information about the project queries and how to translate the database result into Java objects. It is possible to create a Mapper in XML or Interface. Let us see below the Mapper found in the file crud_query.xml: Notice that the file is easy to understand. The first configuration found is a ResultMap that indicates the query result type, and a result class was configured “uai.model.Customer”. In the class we have a attribute with a different name of the database table column, so we need to add a configuration to the ResultMap. All queries need a ID that will be used by MyBatis session. In the beginning of the file it is possible to see a namespace declared that works as a Java package, this package will wrap all the queries and the ResultMaps found in the XML file. We could also use a Interface+Annotation instead of the XML. The Mapper found in the crud_query.xml file could be translated in to a Interface like: Only the Read methods were written in the Interface to make the code smaller, but all the CRUD methods could be written in the Interface. Let us see first how to execute a query found in the XML file: The parsing of the object is automatically and the method is easy to read. To run the query all that is needed is to use the combination “namespace + query id” that we saw in the crud_query.xml code above. If the developer wants to use the Interface approach he could do like below: With the interface query mode we have a clean code and the developer will not need to instantiate the Interface, the session class of the MyBatis will do the work. If you want to update, delete or insert a record in the database the code is very easy: About MyBatis we could say: Excellent Documentation: Every time that I had a doubt I could answer it just by reading its site documentation Flexibility: Allowing XML or Interfaces+Annotations the framework gives a huge flexibility to the developer. Notice that if you choose the Interface approach the database portability will be harder, it is easier to choose which XML to send with the deploy artifact rather than an interface Integration: Has integration with Guice and Spring Dynamic Query: Allows to create queries in Runtime, like the JPA criteria. It is possible to add “IFs” to a query to decide which attribute will be used in the query Transaction: If your project is not using Guice of Spring you will need to manually control the transaction Sormula Sormula is a ORM OpenSource framework, very similar to the JPA/Hibernate. The code of the project in this page can be found in here: https://github.com/uaihebert/SormulaCrud Sormula has a class named Database that works like the JPA EntityManagerFactory, the Database class will be like a bridge between the database and your model classes. To execute the SQL actions we will use the Table class that works like the JPA EntityManager, but the Table class is typed. To run Sormula in a code you will need to create a Database instance: To create a Database instance all that we need is a Java Connection. To read data from the database is very easy, like below: You only need to create a Database instance and a Table instance to execute all kind of SQL actions. How can we map a class attribute name different from the database table column name? Take a look below: We can use annotations to do the database mapping in our classes, very close to the JPA style. To update, delete or create data in the database you can do like below: About Sormula we can say that: Has a good documentation Easy to set up It is not found in the maven repository, it will make harder to attach the source code if needed Has a lot of checked exceptions, you will need to do a try/catch for the invoked actions sql2o This framework works with native SQL and makes easier to transform database data into Java objects. The code of the project in this page can be found in here: https://github.com/uaihebert/sql2oCrud sql2o has a Connection class that is very easy to create: Notice that we have a static Sql2o object that will work like a Connection factory. To read the database data we would do something like: Notice that we have a Native SQL written, but we have named parameters. We are not using positional parameters like ‘?1′ but we gave a name to the parameter like ‘:id’. We can say that named parameters has the advantage that we will not get lost in a query with several parameters; when we forget to pass some parameter the error message will tell us the parameter name that is missing. We can inform in the query the name of the column with a different name, there is no need to create a Mapper/RowMapper. With the return type defined in the query we will not need to instantiate manually the object, sql2o will do it for us. If you want to update, delete or insert data in the database you can do like below: It is a “very easy to use” framework. About the sql2o we can say that: Easy to handle scalar query: the returned values of SUM, COUNT functions are easy to handle Named parameters in query: Will make easy to handle SQL with a lot of parameters Binding functions: bind is a function that will automatically populate the database query parameters through a given object, unfortunately it did not work in this project for a problem with the enum. I did not investigate the problem, but I think that it is something easy to handle Take a look at: jOOQ and Avaje jOOQ jOOQ it is a framework indicated by a lot of people, the users of this frameworks praise it in a lot of sites/forums. Unfortunately the jOOQ did not work in my PC because my database was too old, and I could not download other database when writing this post (I was in an airplane). I noticed that to use the jOOQ you will need to generated several jOOQ classes based in your model. jOOQ has a good documentation in the site and it details how to generate those classes. jOOQ is free to those that uses a free database like: MySQL, Postgre, etc. The paid jOOQ version is needed to those that uses paid databases like: Oracle, SQL Server, etc. www.jooq.org/ Avaje Is a framework quoted in several blogs/forums. It works with the ORM concept and it is easy to execute database CRUD actions. Problems that I found: Not well detailed documentation: its Hello World is not very detailed Configurations: it has a required properties configuration file with a lot of configurations, really boring to those that just want to do a Hello World A Enhancer is needed: enhancement is a method do optimize the class bytecode, but is hard to setup in the beginning and is mandatory to do before the Hello World www.avaje.org Is a Raw JDBC Approach Worth It? The advantages of JDBC are: Best performance: We will not have any framework between the persistence layer and the database. We can get the best performance with a raw JDBC Control over the SQL: The written SQL is the SQL that will be executed in the database, no framework will edit/update/generate the query SQL Native Resource: We could access all natives database resources without a problem, e.g.: functions, stored procedures, hints, etc The disadvantages are: Verbose Code: After receiving the database query result we need to instantiate and populate the object manually, invoking all the required “set” methods. This code will get worse if we have classes relationships like one-to-many. It will be very easy to find a while inside another while. Fragile Code: If a database table column changes its name it will be necessary to edit all the project queries that uses this column. Some project uses constants with the column name to help with this task, e.g. Customer.NAME_COLUMN, with this approach the table column name update would be easier. If a column is removed from the database all the project queries would be updated, even if you have a column constants. Complex Portability: If your project uses more than one database it would be necessary to have almost all queries written for each vendor. For any update in any query it would be necessary to update every vendor query, this could take a lot the time from the developers. I can see only one factor that would make me choose a raw JDBC approach almost instantly: Performance: If your project need to process thousands of transactions per minutes, need to be scalable and with a low memory usage this is the best choice. Usually median/huge projects has all this high performance requirements. It is also possible to have a hybrid solution to the projects; most of the project repository (DAO) will use a framework, and just a small part of it will use JDBC I do like JDBC a lot, I have worked and I still working with it. I just ask you to not think that JDBC is the silver bullet for every problem. If you know any other advantage/disadvantage that is not listed here, just tell me and I will add here with the credits going to you. [= How Can I Choose the Right Framework? We must be careful if you want to change JPA for other project or if you are just looking for other persistence framework. If the solutions in page 3 are not solving your problems the best solution is to change the persistence framework. What should you considerate before changing the persistence framework? Documentation: is the framework well documented? Is easy to understand how it works and can it answer most of your doubts? Community: has the framework an active community of users? Has a forum? Maintenance/Fix Bugs: Is the framework receiving commits to fix bugs or receiving new features? There are fix releases being created? With which frequency? How hard is to find a developer that knows about this framework? I believe that this is the most important issue to be considered. You could add to your project the best framework in the world but without developers that know how to operate it the framework will be useless. If you need to hire a senior developer how hard would be to find one? If you urgently need to hire someone that knows that unknown framework maybe this could be very difficult. Final Thoughts I will say it again: I do not think that JPA could/should be applied to every situation in every project in the world; I do no think that that JPA is useless just because it has disadvantages just like any other framework. I do not want you to be offended if your framework was not listed here, maybe the research words that I used to find persistence frameworks did not lead me to your framework. I hope that this post might help you. If your have any double/question just post it. [= See you soon! \o_

A couple weeks ago I read an article by Michael Church titled “How the Other Half Works: an Adventure in the Low Status of Software Engineers“. It is the story of Bill, who had two very different experiences interviewing for two different positions at two different companies: one as a software engineer, and one as a VP-level manager. Bill’s experience is as you would expect from the title. The article is well worth reading. It was difficult to process the article’s conclusions, because my interview experiences have not been like Bill’s. In fact, in some cases it has been the opposite. I once interviewed for a management-level position that I was woefully unqualified for. Once it was clear I wasn’t the person for the job, they changed the day’s schedule to allow some engineers to unnecessarily hammer me with technical questions. Likewise, I recently interviewed for a software engineer position at a healthcare company with a very small programming team, yet was treated extremely respectfully by everyone, including the CEO. Furthermore, I know I am not guilty of holding software engineers in low status, as anyone I have worked with will tell you. A few days later, after thinking about the article some more, I started to get some flashbacks*. I wanted to make changes to scrum teams, consolidating several smaller teams into fewer larger teams. I was told “if we do this, it must be secret. We cannot discuss team composition with developers. They just gossip and act like children.” I was discussing systematic problems with management structure with a sympathetic senior manager. I was told “I once put forward a proposal that employees should choose their own managers. I was laughed out of the room.” We were considering two senior developers for a second Technical Director position. I was told “you must leave this to me to handle, we do not want them to know the other is being considered.” Of course they were good friends, and eventually made the decision themselves. I raised an objection to a workflow a tools team had put together, concerned it put a large and unnecessary burden on content creators. I was told, “if they don’t want to deal with it, they don’t need to work here.” A team griped frequently about their tools, which were both essential and horrific. Management felt they weren’t sufficiently appreciative when any minor fixes were made. Instead of fixing the tools, they and I were told “there will be no more discussion of these issues, except as initiated by management.” Until now, I thought of these events as incredibly stupid decisions made by unqualified and disconnected individuals. Unfortunately, that’s not the case. These are incredibly predictable decisions made by normal individuals who are produced by a totally unsurprising system. Once the engineers are no longer running the show, they are quick to lose social status. Engineers remaining in management will be assimilated, demoted, or their position removed entirely. There’s no way to reverse this, and I think it’s why culture can so quickly spiral from enjoyable to miserable. Finally, it’s also interesting to think about this in the context of the Silicon Valley “anti poaching” conspiracy which depressed employee salaries. It demonstrates the systematically low status of software engineers better than anything. Management at Apple, Google, Adobe, Intel, and more, saw engineers as mere pawns, while simultaneously acknowledging how vital they were to the success of those companies. What a world! * If you’ve worked with me before, you can probably guess who some of these nameless individuals are. I have not obfuscated things for the sake of protecting the innocent, because I don’t find these shitty managers innocent.

In several recent blogs (in my reviews of the books Java EE 7 Performance Tuning and Optimization and WildFly Performance Tuning in particular), I have referenced my own past blog posts on certain Oracle JDK command-line tools. I was aghast to discover that I had never exclusively addressed the nifty jinfo tool and this post sets to rectify that troubling situation. I suspect that the reasons I chose not to write about jinfo previously include limitations related to jinfo discussed in my post VisualVM: jinfo and So Much More. In the Java SE 8 version of jinfo running on my machine, the primary limitation of jinfo on Windows that I discussed in the post Acquiring JVM Runtime Information has been addressed. In particular, I noted in that post that the -flags option was not supported on Windows version of jinfo at that time. As the next screen snapshot proves, that is no longer the case (note the use of jps to acquire the Java process ID to instruct jinfoto query). As the above screen snapshot demonstrates, the jinfo -flags command and option show the flags the explicitly specified JVM options of the Java process being monitored. If I want to find out about other JVM flags that are in effect implicitly (automatically), I can run java -XX:+PrintFlagsFinal to see all default JVM options. I can then query for any one of these against a running JVM process to find out what that particular JVM is using (same default or overridden different value). The next screen snapshot demonstrates how a small portion of the output provided from running java -XX:+PrintFlagsFinal. Let's suppose I notice a flag called PrintHeapAtGC in the above output and want to know if it's set in my particular Java application (-XX:+PrintHeapAtGC means it's set and -XX:-PrintHeapAtGC means it's not set). I can have jinfo tell me what its setting is (note my choice to use jcmd instead of jps in this case to determine the Java process ID): Because of the subtraction sign (-) instead of an addition sign (+) after the colon and before "PrintHeapAtGC", we know this is turned off for the Java process with the specified ID. It turns out that jinfo does more than let us look; it also let's us touch. The next screen snapshot shows changing this option using jinfo. As the previous screen snapshot indicates, I can turn off and on the boolean-style JVM options by simply using the same command to view the flag's setting but preceding the flag's name with the addition sign (+) to turn it on or with the substraction sign (-) to turn it off. In the example just shown, I turned off thePrintGCDateStamps, turned it back on again, and monitored its setting between those changes. Not all JVM options are boolean conditions. In those cases, their new values are assigned to them by concatenating the equals sign (=) and new value after the flag name. It's also important to note that the target JVM (the one you're trying to peek at and touch with jinfo will not allow you to change all its JVM option settings). In such cases, you'll likely see a stack trace with message "Command failed in target VM." In addition to displaying a currently running JVM's options and allowing the changing of some of these, jinfoalso allows one to see system properties used by that JVM as name/value pairs. This is demonstrated in the next screen snapshot with a small fraction of the output shown. Perhaps the easiest way to run jinfo is to simply provide no arguments other than the PID of the Java process in question and have both JVM options (non-default and command-line) and system properties displayed. Running jinfo -help provides brief usage details. Other important details are found in the Oracle documentation on the jinfo tool. These details includes the common (when it comes to these tools) reminder that this tool is "experimental and unsupported" and "might not be available in future releases of the JDK." We are also warned that jinfo on Windows requires availability of dbgeng.dll or installed Debugging Tools For Windows. Although I have referenced the handy jinfo command line tool previously in posts VisualVM: jinfo and So Much More and Acquiring JVM Runtime Information, it is a handy enough tool to justify a post of its very own. As a command-line tool, it enjoys benefits commonly associated with command-line tools such as being relatively lightweight, working well with scripts, and working in headless environments.

A common problem that people encounter when trying to import CSV files into Neo4j using Cypher’s LOAD CSV command is how to handle empty or ‘null’ entries in said files. For example let’s try and import the following file which has 3 columns, 1 populated, 2 empty: $ cat /tmp/foo.csv a,b,c mark,, load csv with headers from "file:/tmp/foo.csv" as row MERGE (p:Person {a: row.a}) SET p.b = row.b, p.c = row.c RETURN p When we execute that query we’ll see that our Person node has properties ‘b’ and ‘c’ with no value: ==> +-----------------------------+ ==> | p | ==> +-----------------------------+ ==> | Node[5]{a:"mark",b:"",c:""} | ==> +-----------------------------+ ==> 1 row ==> Nodes created: 1 ==> Properties set: 3 ==> Labels added: 1 ==> 26 ms That isn’t what we want – we don’t want those properties to be set unless they have a value. TO achieve this we need to introduce a conditional when setting the ‘b’ and ‘c’ properties. We’ll assume that ‘a’ is always present as that’s the key for our Person nodes. The following query will do what we want: load csv with headers from "file:/tmp/foo.csv" as row MERGE (p:Person {a: row.a}) FOREACH(ignoreMe IN CASE WHEN trim(row.b) <> "" THEN [1] ELSE [] END | SET p.b = row.b) FOREACH(ignoreMe IN CASE WHEN trim(row.c) <> "" THEN [1] ELSE [] END | SET p.c = row.c) RETURN p Since there’s no if or else statements in cypher we create our own conditional statement by using FOREACH. If there’s a value in the CSV column then we’ll loop once and set the property and if not we won’t loop at all and therefore no property will be set. ==> +-------------------+ ==> | p | ==> +-------------------+ ==> | Node[4]{a:"mark"} | ==> +-------------------+ ==> 1 row ==> Nodes created: 1 ==> Properties set: 1 ==> Labels added: 1

We’ve been missing an introduction to using MongoDB from Java for a little while now - there’s plenty of information in the documentation, but we were lacking a step-by-step guide to getting started as a Java developer. I sought to rectify this with a couple of blog posts for the MongoDB official blog: the first, an introduction to using MongoDB from Java, including a non-comprehensive list of some of the libraries you can use; the second, an introductory guide to simple CRUD operations using the Java driver: Getting Started with MongoDB and Java, Part 1 Getting Started with MongoDB and Java, Part 2 This is very much aimed at Java/JVM developers who are new to MongoDB, and want to get a feel for how you use it. These guides are for the current (2.x) driver. When we release 3.x, we’ll release updated guides as well.

Some weeks ago I started creating a small JUnit Runner (Oleaster) that allows you to use the Jasmine way of writing unit tests in JUnit. I learned that creating custom JUnit Runners is actually quite simple. In this post I want to show you how JUnit Runners work internally and how you can use custom Runners to modify the test execution process of JUnit. So what is a JUnit Runner? A JUnit Runner is class that extends JUnit's abstract Runner class. Runners are used for running test classes. The Runner that should be used to run a test can be set using the @RunWith annotation. @RunWith(MyTestRunner.class) public class MyTestClass { @Test public void myTest() { .. } } JUnit tests are started using the JUnitCore class. This can either be done by running it from command line or using one of its various run() methods (this is what your IDE does for you if you press the run test button). JUnitCore.runClasses(MyTestClass.class); JUnitCore then uses reflection to find an appropriate Runner for the passed test classes. One step here is to look for a @RunWith annotation on the test class. If no other Runner is found the default runner (BlockJUnit4ClassRunner) will be used. The Runner will be instantiated and the test class will be passed to the Runner. Now it is Job of the Runner to instantiate and run the passed test class. How do Runners work? Lets look at the class hierarchy of standard JUnit Runners: Runner is a very simple class that implements the Describable interface and has two abstract methods: public abstract class Runner implements Describable { public abstract Description getDescription(); public abstract void run(RunNotifier notifier); } The method getDescription() is inherited from Describable and has to return a Description.Descriptions contain the information that is later being exported and used by various tools. For example, your IDE might use this information to display the test results. run() is a very generic method that runs something (e.g. a test class or a test suite). I think usually Runner is not the class you want to extend (it is just too generous). In ParentRunner things get a bit more specific. ParentRunner is an abstract base class for Runners that have multiple children. It is important to understand here, that tests are structured and executed in a hierarchical order (think of a tree). For example: You might run a test suite which contains other test suites. These test suites then might contain multiple test classes. And finally each test class can contain multiple test methods. ParentRunner has the following three abstract methods: public abstract class ParentRunner extends Runner implements Filterable, Sortable { protected abstract List getChildren(); protected abstract Description describeChild(T child); protected abstract void runChild(T child, RunNotifier notifier); } Subclasses need to return a list of the generic type T in getChildren(). ParentRunner then asks the subclass to create a Description for each child (describeChild()) and finally to run each child (runChild()). Now let's look at two standard ParentRunners: BlockJUnit4ClassRunner and Suite. BlockJUnit4ClassRunner is the default Runner that is used if no other Runner is provided. So this is the Runner that is typically used if you run a single test class. If you look at the source ofBlockJUnit4ClassRunner you will see something like this: public class BlockJUnit4ClassRunner extends ParentRunner { @Override protected List getChildren() { // scan test class for methonds annotated with @Test } @Override protected Description describeChild(FrameworkMethod method) { // create Description based on method name } @Override protected void runChild(final FrameworkMethod method, RunNotifier notifier) { if (/* method not annotated with @Ignore */) { // run methods annotated with @Before // run test method // run methods annotated with @After } } } Of course this is overly simplified, but it shows what is essentially done in BlockJUnit4ClassRunner. The generic type parameter FrameworkMethod is basically a wrapper aroundjava.lang.reflect.Method providing some convenience methods. In getChildren() the test class is scanned for methods annotated with @Test using reflection. The found methods are wrapped in FrameworkMethod objects and returned. describeChildren() creates aDescription from the method name and runChild() finally runs the test method. BlockJUnit4ClassRunner uses a lot of protected methods internally. Depending on what you want to do exactly, it can be a good idea to check BlockJUnit4ClassRunner for methods you can override. You can have a look at the source of BlockJUnit4ClassRunner on GitHub. The Suite Runner is used to create test suites. Suites are collections of tests (or other suites). A simple suite definition looks like this: @RunWith(Suite.class) @Suite.SuiteClasses({ MyJUnitTestClass1.class, MyJUnitTestClass2.class, MyOtherTestSuite.class }) public class MyTestSuite {} A test suite is created by selecting the Suite Runner with the @RunWith annotation. If you look at the implementation of Suite you will see that it is actually very simple. The only thing Suite does, is to create Runner instances from the classes defined using the @SuiteClasses annotation. So getChildren() returns a list of Runners and runChild() delegates the execution to the corresponding runner. Examples With the provided information it should not be that hard to create your own JUnit Runner (at least I hope so). If you are looking for some example custom Runner implementations you can have a look at the following list: Fabio Strozzi created a very simple and straightforward GuiceJUnitRunner project. It gives you the option to inject Guice components in JUnit tests. Source on GitHub Spring's SpringJUnit4ClassRunner helps you test Spring framework applications. It allows you to use dependency injection in test classes or to create transactional test methods. Source on GitHub Mockito provides MockitoJUnitRunner for automatic mock initialization. Source on GitHub Oleaster's Java 8 Jasmine runner. Source on GitHub (shameless self promotion) Conclusion JUnit Runners are highly customizable and give you the option to change to complete test execution process. The cool thing is that can change the whole test process and still use all the JUnit integration points of your IDE, build server, etc. If you only want to make minor changes it is a good idea to have a look at the protected methods of BlockJUnit4Class runner. Chances are high you find an overridable method at the right location. In case you are interested in Olaester, you should have a look at my blog post: An alternative approach of writing JUnit tests.

Less than half of development teams do code reviews and the other half are probably not getting as much out of code reviews as they should.

I’ve been talking with a very nice recruiter over at Google over the last couple weeks, and she has been so kind in keeping me updated about opportunities for evangelism at Google. This is the 3rd round of talks I've had with Google while being the API Evangelist, talks that historically go nowhere because of their programming test, which is a super silly aspect of their HR process. I was straight up with the Google recruiter a couple of weeks ago when she first emailed me, and again when we talked on the phone last week—I do not take programming tests to open up doors for employment conversations, sorry. ;-( It is a waste of my time, and yours, and it doesn’t measure shit. I understand that you have to qualify large number of folks, at your very algorithmic-centric company, but when it comes to measuring what I do, a programming test isn’t a thing. If programming a tic-tac-toe game on a live screen share is what you need to open up a conversation with professionals around evangelizing your platform, you need to look elsewhere. Nowhere in my role as the API Evangelist do I have to code under time pressure with someone else watching, sorry. I would even say, having hacker skills, trumps programming skills in a public facing evangelism role, and speed, quality of code go out the window. This is about making connections through hacker storytelling, something that doesn't always pencil out to producing the best code and is more about helping people understand what is possible using a platform, in the most meaningful way—requiring more focus on the person and their problems, not the code or algorithm. I’ve managed to have man very meaningful conversations with other tech giants like Intel, IBM, large institutions like UC Berkeley, BYU, and establish fruitful relationships with partners like 3Scale and API Spark, and across federal agencies like Department of Education, Energy, NASA, and the White House around APIs--all without taking programming tests. I talk to startups, SMBs, SMEs, organizations, institutions, and government agencies all the time, and I never have to code under pressure in front of an audience. I’m not under the illusion that I will change your hiring practices, Google—you are a very smart, and successful company. All I’m saying is you are probably filtering out some pretty talented folks who are extremely passionate and knowledgeable in what they do, and connected in their space, and when you won’t engage in meaningful conversations without a programming test, your missing out. I actually prefer working with organizations from the outside in. I think it better reflects the essence of API Evangelism. The companies who have trouble working with outside entities that don't have traditional HR processes are probably not going to lead when it comes developing an API driven ecosystem. If your company doesn't have the time to research me, and understand what I bring to the API space, and what my skills are, we probably aren't a fit. Everything about me is available online at API Evangelist, Kin Lane, Twitter, and Github--you just have to look. If you are only looking at resumes, and making people take tests, you will probably get what you are looking for!

HDF5 is a format designed to store large numerical arrays of homogenous type. It cames particularly handy when you need to organize your data models in a hierarchical fashion and you also need a fast way to retrieve the data. Pandas implements a quick and intuitive interface for this format and in this post will shortly introduce how it works. We can create a HDF5 file using the HDFStore class provided by Pandas: import numpy as np from pandas importHDFStore,DataFrame# create (or open) an hdf5 file and opens in append mode hdf =HDFStore('storage.h5') Now we can store a dataset into the file we just created: df =DataFrame(np.random.rand(5,3), columns=('A','B','C'))# put the dataset in the storage hdf.put('d1', df, format='table', data_columns=True) The structure used to represent the hdf file in Python is a dictionary and we can access to our data using the name of the dataset as key: print hdf['d1'].shape (5, 3) The data in the storage can be manipulated. For example, we can append new data to the dataset we just created: hdf.append('d1',DataFrame(np.random.rand(5,3), columns=('A','B','C')), format='table', data_columns=True) hdf.close()# closes the file There are many ways to open a hdf5 storage, we could use again the constructor of the class HDFStorage, but the function read_hdf makes us also able to query the data: from pandas import read_hdf # this query selects the columns A and B# where the values of A is greather than 0.5 hdf = read_hdf('storage.h5','d1',where=['A>.5'], columns=['A','B']) At this point, we have a storage which contains a single dataset. The structure of the storage can be organized using groups. In the following example we add three different datasets to the hdf5 file, two in the same group and another one in a different one: hdf =HDFStore('storage.h5') hdf.put('tables/t1',DataFrame(np.random.rand(20,5))) hdf.put('tables/t2',DataFrame(np.random.rand(10,3))) hdf.put('new_tables/t1',DataFrame(np.random.rand(15,2))) Our hdf5 storage now looks like this: print hdf File path: storage.h5 /d1 frame_table (typ->appendable,nrows->10,ncols->3,indexers->[index],dc->[A,B,C]) /new_tables/t1 frame (shape->[15,2]) /tables/t1 frame (shape->[20,5]) /tables/t2 frame (shape->[10,3]) On the left we can see the hierarchy of the groups added to the storage, in the middle we have the type of dataset and on the right there is the list of attributes attached to the dataset. Attributes are pieces of metadata you can stick on objects in the file and the attributes we see here are automatically created by Pandas in order to describe the information required to recover the data from the hdf5 storage system.

[This article was written by Koby Nachmany.] If you are familiar with configuration management (aka CM) and automation, you probably know a thing or two about Puppet, and the amazing and rich collection of modules it offers. Puppet Forge contains a wealth of third party modules that enable us to do some pretty nifty stuff with almost no effort. Puppet helps deal with the messy parts of CM, like installing binaries and running installation scripts that are tedious to do manually. Tools such as Puppet were originally created for IT operations people, that are for the most part infrastructure-centric, and are best suited for setup and maintenance of hosts in a physical data center. Dealing with applications and certainly managing applications on an elastic virtualized or even cloudified environment, brings a set of new challenges despite the agility and other benefits it provides. Now imagine we can have this goodness coupled with an intelligent orchestration framework for an entire deployment? In this blog post I'd like to demonstrate how a cloud application orchestrator can complement already existing automation processes powered by configuration management tools, in this case we will demonstrate with Puppet. I will use the nodecellar application and the popular WordPress content management framework as examples. This will hopefully provide a good introduction to Cloudify blueprints. Overview So we've seen how Cloudify 3 allows us to easily orchestrate the "nodecellar" application Read about it Cloudify blueprints here. With the "nodecellar" example, Cloudify deploys a complex application using workflows that map deployment lifecycle events to bash scripts using Cloudify's bash runner plugin. Cloudify's Puppet integration now makes this pretty easy. Cloudify 3.0 - Taking Puppet to the Next Level of Orchestration. Check it out. Go The synergy between Cloudify and Puppet not only allows you to enjoy the benefits of your Puppet environment, but it also amplifies its usability by introducing unique advantages that will answer the following common challenges involved with configuration management tools: Agent Installation: Provision your service VMs, install a Puppet agent (if you like) and wires them up with the Puppet Master. Or, if you choose to run standalone, you can install the agent with the appropriate manifests needed for that service, as well. Order of Dependencies: Define the dependencies between application stacks, services and infrastructure resources. Which will then be launched based on that order. Remote Execution and Updates: Other than the basic install/uninstall, Cloudify enables customized application workflows that allow you to execute tools like remote shell scripts on a group of instances that belong to a particular service, or to a specific instance in a group. This feature is useful to run maintenance operations, such as snapshots in the case of a database, or code pushes in a continuous deployment model. In addition, you can run puppet apply whenever you feel it's right for your service. Post Deployment: Once your application is up, Cloudify will be able to glue your monitoring tool of choice, or you can choose to use the built-in one. A robust policy engine, enables auto-healing and even auto-scaling according to your service's required SLA. I'm now going to take a deep dive on my experience with a WordPress example that I feel is a very good representation of how Puppet and Cloudify work in sync. Let's say we want to deploy the popular WordPress application stack on two VMs . Something as follows: The flow is quite simple: -server 3.5.1 with the basic following modules installed: |-- hunner-wordpress (v0.6.0) |-- puppetlabs-apache (v1.0.1) - with php mods enabled |-- puppetlabs-mysql (v2.1.0) Your site.pp file should resemble something like this: node /^apache_web.*/ { include apache class { 'wordpress': create_db => false, create_db_user => false, } } node /^mysql.*/ { class { '::mysql::server': root_password => 'password', override_options => { 'mysqld' => { 'bind_address' => '0.0.0.0' } } } include mysql::client include wordpress } As we can see, we have an Apache PHP application that will likely require a database connection string (IP, port, user and password). This is where Cloudify facilitates the "gluing" of all the pieces together, by allowing us to inject dynamic/static custom facts to the dependent node (Apache server). Cloudify supports both standalone agents and PuppetMaster environments. Step 2: Tweaking the Original WordPress Module. Some minor adaptations to the wordpress init class of the WordPress module will allow us to embed these facts during Puppet agent invocation. Below is a code snippet (With defaults truncated): class wordpress ( $db_host_ip = $cloudify_related_host_ip, $db_user, = $cloudify_properties_db_user, $db_password = $cloudify_properties_db_pw, . . ) And some tweaking to the templates/wp-config.php.erb: /** MySQL hostname */ define('DB_HOST', ''); Let's add some tags for finer control of manifest execution: The MySQL node will not require the application part to run on it, so I've excluded it using a Puppet "tag" (read more about Puppet tags). Cloudify, of course, supports this and will provide the appropriate tags during agent invocation. -> class { 'wordpress::app': tag => ['postconfigure'], install_dir => $install_dir, install_url => $install_url, version => $version, db_name => $db_name, . .} Step 3: Creating the Blueprint In a similar way to the "nodecellar" blueprint, first lets create a folder with the name of "wp_puppet" and create a blueprint.yaml file within it. This file will then serve as the blueprint file. Now let's declare the name of this blueprint. blueprint: name: wp_puppet nodes: Now we can start creating the topology. Step 4: Creating VM Nodes Since, in this case I use the OpenStack provider to create the nodes, let's import the "OpenStack types" plugin. imports: - http://www.getcloudify.org/spec/openstack-plugin/1.0/plugin.yaml Since the VMs are the same, I declared a generic template for a VM host: vm_host: derived_from: cloudify.openstack.server properties: - install_agent: true - worker_config: user: ubuntu port: 22 # example for ssh key file (see `key_name` below) # this file matches the agent key configured during the bootstrap key: ~/.ssh/agent.key # Uncomment and update `management_network_name` when working a n neutron enabled openstack - management_network_name: cfy-mng-network - server: image: 8c096c29-a666-4b82-99c4-c77dc70cfb40 flavor: 102 key_name: cfy-agnt-kp security_groups: ['cfy-agent-default', 'wp_security_group'] # This is how we inject the puppet server's ip userdata: | #!/bin/bash -ex grep -q puppet /etc/hosts || echo "x.x.x.x puppet" | sudo -A tee -a /etc/hosts Create the MySQL and Apache VMs: - name: mysql_db_vm type: vm_host instances: deploy: 1 - name: apache_web_vm type: vm_host instances: deploy: 1 Step 5: Declaring Apache and MySQL Servers Since we are using the Puppet plugin to create those servers, first we have to import it: plugins: puppet_plugin: derived_from: cloudify.plugins.agent_plugin properties: url: https://github.com/cloudify-cosmo/cloudify-puppet-plugin/archive/nightly.zip The plugin defines server types as follows: middleware_server, app_server, db_server, web_server, message_bus_server, app_module. They are virtually the same, but serve the purpose of enabling better readability for the user and GUI visualization A Puppet server type is derived_from: cloudify.types.server type, but includes some puppet-specific properties and lifecycle events. For documentation see: Puppet Types So we now will go ahead and declare the server types: cloudify.types.puppet.web_server: derived_from: cloudify.types.web_server properties: # All Puppet related configuration goes inside # the "puppet_config" property. - puppet_config interfaces: cloudify.interfaces.lifecycle: # Specifically "start" operation. Otherwise tags must be # provided. - start: puppet_plugin.operations.operation cloudify.types.puppet.app_module: derived_from: cloudify.types.app_module properties: - puppet_config interfaces: cloudify.interfaces.lifecycle: - configure: puppet_plugin.operations.operation cloudify.types.puppet.db_server: derived_from: cloudify.types.db_server properties: - puppet_config interfaces: cloudify.interfaces.lifecycle: - start: puppet_plugin.operations.operation Step 6: Instantiating the Apache and MySQL nodes: Here we provide the Puppet configuration and tags and define the relationships between the nodes. Cloudify's agent will use those relationships in order to decide the appropriate facts to inject. - name: apache_web_server type: cloudify.types.puppet.web_server properties: port: 8080 puppet_config: server: puppet environment: wordpress_env relationships: - type: cloudify.relationships.contained_in target: apache_web_vm - name: wordpress_app type: cloudify.types.puppet.app_module properties: db_user: wordpress db_pass: passwd puppet_config: server: puppet tags: ['postconfigure'] environment: wordpress_env relationships: - type: cloudify.relationships.contained_in target: apache_web_server - type: wp_connected_to_mysql target: mysql_db_server - name: mysql_db_server type: cloudify.types.puppet.db_server properties: db_user: wordpress db_pass: passwd puppet_config: server: puppet environment: wordpress_env relationships: - type: cloudify.relationships.contained_in target: mysql_db_vm Step 7: Upload the Blueprint and Create the Deployment (via CLI or GUI) Then execute your deployment (via CLI or GUI). ubuntu@koby-n-cfy3-cli:~/cosmo_cli$ cfy blueprints upload -b wp9 wordpress/blueprint.yaml ubuntu@koby-n-cfy3-cli:~/cosmo_cli$ cfy deployments create -b wp9 -d WordPress_Deployment_1 Step 8: Take a Quick Coffee Break. Step 9: Enjoy your Orchestrated WordPress Stack!

"Lambda Architecture" (introduced by Nathan Marz) has gained a lot of traction recently. Fundamentally, it is a set of design patterns of dealing with Batch and Real time data processing workflow that fuel many organization's business operations. Although I don't realize any novice ideas has been introduced, it is the first time these principles are being outlined in such a clear and unambiguous manner. In this post, I'd like to summarize the key principles of the Lambda architecture, focus more in the underlying design principles and less in the choice of implementation technologies, which I may have a different favors from Nathan. One important distinction of Lambda architecture is that it has a clear separation between the batch processing pipeline (ie: Batch Layer) and the real-time processing pipeline (ie: Real-time Layer). Such separation provides a means to localize and isolate complexity for handling data update. To handle real-time query, Lambda architecture provide a mechanism (ie: Serving Layer) to merge/combine data from the Batch Layer and Real-time Layer and return the latest information to the user. Data Source Entry At the very beginning, data flows in Lambda architecture as follows ... Transaction data starts streaming in from OLTP system during business operations. Transaction data ingestion can be materialized in the form of records in OLTP systems, or text lines in App log files, or incoming API calls, or an event queue (e.g. Kafka) This transaction data stream is replicated and fed into both the Batch Layer and Realtime Layer Here is an overall architecture diagram for Lambda. Batch Layer For storing the ground truth, "Master dataset" is the most fundamental DB that captures all basic event happens. It stores data in the most "raw" form (and hence the finest granularity) that can be used to compute any perspective at any given point in time. As long as we can maintain the correctness of master dataset, every perspective of data view derived from it will be automatically correct. Given maintaining the correctness of master dataset is crucial, to avoid the complexity of maintenance, master dataset is "immutable". Specifically data can only be appended while update and delete are disallowed. By disallowing changes of existing data, it avoids the complexity of handling the conflicting concurrent update completely. Here is a conceptual schema of how the master dataset can be structured. The center green table represents the old, traditional-way of storing data in RDBMS. The surrounding blue tables illustrates the schema of how the master dataset can be structured, with some key highlights Data are partitioned by columns and stored in different tables. Columns that are closely related can be stored in the same table NULL values are not stored Each data record is associated with a time stamp since then the record is valid Notice that every piece of data is tagged with a time stamp at which the data is changed (or more precisely, a change record that represents the data modification is created). The latest state of an object can be retrieved by extracting the version of the object with the largest time stamp. Although master dataset stores data in the finest granularity and therefore can be used to compute result of any query, it usually take a long time to perform such computation if the processing starts with such raw form. To speed up the query processing, various data at intermediate form (called Batch View) that aligns closer to the query will be generated in a periodic manner. These batch views (instead of the original master dataset) will be used to serve the real-time query processing. To generate these batch views, the "Batch Layer" use a massively parallel, brute force approach to process the original master dataset. Notice that since data in master data set is timestamped, the data candidate can be identified simply from those that has the time stamp later than the last round of batch processing. Although less efficient, Lambda architecture advocates that at each round of batch view generation, the previous batch view should just be simply discarded and the new batch view is computed from master dataset. This simple-mind, compute-from-scratch approach has some good properties in stopping error propagation (since error cannot be accumulated), but the processing may not be optimized and may take a longer time to finish. This can increase the "staleness" of the batch view. Real time Layer As discussed above, generating the batch view requires scanning a large volume of master dataset that takes few hours. The batch view will therefore be stale for at least the processing time duration (ie: between the start and end of the Batch processing). But the maximum staleness can be up to the time period between the end of this Batch processing and the end of next Batch processing (ie: the batch cycle). The following diagram illustrate this staleness. Even the batch view is stale period, business operates as usual and transaction data will be streamed in continuously. To answer user's query with the latest, up-to-date information. The business transaction records need to be captured and merged into the real-time view. This is the responsibility of the Real-time Layer. To reduce the latency of latest information availability close to zero, the merge mechanism has to be done in an incremental manner such that no batching delaying the processing will be introduced. This requires the real time view update to be very different from the batch view update, which can tolerate a high latency. The end goal is that the latest information that is not captured in the Batch view will be made available in the Realtime view. The logic of doing the incremental merge on Realtime view is application specific. As a common use case, lets say we want to compute a set of summary statistics (e.g. mean, count, max, min, sum, standard deviation, percentile) of the transaction data since the last batch view update. To compute the sum, we can simply add the new transaction data to the existing sum and then write the new sum back to the real-time view. To compute the mean, we can multiply the existing count with existing mean, adding the transaction sum and then divide by the existing count plus one. To implement this logic, we need to READ data from the Realtime view, perform the merge and WRITE the data back to the Realtime view. This requires the Realtime serving DB (which host the Realtime view) to support both random READ and WRITE. Fortunately, since the realtime view only need to store the stale data up to one batch cycle, its scale is limited to some degree. Once the batch view update is completed, the real-time layer will discard the data from the real time serving DB that has time stamp earlier than the batch processing. This not only limit the data volume of Realtime serving DB, but also allows any data inconsistency (of the realtime view) to be clean up eventually. This drastically reduce the requirement of sophisticated multi-user, large scale DB. Many DB system support multiple user random read/write and can be used for this purpose. Serving Layer The serving layer is responsible to host the batch view (in the batch serving database) as well as hosting the real-time view (in the real-time serving database). Due to very different accessing pattern, the batch serving DB has a quite different characteristic from the real-time serving DB. As mentioned in above, while required to support efficient random read at large scale data volume, the batch serving DB doesn't need to support random write because data will only be bulk-loaded into the batch serving DB. On the other hand, the real-time serving DB will be incrementally (and continuously) updated by the real-time layer, and therefore need to support both random read and random write. To maintain the batch serving DB updated, the serving layer need to periodically check the batch layer progression to determine whether a later round of batch view generation is finished. If so, bulk load the batch view into the batch serving DB. After completing the bulk load, the batch serving DB has contained the latest version of batch view and some data in the real-time view is expired and therefore can be deleted. The serving layer will orchestrate these processes. This purge action is especially important to keep the size of the real-time serving DB small and hence can limit the complexity for handling real-time, concurrent read/write. To process a real-time query, the serving layer disseminates the incoming query into 2 different sub-queries and forward them to both the Batch serving DB and Realtime serving DB, apply application-specific logic to combine/merge their corresponding result and form a single response to the query. Since the data in the real-time view and batch view are different from a timestamp perspective, the combine/merge is typically done by concatenate the results together. In case of any conflict (same time stamp), the one from Batch view will overwrite the one from Realtime view. Final Thoughts By separating different responsibility into different layers, the Lambda architecture can leverage different optimization techniques specifically designed for different constraints. For example, the Batch Layer focuses in large scale data processing using simple, start-from-scratch approach and not worrying about the processing latency. On the other hand, the Real-time Layer covers where the Batch Layer left off and focus in low-latency merging of the latest information and no need to worry about large scale. Finally the Serving Layer is responsible to stitch together the Batch View and Realtime View to provide the final complete picture. The clear demarcation of responsibility also enable different technology stacks to be utilized at each layer and hence can tailor more closely to the organization's specific business need. Nevertheless, using a very different mechanism to update the Batch view (ie: start-from-scratch) and Realtime view (ie: incremental merge) requires two different algorithm implementation and code base to handle the same type of data. This can increase the code maintenance effort and can be considered to be the price to pay for bridging the fundamental gap between the "scalability" and "low latency" need. Nathan's Lambda architecture also introduce a set of candidate technologies which he has developed and used in his past projects (e.g. Hadoop for storing Master dataset, Hadoop for generating Batch view, ElephantDB for batch serving DB, Cassandra for realtime serving DB, STORM for generating Realtime view). The beauty of Lambda architecture is that the choice of technologies is completely decoupled so I intentionally do not describe any of their details in this post. On the other hand, I have my own favorite which is different and that will be covered in my future posts.

There are many reasons to learn an ORM tool like JPA, but it's not a magic bullet that will solve all your problems.

In this article we will discuss how we can perform CRUD operations on a backbone model using a REST based HTTP service. Background Earlier we have discussed about the benefits of using backbone.js and we also looked at the backbone models. Link to complete series: BackBone Tutorial – Part 1: Introduction to Backbone.Js BackBone Tutorial – Part 2: Understanding the basics of Backbone Models BackBone Tutorial – Part 3: More about Backbone Models BackBone Tutorial – Part 4: CRUD Operations on BackboneJs Models using HTTP REST Service[^] BackBone Tutorial – Part 5: Understanding Backbone.js Collections[^] BackBone Tutorial – Part 6: Understanding Backbone.js Views[^] BackBone Tutorial – Part 7: Understanding Backbone.js Routes and History[^] In this article we will look at performing the CRUD operations on backbone models using a REST based web service. Using the code The first thing we will do is that we will create a simple REST based web api that can be used to save the data on the server using our simple backbone application. For this I have created a simple database with a single table as: The ID field is configured to auto increment and this is the primary key of the table. so while creating a new model we don’t have to provide this to the server. Now on top of this model, I have written a simple ASP.NET web api that will provide us the RESTful api. This API is configured to run on my local machine at: http://localhost:51377/. The API details are as follows: Create: POST http://localhost:51377/api/values Read: GET http://localhost:51377/api/values/{id} Update: PUT http://localhost:51377/api/values/{id} Delete: DELETE http://localhost:51377/api/values/{id} Once we have the API running, we can start working on our backbone model. We had create the backbone model in our previous article as: var Book = Backbone.Model.extend({ defaults: { ID: "", BookName: "" }, idAttribute: "ID", initialize: function () { console.log('Book has been initialized'); this.on("invalid", function (model, error) { console.log("Houston, we have a problem: " + error) }); }, constructor: function (attributes, options) { console.log('Book\'s constructor had been called'); Backbone.Model.apply(this, arguments); }, validate: function (attr) { if (!attr.BookName) { return "Invalid BookName supplied." } } }); The backbone models inherently supports saving on the server using a restful web api. To save the model using a HTTP REST service, we need to specify the urlRoot in the backbone model. To actually save the model, we can call the save on the backbone model.The save method will trigger the validations and if the validations are successful, it will try to identify the action to be performed i.e. create or update and based on that action, it will use urlRoot and call the appropriate REST API to perform the operation. Let us specify the URL root to enable this model to use our web api service. var Book = Backbone.Model.extend({ defaults: { ID: "", BookName: "" }, idAttribute: "ID", initialize: function () { console.log('Book has been initialized'); this.on("invalid", function (model, error) { console.log("Houston, we have a problem: " + error) }); }, constructor: function (attributes, options) { console.log('Book\'s constructor had been called'); Backbone.Model.apply(this, arguments); }, validate: function (attr) { if (!attr.BookName) { return "Invalid BookName supplied." } }, urlRoot: 'http://localhost:51377/api/Books' }); Now let us try to perform CRUD operations on this model. Create To create a new entity on the server, we need to populate the non identity fields in the model (other than ID in this case) and then call the Save method on the model. // Lets perform a create operation [CREATE] var book = new Book({ BookName: "Backbone Book 43" }); book.save({}, { success: function (model, respose, options) { console.log("The model has been saved to the server"); }, error: function (model, xhr, options) { console.log("Something went wrong while saving the model"); } }); Read To read a single book entity, we need to create the book entity with the identity attribute populated, i.e., the ID of the book we want to read. Then we need to call the fetch method on the model object. // Now let us try to retrieve a book [READ] var book1 = new Book({ ID: 40 }); book1.fetch({ success: function (bookResponse) { console.log("Found the book: " + bookResponse.get("BookName")); } }); Update Now let’s say we want to update the name of the book retrieved in the earlier fetch call. All we need to do is set the attributes we need to update and call the save method again. // Lets try to update a book [UPDATE] var book1 = new Book({ ID: 40 }); book1.fetch({ success: function (bookResponse) { console.log("Found the book: " + bookResponse.get("BookName")); // Let us update this retreived book now (doing it in the callback) [UPDATE] bookResponse.set("BookName", bookResponse.get("BookName") + "_updated"); bookResponse.save({}, { success: function (model, respose, options) { console.log("The model has been updated to the server"); }, error: function (model, xhr, options) { console.log("Something went wrong while updating the model"); } }); } }); Delete Now to delete a Model, we just need to call the destroy method of the model object. // Let us delete the model with id 13 [DELETE] var book2 = new Book({ ID: 40 }); book2.destroy({ success: function (model, respose, options) { console.log("The model has deleted the server"); }, error: function (model, xhr, options) { console.log("Something went wrong while deleting the model"); } }); Custom URLs to perform CRUD operation on models There are few scenarios where we might want to have provide custom URLs for the individual operations. This can be achieved by overriding the sync function and providing custom URL for each action. Let us create one more model BookEx to see how this can be done. var BookEx = Backbone.Model.extend({ defaults: { ID: "", BookName: "" }, idAttribute: "ID", // Lets create function which will return the custom URL based on the method type getCustomUrl: function (method) { switch (method) { case 'read': return 'http://localhost:51377/api/Books/' + this.id; break; case 'create': return 'http://localhost:51377/api/Books'; break; case 'update': return 'http://localhost:51377/api/Books/' + this.id; break; case 'delete': return 'http://localhost:51377/api/Books/' + this.id; break; } }, // Now lets override the sync function to use our custom URLs sync: function (method, model, options) { options || (options = {}); options.url = this.getCustomUrl(method.toLowerCase()); // Lets notify backbone to use our URLs and do follow default course return Backbone.sync.apply(this, arguments); } }); Now we can perform the CRUD operations on this model in the same way as we did for the previous model. Point of interest In this article we have looked at how to perform CRUD operations on backbone models using HTTP based REST service. This has been written from a beginner’s perspective. I hope this has been informative. Download sample Web API code: WebAPISample Download sample backbone app code: backboneSample

1 Start with Personas The first step towards writing the right user stories is to understand your target users and customers. After all, user stories want to tell a story about the users using the product. If you don’t know who the users are and what problem we want to solve then it’s impossible to write the right stories and you end up with a long wish list rather than a description of the relevant product functionality. Personas offer a great way to capture the users and the customers with their needs. They are fictional characters that have a name and picture; relevant characteristics such as a role, activities, behaviours, and attitudes; and a goal, which is the problem that has to be addressed or the benefit that should be provided. Let’s look at an example. Say we want to create a game for children, which is fun to play and which educates the kids about music and dancing. We would then create at least two personas, one to represent the children, and one for the parents, as the following picture illustrates. The two sample personas above use my simple yet effective persona template. It encourages you to keep your personas concise, to focus on what really matters and to leave out the rest. You can download the template from romanpichler.com/tools/persona-template where more information on writing personas and using the template is available. Once you have created a cast of characters, select a primary persona, the persona you are mainly designing and building the product for. This helps you make the right product decision and get the user experience (UX) right. In the example above, I have chosen Yasmin as the primary persona. 2 Derive Epics from the Persona Goals Once you have created your personas, use their goals personas to identify the product functionality. Ask yourself what the product should do to address the personas’ problems or to create the desired benefits for them, as the following picture shows. Start with your primary persona and capture the functionality as epics, as coarse-grained, high-level stories. Write all the epics necessary to meet the persona goals but keep them rough and sketchy at this stage. For the dance game, we could write the epics below assuming that the game will be initially launched as an iPad app: As the epics above show, the game should allow the players to select different characters, to make them dance, to choose different dance floors and music tracks, to play the game with their friends, and to post a snapshot of their game on Facebook. While epics are great to sketch the product’s functionality, there is more to your product than epics and stories: You should also capture the user interaction and the sequences in which the epics are used, the visual design of your product, and the important nonfunctional qualities such as interoperability and performance. Use, for instance, workflow diagrams, story maps, storyboards, sketches, mock-ups, and constraint cards to describe them. You can find out more about describing the different product aspects in my post “User Stories are Not Enough to Create a Great User Experience”. 3 Progressively Decompose the Epics into User Stories With a holistic but coarse-grained description of your product in place start progressively decomposing your epics. Rather than detailing all epics and writing all user stories in one go, you derive your stories step by step as the following picture shows. As long as there are some significant risks present and you are figuring out what the product should look like and do, it’s best to derive just enough user stories just in time for the next sprint. Use your sprint goal or hypothesis to determine which epics to decompose and which stories to write as the following diagram illustrates. The approach depicted above minimises the amount of detailed items in your product backlog. This makes it easier to integrate new insights derived from exposing product increments or minimum viable products (MVPs) to users and customers. Say that we want to address the risk of creating the wrong game characters by developing an executable prototype that allows us to run a usability test with selected children. We could then write the following user stories: The stories above are derived from the epics “Choose character” and “Play with character”. The resulting prototype only partially implements the two epics – just to the extent of being able to test if the characters resonate with the users. Once you understand better how to meet the customer and user needs, you can start pre-writing user stories and have a larger inventory of detailed items on your product backlog as you are unlikely to experience bigger changes to your epics and your overall backlog. 4 Get the Stories Ready Before the development team starts working on the stories, check that each user story is ready: clear, feasible, and testable. A story is clear if there is a shared understanding between the product owner and the team about its meaning. It is feasible if it can be delivered in the next sprint according to the Definition of Done. This implies that the story is small enough to fit into the sprint but also that the necessary user interface design, test, and documentation work can be carried out. In the case of the sample stories above, we would have to add acceptance criteria, ensure that the stories are small enough to fit into the next sprint, and consider creating some very rough design sketches to indicate what the characters look like. For instance, to get the story “Yas chooses the little girl” ready, we could create the following rough sketch: The sketch above complement the user story and allows the team to implement the entire story including the visual design in the next sprint. With ready user stories in place the development team is in a good position to progress your product in an effective manner. For more details on getting user stories ready please take a look at my post “The Definition of Ready in Scrum”. Learn More You can learn more about writing the right epics and user stories by attending my Writing Great User Stories training course. If you want to learn more about the creating the UX artefacts mentioned in this post, then attend my Agile UX and Scrum training course. Please contact me if you are interested in having the courses delivered at your office. The persona pictures and the Manga girl sketch were created by Ole Størksen. Thanks Ole!

Introduction Now that I described the the basics of JPA and Hibernate flush strategies, I can continue unraveling the surprising behavior of Hibernate’s AUTO flush mode. Not all queries trigger a Session flush Many would assume that Hibernate always flushes the Session before any executing query. While this might have been a more intuitive approach, and probably closer to the JPA’s AUTO FlushModeType, Hibernate tries to optimize that. If the current executed query is not going to hit the pending SQL INSERT/UPDATE/DELETE statements then the flush is not strictly required. As stated in the reference documentation, the AUTO flush strategy may sometimessynchronize the current persistence context prior to a query execution. It would have been more intuitive if the framework authors had chosen to name it FlushMode.SOMETIMES. JPQL/HQL and SQL Like many other ORM solutions, Hibernate offers a limited Entity querying language (JPQL/HQL) that’s very much based on SQL-92 syntax. The entity query language is translated to SQL by the current database dialect and so it must offer the same functionality across different database products. Since most database systems are SQL-92 complaint, the Entity Query Language is an abstraction of the most common database querying syntax. While you can use the Entity Query Language in many use cases (selecting Entities and even projections), there are times when its limited capabilities are no match for an advanced querying request. Whenever we want to make use of some specific querying techniques, such as: Window functions Pivot table Common Table Expressions we have no other option, but to run native SQL queries. Hibernate is a persistence framework. Hibernate was never meant to replace SQL. If some query is better expressed in a native query, then it’s not worth sacrificing application performance on the altar of database portability. AUTO flush and HQL/JPQL First we are going to test how the AUTO flush mode behaves when an HQL query is about to be executed. For this we define the following unrelated entities: The test will execute the following actions: A Person is going to be persisted. Selecting User(s) should not trigger a the flush. Querying for Person, the AUTO flush should trigger the entity state transition synchronization (A person INSERT should be executed prior to executing the select query). 1 2 3 4 Product product = newProduct(); session.persist(product); assertEquals(0L, session.createQuery("select count(id) from User").uniqueResult()); assertEquals(product.getId(), session.createQuery("select p.id from Product p").uniqueResult()); Giving the following SQL output: 1 2 3 4 [main]: o.h.e.i.AbstractSaveEventListener - Generated identifier: f76f61e2-f3e3-4ea4-8f44-82e9804ceed0, using strategy: org.hibernate.id.UUIDGenerator Query:{[selectcount(user0_.id) as col_0_0_ from user user0_][]} Query:{[insert into product (color, id) values (?, ?)][12,f76f61e2-f3e3-4ea4-8f44-82e9804ceed0]} Query:{[selectproduct0_.idas col_0_0_ from product product0_][]} As you can see, the User select hasn’t triggered the Session flush. This is because Hibernate inspects the current query space against the pending table statements. If the current executing query doesn’t overlap with the unflushed table statements, the a flush can be safely ignored. HQL can detect the Product flush even for: Sub-selects 1 2 3 4 5 session.persist(product); assertEquals(0L, session.createQuery( "select count(*) "+ "from User u "+ "where u.favoriteColor in (select distinct(p.color) from Product p)").uniqueResult()); Resulting in a proper flush call: 1 2 Query:{[insert into product (color, id) values (?, ?)][Blue,2d9d1b4f-eaee-45f1-a480-120eb66da9e8]} Query:{[selectcount(*) as col_0_0_ from user user0_ where user0_.favoriteColor in(selectdistinct product1_.color from product product1_)][]} Or theta-style joins 1 2 3 4 5 session.persist(product); assertEquals(0L, session.createQuery( "select count(*) "+ "from User u, Product p "+ "where u.favoriteColor = p.color").uniqueResult()); Triggering the expected flush : 1 2 Query:{[insert into product (color, id) values (?, ?)][Blue,4af0b843-da3f-4b38-aa42-1e590db186a9]} Query:{[selectcount(*) as col_0_0_ from user user0_ cross joinproduct product1_ where user0_.favoriteColor=product1_.color][]} The reason why it works is because Entity Queries are parsed and translated to SQL queries. Hibernate cannot reference a non existing table, therefore it always knows the database tables an HQL/JPQL query will hit. So Hibernate is only aware of those tables we explicitly referenced in our HQL query. If the current pending DML statements imply database triggers or database level cascading, Hibernate won’t be aware of those. So even for HQL, the AUTO flush mode can cause consistency issues. If you enjoy reading this article, you might want to subscribe to my newsletter and get a discount for my book as well. AUTO flush and native SQL queries When it comes to native SQL queries, things are getting much more complicated. Hibernate cannot parse SQL queries, because it only supports a limited database query syntax. Many database systems offer proprietary features that are beyond Hibernate Entity Query capabilities. Querying the Person table, with a native SQL query is not going to trigger the flush, causing an inconsistency issue: 1 2 3 Product product = newProduct(); session.persist(product); assertNull(session.createSQLQuery("select id from product").uniqueResult()); 1 2 3 DEBUG [main]: o.h.e.i.AbstractSaveEventListener - Generated identifier: 718b84d8-9270-48f3-86ff-0b8da7f9af7c, using strategy: org.hibernate.id.UUIDGenerator Query:{[selectidfrom product][]} Query:{[insert into product (color, id) values (?, ?)][12,718b84d8-9270-48f3-86ff-0b8da7f9af7c]} The newly persisted Product was only inserted during transaction commit, because the native SQL query didn’t triggered the flush. This is major consistency problem, one that’s hard to debug or even foreseen by many developers. That’s one more reason for always inspecting auto-generated SQL statements. The same behaviour is observed even for named native queries: 1 2 3 4 @NamedNativeQueries( @NamedNativeQuery(name = "product_ids", query = "select id from product") ) assertNull(session.getNamedQuery("product_ids").uniqueResult()); So even if the SQL query is pre-loaded, Hibernate won’t extract the associated query space for matching it against the pending DML statements. Overruling the current flush strategy Even if the current Session defines a default flush strategy, you can always override it on a query basis. Query flush mode The ALWAYS mode is going to flush the persistence context before any query execution (HQL or SQL). This time, Hibernate applies no optimization and all pending entity state transitions are going to be synchronized with the current database transaction. 1 assertEquals(product.getId(), session.createSQLQuery("select id from product").setFlushMode(FlushMode.ALWAYS).uniqueResult()); Instructing Hibernate which tables should be syncronized You could also add a synchronization rule on your current executing SQL query. Hibernate will then know what database tables need to be syncronzied prior to executing the query. This is also useful for second level caching as well. 1 assertEquals(product.getId(), session.createSQLQuery("select id from product").addSynchronizedEntityClass(Product.class).uniqueResult()); If you enjoyed this article, I bet you are going to love my book as well. Conclusion The AUTO flush mode is tricky and fixing consistency issues on a query basis is a maintainer’s nightmare. If you decide to add a database trigger, you’ll have to check all Hibernate queries to make sure they won’t end up running against stale data. My suggestion is to use the ALWAYS flush mode, even if Hibernate authors warned us that: this strategy is almost always unnecessary and inefficient. Inconsistency is much more of an issue that some occasional premature flushes. While mixing DML operations and queries may cause unnecessary flushing this situation is not that difficult to mitigate. During a session transaction, it’s best to execute queries at the beginning (when no pending entity state transitions are to be synchronized) and towards the end of the transaction (when the current persistence context is going to be flushed anyway). The entity state transition operations should be pushed towards the end of the transaction, trying to avoid interleaving them with query operations (therefore preventing a premature flush trigger).

Pragmatic developers can look at the Internet of Things in two ways: This is amazing. I can only begin to imagine how I can directly improve the world outside the set of networked computer boxes. This is terrifying. If something goes wrong, then it’s on me—and this time the system affected extends outside the set of networked computer boxes. IoT is amazing in the way it bridges physical and virtual environments, but even the phrase “Internet of Things” should give a developer pause. Computers are pretty smart. Things are stupid. IoT tries to put Things online and tries to make them into inter-networked computers. That’s pop-philosophy, but you want to develop in the real world. So what real-world challenges will you face when you shoot for the IoT moon? Two Types of Challenges It seems there are two types of programming challenges for the Internet of Things: Data and control (the comp-sci and networking stuff) Information and business logic (the info-sci and human-computer interaction stuff) For this article, we’re going to talk about the programming problems we can solve around IoT. We’ll start at the bottom (data and control) and work our way up to the big picture (information and business logic). Type 1: Data and Control Challenge 1.1: Power This one is pretty obvious. Many IoT devices are wireless, and no one has invented thumbnail fusion reactors yet. One solution is equally obvious: pick your algorithms carefully. If you can save cycles to perform a given task, then do it. Libraries for implementing power-optimized algorithms will presumably spring up in greater numbers, but even so, you may need to inject some heavy-duty comp-sci know-how into IoT app development. The second solution is more complex than the first. Higher-level developers will have to think more about Dynamic Power Management (DPM), which just means: shutting down devices when they don’t need to be on and starting them up when they do. Normally the operating system worries about this, but an IoT application that orchestrates wearables and phones, for example, will know things that each device’s OS won’t—and therefore will be able to switch things on and off more intelligently than each device’s individual OS. Another option is to write or customize an embedded OS. Challenge 1.2: Latency Latency on IoT sits in two places: at the source and in the pipes. The basic problem is a physical one. Thing-chips often have to be small, which means that the chip can only be as powerful as current transistor technology allows. Another problem is power. Many small devices transmit and receive data in discrete active/sleep cycles (think TDMA) in order to save bandwidth and power, but this increases latency inversely to power saved. Another tradeoff is that network topologies optimized for IoT can involve more hops over slower devices. Mesh networks, for example, are immune to the failure of a few nodes. Similarly, “fog” and “edge” computing paradigms relieve Internet infrastructure by doing as much as possible without hub-nodes. The downside is that each node (a) can’t do very much on its own and (b) can only talk to neighboring nodes. The problem in the pipes is a matter of network infrastructure. Simply: the more Things, the less available bandwidth. Infrastructure technology will get faster, but cell networks won’t catch up overnight. And Things, unlike fancier computers, are often supposed to transmit blindly—that is, without anyone necessarily asking them to. This means there’s a massive potential for wasted bandwidth. Challenge 1.3: Unreliability The third challenge flows from the first two. Devices are unreliable–“Things” even more so. The distributed and decentralized virtues of IoT bring their own reliability problems. Here are just a few: Ubiquitous devices are cheap, so they fail more often. Truly ad-hoc connectivity implies ephemeral SLA, so uptime and recovery time may be unclear. Loosely controlled devices may have better things to do than give you their data (or computing resources), so concurrency may grow very complex. Less-reliable hardware generates less-reliable information (‘does my outlying datapoint just signify device failure?’), so you may need to chew your data more thoroughly at the application level. In a sense, IoT decouples low-level (the sub-session layer) from high-level channel capacity, because the distribution of error-sources on IoT is more heavily weighted toward originating or remote nodes. This means more error-correcting for application developers. Type 2: Information and Business Logic Challenge 2.1: Vast & Thin Data Sensors on smartphones are already generating oceans of raw data. These sensors are pretty sophisticated. Every major mobile OS provides a unified, simple API to access clean sensor and geo data. But start grabbing this data and it’s not immediately clear what to do with it. Try to think of killer applications for barometric data—besides weather and elevation (with GPS)—off the top of your head. Raw sensor data is extremely thin. It doesn’t explain itself, and we haven’t yet produced a complete mapping from physical measurements to business logic—let alone software design. Even if you know what to do with sensor/geo data eventually, you may have to learn new algorithms and data structures to process immediately. Geo-graphs aren’t CS101 graph data structures (for one thing, edge length is a first-class citizen of geo-graphs). The size of data over IoT is itself a problem. Wireless sensors beget tons of data. All the problems (and opportunities) of Big Data cascade naturally from IoT. Massively distributed computing on IoT devices is an exciting thought, but the toolchain for splitting calculations over a thousand idle Fitbits just isn’t here yet. 2. Context-Sensitivity Consider the term “ubiquitous computing,” defined as: what happens when wirelessly connected sensors and actuators, placed more or less everywhere, allow software to interact with much larger swaths of the physical world than just hardware or bare metal. Put ubiquitous computing on the Internet, and IoT makes the software context much larger. This has implications at two basic levels. At a high computer-architectural level: IoT extends the concept of computing environment well outside the von Neumann machine and weakens the concept of peripheral I/O. In an IoT-world interface, sensors are input and actuators are output. As IoT devices process increasingly at the edge (within individual nodes), the devices that appear peripheral to other nodes are actually doing an awful lot of computation. At a high business-logic level: the more stuff outside the computer-box affects the program, the less predictable the program behavior becomes at runtime. The same bizarrely-birthed memory leak might slow down the UI in a smartphone context but contribute to a cascading electrical grid failure in an IoT context. This means that IoT demands more self-monitoring and self-repairing code. Two Types of Solutions Plenty of researchers are working on ambitious solutions to the programming challenges presented by IoT. Two of the more exciting examples include: Abstract Task Graph—a data-driven model that maps the network graph to an application graph [1] Computational REST—replaces content resources with computation resources [2] There are also a few more strategies you can use right now to solve some of the IoT programming challenges mentioned above. Reactive ProgrammingThis general purpose paradigm responds to all major application-level challenges and embraces opportunities presented by IoT. The four definitive attributes of a reactive application are: event-driven, scalable, resilient, and responsive [3]. These four are excellent guiding principles for IoT applications at a high, cross-stack level. Flow-based Programming and the Actor ModelBoth present strongly independent components where only messages can affect processes. Both are deeply amenable to concurrency (for example, shared state is discouraged), nondeterminism, and scaling without exponential complexity growth, because components are black boxes. FBP is a bit more pragmatic and restrictive while the actor model is less restrictive and a bit harder to implement. FBP has already been implemented in Javascript (NoFlo), and the actor model has been implemented in Java (Akka) [4][5][6]. What’s important to remember is that there are already tools and techniques that can help you build IoT applications. FBP, actors, and reactive programming all have key attributes for creating applications that leverage the strengths of IoT to overcome its challenges. [1] https://www.usenix.org/legacy/event/mobisys05/eesr05/tech/full_papers/bakshi/bakshi.pdf [2] http://isr.uci.edu/tech_reports/UCI-ISR-10-3.pdf [3] http://www.reactivemanifesto.org/ [4] http://jpaulmorrison.com/fbp/ [5] http://arxiv.org/ftp/arxiv/papers/1008/1008.1459.pdf [6] http://noflojs.org/ [7] http://akka.io/ 2014 Guide to Internet of Things The 2014 Guide to Internet of Things covers 39 different IoT SDKs, developer programs, and hardware options, plus: Key findings from our survey of over 2,000 developers "How to IoT Your Life: The Complete Shopping List" "The Scale of IoT" Infographic Glossary of common IoT terms Four in-depth articles from industry experts DOWNLOAD NOW

While solving one of my assignment, I realize the age long practice of writing db queries in java class file is really a big pain. They are difficult to read and understand. Further if you want to modify it again a lot of task. Thus, to get rid of the pain, I put the db queries in spring’s context xml. Later I realize using such injections using xml, its not only easier to maintain and understand also we can have module wise segregation of db queries in different or in different "xyzDAO-queries.xml". Further its easier to add more customers to the existing application. Please see the code snippet area for the context.xml called "userDAO-queries.xml". I've created Map with an id. In the Map I put the key as and values as .db queries goes here.... , which contains the query. The java classes are DBQueriesImpl.java and UserDAOImpl.java for fetching the exact query from the "xyzDAO-queries.xml". Please see the comments in the java class. Context XML (userDAO-queries.xml) part : SELECT user_id, user_name, user_password, user_fname, user_mname, user_lname, user_create_time FROM ty_users SELECT user_id, user_name, user_password FROM ty_users WHERE user_name = ? and user_password = ? SELECT user_id, user_name, user_password FROM ty_users WHERE user_id=? INSERT INTO ty_users (user_name, user_password) VALUES (?, ?) Java Code Part: UserDAOImpl.java public class UserDAOImpl implements UserDAO{ private static final Logger logger = LoggerFactory.getLogger(UserDAOImpl.class);// logger DBQueries dbQuery = new DBQueriesImpl("userDAO-queries.xml");// DBConnection dbConn = ConnectionFactory.getInstance().getConnectionMySQL(); @Override public List authenitcUserDetails(String userName, String userPassword) { Map queryMap = null; String query ; List> mappedUserList = null; List listUser = null; try{ listUser = new ArrayList(); queryMap = dbQuery.getQueryBeanFromFactory("VALIDATE_USER_CREDENTIAL"); //passing the query-name or Map Id. query = dbQuery.getQuery(queryMap);//passing the query-map to get the query. //rest of the code is common fetching code. mappedUserList = dbConn.dbQueryRead(query, new Object[]{userName,userPassword}); if(mappedUserList.size()>0){ listUser = new QueryResultSetMapperImpl().mapRersultSetToObject(mappedUserList, User.class); } }catch(Exception ex){ logger.error("fetchAllUser Error:: ", ex); } return listUser; } } DBQueriesImpl.java public class DBQueriesImpl implements DBQueries { private static final Logger logger = LoggerFactory.getLogger(DBQueriesImpl.class); private ApplicationContext queriesCtx ; /** * @Desc: Loading the queryContext xml * @param queryContext */ public DBQueriesImpl(String queryContext) { queriesCtx = new ClassPathXmlApplicationContext(queryContext); } /** * @Desc: Reading from the loaded application context and getting the query-map, . */ @Override public Map getQueryBeanFromFactory(String beanId){ Map queryMap = null; if (queriesCtx != null && beanId != null) { queryMap = (Map)queriesCtx.getBean(beanId); } return queryMap; } /** * @Desc: Getting the exact query from the query-map, . */ @Override public String getQuery(Map queryMap) { String query=null; try{ if(queryMap.containsKey(QueryConstants.QUERY_NODE)){ query = (String) queryMap.get(QueryConstants.QUERY_NODE); queryMap.remove(QueryConstants.QUERY_NODE); }else{ throw new NoSuchFieldError(); } }catch(Exception excp){ excp.printStackTrace(); } return query; } } With this kind of setting I can also solve few other problems. In one of my assignment I used this trick. In the application, various clients have different set of data requirements thus, different set of sql-queries. Now instead of writing java classes to met client requirement we just write the set of spring-context.xmls having sql and used those spring-context at runtime and fetch data.

The objectives of Test Driven Development and unit testing are generally misunderstood. The problem is the word ‘test’, it is much less about testing and much more about specification of requirements, showing your working – as in maths, and the impact it has on design. TDD is much more important than only testing. Robert C Martin has a good analogy, he likens TDD to double entry bookkeeping: Software is a remarkably sensitive discipline. If you reach into a base of code and you change one bit you can crash the software. Go into the memory and twiddle one bit at random and very likely you will elicit some form of crash. Very, very few systems are that sensitive. You could go out to one of these bridges over here, start taking bolts out and they probably wouldn’t fall. I could pull out a gun and start shooting randomly and I probably wouldn’t kill too many people. I might wound a few but — you know — you get a bullet in the leg or a lung and you’d probably survive. People are resilient — they can survive the loss of a leg and so forth. Bridges are resilient — they survive the loss of components. But software isn’t resilient at all: one bit changes and — BANG! — it crashes. Very few disciplines are that sensitive. But there is one other [discipline] that is, and that’s Accounting. The right mistake at exactly the right time on the right spreadsheet — that one-digit error can crash the company and send the offenders off to jail. How do accountants deal with that sensitivity? Well, they have disciplines. And one of the primary disciplines is dual-entry bookkeeping. Everything is said twice. Every transaction is entered two times — once on the credit side and once on the debit side. Those two transactions follow separate mathematical pathways until they end up at this wonderful subtraction on the balance sheet that has to yield to zero. This is what test-driven development is: dual-entry bookkeeping. Everything is said twice — once on the test side and once on the production code side and everything runs in an execution that yields either a green bar or a red bar just like the zero on the balance sheet. It seems like that’s a good practice for us: to manage these sensitivities of our discipline… -Robert C. Martin The sensitivity of software is a good point to reflect upon, there is little in human experience that is so complex and yet so fragile. Without a strong focus on showing your working, no matter how good you are as a developer, if you omit the tests., your software will be worse than it could have been. The double-entry bookkeeping analogy only holds up though if you do test first development. If you write your test after the code it is generally not sufficiently independent to provide a valid “separate path” check. Test first is the idea that your write the test before you write the code that is being tested. This seems like a bizarre idea to many people at first, but actually makes perfect sense. If you write the test first and run it, you get to see it fail, so you are testing the test. If you write the test first then you are expressing what you want of your software from the outside in. It leads you to design for behaviour and so you have less of a tendency to get lost in irrelevant technicalities. This is a much more effective design approach than testing after you have written the code, and as a by product it leads inevitably to software that is easy to test – you have to be pretty dumb to write a test before you have written the code for an idea that can’t be tested! Finally there is a virtuous circle here. Software is easy to test when it is modular. It is easy to test when dependencies are externalised and it is easy to test when there is a clear separation of concerns. Now the software industry is famous for change, but if there is any idea that has remained constant for, literally, decades it is that quality software is modular, has well defined dependencies and clear separation of concerns – sound familiar? This has been how computer science has defined quality since before I started, and that was a very long time ago! Using TDD as a practice makes you produce higher quality software, not because it is well tested (though that is a nice by-product) but because it improves the quality of your designs. Want more detail: http://c2.com/cgi/wiki?TestDrivenDevelopment http://www.agiledata.org/essays/tdd.html http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd http://unitmm.sourceforge.net/fibonacci_example.shtml http://clean-cpp.org/test-driven-development/ http://agile2007.agilealliance.org/downloads/presentations/TDD-Cpp-Agile2007-HandsOnTddInCpp.ppt_801.pdf http://www.growing-object-oriented-software.com/

By William Zola, Lead Technical Support Engineer at MongoDB This is our final stop in this tour of modeling One-to-N relationships in MongoDB. In the first post, I covered the three basic ways to model a One-to-N relationship. Last time, I covered some extensions to those basics: two-way referencing and denormalization. Denormalization allows you to avoid some application-level joins, at the expense of having more complex and expensive updates. Denormalizing one or more fields makes sense if those fields are read much more often than they are updated. Read part two if you’ve missed it. Whoa! Look at All These Choices! So, to recap: You can embed, reference from the “one” side, or reference from the “N” side, or combine a pair of these techniques You can denormalize as many fields as you like into the “one” side or the “N” side Denormalization, in particular, gives you a lot of choices: if there are 8 candidates for denormalization in a relationship, there are 2 8 (1024) different ways to denormalize (including not denormalizing at all). Multiply that by the three different ways to do referencing, and you have over 3,000 different ways to model the relationship. Guess what? You now are stuck in the “paradox of choice” — because you have so many potential ways to model a “one-to-N” relationship, your choice on how to model it just got harder. Lots harder. Rules of Thumb: Your Guide Through the Rainbow Here are some “rules of thumb” to guide you through these indenumberable (but not infinite) choices One: favor embedding unless there is a compelling reason not to Two: needing to access an object on its own is a compelling reason not to embed it Three: Arrays should not grow without bound. If there are more than a couple of hundred documents on the “many” side, don’t embed them; if there are more than a few thousand documents on the “many” side, don’t use an array of ObjectID references. High-cardinality arrays are a compelling reason not to embed. Four: Don’t be afraid of application-level joins: if you index correctly and use the projection specifier (as shown in part 2) then application-level joins are barely more expensive than server-side joins in a relational database. Five: Consider the write/read ratio when denormalizing. A field that will mostly be read and only seldom updated is a good candidate for denormalization: if you denormalize a field that is updated frequently then the extra work of finding and updating all the instances is likely to overwhelm the savings that you get from denormalizing. Six: As always with MongoDB, how you model your data depends — entirely — on your particular application’s data access patterns. You want to structure your data to match the ways that your application queries and updates it. Your Guide To The Rainbow When modeling “One-to-N” relationships in MongoDB, you have a variety of choices, so you have to carefully think through the structure of your data. The main criteria you need to consider are: What is the cardinality of the relationship: is it “one-to-few”, “one-to-many”, or “one-to-squillions”? Do you need to access the object on the “N” side separately, or only in the context of the parent object? What is the ratio of updates to reads for a particular field? Your main choices for structuring the data are: For “one-to-few”, you can use an array of embedded documents For “one-to-many”, or on occasions when the “N” side must stand alone, you should use an array of references. You can also use a “parent-reference” on the “N” side if it optimizes your data access pattern. For “one-to-squillions”, you should use a “parent-reference” in the document storing the “N” side. Once you’ve decided on the overall structure of the data, then you can, if you choose, denormalize data across multiple documents, by either denormalizing data from the “One” side into the “N” side, or from the “N” side into the “One” side. You’d do this only for fields that are frequently read, get read much more often than they get updated, and where you don’t require strong consistency, since updating a denormalized value is slower, more expensive, and is not atomic. Productivity and Flexibility The upshot of all of this is that MongoDB gives you the ability to design your database schema to match the needs of your application. You can structure your data in MongoDB so that it adapts easily to change, and supports the queries and updates that you need to get the most out of your application.

[This article was written by Andreea Borca, developer of patient-empowering solutions for the healthcare industry, co-host of Farstuff: The IoT Podcast, and featured author in DZone's 2014 Guide to Internet of Things]. The creation of the Internet was a significant shift in the way people acquire information, interact with each other, and make decisions. Now, the Internet is expanding its reach to a range of devices that can gather and analyze physical data and react to that data in a variety of applications that we’ve never seen before. This “Internet of Things” marks another dynamic shift in the history of technology. This new stage in the Internet’s evolution is changing it from a tool that we actively need to engage with—deliberately using a browser to access it—to one that passively endows the world around us with a “mind” of its own. We are developing a world where things interact intelligently and cooperate to achieve goals without explicit guidance from human operators. Defining the Internet of Things First, we need to define the Internet of Things (also called “The Internet of Everything” by Cisco). A system falls under the Internet of Things definition if it meets the following criteria, known as the “3 Cs”: 1. It must Connect – to the physical world around itself collecting information, to other things in order to interact with them effectively, to the internet or a network, etc. 2. It must Compute – by processing the inputs it receives in some way and making them meaningful to other systems. 3. It must Communicate – with the network, with other things, and with the user if necessary (more often than not, as you’ll see, communicating to the user may be an unnecessary burden). Challenges for the Internet of Things Efficiency Devices within the Internet of Things (IoT) only need to do the bare minimum necessary to effectively work within the existing ecosystem. Many of the newest products rely heavily on the power of your smartphone to connect to the Internet and orchestrate devices, but there is also extensive pressure to reduce the size, energy consumption, and cost of the processing entities within IoT devices. In order to reduce power consumption and manage node outages, there is a concept of daisy-chaining across a network of devices into a more powerful central hub. This is known as mesh networking, and it’s becoming quite popular for IoT systems. Security, Privacy & the need to Share A core requirement of a well-functioning IoT device is to collect, transfer, and store data from a wide variety of sources. As more sensors arrive in cities and healthcare institutions, that increasingly connected information will unavoidably lead to more concern about security and privacy. The debate is still raging over balancing the clear benefits of new discoveries from processing Big Data with the strong personal fear of losing privacy. With IoT now in the picture, there is concern about devices that continuously and passively collect information on users. One recent clash over always-on sensors came with the release of Microsoft’s Xbox One Kinect console, which has a camera that is constantly pointed at your living room. Although the camera itself is not always on, the backlash over that possibility was fierce [1]. Finding this balance will quickly become a requirement for continued progress. Furthermore, the very nature of IoT and the connectivity network necessary for its success does make it particularly vulnerable in certain instances. Devices are especially vulnerable when connected over WiFi, because low tech sensor nodes with minimal computing power tend to be less secure, making them the ideal point of entry for infiltrators. Standards As with all new technologies, the battle over standards is always a struggle. Nest, the company that developed the most popular smart home thermostat, and its new owner, Google, are now making significant strides trying to establish the Nest platform as the foundation for all consumer-based IoT devices and their software counterparts [2]. Cisco, Qualcomm, IBM, Microsoft, and most other major players have a similar strategy for creating standard models for approaching the Internet of Things. The pressure to standardize is especially clear when new devices are appearing weekly. ZigBee already has extensive reach as an established standard for many household IoT devices. However, as a preferred codebase has yet to emerge as the standard of choice, it is recommended to connect with major standardization organizations like the IEEE, IETF, and the ZigBee Alliance [3]. Currently, the most common sensor networks use protocols such as Bluetooth Low Energy (BLE), RFID tags, ZigBee, and Wi-Fi. There are also iBeacons, which allow devices like smartphones to better identify their location and potential needs with NFC-powered micro-location and GPS technology. Opportunities for the Internet of Things There are numerous prospects to consider when looking to develop IoT products. Given the multi-trillion dollar projections for the future IoT economy, we should take a look at these emerging markets for IoT tech [4]. Consumers The consumer IoT space has bred a small but growing segment of followers that have invested early into “smart” tech. At this year’s CES, we saw everything from the Babolat Tennis Racket that becomes your personal tennis coach to the Kolibree Toothbrush that monitors your gum health while you brush. The fastest growing consumer IoT segment seems to be in smart home technology, with products such as self-managing refrigerators and resident-sensing door locks. Commercial Retailers have already proven adept at collecting a consumer’s shopping history. With the functionality of NFC-powered beacons, these retailers are eager to personalize your shopping experience in a whole new way. Essentially, each physical shopping trip can now be as littered with targeted ads as any typical online search, much like a scene from the 2002 sci-fi film Minority Report. Walk into a store and instantly the advertising screens on the wall change to address your particular demographic, income level, and shopping preferences. If you’ve connected your Google calendar to certain applications, these screens would show outfits targeting your next big event. Signs on clothing racks sense you coming near and change prices, fully leveraging a custom pricing model that would have economists drooling. And as you try on outfits, the smart mirror in the dressing room recommends accessories or comments on alternatives that might be a better fit for your body type. After all of these IoT events have helped you with your purchase, there’s no need to checkout. You’ve registered with the store and there’s a beacon at the exit that registers what you picked up and charges your card automatically as you leave. Healthcare With the recent U.S. mandate that all health records must be digital, there has been an explosion in the marketplace of new, patient-centered, smart health devices. The excitement of a healthcare revolution among top innovative companies, incubators, and startups predicts that this trend is not likely to taper off anytime soon. The key areas of focus so far have been: monitoring technologies like wearables (especially passive monitoring), function improving technologies, education, and notification technologies. Wearables are generally the first consumer touch point in the IoT health sphere. With the popularity of Fitbit pedometers and Withings scales, the market is starting to experiment with internal monitoring and potentially replacing some organs completely in the near future. A study at Boston University has had incredibly positive results creating an artificial pancreas for Type 1 diabetics by inserting an insulin and glucagon pump that responds when an attached glucometer goes below a certain level, just like an actual pancreas. Proteus, a promising startup out of San Francisco, has created an all-natural microchip in a pill that the patient swallows in order to monitor whether they are remembering to take their medication. The pill sends data to an armband that the user is wearing, which then can send notifications to family members regarding the patient’s status. The most impressive feature is the fact that these chips are powered by the energy in the patient’s digestive system. Cities, Infrastructure, and Industry The long-term vision of the future includes technology such as self-driving cars and city lights that alert police when there’s been an accident. In this stage of development, the majority of value is coming from technologies that monitor and collect data in urban settings. From an evolutionary perspective, the IoT city as a whole is still in what many would consider a learning phase. The main objective is to collect as much data as possible, make it available via open APIs, and encourage motivated data analysts to find opportunities for improvement in utility usage, environmental impacts, and service management for larger populations. This is one area where being an industrial country like the U.S. may actually impede the ability to progress as quickly as our less established counterparts. Third world countries that haven’t yet built a solid infrastructure allow for the creativity and flexibility to implement sophisticated solutions unfettered by generations of previous development. Silicon Valley powerhouses like Facebook and Google are actively engaged in projects to create a free global Wi-Fi network, and key locations in Africa have allowed them to experiment with these projects. Being an Early Mover In the very near future, as more and more things connect to the internet, internet connectivity from IoT devices will dwarf the amount of traditional web browsing. The core standards and assumptions that will drive this next revolution in computing technology are still being established and, as a result, building anything that can add value to this exploding industry (software, hardware, devices, sensors, beacons etc.) is a remarkable opportunity for the right developer. Right now is the time to start contributing to the development of these technologies if you want to be an early mover in IoT. 2014 Guide to Internet of Things The 2014 Guide to Internet of Things covers 39 different IoT SDKs, developer programs, and hardware options, plus: Key findings from our survey of over 2,000 developers "How to IoT Your Life: The Complete Shopping List" "The Scale of IoT" Infographic Glossary of common IoT terms Four in-depth articles from industry experts DOWNLOAD NOW