Memoization is a technique used to speed up functions. Memoization may be done manually. It may also be done automatically. We can find many examples of automatic memoization on Internet. In this article, I will show how Java 8 makes it very easy to memoize functions. What is memoization Memoization consist in caching the results of functions in order to speed them up when they are called several times with the same argument. The first call implies computing and storing the result in memory before returning it. Subsequent calls with the same parameter imply only fetching the previously stored value and returning it. How to apply memoization Memoization may be applied manually by hard coding it in every function that may benefit from it. If it takes a long time to compute the return value, memoization will speed up the program. For functions that take less time to evaluate than fetching the previously stored value from memory, memoization is clearly not a good option. Hard coding memoization by hand in each function is not a good option neither because it is repeating the same principle again and again. That is why automatic memoization is desirable. What to memoize Memoization applies to functions. Prior to Java 8, Java had no functions. However, it was perfectly possible to define some. Furthermore, we used to create “functional” methods, that is methods taking an argument and returning a value based only upon this argument. These kind of method may benefit from memoization. By the way, there is a match between functional methods and functions. For example, the following method: Integer doubleValue(Integer x) { return x * 2; } corresponds to: Integer doubleValue(Integer x) { if (cache.containsKey(x)) { return cache.get(x); } else { Integer result = x * 2; cache.put(x, result) ; return result; } } In Java 8, we can make this much cleaner: Map cache = new ConcurrentHashMap<>(); Integer doubleValue(Integer x) { return cache.computeIfAbsent(x, y -> y * 2); } Our function may be modified to use the same technique: Function doubleValue = x -> cache.computeIfAbsent(x, y -> y * 2); This is pretty simple, but it has two main drawbacks: We have to repeat this modification for all functions. The map we use is exposed and could potentially be modified by another thread having nothing to do with the function. The second problem is quite easy to address. We may put the method or the function in a separate class, including the map, with private access. For example, for the method case: public class Doubler { private static Map cache = new ConcurrentHashMap<>(); public static Integer doubleValue(Integer x) { return cache.computeIfAbsent(x, y -> y * 2); } } We may then instantiate that class and use it each time we want to compute a value: Integer y = Doubler.doubleValue(x); With this solution, the map is no longer accessible from outside. We can't do the same for functions because functions are anonymous classes and such classes may not have static members. One possibility would be to pass the map to the function as an additional argument. This may be done through a closure: class Doubler { private static Map cache = new ConcurrentHashMap<>(); public static Function doubleValue = x->cache.computeIfAbsent(x, y -> y * 2); } We can use this function as follows: Integer y = Doubler.doubleValue.apply(x); This gives no advantage compared to the “method” solution. However, we may also use this function in more idiomatic examples, such as: IntStream.range(1, 10).boxed().map(Doubler.doubleValue); This is equivalent to using the method version with the following syntax: IntStream.range(1, 10).boxed().map(ThisClass::doubleValue); The main problem is that while solving the second issue, we have made the first one more acute, which make automatic memoization more desirable. Automatic memoization: requirements What we need is a way to do the following: Function f = x -> x * 2; Function g = Memoizer.memoize(f); so that we may use the memoized function as a drop in replacement for the original one. All values returned by function g will be calculated through the original function f the first time, and returned from the cache for all subsequent accesses. By contrast, if we create a third function: Function f = x -> x * 2; Function g = Memoizer.memoize(f); Function h = Memoizer.memoize(f); the values cached by g will not be returned by h. In other words, g and h will use separate caches. Implementation The Memoizer class is quite simple: public class Memoizer { private final Map cache = new ConcurrentHashMap<>(); private Memoizer() {} private Function doMemoize(final Function function) { return input -> cache.computeIfAbsent(input, function::apply); } public static Function memoize(final Function function) { return new Memoizer().doMemoize(function); } } Using this class is also extremely simple: Integer longCalculation(Integer x) { try { Thread.sleep(1_000); } catch (InterruptedException ignored) { } return x * 2; } Function f = this::longCalculation; Function g = Memoizer.memoize(f); public void automaticMemoizationExample() { long startTime = System.currentTimeMillis(); Integer result1 = g.apply(1); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = g.apply(1); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } Running the automaticMemoizationExample method will produce the following result: 2 2 1000 0 We can now make memoized function out of ordinary ones by just calling a single method! What about functions with several arguments? Short answer: nothing. There are no such things in this world as functions with several arguments. Functions are applications of one set (the source set) to another set (the target set). So, they simply can't have several arguments. But this does not solve our problem. What is the functional equivalent to a method with several arguments? Long answer: what people generally consider as functions with several arguments are in fact either: Functions of tuples Function returning functions returning functions … returning a result In either cases, we are only concerned with functions of one argument, so we can easily use our Memoizer class. Using functions of tuples would probably be the simplest choice... if Java had tuples! We could of course write tuples. But to store tuples in maps, we would have to implement equals and hashcode for them, plus we would have to define tuples for two elements (pairs), tuple for three elements, and so on. Who knows where to stop? The second option is much easier. It is based upon currying, which means applying each argument one after the other instead of applying them as a whole (the tuple). Currying a function is very easy. The only problem, in Java 8, is that writing the types is really cumbersome. Currying a “function of two arguments” (in fact a function of a pair) is easy once you master the type. Java has in fact a shortcut for functions of tuple2 which is called BiFunction. We will take this as an example. The two following functions are equivalent (from the result point of view): BiFunction h = (x, y) -> x + y; Function> hc = x -> y -> x + y; Not considering the types, there are very little differences. In the first case, the two arguments are put between parentheses, separated by a comma, which is, by the way, how tuples are written in most languages which have them! Remove the parentheses and separate the arguments with an arrow and you get the curried version. We can only regret that we have to write the type as: Function> when other languages use a simplified syntax such as: Integer -> Integer -> Integer From this, it is easy to memoized this curried version, although we cant use the same simple form as previously. We have to memoize each function: Function> mhc = Memoizer.memoize(x -> Memoizer.memoize(y -> x + y)); Same thing for a function of (a tuple of) 3 arguments (which by the way has no equivalent in Java): Function>> f3 = x -> y -> z -> x + y - z; Function>> f3m = Memoizer.memoize(x -> Memoizer.memoize(y -> Memoizer.memoize(z -> x + y – z)); Here is an example of using this memoized function “of three arguments”: Function>> f3 = x -> y -> z -> longCalculation(x) + longCalculation(y) - longCalculation(z); Function>> f3m = Memoizer.memoize(x -> Memoizer.memoize(y -> Memoizer.memoize(z -> longCalculation(x) + longCalculation(y) - longCalculation(z)))); public void automaticMemoizationExample2() { long startTime = System.currentTimeMillis(); Integer result1 = f3m.apply(2).apply(3).apply(4); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = f3m.apply(2).apply(3).apply(4); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } This example produces the following output: 2 2 3002 0 showing that the first access to method longCalculation has taken 3000 milliseconds and the second has return immediately. On the other hand, using a function of tuple may seem easier once you have the Tuple class defined. Here is an example of Tuple3: public class Tuple3 { public final T _1; public final U _2; public final V _3; public Tuple3(T t, U u, V v) { _1 = Objects.requireNonNull(t); _2 = Objects.requireNonNull(u); _3 = Objects.requireNonNull(v); } @Override public boolean equals(Object o) { if (!(o instanceof Tuple3)) return false; else { Tuple3 that = (Tuple3) o; return _1.equals(that._1) && _2.equals(that._2) && _3.equals(that._3); } } @Override public int hashCode() { return _1.hashCode() + _2.hashCode() + _3.hashCode(); } } Using this class, we may rewrite the previous example as: Function, Integer> ft = x -> longCalculation(x._1) + longCalculation(x._2) - longCalculation(x._3); Function, Integer> ftm = Memoizer.memoize(ft); public void automaticMemoizationExample3() { long startTime = System.currentTimeMillis(); Integer result1 = ftm.apply(new Tuple3<>(2, 3, 4)); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = ftm.apply(new Tuple3<>(2, 3, 4)); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } Conclusion Memoizing is about maintaining state between function calls. A memoized function is a function which behavior is dependent upon the current state. However, it will always return the same value for the same argument. Only the time needed to return the value will be different. So the memoized function is still a pure function if the original function is pure. However, there is a kind of function that may pose a problem: recursive functions that call themselves several times with the same argument may not be memoized this way. This will be addressed in a next article.

I use Charles Proxy extensively for debugging all kinds of applications, and lately I've been using it more with mobile devices. One of the killer features of Charles is its ability to intercept SSL traffic. This is hard - and rightly so, it should be difficult to inspect SSL traffic! Charles handles this by using the server's SSL certificate for the connection from Charles to the remote server, and then using Charles' own SSL certificate for the "last mile" back to your browser or device. This means that the connection will be flagged as insecure; Charles' certificates aren't trusted by your browser or device - but we can easily change that. Get Charles' Certificate On Windows, you can grab the certificate from the Help menu in Charles, but for OS X or Linux you'll need to download it from their website. It's linked from this excellent documentation page about SSL:http://www.charlesproxy.com/documentation/using-charles/ssl-certificates/. Install it On Your Android Device Copy the ``.crt`` file into the root of your SD card (I found this didn't work in a subdirectory, although some documentation suggests it will). Under "Security" -> "Settings", choose "install from device storage" and it should pick up the certificate you put on the SD card. ... that's it :) Check you have enabled the destination for SSL proxying under "Proxy Settings" in Charles, and you're all set! Note, I have one device that refused to see the .crt file on the SD card, but when I emailed it to myself and then saved it from there, was added successfully. Thought I'd mention it! Further Reading Using Charles To Debug PHP SOAP Wireshark Capture on Remote Server View Only Headers with Curl

this article describes steps required to configure your eclipse for sonarqube such that developers are not required to leave the eclipse ide to manage their source code quality. please feel free to comment/suggest if i missed to mention one or more important points. also, sorry for the typos. following are the key points described later in this article: installing & configuraing sonarqube in eclipse analyzing code using sonarqube installing & configuraing sonarqube in eclipse follow the steps given on following pages to install and configure sonarqube in your eclipse ide. the page on installing sonarqube in eclipse consists of steps required to install sonarqube in the ide. once downloaded, it would prompt to restart the eclipse ide. for proceeding ahead, you must restart the eclipse ide. the page on configuring sonarqube in eclipse consists of steps required to configure eclipse for sonarqube. the key thing to note is the fact that as you click on preferences > sonarqube > servers , you see the default server entry for http://localhost:9000. initially, as per instruction page, seeing the default entry, i went ahead to next step for linking the project to sonarqube server and run the analysis. however, it throw the error such as unknowb version of sonarqube server . after a little research, i figured out that the error was thrown as the sonarqube server was not reachable although i saw it configured as http://localhost:9000. the solution is to go to preferences > sonarqube > servers , click “edit” button and you would see sonarqube server url, username and password. the key is to give username and password and run the analysis once again. analyzing code using sonarqube once you are done with installation and configuration of sonarqube server in eclipse ide, for code analysis, all you need is right click on your project and click sonarqube > analyze and that is it. the violations would appear in sonarqube analysis as shown in the screenshot below. sonarqube analysis in eclipse however, do note that these issues may not appear if you try to access on the browser. for that to happen, you need to once again run “sonar-runner” from the project root.

In my last post, I explained how to do code reviews properly. I recommended taking advantage of static analysis tools like Findbugs, PMD, Klocwork or Fortify to check for common mistakes and bad code before passing the code on to a reviewer, to make the reviewer’s job easier and reviews more effective. Some readers asked whether static analysis tools can be used instead of manual code reviews. Manual code reviews add delays and costs to development, while static analysis tools keep getting better, faster, and more accurate. So can you automate code reviews, in the same way that many teams automate functional testing? Do you need to do manual reviews too, or can you rely on technology to do the job for you? Let’s start by understanding what static analysis bug checking tools are good at, and what they aren’t. What static analysis tools can do – and what they can’t do In this article, Paul Anderson at GrammaTech does a good job of explaining how static analysis bug finding works, the trade-offs between recall (finding all of the real problems), precision (minimizing false positives) and speed, and the practical limitations of using static analysis tools for finding bugs. Static analysis tools are very good at catching certain kinds of mistakes, including memory corruption and buffer overflows (for C/C++), memory leaks, illegal and unsafe operations, null pointers, infinite loops, incomplete code, redundant code and dead code. A static analysis tool knows if you are calling a library incorrectly (as long as it recognizes the function), if you are using the language incorrectly (things that a compiler could find but doesn’t) or inconsistently (indicating that the programmer may have misunderstood something). And static analysis tools can identify code with maintainability problems, code that doesn't follow good practice or standards, is complex or badly structured and a good candidate for refactoring. But these tools can’t tell you when you have got the requirements wrong, or when you have forgotten something or missed something important – because the tool doesn't know what the code is supposed to do. A tool can find common off-by-one mistakes and some endless loops, but it won’t catch application logic mistakes like sorting in descending order instead of ascending order, or dividing when you meant to multiply, referring to buyer when it should have been seller, or lessee instead of lessor. These are mistakes that aren't going to be caught in unit testing either, since the same person who wrote the code wrote the tests, and will make the same mistakes. Tools can’t find missing functions or unimplemented features or checks that should have been made but weren't. They can’t find mistakes or holes in workflows. Or oversights in auditing or logging. Or debugging code left in by accident. Static analysis tools may be able to find some backdoors or trapdoors – simple ones at least. And they might find some concurrency problems – deadlocks, races and mistakes or inconsistencies in locking. But they will miss a lot of them too. Static analysis tools like Findbugs can do security checks for you: unsafe calls and operations, use of weak encryption algorithms and weak random numbers, using hard-coded passwords, and at least some cases of XSS, CSRF, and simple SQL injection. More advanced commercial tools that do inter-procedural and data flow analysis (looking at the sources, sinks and paths between) can find other bugs including injection problems that are difficult and time-consuming to trace by hand. But a tool can’t tell you that you forgot to encrypt an important piece of data, or that you shouldn't be storing some data in the first place. It can’t find logic bugs in critical security features, if sensitive information could be leaked, when you got an access control check wrong, or if the code could fail open instead of closed. And using one static analysis tool on its own to check code may not be enough. Evaluations of static analysis tools, such as NIST's SAMATE project (a series of comparative studies, where many tools are run against the same code), show almost no overlap between the problems found by different tools (outside of a few common areas like buffer errors) even when the tools are supposed to be doing the same kinds of checks. Which means that to get the most out of static analysis, you will need to run two or more tools against the same code (which is what SonarQube, for example, which integrates its own static analysis results with other tools, including popular free tools, does for you). If you’re paying for commercial tools, this could get very expensive fast. Tools vs. Manual Reviews Tools can find cases of bad coding or bad typing – but not bad thinking. These are problems that you will have to find through manual reviews. A 2005 study Comparing Bug Finding Tools with Reviews and Tests used Open Source bug finding tools (including Findbugs and PMD) on 5 different code bases, comparing what the tools found to what was found through code reviews and functional testing. Static analysis tools found only a small subset of the bugs found in manual reviews, although the tools were more consistent – manual reviewers missed a few cases that the tools picked up. Just like manual reviews, the tools found more problems with maintainability than real defects (this is partly because one of the tools evaluated – PMD – focuses on code structure and best practices). Testing (black box – including equivalence and boundary testing – and white box functional testing and unit testing) found fewer bugs than reviews. But different bugs. There was no overlap at all between bugs found in testing and the bugs found by the static analysis tools. Finding problems that could happen - or do happen Static analysis tools are good at finding problems that “could happen”, but not necessarily problems that “do happen”. Researchers at Colorado State University ran static analysis tools against several releases of different Open Source projects, and compared what the tools found against the changes and fixes that developers actually made over a period of a few years – to see whether the tools could correctly predict the fixes that needed to be made and what code needed to be refactored. The tools reported hundreds of problems in the code, but found very few of the serious problems that developers ended up fixing. One simple tool (Jlint) did not find anything that was actually fixed or cleaned up by developers. Of 112 serious bugs that were fixed in one project, only 3 were also found by static analysis tools. In another project, only 4 of 136 bugs that were actually reported and fixed were found by the tools. Many of the bugs that developers did fix were problems like null pointers and incorrect string operations – problems that static analysis tools should be good at catching, but didn’t. The tools did a much better job of predicting what code should be refactored: developers ended up refactoring and cleaning up more than 70% of the code structure and code clarity issues that the tools reported (PMD, a free code checking tool, was especially good for this). Ericsson evaluated different commercial static analysis tools against large, well-tested, mature applications. On one C application, a commercial tool found 40 defects – nothing that could cause a crash, but still problems that needed to be fixed. On another large C code base, 1% of the tool’s findings turned out to be bugs serious enough to fix. On the third project, they ran 2 commercial tools against an old version of a C system with known memory leaks. One tool found 32 bugs, another 16: only 3 of the bugs were found by both tools. Surprisingly, neither tool found the already known memory leaks – all of the bugs found were new ones. And on a Java system with known bugs they tried 3 different tools. None of the tools found any of the known bugs, but one of the tools found 19 new bugs that the team agreed to fix. Ericsson’s experience is that static analysis tools find bugs that are extremely difficult to find otherwise. But it’s rare to find stop-the-world bugs – especially in production code – using static analysis. This is backed up by another study on the use of static analysis (Findbugs) at Google and on the Sun JDK 1.6.0. Using the tool, engineers found a lot of bugs that were real, but not worth the cost of fixing: deliberate errors, masked errors, infeasible situations, code that was already doomed, errors in test code or logging code, errors in old code that was “going away soon” or other relatively unimportant cases. Only around 10% of medium and high priority correctness errors found by the tool were real bugs that absolutely needed to be fixed. The Case for Security So far we've mostly looked at static analysis checking for run-time correctness and general code quality, not security. Although security builds on code quality – vulnerabilities are just bugs that hackers look for and exploit – checking code for correctness and clarity isn’t enough for a secure app. A lot of investment in static analysis technology over the past 5-10 years has been in finding security problems in code, such as common problems listed in OWASP’s Top 10 or the SANS/CWE Top 25 Most Dangerous Software Errors. A couple of studies have looked at the effectiveness of static analysis tools compared to manual reviews in finding security vulnerabilities. The first study was on a large application that had 15 known security vulnerabilities found through a structured manual assessment done by security experts. Two different commercial static analysis tools were run across the code. The tools together found less than half of the known security bugs – only the simplest ones, the bugs that didn't require a deep understanding of the code or the design. And of course the tools reported thousands of other issues that needed to be reviewed and qualified or thrown away as false positives. These other issues including some run-time correctness problems, null pointers and resource leaks, and code quality findings (dead code, unused variables), but no other real security vulnerabilities beyond those already found by the manual security review. But this assumes that you have a security expert around to review the code. To find security vulnerabilities, a reviewer needs to understand the code (the language and the frameworks), and they also need to understand what kind of security problems to look for. Another study shows how difficult this is. Thirty developers were hired to do independent security code reviews of a small web app (some security experts, others web developers). They were not allowed to use static analysis tools. The app had 6 known vulnerabilities. 20% of the reviewers did not find any of the known bugs. None of the reviewers found all of the known bugs, although several found a new XSS vulnerability that the researchers hadn’t known about. On average, 10 reviewers would have had only an 80% chance of finding all of the security bugs. And, not Or Static analysis tools are especially useful for developers working in unsafe languages like C/C++ (where there is a wide choice of tools to find common mistakes) or dynamically typed scripting languages like Javascript or PHP (where unfortunately the tools aren't that good), and for teams starting off learning a new language and framework. Using static analysis is (or should be) a requirement in highly regulated, safety critical environments like medical devices and avionics. And until more developers get more training and understand more about how to write secure software, we will all need to lean on static analysis (and dynamic analysis) security testing tools to catch vulnerabilities. But static analysis isn't a substitute for code reviews. Yes, code reviews take extra time and add costs to development, even if you are smart about how you do them – and being smart includes running static analysis checks before you do reviews. If you want to move fast and write good, high-quality and secure code, you still have to do reviews.You can’t rely on static analysis alone.

There have been a couple of articles on null, NPE's and how to avoid them. They make some point, but could stress the easy, safe, beautiful aspects of Java 8's Optional. This article shows some way of dealing with optional values, without additional utility code. The old way Let's consider this code: String unsafeTypeDirName = project.getApplicationType().getTypeDirName(); System.out.println(unsafeTypeDirName); This can obviously break with NullPointerException if any term is null. A typical way of avoiding this: // safe, ugly, omission-prone if (project != null) { ApplicationType applicationType = project.getApplicationType(); if (applicationType != null) { String typeDirName = applicationType.getTypeDirName(); if (typeDirName != null) { System.out.println(typeDirName); } } } This won't explode, but is just ugly, and it's easy to avoid some null check. Java 8 Let's try with Java 8's Optional: // let's assume you will get this from your model in the future; in the meantime... Optional optionalProject = Optional.ofNullable(project); // safe, java 8, but still ugly and omission-prone if (optionalProject.isPresent()) { ApplicationType applicationType = optionalProject.get().getApplicationType(); Optional optionalApplicationType = Optional.ofNullable(applicationType); if (optionalApplicationType.isPresent()) { String typeDirName = optionalApplicationType.get().getTypeDirName(); Optional optionalTypeDirName = Optional.ofNullable(typeDirName); if (optionalTypeDirName.isPresent()) { System.out.println(optionalTypeDirName); } } As noted in a lot of posts, this isn't a lot better than null checks. Some argue that it makes your intent clear. I don't see any big difference, most null checks being pretty obvious on those kind of situations. Ok, let's use the functional interfaces and get more power from Optional: // safe, prettier Optional optionalTypeDirName = optionalProject .flatMap(project -> project.getApplicationTypeOptional()) .flatMap(applicationType -> applicationType.getTypeDirNameOptional()); optionalTypeDirName.ifPresent(typeDirName -> System.out.println(typeDirName)); flatMap() will always return an Optional, so no nulls possible here, and you avoid having to wrap/unwrap to Optional. Please note that I added *Optional() methods in the types for that. There are other ways to do it (map + flatMap to Optional::ofNullable is one). The best one: only return optional value where it makes sense: if you know the value will always be provided, make it non-optional. By the way, this advice works for old style null checks too. ifPresent() will only run the code if it's there. No default or anything. Let's just use member references to express the same in a tight way: // safe, yet prettier optionalProject .flatMap(Project::getApplicationTypeOptional) .flatMap(ApplicationType::getTypeDirNameOptional) .ifPresent(System.out::println); Or if you know that Project has an ApplicationType anyway: // safe, yet prettier optionalProject .map(Project::getApplicationType) .flatMap(ApplicationType::getTypeDirNameOptional) .ifPresent(System.out::println); Conclusion By using Optional, and never working with null, you could avoid null checks altogether. Since they aren't needed, you also avoid omitting a null check leading to NPEs. Still, make sure that values returned from legacy code (Map, ...), which can be null, are wrapped asap in Optional.

Continue learning about product catalogs in MongoDB as we look at product seraches.

we talked previously about the kind of improvements we have in ravendb 3.0 for the indexing backend. in this post, i want to go over a few features that are much more visible. attachment indexing. this is a feature that i am not so hot about, mostly because we want to move all attachment usages to ravenfs. but in the meantime, you can reference the contents of an attachment during index. that can let you do things like store large text data in an attachment, but still make it available for the indexes. that said, there is no tracking of the attachment, so if it change, the document that referred to it won’t be re-indexed as well. but for the common case where both the attachments and the documents are always changed together, that can be a pretty nice thing to have. optimized new index creation. in ravendb 2.5, creating a new index would force us to go over all of the documents in the database, not just the documents that we have in that collection. in many cases, that surprised users, because they expected there to be some sort of physical separation between the collections. in ravendb 3.0, we changed things so creating a new index on a small collection (by default, less than 131,072 items) will be able to only touch the documents that belong to the collections being covered by that index. this alone represent a pretty significant change in the way we are processing indexes. in practice, this means that creating a new index on a small index would complete much more rapidly. for example, i reset an index on a production instance, it covers about 7,583 documents our of 19,191. ravendb was able to index that in just 690 ms, out of about 3 seconds overall that took for the index reset to take place. what about the cases where we have new indexes on large collections? at this point, in 2.5, we would do round robin indexing between the new index and the existing ones. the problem was that 2.5 was biased toward the new index. that meant that it was busy indexing the new stuff, while the existing indexes (which you are actually using) took longer to run. another problem was that in 2.5 creating a new index would effectively poison a lot of performance heuristics. those were built for the assumptions of all indexes running pretty much in tandem. and when we have one or more that weren’t doing so… well, that caused things to be more expensive. in 3.0, we have changed how this works. we’ll have separate performance optimization pipelines for each group of indexes based on its rough indexing position. that lets us take advantage of batching many indexes together. we are also not going to try to interleave the indexes (running first the new index and then the existing ones). instead, we’ll be running all of them in parallel, to reduce stalls and to increase the speed in which everything comes up to speed. this is using our scheduling engine to ensure that we aren’t actually overloading the machine with computation work (concurrent indexing) or memory (number of items to index at once). i’ve very proud in what we have done here, and even though this is actually a backend feature, it is too important to get lost in the minutia of all the other backend indexing changes we talked about in my previous post. explicit cartesian/fanout indexing. a cartesian index (we usually call them fanout indexes) is an index that output multiple index entries per each document. here is an example of such an index: from postcomment in docs.postcomments from comment in postcomment.comments where comment.isspam == false select new { createdat = comment.createdat, commentid = comment.id, postcommentsid = postcomment.__document_id, postid = postcomment.post.id, postpublishat = postcomment.post.publishat } for a large post, with a lot of comments, we are going to get an entry per comment. that means that a single document can generate hundreds of index entries. now, in this case, that is actually what i want, so that is fine. but there is a problem here. ravendb has no way of knowing upfront how many index entries a document will generate, that means that it is very hard to allocate the appropriate amount of memory reserves for this, and it is possible to get into situations where we simply run out of memory. in ravendb 3.0, we have added explicit instructions for this. an index has a budget, by default, each document is allowed to output up to 15 entries. if it tries to output more than 15 entries, that document indexing is aborted, and it won’t be indexed by this index. you can override this option either globally, or on an index by index basis, to increase the number of index entries per document that are allowed for an index (and old indexes will have a limit of 16,384 items, to avoid breaking existing indexes). the reason that this is done is so either you didn’t specify a value, in which case we are limited to the default 15 index entries per document, or you did specify what you believe is a maximum number of index entries outputted per document, in which case we can take advantage of that when doing capacity planning for memory during indexing. simpler auto indexes. this feature is closely related to the previous one. let us say that we want to find all users that have an admin role and has an unexpired credit card. we do that using the following query: var q = from u in session.query() where u.roles.any(x=>x.name == "admin") && u.creditcards.any(x=>x.expired == false) select u; in ravendb 2.5, we would generate the following index to answer this query: from doc in docs.users from doccreditcardsitem in ((ienumerable)doc.creditcards).defaultifempty() from docrolesitem in ((ienumerable)doc.roles).defaultifempty() select new { creditcards_expired = doccreditcardsitem.expired, roles_name = docrolesitem.name } and in ravendb 3.0 we generate this: from doc in docs.users select new { creditcards_expired = ( from doccreditcardsitem in ((ienumerable)doc.creditcards).defaultifempty() select doccreditcardsitem.expired).toarray(), roles_name = ( from docrolesitem in ((ienumerable)doc.roles).defaultifempty() select docrolesitem.name).toarray() } note the difference between the two. the 2.5 would generate multiple index entries per document, while ravendb 3.0 generate just one. what is worse is that 2.5 would generate a cartesian product, so the number of index entries outputted in 2.5 would be the number of roles for a user times the number of credit cards they have. in ravendb 3.0, we have just one entry, and the overall cost is much reduced. it was a big change, but i think it was well worth it, considering the alternative. in my next post, i’ll talk about the other side of indexing, queries. hang on, we still have a lot to go through.

the gnu arm eclipse plugins from liviu already offer several built-in actions which can be performed at the end of a build: creating flash image , create listing file and printing the code and data size: gnu arm eclipse extra post build steps but what if i need different things, or even more things? post-build steps for this there is the ‘post-build steps’ settings i can use: that command is executed at the end of the build: print size post-build step :!: the post build step is only executed if sources files have been compiled and linked. if you want to enforce that there is always a ‘true’ post build, then you need to delete some files in the pre-build step to enforce a compilation and a link phase. multiple post-build steps but what i need more than one action in the post-build step? i could call a batch or script file, but this is probably an overkill in too many cases, and adds a dependency to that script file. a better approach is to directly execute multiple commands as post-build step. unfortunately, the documentation found about the post-build step with a web-search is misleading (e.g. in the eclipse luna documentation ): “command: specifies one or more commands to execute immediately after the execution of the build. use semicolons to separate multiple commands.” unfortunately, semicolons is plain wrong (at least did not work for me) :-(. the solution is to use ‘ & ‘ (ampersand) to separate multiple commands on windows : :idea: on linux, use the ‘;’ to separate commands as noted in the documentation/help, and use ‘&’ on windows. unfortunately, this makes project not cross-platform. multiple post-build commands and this works for me, at least under windows 7 :-).

CodePro Analytix is the premier Java software testing tool for Eclipse developers.

This post is part of the Product Catalog MongoDB Series, in which we will cover many aspects of building a Product Catalog with MongoDB. This approach has been tested with a varied product catalog of 130 million items running on a single server (EC2 i2.2xlarge). MongoDB seems to be the perfect fit to implement a product catalog since products maps so well to documents. But as we shall see it is not as easy as it seems! The data is fairly complex with many relationships involved. Also almost every other system will want to make use of the catalog instead of making its own copy, so typically a low latency, scalable and geo distributed catalog service is the ideal solution. A product has at least the following information: Item: the overall product info (e.g. Levi’s 501) Variant: a specific variant of an item (e.g. in black size 6) which typically has a specific SKU / UPC Price: price information may vary based on the store, the variant, etc Hierarchy: the item taxonomy Facet: facets to search products by Vendors: a given sku may be available through different vendors if the site is a marketplace A classic pitfall is to try to fit everything into a single document. As a result you end up with something very complex with many nested lists, which makes it difficult to navigate and index. Additionally APIs find themselves sending back massive documents even if only partial info is needed. In certain cases, we’ve seen items with 1000s of variant (e.g. Automotive part) which go beyond 16MB of pure JSON (in which case compression becomes mandatory)! Instead here we are going to model the data in a way that is natural, maps well to the API, at the sweet spot between normalization and denorm. Item Model The item collection has document representing the high level data of a product. Here is a sample item document for a shoe: { "_id": "054VA72303012P", "desc": [ { "lang": "en", "val": "Give your dressy look a lift with these women's Kate high-heel shoes by Metaphor. These playful peep-toe pumps feature satin-wrapped stiletto heels and chiffon pompoms at the toes. Rhinestones on each of the silvertone buckles add just a touch of sparkle to these shoes for a flirty footwear look that's made for your next night out." } ], "name": "Women's Kate Ivory Peep-Toe Stiletto Heel", "lname": "women's kate ivory peep-toe stiletto heel", "category": "/84700/80009/1282094266/1200003270", "brand": { "id": "2483510", "img": { "src": "http://i.sears.com/s/i/bl/image/spin_prod_metadata_168138610" }, "name": "Metaphor" }, "assets": { "imgs": [ { "img": { "height": "1900", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_967112812", "width": "1900" } }, { "img": { "height": "1900", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_945877912", "width": "1900" } }, { "img": { "height": "1900", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_945878012", "width": "1900" } } ] }, "shipping": { "dimensions": { "height": "13.0", "length": "1.8", "width": "26.8" }, "weight": "1.75" }, "specs": [ { "name": "Heel Height (in.)", "val": "3.75" } ], "attrs": [ { "name": "Heel Height", "value": "High (2-1/2 to 4 in.)" }, { "name": "Upper Material", "value": "Synthetic" }, { "name": "Toe", "value": "Open toe" } { "name": "Brand", "value": "Metaphor" } ], "variants": { "cnt": 9, "attrs": [ { "dispType": "COMBOBOX", "name": "Width", }, { "dispType": "DROPDOWN", "name": "Color", }, { "dispType": "DROPDOWN", "name": "Shoe Size", } ] }, "lastUpdated": 1400877254787 } Fields of interest: _id: the product id lastUpdated: useful timestamp to see recently updated category: the category path made up of hierarchy nodes name: the product name lname: a lower-case version of the name. This can be useful for doing case-insensitive matching with an index brand: the brand desc: list of descriptions (website, retail box, etc) assets: list of assets (images, etc) attrs: list of attributes as name-value pairs. Will be used to implement facetting. Note that the brand is also included as one attribute. variants: some information on variants, but not the variants themselves Common queries (indexed): find by id: { _id: "the product id" } find by category prefix: { product.cat: { $regex: "^category prefix" } } find by case-insensitive name prefix: { product.lname: { $regex: "^name prefix" } } Variant Model The Variant documents represent specific variations of a product. Certain products only exist in a unique variant (e.g. XBox, no options to pick) whereas other products may have thousands. Here is a sample variant document for the same shoe: { "_id": "05458452563", "name": "Width:Medium,Color:Ivory,Shoe Size:6.5", "lname": "width:medium,color:ivory,shoe size:6.5", "itemId": "054VA72303012P", "altIds": { "upc": "632576103580" }, "assets": { "imgs": [ { "width": "1900", "height": "1900", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_945348512" }, { "width": "1900", "height": "1900", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_945348612" } ] }, "attrs": [ { "name": "Width", "value": "Medium" }, { "name": "Color", "family": "White", "value": "Ivory" }, { "name": "6.5", "value": "6.5" } ] } Fields of interest: _id: the SKU itemId: the parent item id. attrs: a list of attributes specific to the variant. Note that some of the attributes may have both a specific value (e.g. ivory) and a family value (e.g. white). assets: assets specific to the variant (e.g. image with a specific color). Common query parameters (indexed): find by SKU: { _id: "the sku" } find by item Id: { itemId: "item id" } Hierarchy Model The Hierarchy document represents a node of the hierarchical tree representing product taxonomy. The top level nodes represent departments, while further nodes represent specific categories. { "_id": "1200003270", "name": "Women's Heels & Pumps", "count": 223, "parents": [ "1282094266" ], "facets": [ "Heel Height", "Toe", "Upper Material", "Width", "Shoe Size", "Color" ] } Fields of interest: _id: the category id name: the category name count: the number of items in this category. It can be a useful statistic to display. parents: list of parent nodes. Simpler implementations could make use of a single value. facets: list of facets that exist for this category (e.g. color, size). This info will be used when displaying the facets available in the searching page. Common queries (indexed): find by parent id: { p: "parent id" } find top level departments: { p: null } Facet Model The facet document represent a name/value pair representing a product attribute. { "_id": "accessory type=hosiery", "name": "Accessory Type", "value": "Hosiery", "count": 14 } Fields of interest: _id: the id, which is a concatenation of lower-cased facet name and value. name: the facet name with original casing, e.g. “Accessory Type” value: the facet value with original casing, e.g. “Hosiery”. Important note: here the value should be the family value is possible, e.g. “White” rather than “Ivory”. Those facets will be used for searching items, and the family value is better for that purpose. count: the number of items that have this facet. This count will be important in defining the order of attributes in a query when doing faceted search. Common query parameters (indexed): find a specific facet: { _id: "name_value" } find facets for a name: { _id: { $regex: "^name_" } } Price Model The Price document obviously represents the price of an item, but there is quite a bit more to it. We want to be able to vary the price per variant (e.g. gold color is more expensive) or per store (e.g. online store is cheaper). While we will not touch on the store model in this post, let’s just imagine we have a few thousand stores which are grouped into a dozen store groups (e.g. online, west coast, etc). If we implement this naively, we would end up with 1000 stores x 200m variants = 2 billion price documents! Instead let’s be a bit smarter and make good use of MongoDB’s querying capability, by allowing to price products at different levels as needed, thus keeping the number of documents in the millions. A document looks like: { "_id": "SPM8824542513_1234", "price": "69.99", "sale": { "salePrice": "42.72", "saleEndDate": "2050-12-31 23:59:59" }, "lastUpdated": 1374647707394 } Fields of interest: _id: the id is built in a specific way. It is the concatenation of the item information and store information. The item information is either the item id or the variant id (SKU). The store information is either the store group id or the store id. price: the regular price sale: sales information, optional Common queries (indexed): find all prices by item id: { _id: { "$regex": "^itemId_" } } find all prices by SKU (price could be at item level): { _id: { "$in: [ { "$regex": "^itemId_" }, { "$regex": "^sku_" } ] } find price for a given SKU and store (4 combinations are possible): { _id: { "$in: [ "itemId_storeGroupId", "itemId_storeId", "sku_storeGroupId", "sku_storeId" ] } find items on sale, starting with ones ending soonest: { "sale.saleEndDate": { $ne: null } } with sort by { "sale.saleEndDate": 1 } (sparse index on "sale.saleEndDate") Summary Model The previous documents are now properly modeled, easy to maintain, and can efficiently power an API to serve product details pages. The last, and most difficult issue left to tackle is how to do faceted searching and other kind of browsing. We could leave it off to a full text search system or similar software, but it’s actually doable with MongoDB. For this purpose we face some tough challenges: whatever the search is, need a response within milliseconds returning hundreds of items the search can be a combination of many facets: category, brand, etc facets can be both at the item and variant levels: color, size, etc. If matching a specific variant, we should display that specific image (e.g. red shoes). hundreds of variants of the same item could match, in which case only the parent item should be returned as result efficient sorting on several attributes: price, popularity pagination feature which requires deterministic ordering For this purpose we create a separate collection called Summary in which each document represent the summary information of an item and all its variants. The data is stripped out to the minimum needed to power a browse & search feature. Such a document looks like: { "_id": "3ZZVA46759401P", "name": "Women's Chic - Black Velvet Suede", "lname": "women's chic - black velvet suede", "dep": "84700", "cat": "/84700/80009/1282094266/1200003270", "desc": [ { "lang": "en", "val": "This pointy toe slingback features a high quality upper and a classy, simple silhouette. This heel has a classic shape, an adjustable ankle strap for a vintage feel and a secure fit. The Chic is the perfect combination between dressy and professional." } ], "img": [ { "height": "330", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_591726201", "title": "spin_prod_591726201", "width": "450" } ], "attrs": [ "heel height=mid (1-3/4 to 2-1/4 in.)", "brand=metaphor" ], "sattrs": [ "upper material=synthetic", "toe=open toe" ], "vars": [ { "id": "05497884001", "img": [ { "height": "400", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_591726301", "title": "spin_prod_591726301", "width": "450" } ], "attrs": [ "width=medium", "color=black", "shoe size=6" ] }, { "id": "05497884002", "img": [ { "height": "400", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_591726301", "title": "spin_prod_591726301", "width": "450" } ], "attrs": [ "width=medium", "color=black", "shoe size=6.5" ] }, { "id": "05497884004", "img": [ { "height": "400", "src": "http://c.shld.net/rpx/i/s/i/spin/image/spin_prod_591726301", "title": "spin_prod_591726301", "width": "450" } ], "attrs": [ "width=medium", "color=black", "shoe size=7.5" ] } ] } Fields of interest: _id: the item id name: the item name lname: the item name, lower-case img: list of images, ideally just the thumbnail dep: the department (top level of category). Needs to be separate for proper indexing cat: the category path attrs: the item attributes, to be indexed sattrs: the item secondary attributes, not to be indexed vars: list of variants vars.id: the variant sku vars.attrs: the variant attributes, to be indexed vars.sattrs: the secondary variant attributes, not to be indexed Indices: department + attr + category + _id department + vars.attrs + category + _id department + category + _id department + price + _id department + rating + _id Common queries (indexed): find by department: { dep: "department" } find by category prefix: { dep: "department", cat: { $regex: "^category prefix"} } find by item attribute: { dep: "department", attrs: "name=value" } find by several item attributes: { dep: "department", attrs: { $all: [ "name=value", ... ] } find by variant attribute: { dep: "department", vars.attrs: "name=value" } find by several variant attributes: { dep: "department", vars.attrs: { $all: [ "name=value", ... ] } find by item attributes, variant attributes, category: { dep: "department", attrs: { $all: [ "name=value", ... ], vars.attrs: { $all: [ "name=value", ... ], cat: { $regex: "^category prefix"} } A few interesting notes on indexing / querying: each index starts with the department, which is a convenient way to subdivide our product catalog. It is an acceptable restriction to force the user to pick a department before displaying any kind of search facet (unless we’re displaying a pre-computed list like “most popular”). Hence having the department there will ensure that there is always a large amount of filtering done for cheap by the index :) each index ends with “_id” which is useful for pagination. It will give sorting on _id for free for some common queries. It’s always better to avoid resorting the skip/limit for pagination, which is only fine for a low number of pages. for queries using “$all“ the most restrictive attribute should be specified first (e.g. “color=red”). This information can be inferred from the “facet“ collection described earlier. This piece is critical to make facetted searches efficient and keep them in the few milliseconds. Conclusion In conclusion, we’ve seen here how to model and index a product catalog in MongoDB which will allow high performance, flexibility, and easy maintenance. More details on this topic can be see in the MongoDB World video Product Catalog Stay tuned for more information on our Product Catalog MongoDB solution, including: How to implement full text search within MongoDB or with a connector to an external FTS system More statistics and benchmarking on the faceted search capability Operational considerations: geo distributed for low latency queries, stringent read latency SLA, and spiky catalog updates Also check out another interesting topic: how to log all user activities around the site and run useful analytics on them in the MongoDB World video covering the Insight Component

big data traditionally has referred to a collection of data too massive to be handled efficiently by traditional database tools and methods. this original definition has expanded over the years to identify tools (big data tools) that tackle extremely large datasets (nosql databases, mapreduce, hadoop, newsql, etc.), and to describe the industry challenge posed by having data harvesting abilities that far outstrip the ability to process, interpret, and act on that data. technologists knew that those huge batches of user data and other data types were full of insights that could be extracted by analyzing the data in large aggregates. they just didn’t have any cheap, simple technology for organizing and querying these large batches of raw, unstructured data. the term quickly became a buzzword for every sort of data processing product’s marketing team. big data became a catchall term for anything that handled non-trivial sizes of data. sean owen, a data scientist at cloudera, has suggested that big data is a stage where individual data points are irrelevant and only aggregate analysis matters [1]. but this is true for a 400 person survey as well, and most people wouldn’t consider that very big. the key part missing from that definition is the transformation of unstructured data batches into structured datasets. it doesn’t matter if the database is relational or non-relational. big data is not defined by a number of terabytes, it’s rooted in the push to discoverhidden insights in data that companies used to disregard or throw away. due to the obstacles presented by large scale data management, the goal for developers and data scientists is two-fold: first, systems must be created to handle large scale data, and two, business intelligence and insights should be acquired from analysis of the data. acquiring the tools and methods to meet these goals is a major focus in the data science industry, but it’s a landscape where needs and goals are still shifting. what are the characteristics of big data? tech companies are constantly amassing data from a variety of digital sources that is almost without end—everything from email addresses to digital images, mp3s, social media communication, server traffic logs, purchase history, and demographics. and it’s not just the data itself, but data about the data (metadata). it is a barrage of information on every level. what is it that makes this mountain of data big data? one of the most helpful models for understanding the nature of big data is “the three vs:” volume, velocity, and variety. data volume volumeis the sheer size of the data being collected. there was a point in not-so-distant history where managing gigabytes of data was considered a serious task—now we have web giants like google and facebook handling petabytes of information about users’ digital activities. the size of the data is often seen as the first challenge of characterizing big data storage, but even beyond that is the capability of programs to provide architecture that can not only store but query these massive datasets. one of the most popular models for big data architecture comes from google’s mapreduce concept, which was the basis for apache hadoop, a popular data management solution. data velocity velocityis a problem that flows naturally from the volume characteristics of big data. data velocity is the speed at which data is flowing into a business’ infrastructure and the ability of software solutions to receive and ingest that data quickly. certain types of high-velocity data, such as streaming data, needs to be moved into storage and processed on the fly. this is often referred to as complex event processing (cep). the ability to intercept and analyze data that has a lifespan of milliseconds is a widely sought after. this kind of quick-fire data processing has long been the cornerstone of digital financial transactions, but it is also being used to track live consumer behavior or to bring instant updates to social media feeds. data variety variety refers to the source and type of data that is being collected. this data could be anything from raw image data to sensor readings, audio recordings, social media communication, and metadata. the challenge of data variety is being able to take raw, unstructured data and organize it so that an application can use it. this kind of structure can be achieved through architectural models that traditionally favor relational databases—but there is often a need to tidy up this data before it will even be useful to store in a raw form. sometimes a better option is to use a schema-less, non-relational database. how do you manage big data? the three vs is a great model for getting an initial understanding of what makes big data a challenge for businesses. however, big data is not just about the data itself, but the way that it is handled. a popular way of thinking about these challenges is to look at how a business stores, processes, and accesses their data. · store: can you store the vast amounts of data being collected? · process: can you organize, clean, and analyze the data collected? · access: can you search and query this data in an organized manner? the store, process, and access model is useful for two reasons: it reminds businesses that big data is largely about managing data, and it demonstrates the problem of scale within big data management. “big” is relative. the data batches that challenge some companies could be moved through a single google datacenter in under a minute. the only question a company needs to ask itself is how it will store and access increasingly massive amounts of data for its particular use case. there are several high level approaches that companies have turned to in the last few years. the traditional approach the traditional method for handling most data is to use relational databases. data warehouses are then used to integrate and analyze data from many sources. these databases are structured according to the concept of “early structure binding”—essentially, the database has predetermined “questions” that can be asked based on a schema. relational databases are highly functional, and the goal with this type of data processing is for the database to be fully transactional. although relational databases are the most common persistence type by a large margin (see key findings pg. 4-5), a growing number of use cases are not well-suited for relational schema. relational architectures tend to have difficulty when dealing with the velocity and variety of big data, since their structure is very rigid. when you perform functions such as join on many large data sets, the volume can be a problem as well. instead, businesses are looking to non-relational databases, or a mixture of both types, to meet data demand. the newer approach - mapreduce, hadoop, and nosql databases in the early 2000s, web giant google released two helpful web technologies: google file system (gfs) and mapreduce. both were new and unique approaches to the growing problem of big data, but mapreduce was chief among them, especially when it comes to its role as a major influencer of later solution models. mapreduce is a programming paradigm that allows for low cost data analysis and clustered scale-out processing. mapreduce became the primary architectural influence for the next big thing in big data: the creation of the big data management infrastructure known as hadoop. hadoop’s open source ecosystem and ease of use for handling large-scale data processing operations have secured a large part of the big data marketplace. besides hadoop, there was a host of non-relational (nosql) databases that emerged around 2009 to meet a different set of demands for processing big data. whereas hadoop is used for its massive scalability and parallel processing, nosql databases are especially useful for handling data stored within large multi-structured datasets. this kind of discrete data handling is not traditionally seen as a strong point of relational databases, but it’s also not the same kind of data operations that hadoop is running. the solution for many businesses ends up being a combination of these approaches to data management. finding hidden data insights once you get beyond storage and management, you still have the enormous task of creating actionable business intelligence (bi) from the datasets you’ve collected. this problem of processing and analyzing data is maybe one of the trickiest in the data management lifecycle. the best options for data analytics will favor an approach that is predictive and adaptable to changing data streams. the thing is, there’s so many types of analytic models and different ways of providing infrastructure for this process. your analytics solution should scale, but to what degree? scalability can be an enormous pain in your analytical neck, due to the problem of decreasing performance returns when scaling out an algorithm. ultimately, analytics tools rely on a great deal of reasoning and analysis to extract data patterns and data insights, but this capacity means nothing for a business if they can’t then create actionable intelligence. part of this problem is that many businesses have the infrastructure to accommodate big data, but they aren’t asking questions about what problems they’re going to solve with the data. implementing a big data-ready infrastructure before knowing what questions you want to ask is like putting the cart before the horse. but even if we do know the questions we want to ask, data analysis can always reveal many correlations with no clear causes. as organizations get better at processing and analyzing big data, the next major hurdle will be pinpointing the causes behind the trends by asking the right questions and embracing the complexity of our answers. [1] http://www.quora.com/what-is-big-data 2014 guide to big data this guide explores the meaning of big data, how businesses use it, and uncovers new tools and techniques for the future of big data. this guide includes: detailed profiles on 43 big data vendor solutions in-depth articles written by industry experts results from our survey of 850 it professionals "finding the database for your use case" download now

No one loves Null Pointer Exceptions ! Is there a way we can get rid of them ? Maybe . . . Couple of techniques have been discussed in this post Optional type (new in Java 8) Objects class (old Java 7 stuff ;-) ) Optional type in Java 8 What is it? A new type (class) introduced in Java 8 Meant to act as a ‘wrapper‘ for an object of a specific type or for scenarios where there is no object (null) In plain words, its a better substitute for handling nulls (warning: it might not be very obvious at first !) Basic Usage It’a a type (a class) – so, how do I create an instance of it? Just use three static methods in the Optional class public static Optional stringOptional(String input) { return Optional.of(input); } Plain and simple – create an Optional wrapper containing the value. Beware – will throw NPE in case the value itself is null ! public static Optional stringNullableOptional(String input) { if (!new Random().nextBoolean()) { input = null; } return Optional.ofNullable(input); } Slightly better in my personal opinion. There is no risk of an NPE here – in case of a null input, an empty Optional would be returned public static Optional emptyOptional() { return Optional.empty(); } In case you want to purposefully return an ‘empty’ value. ‘empty’ does not imply null Alright – what about consuming/using an Optional? public static void consumingOptional() { Optional wrapped = Optional.of("aString"); if (wrapped.isPresent()) { System.out.println("Got string - " + wrapped.get()); } else { System.out.println("Gotcha !"); } } A simple way is to check whether or not the Optional wrapper has an actual value (use theisPresent method) – this will make you wonder if its any better than usingif(myObj!=null) ;-) Don’t worry, I’ll explain that as well public static void consumingNullableOptional() { String input = null; if (new Random().nextBoolean()) { input = "iCanBeNull"; } Optional wrapped = Optional.ofNullable(input); System.out.println(wrapped.orElse("default")); } One can use the orElse which can be used to return a default value in case the wrapped value is null – the advantage is obvious. We get to avoid the the obvious verbosity of invoking ifPresent before extracting the actual value public static void consumingEmptyOptional() { String input = null; if (new Random().nextBoolean()) { input = "iCanBeNull"; } Optional wrapped = Optional.ofNullable(input); System.out.println(wrapped.orElseGet( () -> { return "defaultBySupplier"; } )); } I was a little confused with this. Why two separate methods for similar goals ? orElse andorElseGet could well have been overloaded (same name, different parameter) Anyway, the only obvious difference here is the parameter itself – you have the option of providing a Lambda Expression representing instance of a Supplier (a Functional Interface) How is using Optional better than regular null checks???? By and large, the major benefit of using Optional is to be able to express your intent clearly – simply returning a null from a method leaves the consumer in a sea of doubt (when the actual NPE occurs) as to whether or not it was intentional and requires further introspection into the javadocs (if any). With Optional, its crystal clear ! There are ways in which you can completely avoid NPE with Optional – as mentioned in above examples, the use of Optional.ofNullable (during Optional creation) andorElse and orElseGet (during Optional consumption) shield us from NPEs altogether Another savior! (in case you can’t use Java 8) Look at this code snippet package com.abhirockzz.wordpress.npesaviors; import java.util.Map; import java.util.Objects; public class UsingObjects { String getVal(Map aMap, String key) { return aMap.containsKey(key) ? aMap.get(key) : null; } public static void main(String[] args) { UsingObjects obj = new UsingObjects(); obj.getVal(null, "dummy"); } } What can possibly be null? The Map object The key against which the search is being executed The instance on which the method is being called When a NPE is thrown in this case, we can never be sure as to What is null? Enter The Objects class package com.abhirockzz.wordpress.npesaviors; import java.util.Map; import java.util.Objects; public class UsingObjects { String getValSafe(Map aMap, String key) { Map safeMap = Objects.requireNonNull(aMap, "Map is null"); String safeKey = Objects.requireNonNull(key, "Key is null"); return safeMap.containsKey(safeKey) ? safeMap.get(safeKey) : null; } public static void main(String[] args) { UsingObjects obj = new UsingObjects(); obj.getValSafe(null, "dummy"); } } The requireNonNull method Simply returns the value in case its not null Throws a NPE will the specified message in case the value in null Why is this better than if(myObj!=null) The stack trace which you would see will clearly have theObjects.requireNonNull method call. This, along with your custom error message will help you catch bugs faster . . much faster IMO ! You can write your user defined checks as well e.g. implementing a simple check which enforces non-emptiness import java.util.Collections; import java.util.List; import java.util.Objects; import java.util.function.Predicate; public class RandomGist { public static T requireNonEmpty(T object, Predicate predicate, String msgToCaller){ Objects.requireNonNull(object); Objects.requireNonNull(predicate); if (predicate.test(object)){ throw new IllegalArgumentException(msgToCaller); } return object; } public static void main(String[] args) { //Usage 1: an empty string (intentional) String s = ""; System.out.println(requireNonEmpty(Objects.requireNonNull(s), (s1) -> s1.isEmpty() , "My String is Empty!")); //Usage 2: an empty List (intentional) List list = Collections.emptyList(); System.out.println(requireNonEmpty(Objects.requireNonNull(list), (l) -> l.isEmpty(), "List is Empty!").size()); //Usage 3: an empty User (intentional) User user = new User(""); System.out.println(requireNonEmpty(Objects.requireNonNull(user), (u) -> u.getName().isEmpty(), "User is Empty!")); } private static class User { private String name; public User(String name){ this.name = name; } public String getName(){ return name; } } } Don’t let NPEs be a pain in the wrong place. We have more than a decent set of tools at our disposal to better handle NPEs or eradicate them altogether ! Cheers! :-)

this tip sounds very basic, but still: i get asked about this about once a week. so it must be something non-obvious in eclipse then ;-): how to add existing files to an eclipse project. as with many things in eclipse, there is not a single way to do something. there are two basic ways to do this: import drag & drop copy & paste the first is the ‘official’ way in eclipse, the other two are much faster and easier :-). 1. import to import one or multiple files, select the folder/project where i want to add the files, then use the menu file > import : menu file import alternatively, i can use the context menu: import context menu then use general > file system : import from file system in the next dialog, i can browse for the files and select them to be added: importing files from filesystem 2. drag & drop while the ‘import’ method is fine, i rarely use it: too many clicks! a easier way (at least under windows) is to drag & drop the files from the windows explorer: drag and drop to add files 3. copy & paste the third way is even easier: select the files i want to add in the explorer/file manager, and copy them ( ctrl-c on windows) paste them in the project/folder in eclipse ( ctrl-v on windows) that last method does not advanced options like if i want to link to the files, see “ link to files and folders in eclipse “, but is definitely the fastest and easiest way. happy adding :-)

APM solutions give you great visibility into any code you have control over; however, today’s systems are largely a combination of code you write along with off-the-shelf components, sitting on top of VMs/containers, and cloud-based services. Thus, full system-wide visibility requires an ability to look into your APM tool as well as log data produced from the components that you may not be able to instrument. This post offers an outline of how APM solutions work and how you can combine them with your system logs to finally get an end-to-end and top-to-bottom view of your system behavior and performance. How APM Tools Work – Hello APM APM tools give you insight deep into your code and often work using cool techniques like dynamic instrumentation. Dynamic instrumentation essentially allows you to instrument your apps on the fly without any need to modify your application source code. Such techniques have become widely been supported by mainstream programming languages to make it possible for even mere mortals to build their own APM tools. For example, since Java version 5, any Java applications can be instrumented using java.lang.instrument, which allows for the instrumentation of any programs running on the JVM through modification of the byte code of methods. It works by letting you alter the corresponding byte code of a class when it is being loaded, such that you can introduce monitoring capabilities such as execution profiling or event tracing. There’e a great beginner tutorial here by Julien Paoletti on how to write your first APM tool in java. It essentially shows you how you can intercept classes at class load time and then inject code into methods of your choice to record how long it takes for given methods to execute. While building a full APM solution is not for the faint hearted, you can easily begin to build your first ‘Hello APM’ tool, and play around with JVM internals following Julien’s post above. Transaction Tracing For those interested in moving beyond simply recording method execution time, you can begin to trace full transactions using some simple techniques. To do so, you essentially need a unique identifier to be passed along to any methods executed in that transaction. Continuing on from our hello world profiler above, you could do this by injecting a unique ID into the thread at any entry point in the system (e.g. new incoming requests). Java provides ThreadLocal storage that allows you to do just this. Using ThreadLocal you can embed a unique ID that gets recorded as each method executes. Reconstructing a Transaction On every invocation of a method along the transaction data is logged. An example of what might be logged by an APM tool is as follows: unique transaction id sequence number call depth method details performance data You can then easily piece together full transaction traces by ordering all method calls by sequence number. Further analysis can be applied to this information for a number of purposes. For example, by analysing the transactions, developers can easily construct design diagrams that can help quickly deduce overall system structure. Relationships between system components can help understand interdependencies enabling developers to anticipate potential conflicts and to debug problems as well as allowing them to reason about their system design (which in turn can have a major impact of system performance). Tracing Transactions Across the Network The real challenge with transaction tracing can come when you are dealing with distributed components. In such scenarios you need to be able to trace transactions across the network. One approach here is to piggy-back the necessary correlation data (unique transaction ID) onto the request from a client to the remote server. RPC (remote procedure call) systems generally employ a standard mechanism, known as ‘stubs and skeletons‘, to hide the complexities of the network from the client making any remote calls. Stubs and skeletons work as follows: The stub masks the low level networking issues from the client and forwards the request on to a server side proxy object (the skeleton). The skeleton masks the low level networking issues from the distributed component. It also delegates the remote request to the distributed component. The distributed component then handles the request and returns control to the skeleton, which in turn returns control to the stub. The stub, in turn, hands back to the client. One approach to the issue of tracing transactions across the network can be achieved by taking advantage of the stubs and skeletons model. Essentially the stub and skeletons can be modified such that the unique transaction ID piggy backs on the communication and is sent as part of the request to the stub and response from the skeleton. The implementation may differ from platform to platform, but the principles can generally be applied. For example, Remote Method Invocation is used for distributed communication on java platforms and details on how this can be achieved for RMI can be found in one of my older research papers here. RMI with Custom Stub Wrapper and Server Side Interception Point Going Beyond APM The above transaction tracing will give you visibility at a method call level across your distributed application. However sometimes external factors outside your application code (server resource, SAAS components your app communicates with, network speed etc.) will have an impact on your overall system performance. One way of enhancing the information provided by your APM solution is to collect and analyze your log data. Logs provide a very flexible way of gathering information on your system behavior without any requirement for deep instrumentation and any of the complex techniques described above. Furthermore you may not be able to instrument every software component or cloud service that makes up your overall system – yet almost all of these will produce valuable log data containing system usage and performance information. In such scenarios, combining APM and log data will give you the complete picture. Below are some tips that will allow you to map logs to APM transactions or to enhance them with data from additional components such as OS, middleware or network level components: Logging the Transaction ID: Any log data produced by your apps can be easily mapped to transactions produced by your APM tool by logging the transaction ID used to trace the transaction. Client Side Logging & Logging User/Session/Account ID: Logging other unique identifiers such as session ID, user ID or account ID, can also help you assist with tracing transactions across log events, where the transaction ID used by the APM tool is unavailable. This can be particularly useful if you are logging events from the client side as well as from back end components where you want to be able to view the sequence of log events related to a give user action or session for example. Same Time Frame For System Logs: Where unique identifiers have not been logged as part of your log events, viewing logs within the same time frame window as your APM tool will help you narrow down related log events and will give you a view into system behavior during the transaction time frame. Correlating with Other KPIs: Logs will contain key performance and resource usage metrics that can be rolled up into trend lines and charts. Correlating APM transaction traces with performance metrics and server resource usage information can help with investigation can result in quicker root cause than investigating transaction traces in isolation. Build It Yourself? Naturally anybody in their right mind would not actually go about building their own APM solution, it’s almost as hair-brained as rolling your own logging solution The good news is you don’t have to do either – simply take advantage of the new Logentries and New Relic integration such that you can trace transactions from end to end and from top to bottom of your entire distributed software stack.

One of the core data structures provided by Hazelcast is IMap extendingjava.util.concurrent.ConcurrentMap - which is basically a distributed map, often used as cache. You can configure such map to use custom MapLoader - piece of Java code that will be asked every time you try to.get() something from that map (by key) which is not yet there. This is especially useful when you use IMap as a distributed in-memory cache - if client code asks for something that wasn't cached yet, Hazelcast will transparently execute your MapLoader.load(key): public interface MapLoader { V load(K key); Map loadAll(Collection keys); Set loadAllKeys(); } The remaining two methods are used during startup to optionally warm-up cache by loading pre-defined set of keys. Your custom MapLoader can reach out to (No)SQL database, web-service, file-system, you name it. Working with such a cache is much more convenient because you don't have to implement tedious "if not in cache load and put in cache" cycle. Moreover, MapLoader has a fantastic feature - if many clients are asking at the same time for the same key (from different threads, or even different cluster members - thus machines), MapLoader is executed only once. This significantly decreases load on external dependencies, without introducing any complexity. In essence IMap with MapLoader is similar to LoadingCache found in Guava - but distributed. However with great power comes great frustration, especially when you don't understand the peculiarities of API and inherent complexity of a distributed system. First let's see how to configure custom MapLoader. You can use hazelcast.xml for that ( element), but you then have no control over life-cycle of your loader (e.g. you can't use Spring bean). A better idea is to configure Hazelcast directly from code and pass an instance of MapLoader: class HazelcastTest extends Specification { public static final int ANY_KEY = 42 public static final String ANY_VALUE = "Forty two" def 'should use custom loader'() { given: MapLoader loaderMock = Mock() loaderMock.load(ANY_KEY) >> ANY_VALUE def hz = build(loaderMock) IMap emptyCache = hz.getMap("cache") when: def value = emptyCache.get(ANY_KEY) then: value == ANY_VALUE cleanup: hz?.shutdown() } Notice how we obtain an empty map, but when asked for ANY_KEY, we get ANY_VALUE in return. This is not a surprise, this is what our loaderMock was expected to do. I left Hazelcast configuration: def HazelcastInstance build(MapLoader loader) { final Config config = new Config("Cluster") final MapConfig mapConfig = config.getMapConfig("default") final MapStoreConfig mapStoreConfig = new MapStoreConfig() mapStoreConfig.factoryImplementation = {name, props -> loader } as MapStoreFactory mapConfig.mapStoreConfig = mapStoreConfig return Hazelcast.getOrCreateHazelcastInstance(config) } Any IMap (identified by name) can have a different configuration. However special "default" map specifies default configuration for all maps. Let's play a bit with custom loaders and see how they behave when MapLoader returns null or throws an exception: def 'should return null when custom loader returns it'() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap cache = hz.getMap("cache") when: def value = cache.get(ANY_KEY) then: value == null !cache.containsKey(ANY_KEY) cleanup: hz?.shutdown() } public static final String SOME_ERR_MSG = "Don't panic!" def 'should propagate exceptions from loader'() { given: MapLoader loaderMock = Mock() loaderMock.load(ANY_KEY) >> {throw new UnsupportedOperationException(SOME_ERR_MSG)} def hz = build(loaderMock) IMap cache = hz.getMap("cache") when: cache.get(ANY_KEY) then: UnsupportedOperationException e = thrown() e.message.contains(SOME_ERR_MSG) cleanup: hz?.shutdown() } MapLoader is executed in a separate thread So far nothing surprising. The first trap you might encounter is how threads interact here. MapLoader is never executed from client thread, always from a separate thread pool: def 'loader works in a different thread'() { given: MapLoader loader = Mock() loader.load(ANY_KEY) >> {key -> "$key: ${Thread.currentThread().name}"} def hz = build(loader) IMap cache = hz.getMap("cache") when: def value = cache.get(ANY_KEY) then: value != "$ANY_KEY: ${Thread.currentThread().name}" cleanup: hz?.shutdown() } This test passes because current thread is "main" while loading occurs from within something like"hz.Cluster.partition-operation.thread-10". This is an important observation and is actually quite obvious if you remember that when many threads try to access the same absent key, loader is called only once. But more needs to be explained here. Almost every operation on IMap is encapsulated into one of operation objects (see also: Command pattern). This operation is later dispatched to one or all cluster members and executed remotely in a separate thread pool, or even on a different machine. Thus, don't expect loading to occur in the same thread, or even same JVM/server (!) This leads to an interesting situation where you request given key on one machine, but actual loading happens on the other. Or even more epic - machines A, B and C request given key whereas machine D physically loads value for that key. The decision which machine is responsible for loading is made based on consistent hashing algorithm. One final remark - of course you can customize the size of thread pools running these operations, see Advanced Configuration Properties. IMap.remove() calls MapLoader This one is totally surprising and definitely to be expected once you think about it: def 'IMap.remove() on non-existing key still calls loader (!)'() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap emptyCache = hz.getMap("cache") when: emptyCache.remove(ANY_KEY) then: 1 * loaderMock.load(ANY_KEY) cleanup: hz?.shutdown() } Look carefully! All we do is removing absent key from a map. Nothing else. Yet, loaderMock.load() was executed. This is a problem especially when your custom loader is particularly slow or expensive. Why was it executed here? Look up the API of `java.util.Map#remove(): V remove(Object key) [...] Returns the value to which this map previously associated the key, or null if the map contained no mapping for the key. Maybe it's controversial but one might argue that Hazelcast is doing the right thing. If you consider our map withMapLoader attached as sort of like a view to external storage, it makes sense. When removing absent key, Hazelcast actually asks our MapLoader: what could have been a previous value? It pretends as if the map contained every single value returned from MapLoader, but loaded lazily. This is not a bug since there is a special method IMap.delete() that works just like remove(), but doesn't load "previous" value: @Issue("https://github.com/hazelcast/hazelcast/issues/3178") def "IMap.delete() doesn't call loader"() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap cache = hz.getMap("cache") when: cache.delete(ANY_KEY) then: 0 * loaderMock.load(ANY_KEY) cleanup: hz?.shutdown() } Actually, there was a bug: IMap.delete() should not call MapLoader.load(), fixed in 3.2.6 and 3.3. If you haven't upgraded yet, even IMap.delete() will go to MapLoader. If you think IMap.remove() is surprising, check out howput() works! IMap.put() calls MapLoader If you thought remove() loading value first is suspicious, what about explicit put() loading a value for a given key first? After all, we are explicitly putting something into a map by key, why Hazelcast loads this value first via MapLoader? def 'IMap.put() on non-existing key still calls loader (!)'() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap emptyCache = hz.getMap("cache") when: emptyCache.put(ANY_KEY, ANY_VALUE) then: 1 * loaderMock.load(ANY_KEY) cleanup: hz?.shutdown() } Again, let's restore to java.util.Map.put() JavaDoc: V put(K key, V value) [...] Returns: the previous value associated with key, or null if there was no mapping for key. Hazelcast pretends that IMap is just a lazy view over some external source, so when we put() something into an IMapthat wasn't there before, it first loads the "previous" value so that it can return it. Again this is a big issue when MapLoaderis slow or expensive - if we can explicitly put something into the map, why load it first? Luckily there is a straightforward workaround, putTransient(): def "IMap.putTransient() doesn't call loader"() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap cache = hz.getMap("cache") when: cache.putTransient(ANY_KEY, ANY_VALUE, 1, TimeUnit.HOURS) then: 0 * loaderMock.load(ANY_KEY) cleanup: hz?.shutdown() } One caveat is that you have to provide TTL explicitly, rather then relying on configured IMap defaults. But this also means you can assign arbitrary TTL to every map entry, not only globally to whole map - useful. IMap.containsKey() involves MapLoader, can be slow or block Remember our analogy: IMap with backing MapLoader behaves like a view over external source of data. That's why it shouldn't be a surprise that containsKey() on an empty map will call MapLoader: def "IMap.containsKey() calls loader"() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap emptyMap = hz.getMap("cache") when: emptyMap.containsKey(ANY_KEY) then: 1 * loaderMock.load(ANY_KEY) cleanup: hz?.shutdown() } Every time we ask for a key that's not present in a map, Hazelcast will ask MapLoader. Again, this is not an issue as long as your loader is fast, side-effect free and reliable. If this is not the case, this will kill you: def "IMap.get() after IMap.containsKey() calls loader twice"() { given: MapLoader loaderMock = Mock() def hz = build(loaderMock) IMap cache = hz.getMap("cache") when: cache.containsKey(ANY_KEY) cache.get(ANY_KEY) then: 2 * loaderMock.load(ANY_KEY) cleanup: hz?.shutdown() } Despite containsKey() calling MapLoader, it doesn't "cache" loaded value to use it later. That's why containsKey()followed by get() calls MapLoader two times, quite wasteful. Luckily if you call containsKey() on existing key, it runs almost immediately, although most likely will require network hop. What is not so fortunate is the behaviour of keySet(),values(), entrySet() and few other methods before version 3.3 of Hazelcast. These would all block in case any key is being loaded at a time. So if you have a map with thousands of keys and you ask for keySet(), one slowMapLoader.load() invocation will block whole cluster. This was fortunately fixed in 3.3, so that IMap.keySet(),IMap.values(), etc. do not block, even when some keys are being computed at the moment. As you can see IMap + MapLoader combo is powerful, but also filled with traps. Some of them are dictated by the API, osme by distributed nature of Hazelcast, finally some are implementation specific. Be sure you understand them before implementing loading cache feature.

Learn how to resolve the ''failure to transfer'' error encountered in Maven in this quick tutorial.

Gephi comes with tools to analyse properties of a graph.

One of the most frequently voiced criticisms of the Java programming language is the amount of Boilerplate Code it requires. This is especially true for simple classes that should do nothing more than store a few values. You need getters and setters for these values, maybe you also need a constructor, overridingequals() and hashcode() is often required and maybe you want a more useful toString()implementation. In the end you might have 100 lines of code that could be rewritten with 10 lines of Scala or Groovy code. Java IDEs like Eclipse or IntelliJ try to reduce this problem by providing various types of code generation functionality. However, even if you do not have to write the code yourself, you always see it (and get distracted by it) if you open such a file in your IDE. Project Lombok (don't be frightened by the ugly web page) is a small Java library that can help reducing the amount of Boilerplate Code in Java Applications. Project Lombok provides a set of annotations that are processed at development time to inject code into your Java application. The injected code is immediately available in your development environment. Lets have a look at the following Eclipse Screenshot: The defined class is annotated with Lombok's @Data annotation and does not contain any more than three private fields. @Data automatically injects getters, setters (for non final fields), equals(), hashCode(),toString() and a constructor for initializing the final dateOfBirth field. As you can see the generated methods are directly available in Eclipse and shown in the Outline view. Setup To set up Lombok for your application you have to put lombok.jar to your classpath. If you are using Maven you just have to add to following dependency to your pom.xml: org.projectlombok lombok 1.14.8 provided You also need to set up Lombok in the IDE you are using: NetBeans users just have to enable the Enable Annotation Processing in Editor option in their project properties (see: NetBeans instructions). Eclipse users can install Lombok by double clicking lombok.jar and following a quick installation wizard. For IntelliJ a Lombok Plugin is available. Getting started The @Data annotation shown in the introduction is actually a shortcut for various other Lombok annotations. Sometimes @Data does too much. In this case, you can fall back to more specific Lombok annotations that give you more flexibility. Generating only getters and setters can be achieved with @Getter and @Setter: @Getter @Setter public class Person { private final LocalDate birthday; private String firstName; private String lastName; public Person(LocalDate birthday) { this.birthday = birthday; } } Note that getter methods for boolean fields are prefixed with is instead of get (e.g. isFoo() instead ofgetFoo()). If you only want to generate getters and setters for specific fields you can annotate these fields instead of the class. Generating equals(), hashCode() and toString(): @EqualsAndHashCode @ToString public class Person { ... } @EqualsAndHashCode and @ToString also have various properties that can be used to customize their behaviour: @EqualsAndHashCode(exclude = {"firstName"}) @ToString(callSuper = true, of = {"firstName", "lastName"}) public class Person { ... } Here the field firstName will not be considered by equals() and hashCode(). toString() will call super.toString() first and only consider firstName and lastName. For constructor generation multiple annotations are available: @NoArgsConstructor generates a constructor that takes no arguments (default constructor). @RequiredArgsConstructor generates a constructor with one parameter for all non-initialized final fields. @AllArgsConstructor generates a constructor with one parameter for all fields in the class. The @Data annotation is actually an often used shortcut for @ToString, @EqualsAndHashCode, @Getter,@Setter and @RequiredArgsConstructor. If you prefer immutable classes you can use @Value instead of @Data: @Value public class Person { LocalDate birthday; String firstName; String lastName; } @Value is a shortcut for @ToString, @EqualsAndHashCode, @AllArgsConstructor,@FieldDefaults(makeFinal = true, level = AccessLevel.PRIVATE) and @Getter. So, with @Value you get toString(), equals(), hashCode(), getters and a constructor with one parameter for each field. It also makes all fields private and final by default, so you do not have to addprivate or final modifiers. Looking into Lombok's experimental features Besides the well supported annotations shown so far, Lombok has a couple of experimental features that can be found on the Experimental Features page. One of these features I like in particular is the @Builder annotation, which provides an implementation of the Builder Pattern. @Builder public class Person { private final LocalDate birthday; private String firstName; private String lastName; } @Builder generates a static builder() method that returns a builder instance. This builder instance can be used to build an object of the class annotated with @Builder (here Person): Person p = Person.builder() .birthday(LocalDate.of(1980, 10, 5)) .firstName("John") .lastName("Smith") .build(); By the way, if you wonder what this LocalDate class is, you should have a look at my blog post about the Java 8 date and time API ;-) Conclusion Project Lombok injects generated methods, like getters and setters, based on annotations. It provides an easy way to significantly reduce the amount of Boilerplate code in Java applications. Be aware that there is a downside: According to reddit comments (including a comment of the project author), Lombok has to rely on various hacks to get the job done. So, there is a chance that future JDK or IDE releases will break the functionality of project Lombok. On the other hand, these comments where made 5 years ago and Project Lombok is still actively maintained. You can find the source of Project Lombok on GitHub.

In this article I will discuss about the entity mapping procedure in JPA. As for my examples I will use the same schemathat I used in one of my previous articles. In my two previous articles I explained how to set up JPA in a Java SE environment. I do not intend to write the setup procedure for a web application because most of the tutorials on the web do exactly that. So let’s skip over directly to object relational mapping, or entity mapping. Wikipedia defines Object Relational Mapping as follows - Object-relational mapping (ORM, O/RM, and O/R mapping) in computer science is a programming technique for converting data between incompatible type systems in object-oriented programming languages. This creates, in effect, a “virtual object database” that can be used from within the programming language. There are both free and commercial packages available that perform object-relational mapping, although some programmers opt to create their own ORM tools. Typically, mapping is the process through which you provide necessary information about your database to your ORM tool. The tool then uses this information to read/write objects into the database. Usually you tell your ORM tool the table name to which an object of a certain type will be saved. You also provide column names to which an object’s properties will be mapped to. Relation between different object types also need to be specified. All of these seem to be a lot of tasks, but fortunately JPA follows what is known as “Convention over Configuration” approach, which means if you adopt to use the default values provided by JPA, you will have to configure very little parts of your applications. In order to properly map a type in JPA, you will at a minimum need to do the following - Mark your class with the @Entity annotation. These classes are called entities. Mark one of the properties/getter methods of the class with the @Id annotation. And that’s it. Your entities are ready to be saved into the database because JPA configures all other aspects of the mapping automatically. This also shows the productivity gain that you can enjoy by using JPA. You do not need to manually populate your objects each time you query the database, saving you from writing lots of boilerplate code. Let’s see an example. Consider the following Address entity which I have mapped according to the above two rules - import javax.persistence.Entity; import javax.persistence.Id; @Entity public class Address { @Id private Integer id; private String street; private String city; private String province; private String country; private String postcode; /** * @return the id */ public Integer getId() { return id; } /** * @param id the id to set */ public Address setId(Integer id) { this.id = id; return this; } /** * @return the street */ public String getStreet() { return street; } /** * @param street the street to set */ public Address setStreet(String street) { this.street = street; return this; } /** * @return the city */ public String getCity() { return city; } /** * @param city the city to set */ public Address setCity(String city) { this.city = city; return this; } /** * @return the province */ public String getProvince() { return province; } /** * @param province the province to set */ public Address setProvince(String province) { this.province = province; return this; } /** * @return the country */ public String getCountry() { return country; } /** * @param country the country to set */ public Address setCountry(String country) { this.country = country; return this; } /** * @return the postcode */ public String getPostcode() { return postcode; } /** * @param postcode the postcode to set */ public Address setPostcode(String postcode) { this.postcode = postcode; return this; } } Now based on your environment, you may or may not add this entity declaration in your persistence.xml file, which I have explained in my previous article. Ok then, let’s save some object! The following code snippet does exactly that - import com.keertimaan.javasamples.jpaexample.entity.Address; import javax.persistence.EntityManager; import com.keertimaan.javasamples.jpaexample.persistenceutil.PersistenceManager; public class Main { public static void main(String[] args) { EntityManager em = PersistenceManager.INSTANCE.getEntityManager(); Address address = new Address().setId(1) .setCity("Dhaka") .setCountry("Bangladesh") .setPostcode("1000") .setStreet("Poribagh"); em.getTransaction() .begin(); em.persist(address); em.getTransaction() .commit(); System.out.println("addess is saved! It has id: " + address.getId()); Address anotherAddress = new Address().setId(2) .setCity("Shinagawa-ku, Tokyo") .setCountry("Japan") .setPostcode("140-0002") .setStreet("Shinagawa Seaside Area"); em.getTransaction() .begin(); em.persist(anotherAddress); em.getTransaction() .commit(); em.close(); System.out.println("anotherAddress is saved! It has id: " + anotherAddress.getId()); PersistenceManager.INSTANCE.close(); } } Let’s take a step back at this point and think what we needed to do if we had used plain JDBC for persistence. We had to manually write the insert queries and map each of the attributes to the corresponding columns for both cases, which would have required a lot of code. An important point to note about the example is the way I am setting the id of the entities. This approach will only work for short examples like this, but for real applications this is not good. You’d typically want to use, say, auto-incremented id columns or database sequences to generate the id values for your entities. For my example, I am using a MySQL database, and all of my id columns are set to auto increment. To reflect this in my entity model, I can use an additional annotation called @GeneratedValue in the id property. This tells JPA that the id value for this entity will be automatically generated by the database during the insert, and it should fetch that id after the insert using a select command. With the above modifications, my entity class becomes something like this - import javax.persistence.Entity; import javax.persistence.Id; import javax.persistence.GeneratedValue; @Entity public class Address { @Id @GeneratedValue private Integer id; // Rest of the class code........ And the insert procedure becomes this - Address anotherAddress = new Address() .setCity("Shinagawa-ku, Tokyo") .setCountry("Japan") .setPostcode("140-0002") .setStreet("Shinagawa Seaside Area"); em.getTransaction() .begin(); em.persist(anotherAddress); em.getTransaction() .commit(); How did JPA figure out which table to use to save Address entities? Turns out, it’s pretty straight-forward - When no explicit table information is provided with the mapping then JPA tries to find a table whose name matches with the entity name. The name of an entity can be explicitly specified by using the “name” attribute of the @Entity annotation. If no name attribute is found, then JPA assumes a default name for an entity. The default name of an entity is the simple name (not fully qualified name) of the entity class, which in our case is Address. So our entity name is then determined to be “Address”. Since our entity name is “Address”, JPA tries to find if there is a table in the database whose name is “Address” (remember, most of the cases database table names are case-insensitive). From our schema, we can see that this is indeed the case. So how did JPA figure our which columns to use to save property values for address entities? At this point I think you will be able to easily guess that. If you cannot, stay tuned for my next post! Until next time. [ Full working code can be found at github.]

Property based testing is an alternative approach to testing, complementingexample based testing. The latter is what we've been doing all our lives: exercising production code against "examples" - inputs we think are representative. Picking these examples is an art on its own: "ordinary" inputs, edge cases, malformed inputs, etc. But why are we limiting ourselves to just few examples? Why not test hundreds, millions... ALL inputs? There are at least two difficulties with that approach: Scale. A pure function taking just one int input would require 4 billion tests. This means few hundred gigabytes of test source code and several months of execution time. Square it if a function takes two ints. For String it practically goes to infinity. Assume we have these tests, executed on a quantum computer or something. How do you know the expected result for each particular input? You either enter it by hand (good luck) or generate expected output. Bygenerate I mean write a program that produces expected value for every input. But aren't we testing such program already in the first place? Are we suppose to write better, error-free version of code under test just to test it? Also known as ugly mirror antipattern. So you understand testing every single input, although ideal, is just a mental experiment, impossible to implement. That being said property based testing tries to get as close as possible to this testing nirvana. Issue #1 is solved by slamming code under test with hundreds or thousands of random inputs. Not all of them, not even a fraction. But a good, random representation. Issue #2 is surprisingly harder. Property based testing can generate random arguments, but it can't figure out what should be the expected outcome for that random input. Thus we need a different mechanism, giving name to whole philosophy. We have to come up with properties (invariants, behaviours) that code under test exhibits no matter what the input is. This sounds very theoretically, but there are many such properties in various scenarios: Absolute value of any number should never be negative Encoding and decoding any string should yield the same String back for every symmetric encoding Optimized version of some old algorithm should produce the same result as the old one for any input Total money in a bank should remain the same after arbitrary number of intra-bank transactions in any order As you can see there are many properties we can think of that do not mention specific example inputs. This is not exhaustive and strict testing. It's more like sampling and making sure samples are "sane". There are many, many libraries supporting property based testing for virtually every language. In this article we will explore Spock and ScalaCheck later. Spock + custom data generators Spock does not support property based testing out-of-the-box. However with help from data driven testing and 3rd-party data generators we can go quite far. Data tables in Spock can be generalized into so-called data pipes: def 'absolute value of #value should not be negative'() { expect: value.abs() >= 0 where: value << randomInts(100) } private static def List randomInts(int count) { final Random random = new Random() (1..count).collect { random.nextInt() } } Code above will generate 100 random integers and make sure for all of them.abs() is non-negative. You might think this test is quite dumb, but to a great surprise it actually discovers one bug! But first let's kill some boilerplate code. Generating random inputs, especially more complex, is cumbersome and boring. I found two libraries that can help us. spock-genesis: import spock.genesis.Gen def 'absolute value of #value should not be negative'() { expect: value.abs() >= 0 where: value << Gen.int.take(100) } Looks great, but if you want to generate e.g. lists of random integers,net.java.quickcheck has nicer API and is not Groovy-specific: import static net.java.quickcheck.generator.CombinedGeneratorsIterables.someLists import static net.java.quickcheck.generator.PrimitiveGenerators.integers def 'sum of non-negative numbers from #list should not be negative'() { expect: list.findAll{it >= 0}.sum() >= 0 where: list << someLists(integers(), 100) } This test is interesting. It makes sure sum of non-negative numbers is never negative - by generating 100 lists of randoms ints. Sounds reasonable. However multiple tests are failing. First of all due to integer overflow sometimes two positiveints add up to a negative one. Duh! Another type of failure that was discovered is actually frightening. While [1,2,3].sum() is 6, obviously, [].sum() is... null(WAT?) As you can see even silliest and most basic property based tests can be useful in finding unusual corner cases in your data. But wait, I said testing absolute of intdiscovered one bug. Actually it didn't, because of poor (too "random") data generators, not returning known edge values in the first place. We will fix that in the next article.