This article was originally written by Jeff Morris In the introduction to this series, I discussed some of the motivation for rewriting .NET SDK, the goals, objectives and the major features of the upcoming 2.0 release, and we examined the high-level architecture (10,000 feet view) of a Couchbase Server Client SDK. In this post we will go over the design and development of one of the core configuration components of a Couchbase SDK: Server Configuration. Introduction A Couchbase SDK client requires configuration from two sources: the Client Configuration, which defines the IP of the cluster to connect to, number of connections to use and other important information regarding how the client will interact with the cluster, and the Server Configuration, which defines the current state of the cluster (e.g. number of nodes, buckets that are available, etc.), thus driving the internal state of a client (Cluster Map) This post will only discuss the Server Configuration aspects and will largely revolve around implementing several well-defined interfaces or contracts. HTTP Streaming Configuration Currently, most clients use a “bootstrapping” technique via client configuration and a “Streaming Configuration” exposed by the Couchbase REST API. This is supported by versions of Couchbase from 2.2 and back. The usual approach is as follows: Within the “uris” element of a Client Configuration (semantics very per client), a URL is defined for which to start the bootstrapping process: http://[SERVER]:8091/pools The response is then parsed and the a request is made to get the buckets configuration: http://[SERVER]:8091/pools/default?uuid=[UUID] This response is parsed and another request is made to get streaming URL from: http://[SERVER]:8091/pools/default/buckets?v=[VERSION]&uuid=[UUID] Finally, the streaming URL connection is made which is long-lived and raises events in the client with respect to changes in the cluster: http://[SERVER]:8091/pools/default/bucketsStreaming/default?bucket_uuid=[UUID] The client will then change its internal state to match that of the current server configuration. There are some problems with this approach, among others: The “streaming URL” is resource intensive to create and maintain (mainly memory) on the server-side During a rebalance or failover situation, the cluster configuration may change many, many times. Each time this happens the client must tear down all of its resources (socket connections, VBucket mappings) and build its state up again and again, which can leads to reduced throughput, latency, higher than expected memory and CPU usage, and so on and so forth… Operations that are in-flight may be terminated and then re-tried on a new config state – it’s as if the “carpet has been pulled out from underneath them”. Responding to NOT_MY_VBUCKET responses are handled in-efficiently by simple trying the next node in the list – there is no information to help the client in which node to re-direct the operation to. A New Model for Configuration Management: CCCP While the streaming HTTP “bootstrapping” approach has worked reasonably well for most clients, the downsides have begun to outweigh the plusses, thus a new model for updating client configuration has been defined is available starting with the 2.5 version of the Couchbase Server: Client Cluster Configuration Publication or “CCCP”. CCCP introduces a new operation to be used before or after authentication to request configuration as well as a mechanism for returning configuration information when a NOT_MY_VBUCKET response is returned for a failed operation. In this case CCCP supporting SDK, the client will react by using the configuration to update itself before resending the operation. Note that a NOT_MY_VBUCKET is the standard response that is returned by the cluster when the cluster itself has changed (during a rebalance or failover scenario for example) and the client has not yet “synched” up and is using a stale configuration, resulting in an invalid key mapping. Whereas the “bootstrapping” approach is somewhat of a “pull” type operation, CCCP is either “push” or “pull” depending upon whether the request was initiated by the client (via an explicit CMD_GET_CLUSTER_CONFIG operation) or by the server itself (via a NOT_MY_VBUCKET response to an operation). We will go over CCCP in more detail in a later post. File Based Configuration One other semi-supported configuration option exists: file based configuration. File based configuration is primarily useful for testing and development and we will provide an implementation in the test projects to remove some of the dependencies that are difficult to replicate and or cause false positives when running the test suite. Structural Architecture View Internally the Server Configuration component of the client is a provider based model, in which multiple implementations of a configuration provider can be configured in the client and then a strategy can be chosen to determine which provider should be used. The default is a simple linear, fallback approach where the first configured provider is used and then if it fails the next provider in sequence will take its place. Here is a diagram showing the main actor objects and the relationships with some of other key objects within the client which will be discussed in subsequent posts: A description of each follows: ConfigurationProvider: a source which shall yield a new ConfigInfo. It’s the responsibility of the provider to provide the mechanism for fetching the configuration from its source. ConfigurationInformation: the configuration info contains a list of possible nodes and the VBucket map informing clients about which servers within said nodes a given key should be forwarded to. ConfigurationManager: bridge between the client and the providers and the strategy taken to determine which provider to use and what retry logic to apply. A more detailed document of this architecture can be found here. Please note that this, like all development, is an evolutionary process, so expect some changes and revisions over time. Conclusion and Next Steps This post discussed the history (HTTP Streaming) and the future (CCCP) of Couchbase SDK Server Configuration Management. In the next post we will go into detail the implementation of the HTTP Streaming configuration provider which is required for clients targeting pre-2.5 versions of the Couchbase Server.

In this post we will see how to setup a load balanced JBoss or Tomcat environment with the fail-over capability. This post assumes that both Jboss/Tomcat and Apache httpd are setup and running properly. Configure Apache Httpd Step 1: Configure apache’s workers.properties Go to /usr/local/apache2/conf/extra and open workers.properties and configure the properties indicated in bold. # for mapping requests # The configuration directives are valid # for the mod_jk version 1.2.18 and later # worker.list=loadbalancer,status # Define node # modify the host as your host IP or DNS name. worker.node.port=8009 worker.node.host=192.168.0.3 #(IP or DNS name of the server on which Jboss is running) worker.node.type=ajp13 worker.node.lbfactor=1 worker.node2.port=8009 worker.node2.host=192.168.0.4 #(IP or DNS name of the server on which Jboss is running) worker.node2.type=ajp13 worker.node2.lbfactor=1 # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node,node2 worker.loadbalancer.sticky_session=1 worker.status.type=status The property balance_workers is used to specify the nodes to load balance. For example, if we specify only ‘node’, all requests will be routed to the server named ‘node’. Modify this to test that requests are going to both servers. Step 2: Configure mod-jk.conf Go to /usr/local/apache2/conf/extra and open mod-jk.conf and configure the bold properties according to your requirements. #Load mod_jk module # Specify the filename of the mod_jk lib LoadModule jk_module modules/mod_jk.so # Where to find workers.properties JkWorkersFile conf/extra/workers.properties # Where to put jk logs JkLogFile /var/apache/logs/mod_jk.log # Set the jk log level [debug/error/info] JkLogLevel debug # Select the log format JkLogStampFormat "[%a %b %d %H:%M:%S %Y]" # JkOptions indicates to send SSK KEY SIZE # Note: Changed from +ForwardURICompat. # See http://tomcat.apache.org/security-jk.html JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories JkOptions +FlushPackets # JkRequestLogFormat JkRequestLogFormat "%w %V %T" # Mount your applications JkMount /* loadbalancer # You can use external file for mount points. # It will be checked for updates each 60 seconds. # The format of the file is: /url=worker # /examples/*=loadbalancer JkMountFile conf/extra/uriworkermap.properties # Add shared memory. # This directive is present with 1.2.10 and # later versions of mod_jk, and is needed for # for load balancing to work properly # Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to # https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=225452 JkShmFile /var/run/jk.shm # Add jkstatus for managing runtime data JkMount status Order deny,allow Deny from all Allow from 127.0.0.1 The JkMount /* loadbalancer indicates that all requests are to be routed through the load balancer. Step 3: Go to /usr/local/apache2/conf and edit httpd.conf Add Include conf/extra/mod-jk.conf to the end of the file. Configure Tomcat/JBoss These settings need to be made for each of the nodes we intend on placing behind the load balancer. We need to modify the server.xml file which is located at For Tomcat ../apache-tomcat/conf/ For JBoss ../jboss-5.1.0.GA/server/default/deploy/jbossweb.sar Edit the following tag: where the jvmRoute attribute is configured to ‘node’ (which is the name we gave the first worker while configuring workers.properties.) Next, locate the following tag and edit the port to be the same as while configuring the workers.properties. Configure the other node(s) accordingly. Next, Start Apache httpd and the Jboss/tomcat node(s) that you have configured.

In our previous post on MongoDB storage structure and dbStats metrics, we covered how MongoDB stores data and the differences between the dataSize, storageSize and fileSize metrics. We can now apply this knowledge to evaluate strategies for re-using MongoDB disk space. When documents or collections are deleted, empty record blocks within data files arise. MongoDB attempts to reuse this space when possible, but it will never return this space to the file system. This behavior explains why fileSize never decreases despite deletes on a database. If your app frequently deletes or if your fileSize is significantly larger than the size of your data plus indexes, you can use one of the methods below reclaim free space. Getting your free space back Compacting individual collections You can compact individual collections using the compact command. This command rewrites and defragments all data in a collection, as well as all of the indexes on that collection. Important notes on compacting: This operation blocks all other database activity when running and should be used only when downtime for your database is acceptable. If you are running a replica set, you can perform compaction on secondaries in order to avoid blocking the primary and use failover to make the primary a secondary before compacting it. Compacting individual collections will not reduce your storage footprint on disk (i.e., your fileSize) but it will defragment the collections you compact. Compacting one or more databases For a single-node MongoDB deployment, you can use the db.repairDatabase() command to compact all the collections in the database. This operation rewrites all the data and indexes for each collection in the database from scratch and thereby compacts and defragments the entire database. To compact all the databases on your server process, you can stop your mongod process and run it with the “–repair” option. Important notes on running a repair: This operation blocks all other database activity when running and should be used only when downtime for your database is acceptable. Running a repair requires free disk space equal to the size of your current data set plus 2 GB. You can use space in a different volume than the one that your mongod is running in by specifying the “–repairpath” option. Compacting all databases on a server by re-syncing replica set nodes For a multi-node MongoDB deployment, you can resync a secondary from scratch to reclaim space. By resyncing each node in your replica set you effectively rewrite the data files from scratch and thereby defragment your database. Please note that if your cluster is comprised of only two electable nodes, you will sacrifice high availability during the resync because the secondary is completely wiped before syncing. If your app is sensitive to downtime, we recommend a process similar to the one we use here at MongoLab which we call a “rolling node replacement.” This process replaces each node in your cluster in turn by bringing a new node into the cluster, replicating the data to that new node and removing the old node. In this way, your cluster can maintain the same level of redundancy during the compaction as during normal operations. A tip about efficiently using space usePowerOf2Sizes Setting the usePowerof2Sizes option is a proactive approach to reusing space in collections that experience frequent document moves or deletions. This option supersedes the default padding factor mechanism and reduces the impact of fragmentation within the collection by allocating additional space for each document in intervals that follow the powers of 2. Setting this option for a specific collection makes it less likely that documents in that collection need to be moved when they grow in size, less likely that a document will need to be moved more than once in its lifetime, and more likely that space left by moving documents can be reused by new or other moved documents. Thanks for reading! We hope the above strategies help guide you in evaluating options for reusing empty space in your MongoDB.

At the end of last year I was playing around with running scheduled tasks to monitor a Neo4j cluster and one of the problems I ran into was that the monitoring would sometimes exit. I eventually realised that this was because a RuntimeException was being thrown inside the Runnable method and I wasn’t handling it. The following code demonstrates the problem: import java.util.ArrayList; import java.util.List; import java.util.concurrent.*; public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } If we run that code we’ll see the RuntimeException but the executor won’t exit because the thread died without informing it: Exception in thread "main" pool-1-thread-1 -> 1391212558074 java.util.concurrent.ExecutionException: java.lang.RuntimeException: game over at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at RunnableBlog.main(RunnableBlog.java:11) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) Caused by: java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) At the time I ended up adding a try catch block and printing the exception like so: public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } catch (RuntimeException e) { e.printStackTrace(); } } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } This allows the exception to be recognised and as far as I can tell means that the thread executing the Runnable doesn’t die. java.lang.RuntimeException: game over pool-1-thread-1 -> 1391212651955 at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-1 -> 1391212652956 java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-1 -> 1391212653955 java.lang.RuntimeException: game over at RunnableBlog$1.run(RunnableBlog.java:16) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) This worked well and allowed me to keep monitoring the cluster. However, I recently started reading ‘Java Concurrency in Practice‘ (only 6 years after I bought it!) and realised that this might not be the proper way of handling the RuntimeException. public class RunnableBlog { public static void main(String[] args) throws ExecutionException, InterruptedException { ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis()); throw new RuntimeException("game over"); } catch (RuntimeException e) { Thread t = Thread.currentThread(); t.getUncaughtExceptionHandler().uncaughtException(t, e); } } }, 0, 1000, TimeUnit.MILLISECONDS).get(); System.out.println("exit"); executor.shutdown(); } } I don’t see much difference between the two approaches so it’d be great if someone could explain to me why this approach is better than my previous one of catching the exception and printing the stack trace.

As it turns out, doing work on big data sets is quite hard. To start with, you need to get the data, and it is… well, big. So that takes a while. Instead, I decided to test my theory on the following scenario. Given 4 GB of random numbers, let us find how many times we have the number 1. Because I wanted to ensure a consistent answer, I wrote: public static IEnumerable RandomNumbers() { const long count = 1024 * 1024 * 1024L * 1; var random = new MyRand(); for (long i = 0; i < count; i++) { if (i % 1024 == 0) { yield return 1; continue; } var result = random.NextUInt(); while (result == 1) { result = random.NextUInt(); } yield return result; } } /// /// Based on Marsaglia, George. (2003). Xorshift RNGs. /// http://www.jstatsoft.org/v08/i14/paper /// public class MyRand { const uint Y = 842502087, Z = 3579807591, W = 273326509; uint _x, _y, _z, _w; public MyRand() { _y = Y; _z = Z; _w = W; _x = 1337; } public uint NextUInt() { uint t = _x ^ (_x << 11); _x = _y; _y = _z; _z = _w; return _w = (_w ^ (_w >> 19)) ^ (t ^ (t >> 8)); } } I am using a custom Rand function because it is significantly faster than System.Random. This generate 4GB of random numbers, at also ensure that we get exactly 1,048,576 instances of 1. Generating this in an empty loop takes about 30 seconds on my machine. For fun, I run the external sort routine in 32 bits mode, with a buffer of 256MB. It is currently processing things, but I expect it to take a while. Because the buffer is 256 in size, we flush it every 128 MB (while we still have half the buffer free to do more work). The interesting thing is that even though we generate random number, sorting then compressing the values resulted in about 60% compression rate. The problem is that for this particular case, I am not sure if that is a good thing. Because the values are random, we need to select a pretty high degree of compression just to get a good compression rate. And because of that, a significant amount of time is spent just compressing the data. I am pretty sure that for real world scenario, it would be better, but that is something that we’ll probably need to test. Not compressing the data in the random test is a huge help. Next, external sort is pretty dependent on the performance of… sort, of course. And sort isn’t that fast. In this scenario, we are sorting arrays of about 26 million items. And that takes time. Implementing parallel sort cut this down to less than a minute per batch of 26 million. That let us complete the entire process, but then it halts with the merge. The reason for that is that we push all the values into a heap, and there are 1 billion of them. Now, the heap never exceed 40 items, but those are still 1 billion * O(log 40) or about 5.4 billion comparisons that we have to do, and we do this sequentially, which takes time. I tried thinking about ways to parallel, but I am not sure how that can be done. We have 40 sorted files, and we want to merge all of them. Obviously we can sort each 10 files set in parallel, then sort the resulting 4, but the cost we have now is the actual sorting cost, not I/O. I am not sure how to approach this. For what is it worth, you can find the code for this here.

Before starting, you would need the Ultimate version of IDEA to run WebLogic Server (yes, the paid version or the 30 days trial). The Community edition of IDEA will not support Application Server deployment. I also assume you have already setup WebLogic Server and a user domain as per my previous blog instructions. So now let's setup the IDE to boost your development. Create a simple HelloWorld web application in IDEA. For your HelloWorld, you can go into the Project Settings > Artifacts, and add "web:war exploded" entry for your application. You will add this into your app server later. Ensure you have added the Application Server Views plugin with WebLogic Server. (It's under Settings > IDE Settings > Application Server) Click + and enter Name: WebLogic 12.1.2 WebLogic Home: C:\apps\wls12120 Back to your editor, select Menu: Run > Edit Configuration Click + and add "WebLogic Server" > Local Name: WLS On Server tab, ensure DomainPath is set: C:\apps\wls12120\mydomain On Deployment tab, select "web:war exploded" for your HelloWorld project. Click OK Now Menu: Run > "Run WLS" Your WebLogic Server should now start and running your web application inside. You may visit the browser on http://localhost:7001/web_war_exploded Some goodies with Intellij IDEA and WLS are: Redeploy WAR only without restarting server Deploy application in exploded mode and have IDE auto make and sync Debug application with server running within IDE Full control on server settings NOTE: As noted in previous blog, if you do not set MW_HOME as system variable, then you must add this in IDEA's Run Configuration. Or you edit your "mydomain/bin/startWebLogic.cmd" and"stopWebLogic.cmd" scripts directly.

in this post i would like to cover the single responsibility principle ( srp ). i think that this is the basis of any clean and well designed system. what is srp? the term was introduced by robert c. martin . it is the ‘s’ from the solid principles, which are the basis for ood. http://en.wikipedia.org/wiki/solid_(object-oriented_design) here’s the pdf paper for srp by robert c. martin https://docs.google.com/file/d/0byowmqah_nugnhetcu5oekddmkk/ from wikipedia: …in object-oriented programming, the single responsibility principle states that every class should have a single responsibility, and that responsibility should be entirely encapsulated by the class. all its services should be narrowly aligned with that responsibility…. from clean code : a class or module should have one, and only one, reason to change . so if a class (or module) needs to be modified for more than one reason, it does more than one thing. i.e. has more than one responsibility. why srp? organize the code let’s imagine a car mechanic who owns a repair shop. he has many many tools to work with. the tools are divided into types; pliers, screw-drivers (phillips / blade), hammers, wrenches (tubing / hex) and many more. how would it be easier to organize the tools? few drawers with different types in each one of them? or, many small drawers, each containing a specific type?now, imagine the drawer as the module . this is why many small modules (classes) are more organized then few large ones. less fragile when a class has more than one reason to be changed, it is more fragile. a change in one location might lead to some unexpected behavior in totally other places. low coupling more responsibilities lead to higher coupling. the couplings are the responsibilities. higher coupling leads to more dependencies, which is harder to maintain. code changes refactoring is much easier for a single responsibility module. if you want to get the shotgun effect , let your classes have more responsibilities. maintainability it’s obvious that it is much easier to maintain a small single purpose class, then a big monolithic one. testability a test class for a ‘one purpose class’ will have less test cases (branches). if a class has one purpose it will usually have less dependencies, thus less mocking and test preparing. the “self documentation by tests” becomes much clearer. easier debugging since i started doing tdd and test-first approach, i hardly debug. really. but, there come times when i must debug in order to understand what’s going on. in a single responsibility class, finding the bug or the cause of the problem, becomes a much easier task. what needs to have single responsibility? each part of the system. the methods the classes the packages the modules how to recognize a break of the srp? class has too many dependencies a constructor with too many input parameters implies many dependencies (hopefully you do inject dependencies). another way too see many dependencies is by the test class. if you need to mock too many objects, it usually means breaking the srp. method has too many parameters same as the class’s smell. think of the method’s parameters as dependencies. the test class becomes too complicated if the test has too many variants, it might suggest that the class has too many responsibilities. it might suggest that some methods do too much. class / method is long if a method is long, it might suggest it does too much. same goes for a class. my rule of thumb is that a class should not exceed 200-250 loc. imports included descriptive naming if you need to describe what your class / method / package is using with the and world, it probably breaks the srp. class with low cohesion cohesion is an important topic of its own and should have its own post. but cohesion and srp are closely related and it is important to mention it here. in general, if a class (or module) is not cohesive, it probably breaks the srp. a hint for a non-cohesive class: the class has two fields. one field is used by some methods. the other field is used by the other methods. change in one place breaks another if a change in the code to add a new feature or simply refactor broke a test which seems unrelated, it might suggest a breaking the srp. shotgun effect if a small change makes a big ripple in your code. if you need to change many locations it might suggest, among other smells, that the srp is broken. unable to encapsulate a module i will explain using spring, but the concept is important (not the implementation). suppose you use the @configuration or xml configuration. if you can’t encapsulate the beans in that configuration, it should give you a hint of too much responsibility. the configuration should hide any inner bean and expose minimal interfaces. if you need to change the configuration due to more than one reason, then, well, you know… how to make the design compliant with the single responsibility principle the suggestions below can apply to other topics of the solid principles. they are also good for any clean code suggestion. but here they are aimed for the single responsibility principle. awareness this is a general suggestion for clean code. we need to be aware of our code. we need to take care. as for srp, we need to try and catch as early as we can a class that is responsible for too much. we need to always look for a ‘too big method’. testable code write your code in a way that everything can be tested. then, you will surly want that your tests be simple and descriptive. tdd (i am not going to add anything here) code coverage metrics sometimes, when a class does too much, it won’t have 100% coverage at first shot. check the code quality metrics. refactoring and design patterns for srp, we’ll mostly do extract-method, extract-class, move-method. we’ll use composition and strategy instead of conditionals. clear modularization of the system when using a di injector (spring), i think that configuration class (or xml) can pinpoint the modules design. and modules’ single responsibility. i prefer to have several small to medium size of configuration files (xml or java) than having one big file / class. it helps see the responsibility of the module and easier to maintain. i think that the configuration approach of injection has an advantage of annotation approach. simply because the configuration approach put the modules in the spotlight. conclusion as i mentioned in the beginning of this post, i think that single-responsibility-principle is the basis of a good design. if you have this principle in your mind while designing and developing, you will have a simpler more readable code. better design will be followed. one final note as always, one needs to be careful on how to apply practices, code and design. sometimes we might do over-work and make simple things over complex. so a common sense must be applied at any refactor and change.

Learn how to convert XML objects to Java, and vice versa.

Within this blog post I describe how you can use AspectJ to automatically translate one type of exception to another. The problem Sometimes we are in situations where we have to convert an exception (often thrown by a third-party library) to another type of exception. Assume you are using a persistence framework like hibernate and you do not want to leak hibernate specific exceptions out of a certain application layer. Maybe you are using more than one persistence technology and you want to wrap technology specific exceptions into a common base exception. In such situations, one can end with code like this: public class MyRepository { public Object getSomeData() { try { // assume hibernate is used to access some data } catch(HibernateException e) { // wrap hibernate specific exception into a general DataAccessException throw new DataAccessException(e); } } } Of course this becomes ugly if you have to do this every time you access a certain framework. The AspectJ way AspectJ is an aspect oriented programming (AOP) extension for Java. With AspectJ we can define Cross-cutting concerns that take care of the exception translation process for us. To get started we first have to add the AspectJ dependency to our project: org.aspectj aspectjrt 1.7.4 Next we have to set up ajc, the compiler and bytecode weaver for AspectJ. This step depends on the developing environment you are using, so I will not go into details here. Eclipse users should have a look at theAspectJ Development Tools (AJDT) for Eclipse. IntelliJ IDEA users should make sure the AspectJ plugin is enabled. There is also an AspectJ Maven plugin available (check this pom.xml for an example configuration). Now let's define our aspect using AspectJ annotations: @Aspect public class ExceptionTranslationAspect { @Around("execution(* com.mscharhag.exceptiontranslation.repository..*(..))") public Object translateToDataAccessException(ProceedingJoinPoint pjp) throws Throwable { try { return pjp.proceed(); } catch (HibernateException e) { throw new DataAccessException(e); } } } Using the @Aspect annotation we can declare a new aspect. Within this aspect we use the @Aroundannotation to define an advice that is always executed if the passed pointcut is matched. Here, the pointcut execution(* com.mscharhag.exceptiontranslation.repository..*(..)) tells AspectJ to call translateToDataAccessException() every time a method of a class inside thecom.mscharhag.exceptiontranslation.repository package is executed. Within translateToDataAccessException() we can use the passed ProceedingJoinPoint object to proceed the method execution we intercepted. In this example we just add a try/catch block around the method execution. Using the ProceedingJoinPoint instance we could also do more interesting things like analyzing the method signature using pjp.getSignature() or accessing method parameters withpjp.getArgs(). We can now remove the try/catch block from the example repository implementation shown above and use a simple test to verify our aspect is working: public class MyRepositoryTest { private MyRepository repository = new MyRepository(); @Test(expected = DataAccessException.class) public void testExceptionTranslation() { this.repository.getSomeData(); } } Conclusion Using AspectJ we can easily automate the conversion of Java runtime exceptions. This simplifies our code by removing try/catch blocks that would otherwise be required for exception translation. You can find the full source of the example project on GitHub.

Security has become a great topic of discussion in the last few years due to the recent releasing of documents from Edward Snowden and the explosion of hacking against online commerce stores like JC Penny, Sony andTarget. While this post will not give you all of the tools to help prevent the use of illegally sourced data, this post will provide a starting point for building a set of tools and tactics that will help prevent the use of data by other parties. This post will show how to adopt AES encryption for strings in a Java environment. It will talk about creating AES keys and storing AES keys in a JCEKS keystore format. A working example of the code in this blog is located athttps://github.com/mike-ensor/aes-256-encryption-utility It is recommended to read each section in order because each section builds off of the previous section, however, this you might want to just jump quickly jump to a particular section. Setup - Setup and create keys with keytool Encrypt - Encrypt messages using byte[] keys Decrypt - Decrypt messages using same IV and key from encryption Obtain Keys from Keystore - Obtain keys from keystore via an alias What is JCEKS? JCEKS stands for Java Cryptography Extension KeyStore and it is an alternative keystore format for the Java platform. Storing keys in a KeyStore can be a measure to prevent your encryption keys from being exposed. Java KeyStores securely contain individual certificates and keys that can be referenced by an alias for use in a Java program. Java KeyStores are often created using the "keytool" provided with the Java JDK. NOTE: It is strongly recommended to create a complex passcode for KeyStores to keep the contents secure. The KeyStore is a file that is considered to be public, but it is advisable to not give easy access to the file. Setup All encryption is governed by laws of each country and often have restrictions on the strength of the encryption. One example is that in the United States, all encryption over 128-bit is restricted if the data is traveling outside of the boarder. By default, the Java JCE implements a strength policy to comply with these rules. If a stronger encryption is preferred, and adheres to the laws of the country, then the JCE needs to have access to the stronger encryption policy. Very plainly put, if you are planning on using AES 256-bit encryption, you must install theUnlimited Strength Jurisdiction Policy Files. Without the policies in place, 256-bit encryption is not possible. Installation of JCE Unlimited Strength Policy This post is focusing on the keys rather than the installation and setup of the JCE. The installation is rather simple with explicit instructions found here (NOTE: this is for JDK7, if using a different JDK, search for the appropriate JCE policy files). Keystore Setup When using the KeyTool manipulating a keystore is simple. Keystores must be created with a link to a new key or during an import of an existing keystore. In order to create a new key and keystore simply type: keytool -genseckey -keystore aes-keystore.jck -storetype jceks -storepass mystorepass -keyalg AES -keysize 256 -alias jceksaes -keypass mykeypass Important Flags In the example above here are the explanations for the keytool's parameters: Keystore Parameters genseckey Generate SecretKey. This is the flag indicating the creation of a synchronous key which will become our AES key keystore Location of the keystore. If the keystore does not exist, the tool will create a new store. Paths can be relative or absolute but must be local storetype this is the type of store (JCE, PK12, JCEKS, etc). JCEKS is used to store symmetric keys (AES) not contained within a certificate. storepass password related to the keystore. Highly recommended to create a strong passphrase for the keystore Key Parameters keyalg algorithm used to create the key (AES/DES/etc) keysize size of the key (128, 192, 256, etc) alias alias given to the newly created key in which to reference when using the key keypass password protecting the use of the key Encrypt As it pertains to data in Java and at the most basic level, encryption is an algorithmic process used to programmatically obfuscate data through a reversible process where both parties have information pertaining to the data and how the algorithm is used. In Java encryption, this involves the use of a Cipher. A Cipher object in the JCE is a generic entry point into the encryption provider typically selected by the algorithm. This example uses the default Java provider but would also work with Bouncy Castle. Generating a Cipher object Obtaining an instance of Cipher is rather easy and the same process is required for both encryption and decryption. (NOTE: Encryption and Decryption require the same algorithm but do not require the same object instance) Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding"); Once we have an instance of the Cipher, we can encrypt and decrypt data according to the algorithm. Often the algorithm will require additional pieces of information in order to encrypt/decrypt data. In this example, we will need to pass the algorithm the bytes containing the key and an initial vector (explained below). Initialization In order to use the Cipher, we must first initialize the cipher. This step is necessary so we can provide additional information to the algorithm like the AES key and the Initial Vector (aka IV). cipher.init(Cipher.ENCRYPT_MODE, secretKeySpecification, initialVector); Parameters The SecretKeySpecification is an object containing a reference to the bytes forming the AES key. The AES key is nothing more than a specific sized byte array (256-bit for AES 256 or 32 bytes) that is generated by the keytool(see above). Alternative Parameteters There are multiple methods to create keys such as a hash including a salt, username and password (or similar). This method would utilize a SHA1 hash of the concatenated strings, convert to bytes and then truncate result to the desired size. This post will not show the generation of a key using this method or the use of a PBE key method using a password and salt. The password and/or salt usage for the keys is handled by the keytool using the inputs during the creation of new keys. Initialization Vector The AES algorithm also requires a second parameter called the Initialiation Vector. The IV is used in the process to randomize the encrypted message and prevent the key from easy guessing. The IV is considered a publicly shared piece of information, but again, it is not recommended to openly share the information (for example, it wouldn't be wise to post it on your company's website). When encrypting a message, it is not uncommon to prepend the message with the IV since the IV will be a set/known size based on the algorithm. NOTE: the AES algorithm will output the same result if using the same IV, key and message. It is recommended that the IV be randomly created each time an encryption takes place. With the newly initialized Cipher, encrypting a message is simple. Simply call: byte[] encryptedMessageInBytes = Cipher.doFinal((message.getBytes("UTF-8")); String base64EncodedEncryptedMsg = BaseEncoding.base64().encode(encryptedMessageInBytes); String base32EncodedEncryptedMsg = BaseEncoding.base32().encode(encryptedMessageInBytes); Encoding Results Byte arrays are difficult to visualize since they often do not form characters in any charset. The best recommendation to solve this is to represent the bytes in HEX (base-16), Double HEX (base-32) or Base64 format. If the message will be passed via a URL or POST parameter, be sure to use a web-safe Base64 encoding. Google Guava library provides a excellent BaseEncoding utility. NOTE: Remember to decode the encoded message before decrypting. Decrypt Decrypting a message is almost a reverse of the encryption process with a few exceptions. Decryption requires a known initialization vector as a parameter unlike the encryption process generating a random IV. Decryption When decrypting, obtain a cipher object with the same process as the encryption method. The Cipher object will need to utilize the exact same algorithm including the method and padding selections. Once the code has obtained a reference to a Cipher object, the next step is to initialize the cipher for decryption and pass in a reference to a key and the initialization vector. // key is the same byte[] key used in encryption SecretKeySpec secretKeySpecification = new SecretKeySpec(key, "AES"); cipher.init(Cipher.DECRYPT_MODE, secretKeySpecification, initialVector); NOTE: The key is stored in the keystore and obtained by the use of an alias. See below for details on obtaining keys from a keystore Once the cipher has been provided the key, IV and initialized for decryption, the cipher is ready to perform the decryption. byte[] encryptedTextBytes = BaseEncoding.base64().decode(message); byte[] decryptedTextBytes = cipher.doFinal(encryptedTextBytes); String origMessage = new String(decryptedTextBytes); Strategies to keep IV The IV used to encrypt the message is important to decrypting the message therefore the question is raised, how do they stay together. One solution is to Base Encode (see above) the IV and prepend it to the encrypted and encoded message: Base64UrlSafe(myIv) + delimiter + Base64UrlSafe(encryptedMessage). Other possible solutions might be contextual such as including an attribute in an XML file with the IV and one for the alias to the key used. Obtain Key from Keystore The beginning of this post has shown how easy it is to create new AES-256 keys that reference an alias inside of a keystore database. The post then continues on how to encrypt and decrypt a message given a key, but has yet shown how to obtain a reference to the key in a keystore. Solution // for clarity, ignoring exceptions and failures InputStream keystoreStream = new FileInputStream(keystoreLocation); KeyStore keystore = KeyStore.getInstance("JCEKS"); keystore.load(keystoreStream, keystorePass.toCharArray()); if (!keystore.containsAlias(alias)) { thrownew RuntimeException("Alias for key not found"); } Key key = keystore.getKey(alias, keyPass.toCharArray()); Parameters keystoreLocation String - Location to local keystore file location keypass String - Password used when creating or modifying the keystore file with keytool (see above) alias String - Alias used when creating new key with keytool (see above) Conclusion This post has shown how to encrypt and decrypt string based messages using the AES-256 encryption algorithm. The keys to encrypt and decrypt these messages are held inside of a JCEKS formatted KeyStore database created using the JDK provided "keytool" utility. The examples in this post should be considered a solid start to encrypting/decrypting symmetric keys such as AES. This should not be considered the only line of defense when encrypting messages, for example key rotation. Key rotation is a method to mitigate risks in the event of a data breach. If an intruder obtains data and manages to hack a single key, the data contained in multiple files should have used several keys to encrypt the data thus bringing down risk of a total exposure loss. All of the examples in this blog post have been condensed into a simple tool allowing for the viewing of keys inside of a keystore, an operation that is not supported out of the box by the JDK keytool. Each aspect of the steps and topics outlined in this post are available at: https://github.com/mike-ensor/aes-256-encryption-utility. NOTE: The examples, sample code and any reference is to be used at the sole implementers risk and there is no implied warranty or liability, you assume all risks.

In this post I would like to describe how to build a fat JAR using NetBeans IDE.

The Spring Expression Language (SpEL) is a powerful expression language that supports querying, manipulating as well as evaluating logical and mathematical expressions.

Introduction When using Spring to asynchronously execute pieces of your code you typically use the @Async annotation on your Spring @Components methods. When using the @Async annotation Spring will use AOP (Aspect Oriented Programming) to wrap your call in a Runnable. This Runnable will then be scheduled for execution on a TaskExecutor. Exception Handling Spring comes with a number of pre-packaged TaskExecutors which are documented in the Spring documentation here. In practice you will likely use the ThreadPoolTaskExecutor or the SimpleAsyncTaskExecutor. When doing so you might at times wonder why a task has not been executed. This happens when an exception occurs inside the method you are trying to execute asynchronous. The aforementioned task executors do not handle the exceptions so the execution fails silently. This problem can be solved by implementing your own AsyncTaskExecutor which handles the exceptions by logging them (or in any other way you wish). Bellow you'll find an example implementation of such a AsyncTaskExecutor. package nl.jborsje.blog.examples; import java.util.concurrent.Callable; import java.util.concurrent.Future; import org.springframework.core.task.AsyncTaskExecutor; import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; public class ExceptionHandlingAsyncTaskExecutor implements AsyncTaskExecutor { private AsyncTaskExecutor executor; public ExceptionHandlingAsyncTaskExecutor(AsyncTaskExecutor executor) { this.executor = executor; } public void execute(Runnable task) { executor.execute(createWrappedRunnable(task)); } public void execute(Runnable task, long startTimeout) { executor.execute(createWrappedRunnable(task), startTimeout); } public Future submit(Runnable task) { return executor.submit(createWrappedRunnable(task)); } public Future submit(final Callable task) { return executor.submit(createCallable(task)); } private Callable createCallable(final Callable task) { return new Callable() { public T call() throws Exception { try { return task.call(); } catch (Exception ex) { handle(ex); throw ex; } } }; } private Runnable createWrappedRunnable(final Runnable task) { return new Runnable() { public void run() { try { task.run(); } catch (Exception ex) { handle(ex); } } }; } private void handle(Exception ex) { System.err.println("Error during @Async execution: " + ex); } } This task executor decorates another task executor and wraps all Runnables and Callables in versions which handle the exception. In order to be able to use this task executor it has to be configured in the Spring application context as a managed bean and it has to be passed the real executor (the one that does the actual work) in its constructor. An example of such a configuration can be found bellow. JUnit testing In the application context used for JUnit testing Spring beans the asynchronous execution of methods can be cumbersome. You typically want to validate the inner workings of your methods not Springs asynchronicity. To achieve this you can use the SyncTaskExecutor provided by Spring. This task executor runs the tasks it is given immediately on the thread that adds the task instead of using a thread pool and running the tasks asynchronous. This task executor is configured in the test scope application context as follows.

This technical tip shows how .NET developers can export Microsoft Visio diagram to XML inside their own applications using Aspose.Diagram for .NET. Aspose.Diagram for .NET lets you export diagrams to a variety of formats: image formats, HTML, SVG, SWF and XML formats: VDX defines an XML diagram. VTX defines an XML template. VSX defines an XML stencil. The Diagram class' constructors read a diagram and the Save method is used to save, or export, a diagram in a different file format. The code snippets in this article show how to use the Save method to save a Visio file to VDX, VTX and VSX. Exporting VSD to VDX VDX is a schema-based XML file format that lets you save diagrams in a format that products other than Microsoft Visio can read. It's a useful format for transferring diagrams between software applications and retaining editable data. To export a VSD diagram to VDX first create an instance of the Diagram class and call the Diagram class' Save method to write the Visio drawing file to VDX. Exporting from VSD to VSX VSX is an XML format for defining stencils, the basic objects from which a diagram is built up. When a Visio file is converted to VSX, only the stencils are exported. To export a VSD diagram to VSX first you need to create an instance of the Diagram class and then call the Diagram class' Save method to write the Visio drawing file to VSX. //The Sample code shows how to export VSD to VDX //[C# Sample] //Call the diagram constructor to load diagram from a VSD file Diagram diagram = new Diagram("D:\\Drawing1.vsd"); this.Response.Clear(); this.Response.ClearHeaders(); this.Response.ContentType = "application/vnd.ms-visio"; this.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vdx"); this.Response.Flush(); System.IO.Stream vdxStream = this.Response.OutputStream; //Save input VSD as VDX diagram.Save(vdxStream, SaveFileFormat.VDX); this.Response.End(); //[VB.NET Code Sample] 'Call the diagram constructor to load diagram from a VSD file Dim diagram As New Diagram("D:\Drawing1.vsd") Me.Response.Clear() Me.Response.ClearHeaders() Me.Response.ContentType = "application/vnd.ms-visio" Me.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vdx") Me.Response.Flush() Dim vdxStream As System.IO.Stream = Me.Response.OutputStream 'Save inpupt VSD as VDX diagram.Save(vdxStream, SaveFileFormat.VDX) Me.Response.End() //The Sample code shows how to export VSD to VSX format. [C# Code Sample] // Call the diagram constructor to load diagram from a VSD file Diagram diagram = new Diagram("D:\\Drawing1.vsd"); this.Response.Clear(); this.Response.ClearHeaders(); this.Response.ContentType = "application/vnd.ms-visio"; this.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vsx"); this.Response.Flush(); System.IO.Stream vsxStream = this.Response.OutputStream; //Save input VSD as VSX diagram.Save(vsxStream, SaveFileFormat.VSX); this.Response.End() //[VB.NET Code Sample] 'Call the diagram constructor to load diagram from a VSD file Dim diagram As New Diagram("D:\Drawing1.vsd") Me.Response.Clear() Me.Response.ClearHeaders() Me.Response.ContentType = "application/vnd.ms-visio" Me.Response.AppendHeader("Content-Disposition", "attachment; filename=Diagram.vsx") Me.Response.Flush() Dim vsxStream As System.IO.Stream = Me.Response.OutputStream 'Save input VSD as VSX diagram.Save(vsxStream, SaveFileFormat.VSX) Me.Response.End()

Learn about using Mockito to create autowired fields.

When I describe my current workflow I use the TLA RDD, which is short for REPL Driven Development. I've been using REPL Driven Development for all of my production work for awhile now, and I find it to be the most effective workflow I've ever used. RDD differs greatly from any workflow I've used in the past, and (despite my belief that it's superior) I've often had trouble concisely describing what makes the workflow so productive. This entry is an attempt to describe what I consider RDD to be, and to demonstrate why I find it the most effective way to work. RDD Cycle First, I'd like to address the TLA RDD. I use the term RDD because I'm relying on the REPL to drive my development. More specifically, when I'm developing, I create an s-expression that I believe will solve my problem at hand. Once I'm satisfied with my s-expression, I send that s-expression to the REPL for immediate evaluation. The result of sending an s-expression can either be a value that I manually inspect, or it can be a change to a running application. Either way, I'll look at the result, determine if the problem is solved, and repeat the process of crafting an s-expression, sending it to the REPL, and evaluating the result. If that isn't clear, hopefully the video below demonstrates what I'm talking about. If you're unfamiliar with RDD, the previous video might leave you wondering: What's so impressive about RDD? To answer that question, I think it's worth making explicit what the video is: an example of a running application that needs to change, a change taking place, and verification that the application runs as desired. The video demonstrates change and verification; what makes RDD so effective to me is what's missing: (a) restarting the application, (b) running something other than the application to verify behavior, and (c) moving out of the source to execute arbitrary code. Eliminating those 3 steps allows me to focus on what's important, writing and running code that will be executed in production. Feedback I've found that, while writing software, getting feedback is the single largest time thief. Specifically, there are two types of feedback that I want to get as quickly as possible: (1) Is my application doing what I believe it is? (2) What does this arbitrary code return when executed? I believe the above video demonstrates how RDD can significantly reduce the time needed to answer both of those questions. In my career I've spent significant time writing applications in C#, Ruby, & Java. While working in C# and Java, if I wanted to make and verify (in the application) any non-trivial change to an application, I would need to stop the application, rebuild/recompile, & restart the application. I found the slowness of this feedback loop to be unacceptable, and wholeheartedly embraced tools such as NUnit and JUnit. I've never been as enamored with TDD as some of my peers; regardless, I absolutely endorsed it. The Design aspect of TDD was never that enticing to me, but tests did allow me to get feedback at a significantly superior pace. Tests also provide another benefit while working with C# & Java: They're the poorest man's REPL. Need to execute some arbitrary code? Write a test, that you know you're going to immediately delete, and execute away. Of course, tests have other pros and cons. At this moment I'm limiting my discussion around tests to the context of rapid feedback, but I'll address TDD & RDD later in this entry. Ruby provided a more effective workflow (technically, Rails provided a more effective workflow). Rails applications I worked on were similar to my RDD experience: I was able to make changes to a running application, refresh a webpage and see the result of the new behavior. Ruby also provided a REPL, but I always ran the REPL external to my editor (I knew of no other option). This workflow was the closest, in terms of efficiency, that I've ever felt to what I have with RDD; however, there are some minor differences that do add up to an inferior experience: (a) having to switch out of a source file to execute arbitrary code is an unnecessary nuisance and (b) refreshing a webpage destroys any client side state that you've built up. I have no idea if Ruby now has editor & repl integration, if it does, then it's likely on par with the experience I have now. Semantics It's important to distinguish between two meanings of "REPL" - one is a window that you type forms into for immediate evaluation; the other is the process that sits behind it and which you can interact with from not only REPL windows but also from editor windows, debugger windows, the program's user interface, etc. It's important to distinguish between REPL-based development and REPL-driven development: REPL-based development doesn't impose an order on what you do. It can be used with TDD or without TDD. It can be used with top-down, bottom-up, outside-in and inside-out approaches, and mixtures of them. REPL-driven development seems to be about "noodling in the REPL window" and later moving things across to editor buffers (and so source files) as and when you are happy with things. I think it's fair to say that this is REPL-based development using a series of mini-spikes. I think people are using this with a bottom-up approach, but I suspect it can be used with other approaches too. -- Simon Katz I like Simon's description, but I don't believe that we need to break things down to two different TLAs. Quite simply, (sadly) I don't think enough people are developing in this way, and the additional specification causes a bit of confusion among people who aren't familiar with RDD. However, Simon's description is so spot on I felt the need to describe why I'm choosing to ignore his classifications. RDD & TDD RDD and TDD are not in direct conflict with each other. As Simon notes above, you can do TDD backed by a REPL. Many popular testing frameworks have editor specific libraries that provide immediate feedback through REPL interaction. When working on a feature, the short term goal is to have it working in the application as fast as possible. Arbitrary execution, live changes, and only writing what you need are 3 things that can help you complete that short term goal as fast as possible. The video above is the best example I have of how you go from a feature request to software that does what you want in the smallest amount of time. In the video, I only leave the buffer to verify that the application works as intended. If the short term goal was the only goal, RDD without writing tests would likely be the solution. However, we all know that that are many other goals in software. Good design is obviously important. If you think tests give you better design, then you should probably mix both TDD & RDD. Preventing regression is also important, and that can be accomplished by writing tests after you have a working feature that you're satisfied with. Regression tests are great for giving confidence that a feature works as intended and will continue to in the future. REPL Driven Development doesn't need to replace your current workflow, it can also be used to extend your existing TDD workflow.

Java SE 7 type inference I taught an introductory Java session on generics, and of course demonstrated the shorthand introduced in Java SE 7 for instantiating an instance of a generic type: // Java SE 6 List l = new ArrayList(); // Java SE 7 List l = new ArrayList<>(); This inference is very friendly, especially when we get into more complex collections: // This Map> m = new HashMap>(); // Becomes Map> m = new HashMap<>(); Not only the key and value type of the map, but the type of object stored in the collection used for the value type can be inferred. Of course, sometimes this inference breaks down. It so happens I ran across an interesting example of this. Imagine populating a set from a list, so as to speed up random access and remove duplicates. Something like this will work: List list = ...; // From somewhere Set nodup = new HashSet<>(list); However, this runs into trouble if the list could be null. The HashSetconstructor will not just return an empty set but will throwNullPointerException. So we need to guard against null here. Of course, like all good programmers, we seize the chance to use a ternary operator because ternary operators are cool. List list = ...; // From somewhere Set nodup = (null == list) ? new HashSet<>() : new HashSet<>(list); And here’s where inference breaks down. Because this is no longer a simple assignment, the statement new HashSet<>() can no longer use the left hand side in order to infer the type. As a result, we get that friendly error message, “Type mismatch: cannot convert from HashSet to Set”. What’s especially interesting is that inference breaks down even though the compiler knows that an object of typeSet is what is needed in order to gain agreement of types. The rules for inference are written to be conservative by doing nothing when an invalid inference might cause issues, while the compiler’s type checking is also conservative in what it considers to be matching types. Also interesting is that we only get that error message for the new HashSet<>(). The statement new HashSet<>(list) that uses the list to populate the set works just fine. This is because the inference is completed using the listparameter. Here’s the constructor: public class HashSet extends ... implements ... { ... public HashSet(Collection c) { ... } ... } The List that we pass in gets captured as Collection and this means that E is bound to String, so all is well. As a result, we wind up with the perfectly valid, if a little funny looking: List list = ...; // From somewhere Set nodup = (null == list) ? new HashSet() : new HashSet<>(list); Of course, I imagine most Java programmers do what I do, which is try to use the shortcut and then add the type parameter when the compiler complains. Following the rule about not meddling in the affairs of compilers (subtle; quick to anger), normally I would just fix it without trying very hard to understand why the compiler liked or didn’t like things done in a certain way. But this one was such a strange case I figured it was worth a longer look.

Working in and around Silicon Valley and technology, I hear people throwing around the terms “geek”, “dork”, ”nerd” and “dweeb” constantly. They’re thrown around interchangeably, in fact, which is where the problem lies. They’re not the same and knowing that matters a great deal. In fact, using the wrong term gives credit where credit it isn’t due or unfairly labels someone’s better qualities. Venn diagram Finding a Venn diagram to explain the difference was a big moment. I suddenly see how I should interview for different roles and what to look for in partners and employees. I know what to seek and what to avoid. It was an epiphany. Starting from that point, I could see career paths for each and every one (OK, except one). It breaks out like this: Geek, Nerd, Dweeb and Dork in order of value to the organization Geek – Both smart and driven but able to talk about fun things — they’re your leaders and sales people Nerd - Centered in smarts and drive, tempered by some awkwardness — they sustain your company Dweeb – Smart and awkward but probably uncommitted — they won’t stay up all night to solve a problem Dork - Least fun of the bunch — Avoid these people because they waste your time and sap your will to live These may not be everyone’s definitions, but maybe it’s time to standardize in our labels lest we use the terms insensitively. For an interesting take on these categories, I found this on Democratic Underground. Geek: Someone who spends a lot of time and energy in a certain special but conventional area, like computer programming or trouble-shooting, but not necessarily computers or technology. You can apparently have chess geeks, guitar geeks, or cooking geeks. A geek is an outwardly normal person who can relate to others in general but who has taken the time to learn specific technical skills and would rather talk about their special obsession than anything else. They are generally not athletic and enjoy sedentary pursuits like video games, comic books, being on the internet, etc. They usually dress to suit their special interest, which can be flamboyant, such as wearing a tee-shirt describing their special obsession or a hat bearing a logo of their special pursuit. Geeks can be self-confident and proud of their traits. Nerd: Someone with a great interest in academic subjects like math and science and who is socially awkward and has trouble relating to others outside of their fields of academia. Their IQ often exceeds their weight. Science fiction such as The Matrix and Star Wars or LOTR are often their cup of tea, as are hobbies like astronomy or chemistry sets. Nerds usually dress conservatively and are more interested in the mind than their outward appearance, although as both men and women they tend to be tidy, clean-cut, and hygienic. Nerds generally are self-confident in the academic setting and take pride in their intellect and band together with other nerds although their social skills outside of their academic obsession are diminished. Dork: Someone who has special interests like a geek but whose interests and obsessions are less common and odd, such as having an oddball collection of some sort like old Three Stooges bubblegum cards or an uncommon skill like yodeling. Walking talking Star Trek encyclopedic knowledge and convention dress up obsessions can be considered dorky. They can act silly at times and not care what anyone thinks. Dorks are typically more noted for their quirky personality and tend to be loners. Hygiene can sometimes be an issue. Dorks can nonetheless be self-confident and proud of the way they are because they simply don’t care what others think. Dweeb: A person who tends to be regarded as physically wimpish, intellectually challenged, and socially awkward, with little self-confidence. Dweebs tend to be obsessed with unusual pursuits like dorks (tap dancing or ant farms) but are lacking in skill, knowledge, or ability. Dweebs tend to be loners like dorks but understand their shortcomings and lack pride. Hygiene can also be an issue. I’m not a dork, nor dweeb and I don’t see myself as a nerd…I’m clearly a geek :-). Thank you to the Great White Snark, where I found the Venn diagram.

Note: the following article is purely theoretical. I don’t know if it fits a real-life use-case, but the point is just too good to miss Java’s List sorting has two flavors: one follows the natural ordering of collection objects, the other requires an external comparator. In the first case, Java assumes objects are naturally ordered. From a code point of view, this means types of objects in the list must implement the Comparable interface. For example, such is the case for String and Date objects. If this is not the case, or if objects cannot be compared to one another (because perhapsthey belong to incompatible type as both String and Date). The second case happens when the natural order is not relevant and a comparator has to be implemented. For example, strings are sorted according to the character value, meaning case is relevant. When the use-case requires a case-insensitive sort, the following code will do (using Java 8 enhanced syntax): Collections.sort(strings, (s1, s2) -> s1.compareToIgnoreCase(s2)); The Comparable approach is intrinsic, the Comparator extrinsic; the former case rigid, the latter adaptable to the required context. What applies to lists, however, cannot be applied to Java sets. Objects added to sets have to define equals() and hashCode() and both properties (one could say that it’s only one since they are so coupled together) are intrinsic. There is no way to define an equality that can change depending on the context in the JDK. Enters Trove: The Trove library provide primitive collections with similar APIs to the above. This gap in the JDK is often addressed by using the “wrapper” classes (java.lang.Integer, java.lang.Float, etc.) with Object-based collections. For most applications, however, collections which store primitives directly will require less space and yield significant performance gains. Let’s be frank, Trove is under-documented. However, it offers what is missing regarding extrinsic equality: it provides a dedicated set implementation, that accepts its own extrinsic equality abstraction. A sample code would look like that: HashingStrategy strategy = new MyCustomStrategy(); Set dates = new TCustomHashSet(strategy); A big bonus for using Trove is performance, though: It probably is the first argument to use Trove I never tested that in any context To go further, just have a look at Trove for yourself.

I mentioned several times that the entire point of the exercise was to just see how this works, not to actually do anything production worthy. But it is interesting to see how we could do better here. In no particular order, I think that there are at least several things that we could do to significantly improve the time it takes to sort. Right now we defined 2 indexes on top of a 1GB file, and it took under 1 minute to complete. That gives us a runtime of about 10 days over a 15TB file. Well, one of the reason for this performance is that we execute this in a serial fashion, that is, one after another. But we have to completely isolated indexes, there is no reason why we can’t parallelize the work between them. For that matter, we are buffering in memory up to a certain point, then we sort, then we buffer some more, etc. That is pretty inefficient. We can push the actual sorting to a different thread, and continue parsing and adding to a buffer while we are adding to the buffer. We wrote to intermediary files, but we wrote to those using plain file I/O. But it is usually a lot more costly to write to disk than to compress and then write to disk. We are writing sorted data, so it is probably going to compress pretty well. Those are the things that pop to mind. Can you think of additional options?