If you read the MSDN page on the new WPF 4.5 features, you’ll find it under the label “Automatically updating the source of a data binding“. However, it is more precise and understandable to me to say that a ‘Delay’ property has been added to the Binding markup extension! This post is a part of the series on WPF 4.5 new features. By reading it you’ll discover the goal of this property, how to use it and some scenario where it is really useful. What is it ? A binding is done between two objects : the source (where the data comes from) and the target (where the data go to). For example if you bind the Text property of a TextBlock to a property “Name” of your ViewModel, the TextBlock is the target and the source is your ViewModel. A Binding performs synchronisation between two properties. If and only if the binding is done in TwoWay mode then the target is able to update the source value (the TextBlock is able to update the ViewModel). This synchronisation is done immediately : each change of the target value, even a tiny tiny one update the source. Since WPF 4.5, the Binding has a new property named ‘Delay’ which defines a timespan after which the source is updated. Each update of the target value will be done only after this delay and only if none has been done since. MSDN defines it as “The amount of time to wait before updating the binding source“. We will see later when it can be useful. How to use it ? As this is an added property on the Binding, you only have to define it on each Binding you create. The delay is defined in milliseconds and the default value of ’0′ reproduce the classic behavior of a Binding. Here is an example of a binding which sets a delay of 500 milliseconds: The delay is only applied on only one direction : from the target to the source. When to use it ? My first though was that it is not a really useful feature but after a certain amount of time I found some scenario in which this is very useful. Yes, my brain has a Delay too! As pointed out by MSDN, there is some controls like the Slider for which there is no need to update a source value for each pixel that the Slider moves; Refering to MSDN too, you can use this on TextBox to get the typed text by “block” instead of one letter at a time; It can be useful in master/details scenarii when the change of a selection trigger some time-consuming processing on the UI thread; In the same scenario, if you launch an async works on each change, it can strt a lot of them a saturation of the thread pool (or cause too much network use, etc.); Sometime in master/details scenarii, the details view is quite complex and it takes time to render it. Then the delay can trigger this rendering only when it’s useful. A full example The master/detail scenarii are the one where it can be the most useful for me so I have create a demo project to reproduce it. Let’s say that we a have a list of person and the selected one’s details are displayed. We will then have an interface which looks like this : On each change of the Selected Person, a time-consuming process is triggered on the UI thread in the setter. I simulated this by a call of Thread.Sleep : public Person SelectedPerson { get { return _selectedPerson; } set { _selectedPerson = value; //Simulate time consuming operation Thread.Sleep(500); RaisePropertyChanged("SelectedPerson"); } } When the delay is not set, navigation in the list is just slow and freeze the UI on each changes. Just letting the ‘arrow down’ key makes the UI freeze too much for me. With the delay property set to a value of 500 milliseconds, I can easily navigate in the list with the keyboard. Finally, I added a button to trigger a change from the source object just to prove that the Delay works in only one direction. The complete code source of the Demo can be found on my dropbox folder after registration. Again, it’s a discrete addition to the WPF framework but it’s a useful one ! If you want more info on it, you can read the MSDN page of this new property.

With the advent of Continuous Integration and Continuous Delivery, our builds are split into different steps creating the deployment pipeline. Some of these steps can be compiled and run fast tests, run slow tests, run automated acceptance tests, or releasing the application, to cite a few. Most of us are using Jenkins/Hudson to implement Continuous Integration/Delivery, and we manage job orchestration combining some Jenkins plugins like build pipeline, parameterized-build, join or downstream-ext. We have to configure all of them which implies polluting the job configuration through multiple jobs, which , makes the system configuration very complex to maintain. Build Flow enables us to define an upper level flow item to manage job orchestration and link up rules, using a dedicated DSL. Let's see a very simple example: First step is installing the plugin. Go to Jenkins -> Manage Jenkins -> Plugin Manager -> Available and find for CloudBees Build Flowplugin. Then you can go to Jenkins -> New Job and you will see a new kind of job called Build Flow. In this example we are going to name it build-all-yy. And now you only have to program using flow DSL how this job should orchestrate the other jobs. In "Define build flow using flow DSL" input text you can specify the sequence of commands to execute. In current example I have already created two jobs, one executing clean compile goal (yy-compile job name) and the other one executing javadoc goal (yy-javadoc job name). I know that this deployment pipeline is not real in a true environment but for now it is enough. Then we want javadoc job running after project is compiled. To configure this we don't have to create any upstream or downstream actions, simply add next lines at DSL text area: build("yy-compile"); build("yy-javadoc"); Save and execute build-all-yy job and both projects will be built in a sequential way. Now suppose that we add a third job called yy-sonar which runs sonar goal that generates code quality sonar report. In this case it seems obvious that after project is compiled, generation of javadocs and code quality jobs can be run in parallel. So script is changed to: build("yy-compile") parallel ( {build("yy-javadoc")}, {build("yy-sonar")} ) This plugin also supports more operations like retry (similar behaviour of retry-failed-job plugin) or guard-rescue, that it works mostly like a try+finally block. Also you can create parameterized builds, accessing to build execution or printing to Jenkins console. Next example will print build number of yy-compile job execution: b = build("yy-compile") out.println b.build.number And finally you can also have a quick graphical overview of the execution in Status section. It is true that could be improved more, but for now it is acceptable, and can be used without any problem. Build Flow plugin is in its early stages, in fact it is only at version 0.4. But will be a plugin to be considered in future, and I think it is good to know that it exists. Moreover is being developed by CloudBees folks so it is a guarantee of being fully supported by Jenkins. We Keep Learning. Alex. Warning: In order to run parallel tasks with the plugin Anonymous users must have Read Job access (Jenkins -> Manage Jenkins -> Configure System). There is an issue already inserted into Jira to fix this problem.

Mustache is templating system with implementation in many languages including Java and JavaScript . The templates are also supported by various web frameworks and client side JS libraries. Mustache has simple idea of "logic-less" system because it lacks any explicit control statements, like if, else or goto and also it does not have for statement however looping and conditional calculation can be achieved using custom tags that work with lists and lambdas. The name unfortunately has less to do with Tom Selleck but more with the heavy use of curly braces that look like mustache. The similarity is more than comparable. Mustache has implementation for most of the widely used languages like: Java, Javascript, Ruby,Net and many more. The client side template's in JavaScript Let say that you have some REST service and you have created a book view object that has an additional function that appends amazon associates id to the book url: var book = { id : 12, title : "A Game of Thrones", url : "http://www.amazon.com/gp/product/0553573403/", amazonId : "myAwesomeness", associateUrl : function() { return this.url + '?tag=' + this.amazonId; }, author : { name : 'George R. R. Martin', imdbUrl : 'http://www.imdb.com/name/nm0552333/', wikiUrl : 'https://en.wikipedia.org/wiki/George_R._R._Martin' }, haveInStock : true, similarBooks : [{ id : 13, title : "Decision Points" }, { id : 13, title : "Spoken from the Heart" }], comments : [] }; The standard way of rendering data without using templates would be create an output variable and just append everything inside and at the end just place the data where it should be. jQuery(document).ready(function() { var out = '' + book.title + ' is awesome book get it on Amazon'; jQuery('#content-jquery').html(out); }); This is fairly simple but if you for example want to change the span element with div it takes a little bit of time to figure where it should be closed and often you can miss if the element should be in single quotes or double quotes. The bigger issue here is that the content is peaces of strings that need to be easy to styled via CSS and JavaScript. As the code gets bigger this becomes unmanageable and changes to anything become slower especially if you add on top of this jQuery's manipulation functions like appendTo() or prependTo(). This direct use of out+= type of creating the content reminds me a lot of HttpServlet style of using print writer and doing out.print() and for the same reason why this was almost abandoned we should not do this in JavaScript. To simplify work we can add template engine like Mustache that is one of many client side tempting engines. So how does a template in mustache looks like, well for the example above with the book it would look like :

We recently adopted the use of Spring Data. Spring Data provides a nice pattern/API that you can layer on top of JPA to eliminate boiler-plate code. With that adoption, we started looking at the DAO layer we use against Cassandra for some of our operations. Some of the data we store in Cassandra is simple. It does *not* leverage the flexible nature of NoSQL. In other words, we know all the table names, the column names ahead of time, and we don't anticipate them changing all that often. We could have stored this data in an RDBMs, using hibernate to access it, but standing up another persistence mechanism seemed like overkill. For simplicity's sake, we preferred storing this data in Cassandra. That said, we want the flexibility to move this to an RDBMs if we need to. Enter JPA. JPA would provide us a nice layer of abstraction away from the underlying storage mechanism. Wouldn't it be great if we could annotate the objects with JPA annotations, and persist them to Cassandra? Enter Kundera. Kundera is a JPA implementation that supports Cassandra (among other storage mechanisms). OK -- so JPA is great, and would get us what we want, but we had just adopted the use of Spring Data. Could we use both? The answer is "sort of". I forked off SpringSource's spring-data-cassandra: https://github.com/boneill42/spring-data-cassandra And I started hacking on it. I managed to get an implementation of the PagingAndSortingRepository for which I wrote unit tests that worked, but I was duplicating a lot of what should have come for free in the SimpleJpaRepository. When I tried to substitute my CassandraJpaRepository for the SimpleJpaRepository, I ran into some trouble w/ Kundera. Specifically, the MetaModel implementation appeared to be incomplete. MetaModelImpl was returning null for all managedTypes(). SimpleJpa wasn't too happy with this. Instead of wrangling with Kundera, we punted. We can achieve enough of the value leveraging JPA directly. Perhaps more importantly, there is still an impedance mismatch between JPA and NoSQL. In our case, it would have been nice to get at Cassandra through Spring Data using JPA for a few cases in our app, but for the vast majority of the application, a straight up ORM layer whereby we know the tables, rows and column names ahead of time is insufficient. For those cases where we don't know the schema ahead of time, we're going to need to leverage the converters pattern in Spring Data. So, I started hacking on a proper Spring Data layer using Astyanax as the client. Follow along here: https://github.com/boneill42/spring-data-cassandra More to come on that....

Don’t be afraid – Some clarification to common misunderstandings Since version 3.1, Apache Lucene and Solr use MMapDirectory by default on 64bit Windows and Solaris systems; since version 3.3 also for 64bit Linux systems. This change lead to some confusion among Lucene and Solr users, because suddenly their systems started to behave differently than in previous versions. On the Lucene and Solr mailing lists a lot of posts arrived from users asking why their Java installation is suddenly consuming three times their physical memory or system administrators complaining about heavy resource usage. Also consultants were starting to tell people that they should not use MMapDirectory and change their solrconfig.xml to work instead with slow SimpleFSDirectory or NIOFSDirectory (which is much slower on Windows, caused by a JVM bug #6265734). From the point of view of the Lucene committers, who carefully decided that using MMapDirectory is the best for those platforms, this is rather annoying, because they know, that Lucene/Solr can work with much better performance than before. Common misinformation about the background of this change causes suboptimal installations of this great search engine everywhere. In this blog post, I will try to explain the basic operating system facts regarding virtual memory handling in the kernel and how this can be used to largely improve performance of Lucene (“VIRTUAL MEMORY for DUMMIES”). It will also clarify why the blog and mailing list posts done by various people are wrong and contradict the purpose of MMapDirectory. In the second part I will show you some configuration details and settings you should take care of to prevent errors like “mmap failed” and suboptimal performance because of stupid Java heap allocation. Virtual Memory[1] Let’s start with your operating system’s kernel: The naive approach to do I/O in software is the way, you have done this since the 1970s – the pattern is simple: whenever you have to work with data on disk, you execute a syscall to your operating system kernel, passing a pointer to some buffer (e.g. a byte[] array in Java) and transfer some bytes from/to disk. After that you parse the buffer contents and do your program logic. If you don’t want to do too many syscalls (because those may cost a lot processing power), you generally use large buffers in your software, so synchronizing the data in the buffer with your disk needs to be done less often. This is one reason, why some people suggest to load the whole Lucene index into Java heap memory (e.g., by using RAMDirectory). But all modern operating systems like Linux, Windows (NT+), MacOS X, or Solaris provide a much better approach to do this 1970s style of code by using their sophisticated file system caches and memory management features. A feature called “virtual memory” is a good alternative to handle very large and space intensive data structures like a Lucene index. Virtual memory is an integral part of a computer architecture; implementations require hardware support, typically in the form of a memory management unit (MMU) built into the CPU. The way how it works is very simple: Every process gets his own virtual address space where all libraries, heap and stack space is mapped into. This address space in most cases also start at offset zero, which simplifies loading the program code because no relocation of address pointers needs to be done. Every process sees a large unfragmented linear address space it can work on. It is called “virtual memory” because this address space has nothing to do with physical memory, it just looks like so to the process. Software can then access this large address space as if it were real memory without knowing that there are other processes also consuming memory and having their own virtual address space. The underlying operating system works together with the MMU (memory management unit) in the CPU to map those virtual addresses to real memory once they are accessed for the first time. This is done using so called page tables, which are backed by TLBs located in the MMU hardware (translation lookaside buffers, they cache frequently accessed pages). By this, the operating system is able to distribute all running processes’ memory requirements to the real available memory, completely transparent to the running programs. Schematic drawing of virtual memory (image from Wikipedia [1], http://en.wikipedia.org/wiki/File:Virtual_memory.svg, licensed by CC BY-SA 3.0) By using this virtualization, there is one more thing, the operating system can do: If there is not enough physical memory, it can decide to “swap out” pages no longer used by the processes, freeing physical memory for other processes or caching more important file system operations. Once a process tries to access a virtual address, which was paged out, it is reloaded to main memory and made available to the process. The process does not have to do anything, it is completely transparent. This is a good thing to applications because they don’t need to know anything about the amount of memory available; but also leads to problems for very memory intensive applications like Lucene. Lucene & Virtual Memory Let’s take the example of loading the whole index or large parts of it into “memory” (we already know, it is only virtual memory). If we allocate a RAMDirectory and load all index files into it, we are working against the operating system: The operating system tries to optimize disk accesses, so it caches already all disk I/O in physical memory. We copy all these cache contents into our own virtual address space, consuming horrible amounts of physical memory (and we must wait for the copy operation to take place!). As physical memory is limited, the operating system may, of course, decide to swap out our large RAMDirectory and where does it land? – On disk again (in the OS swap file)! In fact, we are fighting against our O/S kernel who pages out all stuff we loaded from disk [2]. So RAMDirectory is not a good idea to optimize index loading times! Additionally, RAMDirectory has also more problems related to garbage collection and concurrency. Because the data residing in swap space, Java’s garbage collector has a hard job to free the memory in its own heap management. This leads to high disk I/O, slow index access times, and minute-long latency in your searching code caused by the garbage collector driving crazy. On the other hand, if we don’t use RAMDirectory to buffer our index and use NIOFSDirectory or SimpleFSDirectory, we have to pay another price: Our code has to do a lot of syscalls to the O/S kernel to copy blocks of data between the disk or filesystem cache and our buffers residing in Java heap. This needs to be done on every search request, over and over again. Memory Mapping Files The solution to the above issues is MMapDirectory, which uses virtual memory and a kernel feature called “mmap” [3] to access the disk files. In our previous approaches, we were relying on using a syscall to copy the data between the file system cache and our local Java heap. How about directly accessing the file system cache? This is what mmap does! Basically mmap does the same like handling the Lucene index as a swap file. The mmap() syscall tells the O/S kernel to virtually map our whole index files into the previously described virtual address space, and make them look like RAM available to our Lucene process. We can then access our index file on disk just like it would be a large byte[] array (in Java this is encapsulated by a ByteBuffer interface to make it safe for use by Java code). If we access this virtual address space from the Lucene code we don’t need to do any syscalls, the processor’s MMU and TLB handles all the mapping for us. If the data is only on disk, the MMU will cause an interrupt and the O/S kernel will load the data into file system cache. If it is already in cache, MMU/TLB map it directly to the physical memory in file system cache. It is now just a native memory access, nothing more! We don’t have to take care of paging in/out of buffers, all this is managed by the O/S kernel. Furthermore, we have no concurrency issue, the only overhead over a standard byte[] array is some wrapping caused by Java’s ByteBuffer interface (it is still slower than a real byte[] array, but that is the only way to use mmap from Java and is much faster than all other directory implementations shipped with Lucene). We also waste no physical memory, as we operate directly on the O/S cache, avoiding all Java GC issues described before. What does this all mean to our Lucene/Solr application? We should not work against the operating system anymore, so allocate as less as possible heap space (-Xmx Java option). Remember, our index accesses rely on passed directly to O/S cache! This is also very friendly to the Java garbage collector. Free as much as possible physical memory to be available for the O/S kernel as file system cache. Remember, our Lucene code works directly on it, so reducing the number of paging/swapping between disk and memory. Allocating too much heap to our Lucene application hurts performance! Lucene does not require it with MMapDirectory. Why does this only work as expected on operating systems and Java virtual machines with 64bit? One limitation of 32bit platforms is the size of pointers, they can refer to any address within 0 and 232-1, which is 4 Gigabytes. Most operating systems limit that address space to 3 Gigabytes because the remaining address space is reserved for use by device hardware and similar things. This means the overall linear address space provided to any process is limited to 3 Gigabytes, so you cannot map any file larger than that into this “small” address space to be available as big byte[] array. And when you mapped that one large file, there is no virtual space (address like “house number”) available anymore. As physical memory sizes in current systems already have gone beyond that size, there is no address space available to make use for mapping files without wasting resources (in our case “address space”, not physical memory!). On 64bit platforms this is different: 264-1 is a very large number, a number in excess of 18 quintillion bytes, so there is no real limit in address space. Unfortunately, most hardware (the MMU, CPU’s bus system) and operating systems are limiting this address space to 47 bits for user mode applications (Windows: 43 bits) [4]. But there is still much of addressing space available to map terabytes of data. Common misunderstandings If you have read carefully what I have told you about virtual memory, you can easily verify that the following is true: MMapDirectory does not consume additional memory and the size of mapped index files is not limited by the physical memory available on your server. By mmap() files, we only reserve address space not memory! Remember, address space on 64bit platforms is for free! MMapDirectory will not load the whole index into physical memory. Why should it do this? We just ask the operating system to map the file into address space for easy access, by no means we are requesting more. Java and the O/S optionally provide the option to try loading the whole file into RAM (if enough is available), but Lucene does not use that option (we may add this possibility in a later version). MMapDirectory does not overload the server when “top” reports horrible amounts of memory. “top” (on Linux) has three columns related to memory: “VIRT”, “RES”, and “SHR”. The first one (VIRT, virtual) is reporting allocated virtual address space (and that one is for free on 64 bit platforms!). This number can be multiple times of your index size or physical memory when merges are running in IndexWriter. If you have only one IndexReader open it should be approximately equal to allocated heap space (-Xmx) plus index size. It does not show physical memory used by the process. The second column (RES, resident) memory shows how much (physical) memory the process allocated for operating and should be in the size of your Java heap space. The last column (SHR, shared) shows how much of the allocated virtual address space is shared with other processes. If you have several Java applications using MMapDirectory to access the same index, you will see this number going up. Generally, you will see the space needed by shared system libraries, JAR files, and the process executable itself (which are also mmapped). How to configure my operating system and Java VM to make optimal use of MMapDirectory? First of all, default settings in Linux distributions and Solaris/Windows are perfectly fine. But there are some paranoid system administrators around, that want to control everything (with lack of understanding). Those limit the maximum amount of virtual address space that can be allocated by applications. So please check that “ulimit -v” and “ulimit -m” both report “unlimited”, otherwise it may happen that MMapDirectory reports “mmap failed” while opening your index. If this error still happens on systems with lot’s of very large indexes, each of those with many segments, you may need to tune your kernel parameters in /etc/sysctl.conf: The default value of vm.max_map_count is 65530, you may need to raise it. I think, for Windows and Solaris systems there are similar settings available, but it is up to the reader to find out how to use them. For configuring your Java VM, you should rethink your memory requirements: Give only the really needed amount of heap space and leave as much as possible to the O/S. As a rule of thumb: Don’t use more than ¼ of your physical memory as heap space for Java running Lucene/Solr, keep the remaining memory free for the operating system cache. If you have more applications running on your server, adjust accordingly. As usual the more physical memory the better, but you don’t need as much physical memory as your index size. The kernel does a good job in paging in frequently used pages from your index. A good possibility to check that you have configured your system optimally is by looking at both "top" (and correctly interpreting it, see above) and the similar command "iotop" (can be installed, e.g., on Ubuntu Linux by "apt-get install iotop"). If your system does lots of swap in/swap out for the Lucene process, reduce heap size, you possibly used too much. If you see lot's of disk I/O, buy more RUM (Simon Willnauer) so mmapped files don't need to be paged in/out all the time, and finally: buy SSDs. Happy mmapping! Bibliography [1] http://en.wikipedia.org/wiki/Virtual_memory [2] https://www.varnish-cache.org/trac/wiki/ArchitectNotes [3] http://en.wikipedia.org/wiki/Memory-mapped_file [4] http://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details

Spring core comes out-of-the-box with two scopes: singletons and prototypes. Singletons implement the Singleton pattern, meaning there's only a single instance at runtime (in a JVM). Spring instantiates them during context creation, caches them in the context, and serves them from the cache when needed (or something like that). Prototypes are instantiated each time you access the context to get the bean. Problems arise when you need to inject a prototype-scoped bean in a singleton-scoped bean. Since singletons are created (and then injected) during context creation: it's the only time the Spring context is accessed and thus prototype-scoped beans are injected only once, thus defeating their purpose. In order to inejct prototypes into singletons, and side-by-syde with setter and constructor injection, Spring proposes another way for injection, called method injection. It works in the following way: since singletons are instantiated at context creation, it changes the way prototype-scoped are handled, from injection to created by an abstract method. The following snippet show the unsuccessful way to achieve injection: public class Singleton { private Prototype prototype; public Singleton(Prototype prototype) { this.prototype = prototype; } public void doSomething() { prototype.foo(); } public void doSomethingElse() { prototype.bar(); } } The next snippet displays the correct code: public abstract class Singleton { protected abstract Prototype createPrototype(); public void doSomething() { createPrototype().foo(); } public void doSomethingElse() { createPrototype().bar(); } } As you noticed, code doesn't specify the createPrototype() implementation. This responsibility is delegated to Spring, hence the following needed configuration: Note that an alternative to method injection would be to explicitly access the Spring context to get the bean yourself. It's a bad thing to do since it completely defeats the whole Inversion of Control pattern, but it works (and is essentially the only option when a nasty bug happens on the server - see below).However, using method injection has several main limitations: Spring achieves this black magic by changing bytecode. Thus, you'll need to have the CGLIB libraryon the classpath. The feature is only available by XML configuration, no annotations (see this JIRAfor more information) Finally, some application servers have bugs related to CGLIB (such as this one) To go further: Spring's documentation on method injection

Here is a quick example of how to programmatically access Camel MBeans to monitor and manipulate routes... first, get a connection to a JMX server (assumes localhost, port 1099, no auth) note, always cache the connection for subsequent requests (can cause memory utilization issues otherwise) JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi"); JMXConnector jmxc = JMXConnectorFactory.connect(url); MBeanServerConnection server = jmxc.getMBeanServerConnection(); use the following to iterate over all routes and retrieve statistics (state, exchanges, etc)... ObjectName objName = new ObjectName("org.apache.camel:type=routes,*"); List cacheList = new LinkedList(server.queryNames(objName, null)); for (Iterator iter = cacheList.iterator(); iter.hasNext();) { objName = iter.next(); String keyProps = objName.getCanonicalKeyPropertyListString(); ObjectName objectInfoName = new ObjectName("org.apache.camel:" + keyProps); String routeId = (String) server.getAttribute(objectInfoName, "RouteId"); String description = (String) server.getAttribute(objectInfoName, "Description"); String state = (String) server.getAttribute(objectInfoName, "State"); ... } use the following to execute operations against a Camel route (stop,start, etc) ObjectName objName = new ObjectName("org.apache.camel:type=routes,*"); List cacheList = new LinkedList(server.queryNames(objName, null)); for (Iterator iter = cacheList.iterator(); iter.hasNext();) { objName = iter.next(); String keyProps = objName.getCanonicalKeyPropertyListString(); if(keyProps.contains(routeID)) { ObjectName objectRouteName = new ObjectName("org.apache.camel:" + keyProps); Object[] params = {}; String[] sig = {}; server.invoke(objectRouteName, operationName, params, sig); return; } } summary These APIs can easily be used to build a web or command line based tool to support remote Camel management features. All of these features are available via the JMX console and Camel does provide a web console to support some management/monitoring tasks. See these pages for more information... http://camel.apache.org/camel-jmx.html http://camel.apache.org/web-console.html

Once you’ve been working with Jenkins and uberSVN for a while, you may find yourself in a situation where you have several jobs that need to run in a specific order, for example: Job 1 and Job 3 can run simultaneously. BUT Job 2 should only start when Job 1 and Job 3 have finished running. AND Job 4 should only start when Job 2 has finished. How can you implement this complicated setup? This is where Jenkins’ ‘Advanced Project Options’ and build triggers come in handy. In this tutorial, we’ll walk through the different options for scheduling jobs using Jenkins and uberSVN, the free ALM platform for Apache Subversion. Note, this tutorial assumes you have already created a job and configured it to automatically poll your Subversion repository. 1) Open the Jenkins tab of your uberSVN installation and select a job. 2) Click the ‘Configure’ option from the left-hand menu. 3) In the ‘Advanced Project Options’ tab, select the ‘Advanced…’ button 4) This contains two options that are useful for ordering your jobs: Block build when upstream project is building – blocks builds when a dependency is in the queue, or building. Note, these dependencies include both direct and transitive dependencies. Block build when downstream project is building – blocks builds when a child of the project is in the queue, or building. This applies to both direct and transitive children. If this option doesn’t meet your needs, you can explicitly name a project (or projects) that must be built before your job is allowed to run. To set this: 1) Scroll down to the ‘Build triggers’ tab on the configure page. 2) Select the ‘Build after other projects are built’ checkbox. This will bring up a text box where you can list any number of projects. Utilized properly, the build triggers and advanced project options should allow you to organize your jobs into a schedule. Tip, if you need even more control over your build schedule, there are plenty of scheduling plugins available. To add plugins to Jenkins, simply: 1) Open the ‘Manage Jenkins’ screen. 2) Click the ‘Manage Plugins’ link. 3) Open the ‘Available’ tab and select the appropriate plugins from the list.

This blog post explores JavaScript’s eval function by implementing the foundation for an interactive command line. As a bonus, you’ll get to work with ECMAScript.next’s generators (which can already be tried out on current Firefox versions). Writing an evaluator Let’s say you want to implement an interactive command line for JavaScript (such as [1]). On one hand, you would need to get the graphical user interface right: The user inputs JavaScript code, the command line evaluates the code and displays the result. On the other hand, you would have to implement the evaluation. That’s what we will take on here. It is more complex that it initially seems and teaches us a lot about eval. For starters, let’s write a constructor Evaluator: function Evaluator() { } Evaluator.prototype.evaluate = function (str) { return JSON.stringify(eval(str)); }; To use the evaluator, we create an instance and send JavaScript code to it: > var e = new Evaluator(); > e.evaluate("Math.pow(2, 53)") '9007199254740992' > e.evaluate("3 * 7") '21' > e.evaluate("'foo'+'bar'") '"foobar"' JSON.stringify is used so that the evaluation results can be shown to the user and look like the input. Without stringify, things look as follows: > console.log(123) // OK 123 > console.log("abc") // not OK abc With stringify, everything looks OK: > console.log(JSON.stringify(123)) 123 > console.log(JSON.stringify("abc")) "abc" Note that undefined is not valid JSON, but stringify converts it to undefined (the value, not the string), which is fine for our purposes. What we have implemented so far works for basic things, but still has several problems. Let’s tackle them one at a time. Problem: declarations You can evaluate variable and function declarations, but they are forgotten immediately afterwards: > e.evaluate("var x = 12;") undefined > e.evaluate("x") ReferenceError: x is not defined How do we fix this? The following code is a solution: function Evaluator() { this.env = {}; } Evaluator.prototype.evaluate = function (str) { str = rewriteDeclarations(str); var __environment__ = this.env; // (1) with (__environment__) { // (2) return JSON.stringify(eval(str)); } }; function rewriteDeclarations(str) { // Prefix a newline so that search and replace is simpler str = "\n" + str; str = str.replace(/\nvar\s+(\w+)\s*=/g, "\n__environment__.$1 ="); // (3) str = str.replace(/\nfunction\s+(\w+)/g, "\n__environment__.$1 = function"); return str.slice(1); // remove prefixed newline } this.env holds all variable declarations and function declarations in its properties. We make it accessible to the input in two steps. Step 1 – declare: We assign this.env to __environment__ (1) and rewrite the input so that, among other things, each var declaration assigns to __environment__ (3). That demonstrates one important aspect of eval: it sees all variables in surrounding scopes. That is, if you invoke eval inside your function, you expose all of its internals. The only way to keep those internals secret is to put the eval call in a separate function and call that function. Step 2 – access: Use a with statement so that the properties of __environment__ appear as variables to the eval-ed code. This is not an ideal solution, more of a compromise: with should be avoided [2] and can’t be used in the advantageous strict mode [3]. But it is a quick solution for us now. A work-around is quite complex [4]. > var e = new Evaluator(); > e.evaluate("var x = 123;") '123' > e.evaluate("x") '123' Minor drawback: Normal var declarations have the result undefined; due to our rewriting we now get the value that is assigned to the variable. Problem: exceptions Right now, throwing an exception in evaluate’s input means that the method will throw: > e.evaluate("* 3") SyntaxError: Unexpected token * That is obviously unacceptable: In a graphical user interface, we want to report errors back to the user, not (invisibly) throw an exception. Here is one simple way of doing so: Evaluator.prototype.evaluate = function (str) { try { str = rewriteDeclarations(str); var __environment__ = this.env; with (__environment__) { return JSON.stringify(eval(str)); } } catch (e) { return e.toString(); } }; There is nothing surprising in this code, we simply use try-catch and report back what happened. More sophisticated solutions will want to do more, e.g. display the exception’s stack trace. The new evaluator in action: > var e = new Evaluator(); > e.evaluate("* 3") 'SyntaxError: Unexpected token *' Problem: console.log How do we handle calls to console.log in the input? Logged messages should be shown to the user, not be sent to the browser’s console. The solution is surprisingly easy: function Evaluator(cons) { this.env = {}; this.cons = cons; } Evaluator.prototype.evaluate = function (str) { try { str = rewriteDeclarations(str); var __environment__ = this.env; var console = this.cons; with (__environment__) { return JSON.stringify(eval(str)); } } catch (e) { return e.toString(); } }; The constructor now receives a custom implementation of console and assigns it to this.cons. By assigning that object to a local variable named console (1), we temporarily shadow the global console for eval, there is no need to replace it. Beware that that shadowing affects all of the function, you won’t be able to use the browser’s console anywhere in evaluate. The new evaluator in action: > var cons = { log: function (m) { console.log("### "+m) } }; > var e = new Evaluator(cons); > e.evaluate("console.log('hello')") ### hello undefined Problem: eval creates bindings inside the function One scary feature of eval is that it creates variable bindings inside the function that invokes it: > (function () { eval("var x=3"); return x }()) 3 Fortunately, the fix is easy: use strict mode. > (function () { "use strict"; eval("var x=3"); return x }()) ReferenceError: x is not defined You can’t use with in strict mode, so you’ll have to replace it with a work-around [4]. Keeping declarations in an environment An environment is where JavaScript keeps the parameters and variables of a function. It maps variable names to values and is thus similar to an object. We might be able to avoid rewriting the input and manage declarations via environments. The idea is as follows. eval puts declarations in some environment: Non-strict mode: the environment of the surrounding function. Strict mode: a newly created environment. What if we could reuse that environment for the next invocation of eval, instead of throwing it away? Then eval would properly remember prior declarations. Strict mode gives us no way to access the temporary environment it creates for each invocation. However, in non-strict mode, we might be able to keep the environment of the surrounding function around. The following subsections explore two ways of doing so. Declarations via nested scopes If you create a function g inside another function f, then g permanently retains a reference to f’s current environment envf. Whenever g is called, a new g-specific environment envg is created. But envg points to its parent environment envf. Variables that can’t be found in g’s scope (as managed via envg), are looked up in f’s scope (via envf). Thus, envf is not lost, as long as g exists. That gives us a strategy for keeping the environment of the function that calls eval around. In the following code that function is called evalHelper and creates a new function that has to be used for the next call of eval. Hence, declarations made in the former function are accessible in the later function. function Evaluator() { var that = this; that.evalHelper = function (str) { that.evalHelper = function (str) { return eval(str); }; return eval(str); }; } Evaluator.prototype.evaluate = function (str) { return this.evalHelper(str); }; The fatal problem of this implementation is that you cannot nest to arbitrary depth. But, for the above depth of 2, it works perfectly: > var e = new Evaluator(); > e.evaluate("var x = 7;"); undefined > e.evaluate("x * 3") 21 Declarations via a generator It would be great if we could “restart” the function that calls eval, re-enter it with its previous environment still in place. ECMAScript.next’s generators [5] let you do that. Current versions of Firefox already support generators. Here is a demonstration of how they work in these versions (in ECMAScript.next, you will have to write function*, but apart from that, the code is the same): function mygen() { console.log((yield 0) + " @ 0"); console.log((yield 1) + " @ 1"); console.log((yield 2) + " @ 2"); } The above is a generator function. Invoke it and it will create a generator object. On that object, you first need to invoke the next() method to start execution. A yield x inside the code pauses execution and returns x to the the previously called generator object method. After the first next(), you can either call next() or send(y). The latter means that the currently paused yield will continue and produce the value y. The former is equivalent to send(undefined). The following interaction shows mygen in use: > var g = mygen(); > g.next() // can’t use send() the first time 0 > g.send("a") // continue after yield 0, pause again a @ 0 1 > g.send("b") b @ 1 2 The following is an implementation of Evaluator that calls eval via the generator evalGenerator. Because of that, eval always sees the same environment and remembers declarations. function evalGenerator(console) { var str = yield; while(true) { try { var result = JSON.stringify(eval(str)); str = yield result; } catch (e) { str = yield e.toString(); } } } function Evaluator(cons) { this.evalGen = evalGenerator(cons); this.evalGen.next(); // start } Evaluator.prototype.evaluate = function (str) { return this.evalGen.send(str); }; The new evaluator works as expected. > var e = new Evaluator(); > e.evaluate("var x = 7;") undefined > e.evaluate("x * 2") "14" > e.evaluate("* syntax_error") "SyntaxError: missing ; before statement" The biggest problem with this solution is that it uses the deprecated features non-strict eval together with the new feature generators. There will probably be a way in ECMAScript.next to make this combination work, but it will be a hack and should thus be avoided. Conclusion We have used eval to implement a helper type for a command line. While doing so, we learned a few interesting things about eval: Letting it remember declarations between invocations is complicated; it can access all variables in the scopes surrounding its invocation; and in non-strict mode, it can even create new variables inside the invoking function. The best solution for remembering declarations would be for eval to have an optional parameter for an environment (to be reused), but that is not in the cards. Therefore, the only truly safe solution in pure JavaScript is to use a full-featured JavaScript parser such as esprima to rewrite critical parts of the input code. That is left as an exercise to the reader. References Combining code editing with a command line JavaScript’s with statement and why it’s deprecated JavaScript’s strict mode: a summary Handing variables to eval Asynchronous programming and continuation-passing style in JavaScript

it can be finicky business to keep a production server running. you have to ensure the operating system and any other dependent software is properly patched to keep it up to date. hosted applications need to be upgraded regularly. configuration changes are regularly needed to tweak the environment so that it runs efficiently and communicates properly with other systems. this requires some mix of command-line invocations, jumping between gui screens, and editing text files. the result is a unique snowflake - good for a ski resort, bad for a data center. the first problem with a snowflake server is that it's difficult to reproduce. should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions. if you need to run a cluster, you get difficulties keeping all of the instances of the cluster in sync. you can't easily mirror your production environment for testing. when you get production faults, you can't investigate them by reproducing the transaction execution in a development environment. [1] making disk images of the snowflake can help to some extent with this. but such images easily gather cruft as unnecessary elements of the configuration, not to mention mistakes, perpetuate. the true fragility of snowflakes, however, comes when you need to change them. snowflakes soon become hard to understand and modify. upgrades of one bit software cause unpredictable knock-on effects. you're not sure what parts of the configuration are important, or just the way it came out of the box many years ago. their fragility leads to long, stressful bouts of debugging. you need manual processes and documentation to support any audit requirements. this is one reason why you often see important software running on ancient operating systems. a good way to avoid snowflakes is to hold the entire operating configuration of the server in some form of automated recipe. two tools that have become very popular for this recently are puppet and chef . both allow you to define the operating environment in a form of domainspecificlanguage , and easily apply it to a given system. the point of using a recipe is not just that you can easily rebuild the server (which you could also do with imaging) but you can also easily understand its configuration and thus modify it more easily. furthermore, since this configuration is a text file, you can keep it in version control with all the advantages that brings. if you disable any direct shell access to the server and force all configuration changes to be applied by running the recipe from version control, you have an excellent audit mechanism that ensures every change to the environment is logged. this approach can be very welcome in regulated environments. application deployment should follow a similar approach: fully automated, all changes in version control. by avoiding snowflakes, it's much easier to have test environments be true clones of production, reducing production bugs caused by configuration differences. a good way of ensuring you are avoiding snowflakes is to use phoenixservers . using version-controlled recipes to define server configurations is an important part of continuous delivery . further reading the visible ops handbook is the pioneering book that talked about the dangers of snowflakes and how to avoid them. continuous delivery talks about how this approach is a necessary part of a sane build and delivery process. true artists, however, prefer snowflakes . 1: another metaphor i've heard for this is that you should treat your servers like cattle and not like pets. although i confess i find it odd when this metaphor is used by my vegetarian colleagues.

the great thing with blogging is: i receive great comments, questions and ideas. the great thing with eclipse/codewarrior is that the extensions are almost unlimited . for my earlier post on hiding the toolbar i received a tip for another way which even is better: a plugin to switch eclipse into full screen mode. here is how to install it and how it looks… the plugin is available from http://code.google.com/p/eclipse-fullscreen/ . download the zip file. the zip file has the cn.pande.eclipsex.fullscreen_.jar file. copy that file inside the codewarrrior or eclipse eclipse\plugins folder. after launching eclipse there is new entry in the window menu: full screen menu item this switches eclipse into full screen mode: eclipse in full screen mode this hides the toolbar, menu and status bar, and i have more space available for what matters how to get out of full screen mode: press esc key to exit full screen mode. ctrl+alt+z is the default shortcut to toggle between ‘normal’ and ‘full screen’ mode. happy full screening

In a previous post, I gave an introduction to gevent to show some of the benefits your application might get from using gevent greenlets instead of threads. Some people, however, took issue with my benchmark code, saying that the threaded example was contrived. In this post, I'll try to answer some of the objections. (It actually turns out that there was a bug in the version of ab I was using to test, as well, so I re-ran the tests from the previous post, too.) Threads versus Greenlets Initially, I had proposed a dummy webserver that handled incoming requests by creating a thread and delegating communication to that thread. The code in question is below: def threads(port): s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() t = threading.Thread(target=handle_request, args=(cli, time.sleep)) t.daemon = True t.start() When I could get the code above ot actually run the full benchmark (which it didn't often do) it ended up getting around 1300-1400 requests per second. The gevent version looked very similar: import gevent def greenlet(port): from gevent import socket s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() gevent.spawn(handle_request, cli, gevent.sleep) This code was able to handle closer to 1600 requests per second. Maybe I should have called it out better, but the fact that the gevent version performed better than the threaded version does point out an important aspect of gevent: Greenlets are significantly lighter-weight than true threads, particularly when creating them. However, the folks objected by pointing out that you just don't do that with threads. Nobody does. It's a dumb way to design a server. I agree with all these points, though that wasn't really the point I was going for. One thing I will point out is that: The reason you don't design threaded servers so they fork a thread each time you get a connection is that threads are expensive to fork, unlike greenlets. Fixing the benchmark So anyway, to "fix" the benchmark so it's a little more fair to threads, we'll use a thread pool to create all the threads up-front and then use a Queue.Queue to send work to them. Our server core now looks like this: def threads(port, N=10): s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) q = Queue() for x in xrange(N): t = threading.Thread(target=thread_worker, args=(q,)) t.daemon = True t.start() print 'Ready and waiting with %d threads on port %d' % ( N, port) while True: cli, addr = s.accept() q.put(cli) def thread_worker(q): while True: sock = q.get() handle_request(sock, time.sleep) If I now run this with a thread pool of 200 threads, I can indeed finish the benchmark (ApacheBench as ab -r -n 2000 -c 200... with around 1300 requests per second (a little less, probably due to the synchronization overhead of the Queue). So updating the benchmark to use a thread pool did not improve the performance. The equivalent gevent code uses gevent.pool.Pool: def greenlet(port, N=10): from gevent.pool import Pool from gevent import socket, sleep pool = Pool(N) s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() pool.spawn(handle_request, cli, sleep) Running ab with the same parameters I now get... around 1200-1400 requests per second. So why use gevent, again? So yes, if I had designed the benchmark to omit the thread/greenlet creation entirely, threads and greenlets do indeed perform about the same. The big win for greenlets is when your thread pool isn't big enough to handle the concurrent connections. It turns out that there's a clever denial-of-service attack on web servers known as slowloris that consumes threads from your thread pool quickly. Once your server's threads are all busy handling the slowloris requests, no further work can be done, and you end up with a very lightly loaded but still unresponsive server. To illustrate this, we can try running our benchmark with the thread pool, but only running 20 threads in the pool, but modifying our request handler to take five seconds to handle a request. We'll go ahead and modify the benchmark line to allow more time for responses as well: $ ab -n 2000 -c 200 -r -t 60 http://127.0.0.1:... Now our threaded example ends up timing out connections as it tries to service 200 concurrent connections, each taking five seconds, with only 20 worker threads. If we go back to our naive (un-pooled) gevent example, however, we're able to achieve 47 requests per second, which is close to the theoretical maximum of 50 requests per second, with a very light server load. The point? A slowloris attack will be able to eat up all the threads in your (finite-sized) thread pool, regardless of how big that pool is. Spawning a greenlet each time you receive a connection means you don't waste (almost) any resources waiting on IO. Conclusion There's a good bit more to gevent that I'd like to cover in future posts, but for now the points I'd like to leave you with are the following: You shouldn't be spawning something expensive like a thread for each incoming connection. It eats up various types of server resources. You shouldn't rely on thread pools to protect you from resource exhaustion, because they can fall victim to the slowloris attack. Gevent greenlets are lightweight enough that you can spawn one for each connection, and you don't have to rely on a pool (which can become exhausted in a slowloris type attack). So what do you think? Have I convinced you? I'd love to hear your reaction in the comments below!

Riak is one databases that uses vector clocks for conflict resolution. I came across these two blog posts on Basho.com, company which develops Riak, and these posts are great at explaining the basics of Vector Clocks - definitely a must read if you're into distributed systems: Why vector clocks are easy? Why vector clocks are hard? Voldermort DB (by LinkedIn) is another DB that uses Vector Clocks, as explained below. Not surprisingly, it also takes the idea from Amazon's Dynamo (like Riak): The redundancy of storage makes the system more resilient to server failure. Since each value is stored N times, you can tolerate as many as N – 1 machine failures without data loss. This causes other problems, though. Since each value is stored in multiple places it is possible that one of these servers will not get updated (say because it is crashed when the update occurs). To help solve this problem Voldemort uses a data versioning mechanism called Vector Clocks that are common in distributed programming. This is an idea we took from Amazon’s Dynamo system. This data versioning allows the servers to detect stale data when it is read and repair it. Voldermort's code in Java can be on code.google.com. Finally, before I end this post, you may be asking "why complicate so much?" (if I could get a penny every time I heard that when discussing distributed systems... :-). But in this case, it's a good and typical question: can't we just use timestamp and last one wins? The problem, though, is that it requires times to be perfectly synchronized - which is very difficult and oftentimes impossible. By using vector clocks, you don't have this requirement on the system.

we wanted to set up a nightly integration build with our projects so that we could run unit and integration tests on the latest version of our applications and their underlying libraries. we have a number of libraries that are shared across multiple projects and we wanted this build to run every night and use the latest versions of those libraries even if our applications had a specific release version defined in their maven pom file. in this way we would be alerted early if someone added a change to one of the dependency libraries that could potentially break an application when the developer upgraded the dependent library in a future version of the application. the chart below illustrates our dependencies between our libraries and our applications. updating versions nightly both the crossdock-shared and messaging-shared libraries depend on the siesta framework library. the crossdock web service and crossdockmessaging applications both depend on the crossdock-shared and messaging-shared libraries. because of the dependency structure, we wanted the siestaframework library built first. the crossdock-shared and messaging-shared libraries could be built in parallel, but we didn’t want the builds for the crossdock web service and crossdockmessaging applications to begin until all the libraries had finished building. we also wanted the nightly build to tag a subversion with the build date as well as upload the artifact to our nexus “nightly build” repository. the resulting artifact would look something like siestaframework-20120720.jar also as i had mentioned, even though the crossdockmessaging app may specify in its pom file it depends on version 5.0.4 of the siestaframework library. for the purposes of the nightly build, we wanted it to use the freshly built siestaframework-nightly-20120720.jar version of the library. the first problem to tackle was getting the current date into the project’s version number. for this i started with the jenkins zentimestamp plugin . with this plugin the format of jenkin’s build_id timestamp can be changed. i used this to specify using the format of yyyymmdd for the timestamp. the next step was to get the timestamp into the version number of the project. i was able to accomplish this by using the maven versions plugin. one thing the versions plugin can do is allow you to dynamically override the version number in the pom file at build time. the code snippet from the siestaframework’s pom file is below. org.codehaus.mojo versions-maven-plugin 1.3.1 at this point the jenkins job can be configured to invoke the “versions;set” goal, passing in the new version string to use. the ${build_id} jenkins variable will have the newly formatted date string. this will produce an artifact with the name siestaframework-nightly-20120720.jar uploading artifacts to a nightly repository since this job needed to upload the artifact to a different repository from our release repository that's defined in our project pom files, the “altdeploymentrepository” property was used to pass in the location of the nightly repository. the deployment portion of the siestaframework job specifies the location of the nightly repository where ${lynden_nightly_repo} is a jenkins variable containing the nightly repo url. tagging subversion finally, the jenkins subversion tagging plugin was used to tag svn if the project was successfully built. the plugin provides a post-build action for the job with the configuration section shown below. dynamically updating dependencies so now that the main project is set up, the dependent projects are set up in a similar way, but need to be configured to use the siestaframework-nightly-20120720 of the dependency rather than whatever version they currently have specified in their pom file. this can be accomplished by changing the pom to use a property for the version number of the dependency. for example, if the snippet below was the original pom file— com.lynden siestaframework 5.0.1 —changing it to the following would allow the siestaframework version to be set dynamically: 5.0.1 com.lynden siestaframework ${siesta.version} this version can then be overriden by the jenkins job. the example below shows the jenkins configuration for the crossdock-shared build. enforcing build order the final step in this process is setting up a structure to enforce the build order of the projects. the dependencies are set up in such a way that siestaframework needs to be built first, and the crossdock-shared and messaging-shared libraries can be run concurrently once siestaframework finishes. the crossdock web service and crossdockmessaging application jobs can be run concurrently, too, but not until after both shared libraries have finished. setting up the crossdock-shared and messaging-shared jobs to be built after the siestaframework finishes is pretty straightforward. in the jenkins job configuration for both the shared libraries, the following build trigger is added: to satisfy the requirement that the apps build only after all libraries have built, i enlisted the help of the join plugin . the join plugin can be used to execute a job once all “downstream” jobs have completed. what does this mean exactly? looking at the diagram below, the crossdock-shared and the messaging-shared jobs are “downstream” from the siestaframework job. once both of these jobs complete, a join trigger can be used to start other jobs. in this case, rather than having the join trigger kick off other app jobs directly, i created a dummy join job. in this way, as we add more application builds, we don’t need to keep modifying the siestaframework job with the new application job we just added. to illustrate the configuration, siestaframework has a new post-build action (below): join-build is a jenkins job i configured that does not do anything when executed. then our crossdock web service and crossdockmessaging applications define their builds to trigger as soon as join-build has completed. in this way we are able to run builds each night that will update to the latest version of our dependencies as well as tag svn and archive the binaries to nexus. i’d love to hear feedback from anyone who is handling nightly builds via jenkins, and how they have handled the configuration and build issues.

I’ve been playing with Hive recently and liking what I’ve found. In theory at least it provides a very nice, simple way of getting into analysing large data sets. To make it even easier to show other people what you’re up to Hive has a nascent web interface with a little documentation on the wiki On the one hand it’s rather simple at this point, but that should be easily enought to prettify given a bit of time. The bigger problem was getting it working in the first place. What follows worked for me using the latest cloudera packages on debian testing. I’m assuming you already have Hive and Hadoop installed, the basic packages worked fine for me here. Next up you’ll need the JDK (not just the JRE) as their is some compilation that will go on the first time you run the web interface. apt-get install ant sun-java6-jdk Next up I had to modify the installed /etc/hive/conf/hive-site.xml file as follows: I changed this: hive.metastore.uris file:///var/lib/hivevar/metastore/metadb/ Comma separated list of URIs of metastore servers. The first server that can be connected to will be used. To this. Note the hivevar path doesn’t exist so I’m not sure if this was a typo in the source. hive.metastore.uris file:///var/lib/hive/var/metastore/metadb/ Comma separated list of URIs of metastore servers. The first server that can be connected to will be used. I also change the following section regarding the metastore name: javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=/var/lib/hive/metastore/${user.name}_db;create=true JDBC connect string for a JDBC metastore To this, with a fixed name. When using the above confirguration the file was actually called ${user.name} rather than my username being subsituted in. Elsewhere this seems to work fine. javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true JDBC connect string for a JDBC metastore I’m not convinced the above two changes are needed but have left them here just in case. The main tricky part is making sure a load of environment variables are correctly set. The following worked for me: export ANT_LIB=/usr/share/ant/lib export HIVE_HOME=/usr/lib/hive export HADOOP_HOME=/usr/lib/hadoop export PATH=$PATH:$HADOOP_HOME/bin export JAVA_HOME=/usr/lib/jvm/java-6-sun All being well that should allow you to run the hive command with the web interface like so: hive --service hwi That should bring up a webserver on port 9999 where you should see something similar to the screenshot above.

Konrad recently shared on our company’s technical room an interesting article on how caching is done is a big polish social network, nk.pl. One of the central concepts in the algorithm is generational caching (see here or here). The basic idea is that for cache keys you use some entity-specific string + version number. The version number increases whenever data changes, thus invalidating any old cache entries, and preventing stale data reads. This makes the assumption that the cache has some garbage collection, e.g. it may simply be a LRU cache. Of course on each request we must know the version number – that’s why it must be stored in a global cache (but depending on our consistency requirements, it also may be distributed across the cluster asynchronously). However the data itself can be stored in local caches. So if our system is read-most, the only “expensive” operation that we will have to do per request is retrieve the version numbers for the entities we are interested in. And this is usually very simple information, which can be kept entirely in-memory. Depending on the type of data and the usage patterns, you can cache individual entities (e.g. for a Person entity, the cache key could be person-9128-123, 9128 being the id, 123 the version number), or the whole lot (e.g. for a Countries entity, the cache key could be countries-8, 8 being the version number). Moreover in the global cache you can keep the latest version number per-id or per-entity; meaning that when the version changes, you invalidate a specific entity or all of them. Having written most of Envers, it quite naturally occurred to me that you may use the entity revision numbers as the cache versions. Subsequent Envers revisions are monotonically increasing numbers, for each transaction you get the next one. So whenever a cached entity changes, you would have to populate the global cache with the latest revision number. Envers provides several ways to get the revision numbers. During the transaction, you can call AuditReader.getCurrentRevision() method, which will give you the revision metadata, including the revision number. If you want more fine-grained control, you may implement your own listener (EntityTrackingRevisionListener), see the docs), and get notified whenever an entity is changed, and update the global cache in there. You can also register an after-transaction-completed callback, and update the cache outside of the transaction boundaries. Or, if you know the entity ids, you may lookup the maximum revision number using either AuditReader.getRevisions or an AuditQueryCreator. As you can obtain the current revision number during a transaction, you may even update the version/revision in the global cache atomically, if you use a transactional cache such as Infinispan. All of that of course in addition to auditing, which is still the main purpose of Envers :)

Transaction management is a subject that is generally left to the tender care of a senior developer (or architect). Given the messages coming from some actors of the JavaEE community that with newer versions of JavaEE you don't need Spring anymore, I was interested in some fact-checking on how transaction management was handled in both technologies. Note: these messages were already sent one year and a half ago and prompted me to write this article. Transaction demarcation Note that although both technologies provide programmatic transaction demarcation (start transaction then commit/rollback), we'll focus on declarative demarcation since they are easier to use in real life. In EJB3, transactions are delimited by the @TransactionAttribute annotation. The annotation can be set on the class, in which case every method will have the transactional attribute, or per method. Annotation at the method level can override the annotation at the class level. Annotations are found automatically by the JavaEE-compliant application server. In Spring, the former annotation is replaced by the proprietary @Transactional annotation. The behaviour is exactly the same (annotation at method/class level and possibility of overriding). Annotations can only be found when the Spring bean definition file contains the http://www.springframework.org/schema/tx namespace as well as the following snippet: Alternatively, both technologies provide an orthogonal way to set transaction demarcation: in EJB3, one can use the EJB JAR deployment descriptor (ejb-jar.xml) while in Spring, any Spring configuration file will do. Transactions propagation In EJB3, there are exactly 6 possible propagation values: MANDATORY, REQUIRED (default), REQUIRES_NEW, SUPPORTS, NOT_SUPPORTED and NEVER. Spring adds support for NESTED (see below). Nested transactions Nested transactions are transactions that are started then commited/rollbacked during the execution of the root transaction. Nested transaction results are limited to the scope of this transaction only (it has no effect on the umbrella transaction). Nested transactions are not allowed in EJB3; they are in Spring. Read-only transaction Read-only transactions are best used with certain databases or ORM frameworks like Hibernate. In the latter case, Hibernate optimizes sessions so that they never flush (i.e. never push changes from the cache to the underlying database). I haven't found a way (yet) to qualify a transaction as read-only in EJB3 (help welcome). In Spring, @Transactional has a readOnly parameter to implement read-only transactions: @Transactional(readOnly = true) Local method calls In local method calls, a bean's method A() calls another method B() on the same instance. The expected behavior should be that transaction attributes of method B() is taken into account. It's not the case in neither technologies, but both offer some workaround. In EJB3, you have to inject an instance of the same class and call the method on this instance in order to use transaction attributes. For example: @Stateless public MyServiceBean implements MyServiceLocal { @EJB private MyServiceLocal service; @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public void A() { service.B(); } @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) public void B() { ... } } In Spring, by default, transaction management is handled through pure Java proxies. If you want to have transaction management in local method calls, you'll have to turn on AspectJ. This is easily done in the Spring beans definition file (but beware the side-effects, see the Spring documentation for more details): Exception handling and rollback Exception handling is the area with the greater differences between Spring and EJB3. In EJB3, only runtime exceptions thrown from a method rollback a transaction delimited around this method by default. In order to mimic this behavior for checked exceptions, you have to annotate the exception class with @ApplicationException(rollback = true). Likewise, if you wish to discard this behavior for runtime exceptions, annotate your exception class with @ApplicationException(rollback = false). This has the disadvantage of not being able to use the same exception class to rollback in a method and to still to commit despite the exception in another method. In order to achieve this, you have to manage your transaction programmatically: @Stateless public MyServiceBean implements MyServiceLocal { @Resource private SessionContext context; public void A() { try { ... } catch (MyException e) { context.setRollbackOnly(); } } } In Spring, runtime exceptions also cause transaction rollback. In order to change this behavior, use the rollbackFor or noRollbackFor attributes of @Transactional: public MyServiceImpl { @Resource private SessionContext context; @Transactional(rollbackFor = MyException.class) public void A() { ... } } Conclusion There's no denying that JavaEE has made giant steps in the right direction with its 6th version. And yet, small details keep pointing me toward Spring. If you know only about one - Spring or JavaEE 6, I encourage you to try "the other" and see for yourself which one you're more comfortable with. To go further: Transaction strategies: Understanding transaction pitfalls Spring's documentation in regard to transaction management Oracle's documentation for EJB 3.0 transaction management

A document-oriented database is a designed for storing, retrieving, and managing document-oriented, or semi structured data. Document-oriented databases are one of the main categories of NoSQL databases. The central concept of a document-oriented database is the notion of a Document. While each document-oriented database implementation differs on the details of this definition, in general, they all assume documents encapsulate and encode data (or information) in some standard format(s) (or encoding(s)). Encodings in use include XML, YAML, JSON and BSON, as well as binary forms like PDF and Microsoft Office documents (MS Word, Excel, and so on). MongoDB: MongoDB is a collection-oriented, schema-free document database. Data is grouped into sets that are called ‘collections’. Each collection has a unique name in the database, and can contain an unlimited number of documents. Collections are analogous to tables in a RDBMS, except that they don’t have any defined schema. It store data (which is in BASON – “Binary Serialized dOcument Notation” format) that is a structured collection of key-value pairs, where keys are strings, and values are any of a rich set of data types, including arrays and documents. Home: http://www.mongodb.org/ Quick Start: http://www.mongodb.org/display/DOCS/Quickstart Download: http://www.mongodb.org/downloads CouchDB: CouchDB is a document database server, accessible via a RESTful JSON API. It is Ad-hoc and schema-free with a flat address space. Its Query-able and index-able, featuring a table oriented reporting engine that uses JavaScript as a query language. A CouchDB document is an object that consists of named fields. Field values may be strings, numbers, dates, or even ordered lists and associative maps. Home: http://couchdb.apache.org/ Quick Start: http://couchdb.apache.org/docs/intro.html Download: http://couchdb.apache.org/downloads.html Terrastore: Terrastore is a modern document store which provides advanced scalability and elasticity features without sacrificing consistency. It is based on Terracotta, so it relies on an industry-proven, fast clustering technology. Home: http://code.google.com/p/terrastore/ Quick Start: http://code.google.com/p/terrastore/wiki/Documentation Download: http://code.google.com/p/terrastore/downloads/list RavenDB: Raven is a .NET Linq enabled Document Database, focused on providing high performance, schema-less, flexible and scalable NoSQL data store for the .NET and Windows platforms. Raven store any JSON document inside the database. It is schema-less database where you can define indexes using C#’s Linq syntax. Home: http://ravendb.net/ Quick Start: http://ravendb.net/tutorials Download: http://ravendb.net/download OrientDB: OrientDB is an open source NoSQL database management system written in Java. Even if it is a document-based database, the relationships are managed as in graph databases with direct connections between records. It supports schema-less, schema-full and schema-mixed modes. It has a strong security profiling system based on users and roles and supports SQL as a query languages. Home: http://www.orientechnologies.com/ Quick Start: http://code.google.com/p/orient/wiki/Tutorials Download: http://code.google.com/p/orient/wiki/Download ThruDB: Thrudb is a set of simple services built on top of the Apache Thrift framework that provides indexing and document storage services for building and scaling websites. Its purpose is to offer web developers flexible, fast and easy-to-use services that can enhance or replace traditional data storage and access layers. It supports multiple storage backends such as BerkeleyDB, Disk, MySQL and also having Memcache and Spread integration. Home: http://code.google.com/p/thrudb/ Quick Start: http://thrudb.googlecode.com/svn/trunk/doc/Thrudb.pdf Download: http://code.google.com/p/thrudb/source/checkout SisoDB: SisoDb is a document-oriented db-provider for Sql-Server written in C#. It lets you store object graphs of POCOs (plain old clr objects) without having to configure any mappings. Each entity is treated as an aggregate root and will get separate tables created on the fly. Home: http://www.sisodb.com Quick Start: http://www.sisodb.com/Wiki Download: https://github.com/danielwertheim/SisoDb-Provider/ RaptorDB: RaptorDB is a extremely small size and fast embedded, noSql, persisted dictionary database using b+tree or MurMur hash indexing. It was primarily designed to store JSON data (see my fastJSON implementation), but can store any type of data that you give it. Home: http://www.codeproject.com/KB/database/RaptorDB.aspx Quick Start: http://www.codeproject.com/KB/database/RaptorDB.aspx Download: http://www.codeproject.com/KB/database/RaptorDB.aspx CloudKit: CloudKit provides schema-free, auto-versioned, RESTful JSON storage with optional OpenID and OAuth support, including OAuth Discovery. Home: http://getcloudkit.com/ Quick Start: http://getcloudkit.com/api/ Download: https://github.com/jcrosby/cloudkit Perservere: Persevere is an open source set of tools for persistence and distributed computing using an intuitive standards-based JSON interfaces of HTTP REST, JSON-RPC, JSONPath, and REST Channels. The core of the Persevere project is the Persevere Server. The Persevere server includes a Persevere JavaScript client, but the standards-based interface is intended to be used with any framework or client. Home: http://code.google.com/p/persevere-framework/ Quick Start: http://code.google.com/p/persevere-framework/w/list Download: http://code.google.com/p/persevere-framework/downloads/list Jackrabbit: The Apache Jackrabbit™ content repository is a fully conforming implementation of the Content Repository for Java Technology API (JCR, specified in JSR 170 and 283). A content repository is a hierarchical content store with support for structured and unstructured content, full text search, versioning, transactions, observation, and more. Home: http://jackrabbit.apache.org Quick Start: http://jackrabbit.apache.org/getting-started-with-apache-jackrabbit.html Download: http://jackrabbit.apache.org/downloads.html Conclusion: Document databases store and retrieve documents and basic atomic stored unit is a document. As always your requirement leads into the decision. You need to think about your data-access patterns / use-cases to create a smart document-model. When your domain model can be split and partitioned across some documents, a document-database will be a suitable one for you. For example for a blog-software, a CMS or a wiki-software a document-db works extremely well. But at the same time a non-relational database is not better than a relational one in some cases where your database have a lot of relations and normalization. Just check the following link from stackoverflow also to cover the pros/cons of Relational Vs Document based databases. http://stackoverflow.com/questions/337344/pros-cons-of-document-based-databases-vs-relational-databases

srgolla - SQL Server Scheduling Flowchart This is a basic flowchart explaining SQL Server Scheduling at a very high level. This will appeal to a limited audience, but is still something I thought very informative and not something I see flowcharted every day (week/month/year).

Introduction When the web was founded, it was designed as a system to distribute and update data. If you look at the HTTP protocol you can clearly see these origins. It has commands to GET, UPDATE, POST or DELETE data, but it is always the client (and if we ignore REST, this means in most cases the browser) who takes the initiative. However, the web is changing. It has evolved from a pure data distribution system to an application distribution system. Today, the magic word of IT vendors for this is the ‘cloud’, but in fact this shift to an application distribution system has already started some years ago. In order to start massively replacing traditional desktop applications, the web needs a new trick: two way communication between browser and server. The back-end must be able to update parts of a web page without user initiative. A definitive technical solution is underway in the form of web sockets, but do we really need to wait until this technology is broadly supported by the browsers of most users? In my opinion, the answer should be a clear NO. Where SignalR comes in When there are limitations, people become creative and are able to circumvent the issues blocking them from developing great products. Many popular web applications/sits are already capable of updating their content dynamically, yet they do not rely on web sockets. How is this possible? Because they rely on patterns such as ‘long polling’. With long polling, the browser sends a request for information to the web server with a huge timeout. The web server does not immediately sends data back to the browser, but waits until it has data to send back. When the client receives back data from the server, it will immediately resend a new request to the server. This long polling pattern and similar patterns give the user the illusion that a persistent two way connection exists between the browser and the web server, but it causes some unnecessary hard work for applicative developers and this is where SignalR comes in for .NET developers. It makes an abstraction of the long polling pattern and gives applicative developers the same illusion as their end users: a persistent two way connection between browser and web server. SignalR takes care of all the details and allows developers to focus on their most important task: building a great application for users. Because SignalR abstracts the underlying communication protocol, it can both support WebSockets and patterns, such as long polling. This makes an upgrade to WebSockets fairly easy when your organization adopts Windows Server 2012 and your users have moved to modern browsers such as Internet Explorer 10 or recent versions of Firefox and Google Chrome. Conclusion As I already wanted to emphasize in the title, comparing WebSockets with SignalR is pointless. Yes, WebSockets is technically superior and will probably give you some extra performance on your server side. But SignalR makes it possible to start developing today the web applications of tomorrow. When WebSockets become broadly available, SignalR will make it possible for you to move away from long polling without a lot of impact on your code.