Generate a random alpha numeric string whose length is the number of characters specified. Characters will be chosen from the set of alpha-numeric characters. Count is the length of random string to create. private static final String ALPHA_NUMERIC_STRING = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; public static String randomAlphaNumeric(int count) { StringBuilder builder = new StringBuilder(); while (count-- != 0) { int character = (int)(Math.random()*ALPHA_NUMERIC_STRING.length()); builder.append(ALPHA_NUMERIC_STRING.charAt(character)); } return builder.toString(); }

From theses to reports and books, LaTeX is a pretty useful document markup language. Learn how to install the Tex Live distribution here.

This article is based on my answer on StackOverflow. I am trying to explain how String class stores the texts, how interning and constant pool works. The main point to understand here is the distinction between String Java object and its contents - char[] under private value field. String is basically a wrapper around char[] array, encapsulating it and making it impossible to modify so the String can remain immutable. Also the String class remembers which parts of this array is actually used (see below). This all means that you can have two different String objects (quite lightweight) pointing to the same char[]. I will show you few examples, together with hashCode() of each String and hashCode() of internal char[] value field (I will call it text to distinguish it from string). Finally I'll show javap -c -verbose output, together with constant pool for my test class. Please do not confuse class constant pool with string literal pool. They are not quite the same. See also Understanding javap's output for the Constant Pool Prerequisites For the purpose of testing I created such a utility method that breaks String encapsulation: private int showInternalCharArrayHashCode(String s) { final Field value = String.class.getDeclaredField("value"); value.setAccessible(true); return value.get(s).hashCode(); } It will print hashCode() of char[] value, effectively helping us understand whether this particular String points to the same char[] text or not. Two string literals in a class Let's start from the simplest example. Java code String one = "abc"; String two = "abc"; BTW if you simply write "ab" + "c", Java compiler will perform concatenation at compile time and the generated code will be exactly the same. This only works if all strings are known at compile time. Class constant pool Each class has its own constant pool - a list of constant values that can be reused if they occur several times in the source code. It includes common strings, numbers, method names, etc. Here are the contents of the constant pool in our example above: const #2 = String #38; // abc //... const #38 = Asciz abc; The important thing to note is the distinction between String constant object (#2) and Unicode encoded text "abc" (#38) that the string points to. Byte code Here is generated byte code. Note that both one and two references are assigned with the same #2 constant pointing to "abc" string: ldc #2; //String abc astore_1 //one ldc #2; //String abc astore_2 //two Output For each example I am printing the following values: System.out.println("one.value: " + showInternalCharArrayHashCode(one)); System.out.println("two.value: " + showInternalCharArrayHashCode(two)); System.out.println("one" + System.identityHashCode(one)); System.out.println("two" + System.identityHashCode(two)); No surprise that both pairs are equal: one.value: 23583040 two.value: 23583040 one: 8918249 two: 8918249 Which means that not only both objects point to the same char[] (the same text underneath) so equals() test will pass. But even more, one and two are the exact same references! So one == two is true as well. Obviously if one and two point to the same object then one.value and two.value must be equal. Literal and new String() Java code Now the example we all waited for - one string literal and one new String using the same literal. How will this work? String one = "abc"; String two = new String("abc"); The fact that "abc" constant is used two times in the source code should give you some hint... Class constant pool Same as above. Byte code ldc #2; //String abc astore_1 //one new #3; //class java/lang/String dup ldc #2; //String abc invokespecial #4; //Method java/lang/String."":(Ljava/lang/String;)V astore_2 //two Look carefully! The first object is created the same way as above, no surprise. It just takes a constant reference to already created String (#2) from the constant pool. However the second object is created via normal constructor call. But! The first String is passed as an argument. This can be decompiled to: String two = new String(one); Output The output is a bit surprising. The second pair, representing references to String object is understandable - we created two String objects - one was created for us in the constant pool and the second one was created manually for two. But why, on earth the first pair suggests that both String objects point to the same char[] value array?! one.value: 41771 two.value: 41771 one: 8388097 two: 16585653 It becomes clear when you look at how String(String) constructor works (greatly simplified here): public String(String original) { this.offset = original.offset; this.count = original.count; this.value = original.value; } See? When you are creating new String object based on existing one, it reuses char[] value. Strings are immutable, there is no need to copy data structure that is known to be never modified. Moreover, since new String(someString) creates an exact copy of existing string and strings are immutable, there is clearly no reason for the two to exist at the same time. I think this is the clue of some misunderstandings: even if you have two String objects, they might still point to the same contents. And as you can see the String object itself is quite small. Runtime modification and intern() Java code Let's say you initially used two different strings but after some modifications they are all the same: String one = "abc"; String two = "?abc".substring(1); //also two = "abc" The Java compiler (at least mine) is not clever enough to perform such operation at compile time, have a look: Class constant pool Suddenly we ended up with two constant strings pointing to two different constant texts: const #2 = String #44; // abc const #3 = String #45; // ?abc const #44 = Asciz abc; const #45 = Asciz ?abc; Byte Code ldc #2; //String abc astore_1 //one ldc #3; //String ?abc iconst_1 invokevirtual #4; //Method String.substring:(I)Ljava/lang/String; astore_2 //two The fist string is constructed as usual. The second is created by first loading the constant "?abc" string and then calling substring(1) on it. Output No surprise here - we have two different strings, pointing to two different char[] texts in memory: one.value: 27379847 two.value: 7615385 one: 8388097 two: 16585653 Well, the texts aren't really different, equals() method will still yield true. We have two unnecessary copies of the same text. Now we should run two exercises. First, try running: two = two.intern(); before printing hash codes. Not only both one and two point to the same text, but they are the same reference! one.value: 11108810 two.value: 11108810 one: 15184449 two: 15184449 This means both one.equals(two) and one == two tests will pass. Also we saved some memory because "abc" text appears only once in memory (the second copy will be garbage collected). The second exercise is slightly different, check out this: String one = "abc"; String two = "abc".substring(1); Obviously one and two are two different objects, pointing to two different texts. But how come the output suggests that they both point to the same char[] array?!? one.value: 23583040two.value: 23583040one: 11108810two: 8918249 I'll leave the answer to you. It'll teach you how substring() works, what are the advantages of such approach and when it can lead to big troubles. Lessons learnt String object itself is rather cheap. It's the text it points to that consumes most of the memory String is just a thin wrapper around char[] to preserve immutability new String("abc") isn't really that expensive as the internal text representation is reused. But still avoid such construct. When String is concatenated from constant values known at compile time, concatenation is done by the compiler, not by the JVM substring() is tricky, but most importantly, it is very cheap, both in terms of used memory and run time (constant in both cases)

In a previous post I wrote how to install Apache thrift on Windows. In this post I will guide you how to install Apache Thrift on Ubuntu 12.04. Install dependencies using the following command sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev Download tar.gz archive from the this link, extract it in your home directory using this command tar -xvzf thrift-0.8.0.tar.gz Change into the Thrift installation directory(the one extracted) and carry on the following command ./configure Run the following command within Thrift installation directory make After that carry on the following command sudo make install To verify Thrift has installed properly, use the following command thrift -version

It has been a while since I wanted to have CDI running with JavaFX2. Some people already blogged on how to proceed by getting Guice injection [1] to work with JavaFX & FXML. Well, now it's my turn to provide a way to empower JavaFX with CDI, using Weld as the implementation. My goal was just to have CDI working, no matter how I was using JavaFX, by directly coding in plain Java or using FXML. Ready? Let's go!!! Bootstrap JavaFX & Weld/CDI The launcher class will be the only place where we will have Weld-specific code—all the rest will be totally CDI compliant. The only trick here is to make the application parameters available as a CDI-compliant object so we can reuse them afterwards. Notice also that we use the CDI event mechanism to start up our real application code. public class WeldJavaFXLauncher extends Application { /** * Nothing special, we just use the JavaFX Application methods to boostrap * JavaFX */ public static void main(String[] args) { Application.launch(WeldJavaFXLauncher.class, args); } @SuppressWarnings("serial") @Override public void start(final Stage primaryStage) throws Exception { // Let's initialize CDI/Weld. WeldContainer weldContainer = new Weld().initialize(); // Make the application parameters injectable with a standard CDI // annotation weldContainer.instance().select(ApplicationParametersProvider.class).get().setParameters(getParameters()); // Now that JavaFX thread is ready // let's inform whoever cares using standard CDI notification mechanism: // CDI events weldContainer.event().select(Stage.class, new AnnotationLiteral() {}).fire(primaryStage); } } Start our real JavaFX application Here we start our real application code. We're just listening to the previously fired event (containing the Scene object to render into) so we can start showing our application. In the following example, we load an FXML GUI, but it might have been any node created in any way. public class LoginApplicationStarter { // Let's have a FXMLLoader injected automatically @Inject FXMLLoader fxmlLoader; // Our CDI entry point, we just listen to an event providing the startup scene public void launchJavaFXApplication(@Observes @StartupScene Stage s) { InputStream is = null; try { is = getClass().getResourceAsStream("login.fxml"); // we just load our FXML form (including controler and so on) Parent root = (Parent) fxmlLoader.load(is); s.setScene(new Scene(root, 300, 275)); s.show(); // let's show the scene } catch (IOException e) { throw new IllegalStateException("cannot load FXML login screen", e); } finally { // omitted is cleanup } } } But what about the FXML controller? First let's have a look at the controller we want to use inside our application. It is a pure POJO class annotated with both JavaFX & CDI annotations. // Simple application controller that uses injected fields // to delegate login process and to get default values from the command line using: --user=SomeUser public class LoginController implements Initializable { // Standard FXML injected fields @FXML TextField loginField; @FXML PasswordField passwordField; @FXML Text feedback; // CDI Injected service @Inject LoginService loginService; // Default application parameters retrieved using CDI @Inject Parameters applicationParameters; @FXML protected void handleSubmitButtonAction(ActionEvent event) { feedback.setText(loginService.login(loginField.getText(), passwordField.getText())); } @Override public void initialize(URL location, ResourceBundle resources) { loginField.setText(applicationParameters.getNamed().get("user")); } } In order to have injection working inside the FXML controller, we need to set up JavaFX so that controller objects are created by CDI. As we are in a CDI environment we can also have the FXMLLoader classes injected (that's exactly what we did in the previous LoginApplicationStarter class). How can we achieve this? We just have to provide a Producer class whose responsibility will be to create FXMLLoader instances that are able to load FXML GUIs and instantiate controllers using CDI. The only part that's a little tricky there is that the controller instantiation depends on the required class or interface (using fx:controller in your fxml file). In order to have such a runtime injection/resolution available we use a CDI Instance Object. public class FXMLLoaderProducer { @Inject Instance, Object>() { @Override public Object call(Class param) { return instance.select(param).get(); } }); return loader; } } I hope you found the article interesting and you do not hesitate to comment if you see some errors or possible enhancements. Finally, if you are interested you can find the full source code here. [1] http://andrewtill.blogspot.be/2012/07/creating-javafx-controllers-using-guice.htm

The ExecutorService feature came with Java 5. It extends the Executor interface and provides a thread pool feature to execute asynchronous short tasks. There are five ways to execute the tasks asyncronously by using the ExecutorService interface provided Java 6. ExecutorService execService = Executors.newCachedThreadPool(); This approach creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for 60 seconds are terminated and removed from the cache. ExecutorService execService = Executors.newFixedThreadPool(10); This approach creates a thread pool that reuses a fixed number of threads. Created nThreads will be active at the runtime. If additional tasks are submitted when all threads are active, they will wait in the queue until a thread is available. ExecutorService execService = Executors.newSingleThreadExecutor(); This approach creates an Executor that uses a single worker thread operating off an unbounded queue. Tasks are guaranteed to execute sequentially, and no more than one task will be active at any given time. Methods of the ExecutorService : execute(Runnable) : Executes the given command at some time in the future. submit(Runnable) : Submit method returns a Future Object which represents executed task. Future Object returns null if the task has finished correctly. shutdown() : Initiates an orderly shutdown in which previously submitted tasks are executed, but no new tasks will be accepted. Invocation has no additional effect if already shut down. shutdownNow() : Attempts to stop all actively executing tasks, halts the processing of waiting tasks, and returns a list of the tasks that were awaiting execution. There are no guarantees beyond best-effort attempts to stop processing actively executing tasks. For example, typical implementations will cancel via Thread.interrupt, so any task that fails to respond to interrupts may never terminate. A sample application is below : STEP 1 : CREATE MAVEN PROJECT A maven project is created as below. (It can be created by using Maven or IDE Plug-in). STEP 2 : CREATE A NEW TASK A new task is created by implementing the Runnable interface(creating Thread) as below. TestTask Class specifies business logic which will be executed. package com.otv.task; import org.apache.log4j.Logger; /** * @author onlinetechvision.com * @since 24 Sept 2011 * @version 1.0.0 * */ public class TestTask implements Runnable { private static Logger log = Logger.getLogger(TestTask.class); private String taskName; public TestTask(String taskName) { this.taskName = taskName; } public void run() { try { log.debug(this.taskName + " is sleeping..."); Thread.sleep(3000); log.debug(this.taskName + " is running..."); } catch (InterruptedException e) { e.printStackTrace(); } } STEP 3 : CREATE TestExecutorService by using newCachedThreadPool TestExecutorService is created by using the method newCachedThreadPool. In this case, created thread count is specified at the runtime. package com.otv; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import com.otv.task.TestTask; /** * @author onlinetechvision.com * @since 24 Sept 2011 * @version 1.0.0 * */ public class TestExecutorService { public static void main(String[] args) { ExecutorService execService = Executors.newCachedThreadPool(); execService.execute(new TestTask("FirstTestTask")); execService.execute(new TestTask("SecondTestTask")); execService.execute(new TestTask("ThirdTestTask")); execService.shutdown(); } } When TestExecutorService is run, the output will be seen as below : 24.09.2011 17:30:47 DEBUG (TestTask.java:21) - SecondTestTask is sleeping... 24.09.2011 17:30:47 DEBUG (TestTask.java:21) - ThirdTestTask is sleeping... 24.09.2011 17:30:47 DEBUG (TestTask.java:21) - FirstTestTask is sleeping... 24.09.2011 17:30:50 DEBUG (TestTask.java:23) - ThirdTestTask is running... 24.09.2011 17:30:50 DEBUG (TestTask.java:23) - FirstTestTask is running... 24.09.2011 17:30:50 DEBUG (TestTask.java:23) - SecondTestTask is running... STEP 4 : CREATE TestExecutorService by using newFixedThreadPool TestExecutorService is created by using the method newFixedThreadPool. In this case, required thread count has to be set as the following : package com.otv; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import com.otv.task.TestTask; /** * @author onlinetechvision.com * @since 24 Sept 2011 * @version 1.0.0 * */ public class TestExecutorService { public static void main(String[] args) { ExecutorService execService = Executors.newFixedThreadPool(2); execService.execute(new TestTask("FirstTestTask")); execService.execute(new TestTask("SecondTestTask")); execService.execute(new TestTask("ThirdTestTask")); execService.shutdown(); } } When TestExecutorService is run, ThirdTestTask is executed after FirstTestTask and SecondTestTask’ s executions are completed. The output will be seen as below: 24.09.2011 17:33:38 DEBUG (TestTask.java:21) - FirstTestTask is sleeping... 24.09.2011 17:33:38 DEBUG (TestTask.java:21) - SecondTestTask is sleeping... 24.09.2011 17:33:41 DEBUG (TestTask.java:23) - FirstTestTask is running... 24.09.2011 17:33:41 DEBUG (TestTask.java:23) - SecondTestTask is running... 24.09.2011 17:33:41 DEBUG (TestTask.java:21) - ThirdTestTask is sleeping... 24.09.2011 17:33:44 DEBUG (TestTask.java:23) - ThirdTestTask is running... STEP 5 : CREATE TestExecutorService by using newSingleThreadExecutor TestExecutorService is created by using the method newSingleThreadExecutor. In this case, only one thread is created and tasks are executed sequentially. package com.otv; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import com.otv.task.TestTask; /** * @author onlinetechvision.com * @since 24 Sept 2011 * @version 1.0.0 * */ public class TestExecutorService { public static void main(String[] args) { ExecutorService execService = Executors.newSingleThreadExecutor(); execService.execute(new TestTask("FirstTestTask")); execService.execute(new TestTask("SecondTestTask")); execService.execute(new TestTask("ThirdTestTask")); execService.shutdown(); } } When TestExecutorService is run, SecondTestTask and ThirdTestTask is executed after FirstTestTask’ s execution is completed. The output will be seen as below : 24.09.2011 17:38:21 DEBUG (TestTask.java:21) - FirstTestTask is sleeping... 24.09.2011 17:38:24 DEBUG (TestTask.java:23) - FirstTestTask is running... 24.09.2011 17:38:24 DEBUG (TestTask.java:21) - SecondTestTask is sleeping... 24.09.2011 17:38:27 DEBUG (TestTask.java:23) - SecondTestTask is running... 24.09.2011 17:38:27 DEBUG (TestTask.java:21) - ThirdTestTask is sleeping... 24.09.2011 17:38:30 DEBUG (TestTask.java:23) - ThirdTestTask is running... STEP 6 : REFERENCES http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html http://tutorials.jenkov.com/java-util-concurrent/executorservice.html

In my office laptop, I have installed two versions of JDK. For the office work, I need JDK6 because the internal framework needs it. I’m using JDK7 for my personal projects and exploring the latest and greatest in Java. I have two versions of Eclipse too (one for office work and one is the latest Juno). But, the tricky thing is to manage these multiple JDKs and IDEs. It’s a piece of cake if I just use Eclipse for compiling my code, because the IDE allows me to configure multiple versions of Java runtime. Unfortunately (or fortunately), I have to use the command line/shell to build my code. So, it is important that I have the right version of JDK present in the PATH and other related environment variables (such as JAVA_HOME). Manually modifying the environment variables every time I want to switch between JDKs, isn’t a happy task. But, thanks to Windows Powershell, I’m able to write a scriplet that can do the heavy-lifting for me. Basically, what I want to achieve is to set PATH variable to add Java bin folder and set the JAVA_HOME environment variable and then launch the correct Eclipse IDE. And, I want to do this with a single command. Let’s do it. Open a Windows Powershell. I prefer writing custom Windows scripts in my profile file so that it is available to run when ever I open the shell. To edit the profile, run this command: notepad.exe $profile - the $profile is a special variable that points to your profile file. Write the below script in the profile file and save it. function myIDE{ $env:Path += "C:\vraa\java\jdk7\bin;" $env:JAVA_HOME = "C:\vraa\java\jdk7" C:\vraa\ide\eclipse\eclipse set-location C:\vraa\workspace\myproject play } function officeIDE{ $env:Path += "C:\vraa\java\jdk6\bin;" $env:JAVA_HOME = "C:\vraa\java\jdk6" C:\office\eclipse\eclipse } Close and restart the Powershell. Now you can issue the command myIDE which will set the proper PATH and environment variables and then launch the eclipse IDE. As you can see, there are two functions with different configurations. Just call the function name that you want to launch from the Powershell command line (myIDE or officeIDE).

There appears to be a misconception that annotations are required on the model in order to use a JAXB (JSR-222) implementation. The truth is that JAXB is configuration by exception, so annotations are only required when you want to override default behaviour. In this example I'll demonstrate how to use JAXB without providing any metadata. Domain Model I will use the following domain model for this example. Note how there are no annotations of any kind. Customer Customer is the root object in this example. Normally we would annotate it with @XmlRootElement. Later in the demo code you will see how we can use an instance of JAXBElement instead. package blog.defaults; import java.util.List; public class Customer { private String firstName; private String lastName; private List phoneNumbers; public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } public List getPhoneNumbers() { return phoneNumbers; } public void setPhoneNumbers(List phoneNumbers) { this.phoneNumbers = phoneNumbers; } } PhoneNumber I have purposefully given the fields in this class nonsense names, so that later when we look at the XML you will be able to see that by default the element names are derived from the properties and not the fields. package blog.defaults; public class PhoneNumber { private String foo; private String bar; public String getType() { return foo; } public void setType(String type) { this.foo = type; } public String getNumber() { return bar; } public void setNumber(String number) { this.bar = number; } } Demo Code Since we haven't used @XmlRootElement (or @XmlElementDecl) to associate a root element with our Customer class we will need to tell JAXB what class we want to unmarshal the XML document to. This is done by using one of the unmarshal methods that take a Class parameter (line 14). This will return a JAXBElement, the Customer object is then accessed by calling getValue on it (line 15). To marshal the object back to XML we need to ensure that it is wrapped in a JAXBElement to supply the root element information (line 17). package blog.defaults; import javax.xml.bind.*; import javax.xml.namespace.QName; import javax.xml.transform.stream.StreamSource; public class Demo { public static void main(String[] args) throws Exception { JAXBContext jc = JAXBContext.newInstance(Customer.class); StreamSource xml = new StreamSource("src/blog/defaults/input.xml"); Unmarshaller unmarshaller = jc.createUnmarshaller(); JAXBElement je1 = unmarshaller.unmarshal(xml, Customer.class); Customer customer = je1.getValue(); JAXBElement je2 = new JAXBElement(new QName("customer"), Customer.class, customer); Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); marshaller.marshal(je2, System.out); } } input.xml/Output The following is the input to and output from running the demo code. The first thing we see is that it is a very reasonable XML representation of the data, there aren't any JAXB artifacts. By default JAXB will marshal everything as XML elements, and based on our PhoneNumber class we see that the element names were derived from the property names. Jane Doe 555-1111 work 555-2222 home Further Reading If you enjoyed this post then you may also be interested in: The majority of the articles on this blog describe how to leverage the power of JAXB's metadata to support different use cases, I invite you to check them out: http://blog.bdoughan.com/search/label/JAXB If you are interested in specifying metadata without using annotations, you may be interested in EclipseLink JAXB (MOXy)'s external mapping document: Extending JAXB - Representing Metadata as XML Extending JAXB - Representing Metadata as JSON

With the advent of Continuous Integration and Continuous Delivery, our builds are split into different steps creating the deployment pipeline. Some of these steps can be compiled and run fast tests, run slow tests, run automated acceptance tests, or releasing the application, to cite a few. Most of us are using Jenkins/Hudson to implement Continuous Integration/Delivery, and we manage job orchestration combining some Jenkins plugins like build pipeline, parameterized-build, join or downstream-ext. We have to configure all of them which implies polluting the job configuration through multiple jobs, which , makes the system configuration very complex to maintain. Build Flow enables us to define an upper level flow item to manage job orchestration and link up rules, using a dedicated DSL. Let's see a very simple example: First step is installing the plugin. Go to Jenkins -> Manage Jenkins -> Plugin Manager -> Available and find for CloudBees Build Flowplugin. Then you can go to Jenkins -> New Job and you will see a new kind of job called Build Flow. In this example we are going to name it build-all-yy. And now you only have to program using flow DSL how this job should orchestrate the other jobs. In "Define build flow using flow DSL" input text you can specify the sequence of commands to execute. In current example I have already created two jobs, one executing clean compile goal (yy-compile job name) and the other one executing javadoc goal (yy-javadoc job name). I know that this deployment pipeline is not real in a true environment but for now it is enough. Then we want javadoc job running after project is compiled. To configure this we don't have to create any upstream or downstream actions, simply add next lines at DSL text area: build("yy-compile"); build("yy-javadoc"); Save and execute build-all-yy job and both projects will be built in a sequential way. Now suppose that we add a third job called yy-sonar which runs sonar goal that generates code quality sonar report. In this case it seems obvious that after project is compiled, generation of javadocs and code quality jobs can be run in parallel. So script is changed to: build("yy-compile") parallel ( {build("yy-javadoc")}, {build("yy-sonar")} ) This plugin also supports more operations like retry (similar behaviour of retry-failed-job plugin) or guard-rescue, that it works mostly like a try+finally block. Also you can create parameterized builds, accessing to build execution or printing to Jenkins console. Next example will print build number of yy-compile job execution: b = build("yy-compile") out.println b.build.number And finally you can also have a quick graphical overview of the execution in Status section. It is true that could be improved more, but for now it is acceptable, and can be used without any problem. Build Flow plugin is in its early stages, in fact it is only at version 0.4. But will be a plugin to be considered in future, and I think it is good to know that it exists. Moreover is being developed by CloudBees folks so it is a guarantee of being fully supported by Jenkins. We Keep Learning. Alex. Warning: In order to run parallel tasks with the plugin Anonymous users must have Read Job access (Jenkins -> Manage Jenkins -> Configure System). There is an issue already inserted into Jira to fix this problem.

Mustache is templating system with implementation in many languages including Java and JavaScript . The templates are also supported by various web frameworks and client side JS libraries. Mustache has simple idea of "logic-less" system because it lacks any explicit control statements, like if, else or goto and also it does not have for statement however looping and conditional calculation can be achieved using custom tags that work with lists and lambdas. The name unfortunately has less to do with Tom Selleck but more with the heavy use of curly braces that look like mustache. The similarity is more than comparable. Mustache has implementation for most of the widely used languages like: Java, Javascript, Ruby,Net and many more. The client side template's in JavaScript Let say that you have some REST service and you have created a book view object that has an additional function that appends amazon associates id to the book url: var book = { id : 12, title : "A Game of Thrones", url : "http://www.amazon.com/gp/product/0553573403/", amazonId : "myAwesomeness", associateUrl : function() { return this.url + '?tag=' + this.amazonId; }, author : { name : 'George R. R. Martin', imdbUrl : 'http://www.imdb.com/name/nm0552333/', wikiUrl : 'https://en.wikipedia.org/wiki/George_R._R._Martin' }, haveInStock : true, similarBooks : [{ id : 13, title : "Decision Points" }, { id : 13, title : "Spoken from the Heart" }], comments : [] }; The standard way of rendering data without using templates would be create an output variable and just append everything inside and at the end just place the data where it should be. jQuery(document).ready(function() { var out = '' + book.title + ' is awesome book get it on Amazon'; jQuery('#content-jquery').html(out); }); This is fairly simple but if you for example want to change the span element with div it takes a little bit of time to figure where it should be closed and often you can miss if the element should be in single quotes or double quotes. The bigger issue here is that the content is peaces of strings that need to be easy to styled via CSS and JavaScript. As the code gets bigger this becomes unmanageable and changes to anything become slower especially if you add on top of this jQuery's manipulation functions like appendTo() or prependTo(). This direct use of out+= type of creating the content reminds me a lot of HttpServlet style of using print writer and doing out.print() and for the same reason why this was almost abandoned we should not do this in JavaScript. To simplify work we can add template engine like Mustache that is one of many client side tempting engines. So how does a template in mustache looks like, well for the example above with the book it would look like :

We recently adopted the use of Spring Data. Spring Data provides a nice pattern/API that you can layer on top of JPA to eliminate boiler-plate code. With that adoption, we started looking at the DAO layer we use against Cassandra for some of our operations. Some of the data we store in Cassandra is simple. It does *not* leverage the flexible nature of NoSQL. In other words, we know all the table names, the column names ahead of time, and we don't anticipate them changing all that often. We could have stored this data in an RDBMs, using hibernate to access it, but standing up another persistence mechanism seemed like overkill. For simplicity's sake, we preferred storing this data in Cassandra. That said, we want the flexibility to move this to an RDBMs if we need to. Enter JPA. JPA would provide us a nice layer of abstraction away from the underlying storage mechanism. Wouldn't it be great if we could annotate the objects with JPA annotations, and persist them to Cassandra? Enter Kundera. Kundera is a JPA implementation that supports Cassandra (among other storage mechanisms). OK -- so JPA is great, and would get us what we want, but we had just adopted the use of Spring Data. Could we use both? The answer is "sort of". I forked off SpringSource's spring-data-cassandra: https://github.com/boneill42/spring-data-cassandra And I started hacking on it. I managed to get an implementation of the PagingAndSortingRepository for which I wrote unit tests that worked, but I was duplicating a lot of what should have come for free in the SimpleJpaRepository. When I tried to substitute my CassandraJpaRepository for the SimpleJpaRepository, I ran into some trouble w/ Kundera. Specifically, the MetaModel implementation appeared to be incomplete. MetaModelImpl was returning null for all managedTypes(). SimpleJpa wasn't too happy with this. Instead of wrangling with Kundera, we punted. We can achieve enough of the value leveraging JPA directly. Perhaps more importantly, there is still an impedance mismatch between JPA and NoSQL. In our case, it would have been nice to get at Cassandra through Spring Data using JPA for a few cases in our app, but for the vast majority of the application, a straight up ORM layer whereby we know the tables, rows and column names ahead of time is insufficient. For those cases where we don't know the schema ahead of time, we're going to need to leverage the converters pattern in Spring Data. So, I started hacking on a proper Spring Data layer using Astyanax as the client. Follow along here: https://github.com/boneill42/spring-data-cassandra More to come on that....

Don’t be afraid – Some clarification to common misunderstandings Since version 3.1, Apache Lucene and Solr use MMapDirectory by default on 64bit Windows and Solaris systems; since version 3.3 also for 64bit Linux systems. This change lead to some confusion among Lucene and Solr users, because suddenly their systems started to behave differently than in previous versions. On the Lucene and Solr mailing lists a lot of posts arrived from users asking why their Java installation is suddenly consuming three times their physical memory or system administrators complaining about heavy resource usage. Also consultants were starting to tell people that they should not use MMapDirectory and change their solrconfig.xml to work instead with slow SimpleFSDirectory or NIOFSDirectory (which is much slower on Windows, caused by a JVM bug #6265734). From the point of view of the Lucene committers, who carefully decided that using MMapDirectory is the best for those platforms, this is rather annoying, because they know, that Lucene/Solr can work with much better performance than before. Common misinformation about the background of this change causes suboptimal installations of this great search engine everywhere. In this blog post, I will try to explain the basic operating system facts regarding virtual memory handling in the kernel and how this can be used to largely improve performance of Lucene (“VIRTUAL MEMORY for DUMMIES”). It will also clarify why the blog and mailing list posts done by various people are wrong and contradict the purpose of MMapDirectory. In the second part I will show you some configuration details and settings you should take care of to prevent errors like “mmap failed” and suboptimal performance because of stupid Java heap allocation. Virtual Memory[1] Let’s start with your operating system’s kernel: The naive approach to do I/O in software is the way, you have done this since the 1970s – the pattern is simple: whenever you have to work with data on disk, you execute a syscall to your operating system kernel, passing a pointer to some buffer (e.g. a byte[] array in Java) and transfer some bytes from/to disk. After that you parse the buffer contents and do your program logic. If you don’t want to do too many syscalls (because those may cost a lot processing power), you generally use large buffers in your software, so synchronizing the data in the buffer with your disk needs to be done less often. This is one reason, why some people suggest to load the whole Lucene index into Java heap memory (e.g., by using RAMDirectory). But all modern operating systems like Linux, Windows (NT+), MacOS X, or Solaris provide a much better approach to do this 1970s style of code by using their sophisticated file system caches and memory management features. A feature called “virtual memory” is a good alternative to handle very large and space intensive data structures like a Lucene index. Virtual memory is an integral part of a computer architecture; implementations require hardware support, typically in the form of a memory management unit (MMU) built into the CPU. The way how it works is very simple: Every process gets his own virtual address space where all libraries, heap and stack space is mapped into. This address space in most cases also start at offset zero, which simplifies loading the program code because no relocation of address pointers needs to be done. Every process sees a large unfragmented linear address space it can work on. It is called “virtual memory” because this address space has nothing to do with physical memory, it just looks like so to the process. Software can then access this large address space as if it were real memory without knowing that there are other processes also consuming memory and having their own virtual address space. The underlying operating system works together with the MMU (memory management unit) in the CPU to map those virtual addresses to real memory once they are accessed for the first time. This is done using so called page tables, which are backed by TLBs located in the MMU hardware (translation lookaside buffers, they cache frequently accessed pages). By this, the operating system is able to distribute all running processes’ memory requirements to the real available memory, completely transparent to the running programs. Schematic drawing of virtual memory (image from Wikipedia [1], http://en.wikipedia.org/wiki/File:Virtual_memory.svg, licensed by CC BY-SA 3.0) By using this virtualization, there is one more thing, the operating system can do: If there is not enough physical memory, it can decide to “swap out” pages no longer used by the processes, freeing physical memory for other processes or caching more important file system operations. Once a process tries to access a virtual address, which was paged out, it is reloaded to main memory and made available to the process. The process does not have to do anything, it is completely transparent. This is a good thing to applications because they don’t need to know anything about the amount of memory available; but also leads to problems for very memory intensive applications like Lucene. Lucene & Virtual Memory Let’s take the example of loading the whole index or large parts of it into “memory” (we already know, it is only virtual memory). If we allocate a RAMDirectory and load all index files into it, we are working against the operating system: The operating system tries to optimize disk accesses, so it caches already all disk I/O in physical memory. We copy all these cache contents into our own virtual address space, consuming horrible amounts of physical memory (and we must wait for the copy operation to take place!). As physical memory is limited, the operating system may, of course, decide to swap out our large RAMDirectory and where does it land? – On disk again (in the OS swap file)! In fact, we are fighting against our O/S kernel who pages out all stuff we loaded from disk [2]. So RAMDirectory is not a good idea to optimize index loading times! Additionally, RAMDirectory has also more problems related to garbage collection and concurrency. Because the data residing in swap space, Java’s garbage collector has a hard job to free the memory in its own heap management. This leads to high disk I/O, slow index access times, and minute-long latency in your searching code caused by the garbage collector driving crazy. On the other hand, if we don’t use RAMDirectory to buffer our index and use NIOFSDirectory or SimpleFSDirectory, we have to pay another price: Our code has to do a lot of syscalls to the O/S kernel to copy blocks of data between the disk or filesystem cache and our buffers residing in Java heap. This needs to be done on every search request, over and over again. Memory Mapping Files The solution to the above issues is MMapDirectory, which uses virtual memory and a kernel feature called “mmap” [3] to access the disk files. In our previous approaches, we were relying on using a syscall to copy the data between the file system cache and our local Java heap. How about directly accessing the file system cache? This is what mmap does! Basically mmap does the same like handling the Lucene index as a swap file. The mmap() syscall tells the O/S kernel to virtually map our whole index files into the previously described virtual address space, and make them look like RAM available to our Lucene process. We can then access our index file on disk just like it would be a large byte[] array (in Java this is encapsulated by a ByteBuffer interface to make it safe for use by Java code). If we access this virtual address space from the Lucene code we don’t need to do any syscalls, the processor’s MMU and TLB handles all the mapping for us. If the data is only on disk, the MMU will cause an interrupt and the O/S kernel will load the data into file system cache. If it is already in cache, MMU/TLB map it directly to the physical memory in file system cache. It is now just a native memory access, nothing more! We don’t have to take care of paging in/out of buffers, all this is managed by the O/S kernel. Furthermore, we have no concurrency issue, the only overhead over a standard byte[] array is some wrapping caused by Java’s ByteBuffer interface (it is still slower than a real byte[] array, but that is the only way to use mmap from Java and is much faster than all other directory implementations shipped with Lucene). We also waste no physical memory, as we operate directly on the O/S cache, avoiding all Java GC issues described before. What does this all mean to our Lucene/Solr application? We should not work against the operating system anymore, so allocate as less as possible heap space (-Xmx Java option). Remember, our index accesses rely on passed directly to O/S cache! This is also very friendly to the Java garbage collector. Free as much as possible physical memory to be available for the O/S kernel as file system cache. Remember, our Lucene code works directly on it, so reducing the number of paging/swapping between disk and memory. Allocating too much heap to our Lucene application hurts performance! Lucene does not require it with MMapDirectory. Why does this only work as expected on operating systems and Java virtual machines with 64bit? One limitation of 32bit platforms is the size of pointers, they can refer to any address within 0 and 232-1, which is 4 Gigabytes. Most operating systems limit that address space to 3 Gigabytes because the remaining address space is reserved for use by device hardware and similar things. This means the overall linear address space provided to any process is limited to 3 Gigabytes, so you cannot map any file larger than that into this “small” address space to be available as big byte[] array. And when you mapped that one large file, there is no virtual space (address like “house number”) available anymore. As physical memory sizes in current systems already have gone beyond that size, there is no address space available to make use for mapping files without wasting resources (in our case “address space”, not physical memory!). On 64bit platforms this is different: 264-1 is a very large number, a number in excess of 18 quintillion bytes, so there is no real limit in address space. Unfortunately, most hardware (the MMU, CPU’s bus system) and operating systems are limiting this address space to 47 bits for user mode applications (Windows: 43 bits) [4]. But there is still much of addressing space available to map terabytes of data. Common misunderstandings If you have read carefully what I have told you about virtual memory, you can easily verify that the following is true: MMapDirectory does not consume additional memory and the size of mapped index files is not limited by the physical memory available on your server. By mmap() files, we only reserve address space not memory! Remember, address space on 64bit platforms is for free! MMapDirectory will not load the whole index into physical memory. Why should it do this? We just ask the operating system to map the file into address space for easy access, by no means we are requesting more. Java and the O/S optionally provide the option to try loading the whole file into RAM (if enough is available), but Lucene does not use that option (we may add this possibility in a later version). MMapDirectory does not overload the server when “top” reports horrible amounts of memory. “top” (on Linux) has three columns related to memory: “VIRT”, “RES”, and “SHR”. The first one (VIRT, virtual) is reporting allocated virtual address space (and that one is for free on 64 bit platforms!). This number can be multiple times of your index size or physical memory when merges are running in IndexWriter. If you have only one IndexReader open it should be approximately equal to allocated heap space (-Xmx) plus index size. It does not show physical memory used by the process. The second column (RES, resident) memory shows how much (physical) memory the process allocated for operating and should be in the size of your Java heap space. The last column (SHR, shared) shows how much of the allocated virtual address space is shared with other processes. If you have several Java applications using MMapDirectory to access the same index, you will see this number going up. Generally, you will see the space needed by shared system libraries, JAR files, and the process executable itself (which are also mmapped). How to configure my operating system and Java VM to make optimal use of MMapDirectory? First of all, default settings in Linux distributions and Solaris/Windows are perfectly fine. But there are some paranoid system administrators around, that want to control everything (with lack of understanding). Those limit the maximum amount of virtual address space that can be allocated by applications. So please check that “ulimit -v” and “ulimit -m” both report “unlimited”, otherwise it may happen that MMapDirectory reports “mmap failed” while opening your index. If this error still happens on systems with lot’s of very large indexes, each of those with many segments, you may need to tune your kernel parameters in /etc/sysctl.conf: The default value of vm.max_map_count is 65530, you may need to raise it. I think, for Windows and Solaris systems there are similar settings available, but it is up to the reader to find out how to use them. For configuring your Java VM, you should rethink your memory requirements: Give only the really needed amount of heap space and leave as much as possible to the O/S. As a rule of thumb: Don’t use more than ¼ of your physical memory as heap space for Java running Lucene/Solr, keep the remaining memory free for the operating system cache. If you have more applications running on your server, adjust accordingly. As usual the more physical memory the better, but you don’t need as much physical memory as your index size. The kernel does a good job in paging in frequently used pages from your index. A good possibility to check that you have configured your system optimally is by looking at both "top" (and correctly interpreting it, see above) and the similar command "iotop" (can be installed, e.g., on Ubuntu Linux by "apt-get install iotop"). If your system does lots of swap in/swap out for the Lucene process, reduce heap size, you possibly used too much. If you see lot's of disk I/O, buy more RUM (Simon Willnauer) so mmapped files don't need to be paged in/out all the time, and finally: buy SSDs. Happy mmapping! Bibliography [1] http://en.wikipedia.org/wiki/Virtual_memory [2] https://www.varnish-cache.org/trac/wiki/ArchitectNotes [3] http://en.wikipedia.org/wiki/Memory-mapped_file [4] http://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details

Spring core comes out-of-the-box with two scopes: singletons and prototypes. Singletons implement the Singleton pattern, meaning there's only a single instance at runtime (in a JVM). Spring instantiates them during context creation, caches them in the context, and serves them from the cache when needed (or something like that). Prototypes are instantiated each time you access the context to get the bean. Problems arise when you need to inject a prototype-scoped bean in a singleton-scoped bean. Since singletons are created (and then injected) during context creation: it's the only time the Spring context is accessed and thus prototype-scoped beans are injected only once, thus defeating their purpose. In order to inejct prototypes into singletons, and side-by-syde with setter and constructor injection, Spring proposes another way for injection, called method injection. It works in the following way: since singletons are instantiated at context creation, it changes the way prototype-scoped are handled, from injection to created by an abstract method. The following snippet show the unsuccessful way to achieve injection: public class Singleton { private Prototype prototype; public Singleton(Prototype prototype) { this.prototype = prototype; } public void doSomething() { prototype.foo(); } public void doSomethingElse() { prototype.bar(); } } The next snippet displays the correct code: public abstract class Singleton { protected abstract Prototype createPrototype(); public void doSomething() { createPrototype().foo(); } public void doSomethingElse() { createPrototype().bar(); } } As you noticed, code doesn't specify the createPrototype() implementation. This responsibility is delegated to Spring, hence the following needed configuration: Note that an alternative to method injection would be to explicitly access the Spring context to get the bean yourself. It's a bad thing to do since it completely defeats the whole Inversion of Control pattern, but it works (and is essentially the only option when a nasty bug happens on the server - see below).However, using method injection has several main limitations: Spring achieves this black magic by changing bytecode. Thus, you'll need to have the CGLIB libraryon the classpath. The feature is only available by XML configuration, no annotations (see this JIRAfor more information) Finally, some application servers have bugs related to CGLIB (such as this one) To go further: Spring's documentation on method injection

Once you’ve been working with Jenkins and uberSVN for a while, you may find yourself in a situation where you have several jobs that need to run in a specific order, for example: Job 1 and Job 3 can run simultaneously. BUT Job 2 should only start when Job 1 and Job 3 have finished running. AND Job 4 should only start when Job 2 has finished. How can you implement this complicated setup? This is where Jenkins’ ‘Advanced Project Options’ and build triggers come in handy. In this tutorial, we’ll walk through the different options for scheduling jobs using Jenkins and uberSVN, the free ALM platform for Apache Subversion. Note, this tutorial assumes you have already created a job and configured it to automatically poll your Subversion repository. 1) Open the Jenkins tab of your uberSVN installation and select a job. 2) Click the ‘Configure’ option from the left-hand menu. 3) In the ‘Advanced Project Options’ tab, select the ‘Advanced…’ button 4) This contains two options that are useful for ordering your jobs: Block build when upstream project is building – blocks builds when a dependency is in the queue, or building. Note, these dependencies include both direct and transitive dependencies. Block build when downstream project is building – blocks builds when a child of the project is in the queue, or building. This applies to both direct and transitive children. If this option doesn’t meet your needs, you can explicitly name a project (or projects) that must be built before your job is allowed to run. To set this: 1) Scroll down to the ‘Build triggers’ tab on the configure page. 2) Select the ‘Build after other projects are built’ checkbox. This will bring up a text box where you can list any number of projects. Utilized properly, the build triggers and advanced project options should allow you to organize your jobs into a schedule. Tip, if you need even more control over your build schedule, there are plenty of scheduling plugins available. To add plugins to Jenkins, simply: 1) Open the ‘Manage Jenkins’ screen. 2) Click the ‘Manage Plugins’ link. 3) Open the ‘Available’ tab and select the appropriate plugins from the list.

This blog post explores JavaScript’s eval function by implementing the foundation for an interactive command line. As a bonus, you’ll get to work with ECMAScript.next’s generators (which can already be tried out on current Firefox versions). Writing an evaluator Let’s say you want to implement an interactive command line for JavaScript (such as [1]). On one hand, you would need to get the graphical user interface right: The user inputs JavaScript code, the command line evaluates the code and displays the result. On the other hand, you would have to implement the evaluation. That’s what we will take on here. It is more complex that it initially seems and teaches us a lot about eval. For starters, let’s write a constructor Evaluator: function Evaluator() { } Evaluator.prototype.evaluate = function (str) { return JSON.stringify(eval(str)); }; To use the evaluator, we create an instance and send JavaScript code to it: > var e = new Evaluator(); > e.evaluate("Math.pow(2, 53)") '9007199254740992' > e.evaluate("3 * 7") '21' > e.evaluate("'foo'+'bar'") '"foobar"' JSON.stringify is used so that the evaluation results can be shown to the user and look like the input. Without stringify, things look as follows: > console.log(123) // OK 123 > console.log("abc") // not OK abc With stringify, everything looks OK: > console.log(JSON.stringify(123)) 123 > console.log(JSON.stringify("abc")) "abc" Note that undefined is not valid JSON, but stringify converts it to undefined (the value, not the string), which is fine for our purposes. What we have implemented so far works for basic things, but still has several problems. Let’s tackle them one at a time. Problem: declarations You can evaluate variable and function declarations, but they are forgotten immediately afterwards: > e.evaluate("var x = 12;") undefined > e.evaluate("x") ReferenceError: x is not defined How do we fix this? The following code is a solution: function Evaluator() { this.env = {}; } Evaluator.prototype.evaluate = function (str) { str = rewriteDeclarations(str); var __environment__ = this.env; // (1) with (__environment__) { // (2) return JSON.stringify(eval(str)); } }; function rewriteDeclarations(str) { // Prefix a newline so that search and replace is simpler str = "\n" + str; str = str.replace(/\nvar\s+(\w+)\s*=/g, "\n__environment__.$1 ="); // (3) str = str.replace(/\nfunction\s+(\w+)/g, "\n__environment__.$1 = function"); return str.slice(1); // remove prefixed newline } this.env holds all variable declarations and function declarations in its properties. We make it accessible to the input in two steps. Step 1 – declare: We assign this.env to __environment__ (1) and rewrite the input so that, among other things, each var declaration assigns to __environment__ (3). That demonstrates one important aspect of eval: it sees all variables in surrounding scopes. That is, if you invoke eval inside your function, you expose all of its internals. The only way to keep those internals secret is to put the eval call in a separate function and call that function. Step 2 – access: Use a with statement so that the properties of __environment__ appear as variables to the eval-ed code. This is not an ideal solution, more of a compromise: with should be avoided [2] and can’t be used in the advantageous strict mode [3]. But it is a quick solution for us now. A work-around is quite complex [4]. > var e = new Evaluator(); > e.evaluate("var x = 123;") '123' > e.evaluate("x") '123' Minor drawback: Normal var declarations have the result undefined; due to our rewriting we now get the value that is assigned to the variable. Problem: exceptions Right now, throwing an exception in evaluate’s input means that the method will throw: > e.evaluate("* 3") SyntaxError: Unexpected token * That is obviously unacceptable: In a graphical user interface, we want to report errors back to the user, not (invisibly) throw an exception. Here is one simple way of doing so: Evaluator.prototype.evaluate = function (str) { try { str = rewriteDeclarations(str); var __environment__ = this.env; with (__environment__) { return JSON.stringify(eval(str)); } } catch (e) { return e.toString(); } }; There is nothing surprising in this code, we simply use try-catch and report back what happened. More sophisticated solutions will want to do more, e.g. display the exception’s stack trace. The new evaluator in action: > var e = new Evaluator(); > e.evaluate("* 3") 'SyntaxError: Unexpected token *' Problem: console.log How do we handle calls to console.log in the input? Logged messages should be shown to the user, not be sent to the browser’s console. The solution is surprisingly easy: function Evaluator(cons) { this.env = {}; this.cons = cons; } Evaluator.prototype.evaluate = function (str) { try { str = rewriteDeclarations(str); var __environment__ = this.env; var console = this.cons; with (__environment__) { return JSON.stringify(eval(str)); } } catch (e) { return e.toString(); } }; The constructor now receives a custom implementation of console and assigns it to this.cons. By assigning that object to a local variable named console (1), we temporarily shadow the global console for eval, there is no need to replace it. Beware that that shadowing affects all of the function, you won’t be able to use the browser’s console anywhere in evaluate. The new evaluator in action: > var cons = { log: function (m) { console.log("### "+m) } }; > var e = new Evaluator(cons); > e.evaluate("console.log('hello')") ### hello undefined Problem: eval creates bindings inside the function One scary feature of eval is that it creates variable bindings inside the function that invokes it: > (function () { eval("var x=3"); return x }()) 3 Fortunately, the fix is easy: use strict mode. > (function () { "use strict"; eval("var x=3"); return x }()) ReferenceError: x is not defined You can’t use with in strict mode, so you’ll have to replace it with a work-around [4]. Keeping declarations in an environment An environment is where JavaScript keeps the parameters and variables of a function. It maps variable names to values and is thus similar to an object. We might be able to avoid rewriting the input and manage declarations via environments. The idea is as follows. eval puts declarations in some environment: Non-strict mode: the environment of the surrounding function. Strict mode: a newly created environment. What if we could reuse that environment for the next invocation of eval, instead of throwing it away? Then eval would properly remember prior declarations. Strict mode gives us no way to access the temporary environment it creates for each invocation. However, in non-strict mode, we might be able to keep the environment of the surrounding function around. The following subsections explore two ways of doing so. Declarations via nested scopes If you create a function g inside another function f, then g permanently retains a reference to f’s current environment envf. Whenever g is called, a new g-specific environment envg is created. But envg points to its parent environment envf. Variables that can’t be found in g’s scope (as managed via envg), are looked up in f’s scope (via envf). Thus, envf is not lost, as long as g exists. That gives us a strategy for keeping the environment of the function that calls eval around. In the following code that function is called evalHelper and creates a new function that has to be used for the next call of eval. Hence, declarations made in the former function are accessible in the later function. function Evaluator() { var that = this; that.evalHelper = function (str) { that.evalHelper = function (str) { return eval(str); }; return eval(str); }; } Evaluator.prototype.evaluate = function (str) { return this.evalHelper(str); }; The fatal problem of this implementation is that you cannot nest to arbitrary depth. But, for the above depth of 2, it works perfectly: > var e = new Evaluator(); > e.evaluate("var x = 7;"); undefined > e.evaluate("x * 3") 21 Declarations via a generator It would be great if we could “restart” the function that calls eval, re-enter it with its previous environment still in place. ECMAScript.next’s generators [5] let you do that. Current versions of Firefox already support generators. Here is a demonstration of how they work in these versions (in ECMAScript.next, you will have to write function*, but apart from that, the code is the same): function mygen() { console.log((yield 0) + " @ 0"); console.log((yield 1) + " @ 1"); console.log((yield 2) + " @ 2"); } The above is a generator function. Invoke it and it will create a generator object. On that object, you first need to invoke the next() method to start execution. A yield x inside the code pauses execution and returns x to the the previously called generator object method. After the first next(), you can either call next() or send(y). The latter means that the currently paused yield will continue and produce the value y. The former is equivalent to send(undefined). The following interaction shows mygen in use: > var g = mygen(); > g.next() // can’t use send() the first time 0 > g.send("a") // continue after yield 0, pause again a @ 0 1 > g.send("b") b @ 1 2 The following is an implementation of Evaluator that calls eval via the generator evalGenerator. Because of that, eval always sees the same environment and remembers declarations. function evalGenerator(console) { var str = yield; while(true) { try { var result = JSON.stringify(eval(str)); str = yield result; } catch (e) { str = yield e.toString(); } } } function Evaluator(cons) { this.evalGen = evalGenerator(cons); this.evalGen.next(); // start } Evaluator.prototype.evaluate = function (str) { return this.evalGen.send(str); }; The new evaluator works as expected. > var e = new Evaluator(); > e.evaluate("var x = 7;") undefined > e.evaluate("x * 2") "14" > e.evaluate("* syntax_error") "SyntaxError: missing ; before statement" The biggest problem with this solution is that it uses the deprecated features non-strict eval together with the new feature generators. There will probably be a way in ECMAScript.next to make this combination work, but it will be a hack and should thus be avoided. Conclusion We have used eval to implement a helper type for a command line. While doing so, we learned a few interesting things about eval: Letting it remember declarations between invocations is complicated; it can access all variables in the scopes surrounding its invocation; and in non-strict mode, it can even create new variables inside the invoking function. The best solution for remembering declarations would be for eval to have an optional parameter for an environment (to be reused), but that is not in the cards. Therefore, the only truly safe solution in pure JavaScript is to use a full-featured JavaScript parser such as esprima to rewrite critical parts of the input code. That is left as an exercise to the reader. References Combining code editing with a command line JavaScript’s with statement and why it’s deprecated JavaScript’s strict mode: a summary Handing variables to eval Asynchronous programming and continuation-passing style in JavaScript

the great thing with blogging is: i receive great comments, questions and ideas. the great thing with eclipse/codewarrior is that the extensions are almost unlimited . for my earlier post on hiding the toolbar i received a tip for another way which even is better: a plugin to switch eclipse into full screen mode. here is how to install it and how it looks… the plugin is available from http://code.google.com/p/eclipse-fullscreen/ . download the zip file. the zip file has the cn.pande.eclipsex.fullscreen_.jar file. copy that file inside the codewarrrior or eclipse eclipse\plugins folder. after launching eclipse there is new entry in the window menu: full screen menu item this switches eclipse into full screen mode: eclipse in full screen mode this hides the toolbar, menu and status bar, and i have more space available for what matters how to get out of full screen mode: press esc key to exit full screen mode. ctrl+alt+z is the default shortcut to toggle between ‘normal’ and ‘full screen’ mode. happy full screening

In a previous post, I gave an introduction to gevent to show some of the benefits your application might get from using gevent greenlets instead of threads. Some people, however, took issue with my benchmark code, saying that the threaded example was contrived. In this post, I'll try to answer some of the objections. (It actually turns out that there was a bug in the version of ab I was using to test, as well, so I re-ran the tests from the previous post, too.) Threads versus Greenlets Initially, I had proposed a dummy webserver that handled incoming requests by creating a thread and delegating communication to that thread. The code in question is below: def threads(port): s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() t = threading.Thread(target=handle_request, args=(cli, time.sleep)) t.daemon = True t.start() When I could get the code above ot actually run the full benchmark (which it didn't often do) it ended up getting around 1300-1400 requests per second. The gevent version looked very similar: import gevent def greenlet(port): from gevent import socket s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() gevent.spawn(handle_request, cli, gevent.sleep) This code was able to handle closer to 1600 requests per second. Maybe I should have called it out better, but the fact that the gevent version performed better than the threaded version does point out an important aspect of gevent: Greenlets are significantly lighter-weight than true threads, particularly when creating them. However, the folks objected by pointing out that you just don't do that with threads. Nobody does. It's a dumb way to design a server. I agree with all these points, though that wasn't really the point I was going for. One thing I will point out is that: The reason you don't design threaded servers so they fork a thread each time you get a connection is that threads are expensive to fork, unlike greenlets. Fixing the benchmark So anyway, to "fix" the benchmark so it's a little more fair to threads, we'll use a thread pool to create all the threads up-front and then use a Queue.Queue to send work to them. Our server core now looks like this: def threads(port, N=10): s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) q = Queue() for x in xrange(N): t = threading.Thread(target=thread_worker, args=(q,)) t.daemon = True t.start() print 'Ready and waiting with %d threads on port %d' % ( N, port) while True: cli, addr = s.accept() q.put(cli) def thread_worker(q): while True: sock = q.get() handle_request(sock, time.sleep) If I now run this with a thread pool of 200 threads, I can indeed finish the benchmark (ApacheBench as ab -r -n 2000 -c 200... with around 1300 requests per second (a little less, probably due to the synchronization overhead of the Queue). So updating the benchmark to use a thread pool did not improve the performance. The equivalent gevent code uses gevent.pool.Pool: def greenlet(port, N=10): from gevent.pool import Pool from gevent import socket, sleep pool = Pool(N) s = socket.socket() s.bind(('0.0.0.0', port)) s.listen(500) while True: cli, addr = s.accept() pool.spawn(handle_request, cli, sleep) Running ab with the same parameters I now get... around 1200-1400 requests per second. So why use gevent, again? So yes, if I had designed the benchmark to omit the thread/greenlet creation entirely, threads and greenlets do indeed perform about the same. The big win for greenlets is when your thread pool isn't big enough to handle the concurrent connections. It turns out that there's a clever denial-of-service attack on web servers known as slowloris that consumes threads from your thread pool quickly. Once your server's threads are all busy handling the slowloris requests, no further work can be done, and you end up with a very lightly loaded but still unresponsive server. To illustrate this, we can try running our benchmark with the thread pool, but only running 20 threads in the pool, but modifying our request handler to take five seconds to handle a request. We'll go ahead and modify the benchmark line to allow more time for responses as well: $ ab -n 2000 -c 200 -r -t 60 http://127.0.0.1:... Now our threaded example ends up timing out connections as it tries to service 200 concurrent connections, each taking five seconds, with only 20 worker threads. If we go back to our naive (un-pooled) gevent example, however, we're able to achieve 47 requests per second, which is close to the theoretical maximum of 50 requests per second, with a very light server load. The point? A slowloris attack will be able to eat up all the threads in your (finite-sized) thread pool, regardless of how big that pool is. Spawning a greenlet each time you receive a connection means you don't waste (almost) any resources waiting on IO. Conclusion There's a good bit more to gevent that I'd like to cover in future posts, but for now the points I'd like to leave you with are the following: You shouldn't be spawning something expensive like a thread for each incoming connection. It eats up various types of server resources. You shouldn't rely on thread pools to protect you from resource exhaustion, because they can fall victim to the slowloris attack. Gevent greenlets are lightweight enough that you can spawn one for each connection, and you don't have to rely on a pool (which can become exhausted in a slowloris type attack). So what do you think? Have I convinced you? I'd love to hear your reaction in the comments below!

we wanted to set up a nightly integration build with our projects so that we could run unit and integration tests on the latest version of our applications and their underlying libraries. we have a number of libraries that are shared across multiple projects and we wanted this build to run every night and use the latest versions of those libraries even if our applications had a specific release version defined in their maven pom file. in this way we would be alerted early if someone added a change to one of the dependency libraries that could potentially break an application when the developer upgraded the dependent library in a future version of the application. the chart below illustrates our dependencies between our libraries and our applications. updating versions nightly both the crossdock-shared and messaging-shared libraries depend on the siesta framework library. the crossdock web service and crossdockmessaging applications both depend on the crossdock-shared and messaging-shared libraries. because of the dependency structure, we wanted the siestaframework library built first. the crossdock-shared and messaging-shared libraries could be built in parallel, but we didn’t want the builds for the crossdock web service and crossdockmessaging applications to begin until all the libraries had finished building. we also wanted the nightly build to tag a subversion with the build date as well as upload the artifact to our nexus “nightly build” repository. the resulting artifact would look something like siestaframework-20120720.jar also as i had mentioned, even though the crossdockmessaging app may specify in its pom file it depends on version 5.0.4 of the siestaframework library. for the purposes of the nightly build, we wanted it to use the freshly built siestaframework-nightly-20120720.jar version of the library. the first problem to tackle was getting the current date into the project’s version number. for this i started with the jenkins zentimestamp plugin . with this plugin the format of jenkin’s build_id timestamp can be changed. i used this to specify using the format of yyyymmdd for the timestamp. the next step was to get the timestamp into the version number of the project. i was able to accomplish this by using the maven versions plugin. one thing the versions plugin can do is allow you to dynamically override the version number in the pom file at build time. the code snippet from the siestaframework’s pom file is below. org.codehaus.mojo versions-maven-plugin 1.3.1 at this point the jenkins job can be configured to invoke the “versions;set” goal, passing in the new version string to use. the ${build_id} jenkins variable will have the newly formatted date string. this will produce an artifact with the name siestaframework-nightly-20120720.jar uploading artifacts to a nightly repository since this job needed to upload the artifact to a different repository from our release repository that's defined in our project pom files, the “altdeploymentrepository” property was used to pass in the location of the nightly repository. the deployment portion of the siestaframework job specifies the location of the nightly repository where ${lynden_nightly_repo} is a jenkins variable containing the nightly repo url. tagging subversion finally, the jenkins subversion tagging plugin was used to tag svn if the project was successfully built. the plugin provides a post-build action for the job with the configuration section shown below. dynamically updating dependencies so now that the main project is set up, the dependent projects are set up in a similar way, but need to be configured to use the siestaframework-nightly-20120720 of the dependency rather than whatever version they currently have specified in their pom file. this can be accomplished by changing the pom to use a property for the version number of the dependency. for example, if the snippet below was the original pom file— com.lynden siestaframework 5.0.1 —changing it to the following would allow the siestaframework version to be set dynamically: 5.0.1 com.lynden siestaframework ${siesta.version} this version can then be overriden by the jenkins job. the example below shows the jenkins configuration for the crossdock-shared build. enforcing build order the final step in this process is setting up a structure to enforce the build order of the projects. the dependencies are set up in such a way that siestaframework needs to be built first, and the crossdock-shared and messaging-shared libraries can be run concurrently once siestaframework finishes. the crossdock web service and crossdockmessaging application jobs can be run concurrently, too, but not until after both shared libraries have finished. setting up the crossdock-shared and messaging-shared jobs to be built after the siestaframework finishes is pretty straightforward. in the jenkins job configuration for both the shared libraries, the following build trigger is added: to satisfy the requirement that the apps build only after all libraries have built, i enlisted the help of the join plugin . the join plugin can be used to execute a job once all “downstream” jobs have completed. what does this mean exactly? looking at the diagram below, the crossdock-shared and the messaging-shared jobs are “downstream” from the siestaframework job. once both of these jobs complete, a join trigger can be used to start other jobs. in this case, rather than having the join trigger kick off other app jobs directly, i created a dummy join job. in this way, as we add more application builds, we don’t need to keep modifying the siestaframework job with the new application job we just added. to illustrate the configuration, siestaframework has a new post-build action (below): join-build is a jenkins job i configured that does not do anything when executed. then our crossdock web service and crossdockmessaging applications define their builds to trigger as soon as join-build has completed. in this way we are able to run builds each night that will update to the latest version of our dependencies as well as tag svn and archive the binaries to nexus. i’d love to hear feedback from anyone who is handling nightly builds via jenkins, and how they have handled the configuration and build issues.

I’ve been playing with Hive recently and liking what I’ve found. In theory at least it provides a very nice, simple way of getting into analysing large data sets. To make it even easier to show other people what you’re up to Hive has a nascent web interface with a little documentation on the wiki On the one hand it’s rather simple at this point, but that should be easily enought to prettify given a bit of time. The bigger problem was getting it working in the first place. What follows worked for me using the latest cloudera packages on debian testing. I’m assuming you already have Hive and Hadoop installed, the basic packages worked fine for me here. Next up you’ll need the JDK (not just the JRE) as their is some compilation that will go on the first time you run the web interface. apt-get install ant sun-java6-jdk Next up I had to modify the installed /etc/hive/conf/hive-site.xml file as follows: I changed this: hive.metastore.uris file:///var/lib/hivevar/metastore/metadb/ Comma separated list of URIs of metastore servers. The first server that can be connected to will be used. To this. Note the hivevar path doesn’t exist so I’m not sure if this was a typo in the source. hive.metastore.uris file:///var/lib/hive/var/metastore/metadb/ Comma separated list of URIs of metastore servers. The first server that can be connected to will be used. I also change the following section regarding the metastore name: javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=/var/lib/hive/metastore/${user.name}_db;create=true JDBC connect string for a JDBC metastore To this, with a fixed name. When using the above confirguration the file was actually called ${user.name} rather than my username being subsituted in. Elsewhere this seems to work fine. javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true JDBC connect string for a JDBC metastore I’m not convinced the above two changes are needed but have left them here just in case. The main tricky part is making sure a load of environment variables are correctly set. The following worked for me: export ANT_LIB=/usr/share/ant/lib export HIVE_HOME=/usr/lib/hive export HADOOP_HOME=/usr/lib/hadoop export PATH=$PATH:$HADOOP_HOME/bin export JAVA_HOME=/usr/lib/jvm/java-6-sun All being well that should allow you to run the hive command with the web interface like so: hive --service hwi That should bring up a webserver on port 9999 where you should see something similar to the screenshot above.

Transaction management is a subject that is generally left to the tender care of a senior developer (or architect). Given the messages coming from some actors of the JavaEE community that with newer versions of JavaEE you don't need Spring anymore, I was interested in some fact-checking on how transaction management was handled in both technologies. Note: these messages were already sent one year and a half ago and prompted me to write this article. Transaction demarcation Note that although both technologies provide programmatic transaction demarcation (start transaction then commit/rollback), we'll focus on declarative demarcation since they are easier to use in real life. In EJB3, transactions are delimited by the @TransactionAttribute annotation. The annotation can be set on the class, in which case every method will have the transactional attribute, or per method. Annotation at the method level can override the annotation at the class level. Annotations are found automatically by the JavaEE-compliant application server. In Spring, the former annotation is replaced by the proprietary @Transactional annotation. The behaviour is exactly the same (annotation at method/class level and possibility of overriding). Annotations can only be found when the Spring bean definition file contains the http://www.springframework.org/schema/tx namespace as well as the following snippet: Alternatively, both technologies provide an orthogonal way to set transaction demarcation: in EJB3, one can use the EJB JAR deployment descriptor (ejb-jar.xml) while in Spring, any Spring configuration file will do. Transactions propagation In EJB3, there are exactly 6 possible propagation values: MANDATORY, REQUIRED (default), REQUIRES_NEW, SUPPORTS, NOT_SUPPORTED and NEVER. Spring adds support for NESTED (see below). Nested transactions Nested transactions are transactions that are started then commited/rollbacked during the execution of the root transaction. Nested transaction results are limited to the scope of this transaction only (it has no effect on the umbrella transaction). Nested transactions are not allowed in EJB3; they are in Spring. Read-only transaction Read-only transactions are best used with certain databases or ORM frameworks like Hibernate. In the latter case, Hibernate optimizes sessions so that they never flush (i.e. never push changes from the cache to the underlying database). I haven't found a way (yet) to qualify a transaction as read-only in EJB3 (help welcome). In Spring, @Transactional has a readOnly parameter to implement read-only transactions: @Transactional(readOnly = true) Local method calls In local method calls, a bean's method A() calls another method B() on the same instance. The expected behavior should be that transaction attributes of method B() is taken into account. It's not the case in neither technologies, but both offer some workaround. In EJB3, you have to inject an instance of the same class and call the method on this instance in order to use transaction attributes. For example: @Stateless public MyServiceBean implements MyServiceLocal { @EJB private MyServiceLocal service; @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public void A() { service.B(); } @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) public void B() { ... } } In Spring, by default, transaction management is handled through pure Java proxies. If you want to have transaction management in local method calls, you'll have to turn on AspectJ. This is easily done in the Spring beans definition file (but beware the side-effects, see the Spring documentation for more details): Exception handling and rollback Exception handling is the area with the greater differences between Spring and EJB3. In EJB3, only runtime exceptions thrown from a method rollback a transaction delimited around this method by default. In order to mimic this behavior for checked exceptions, you have to annotate the exception class with @ApplicationException(rollback = true). Likewise, if you wish to discard this behavior for runtime exceptions, annotate your exception class with @ApplicationException(rollback = false). This has the disadvantage of not being able to use the same exception class to rollback in a method and to still to commit despite the exception in another method. In order to achieve this, you have to manage your transaction programmatically: @Stateless public MyServiceBean implements MyServiceLocal { @Resource private SessionContext context; public void A() { try { ... } catch (MyException e) { context.setRollbackOnly(); } } } In Spring, runtime exceptions also cause transaction rollback. In order to change this behavior, use the rollbackFor or noRollbackFor attributes of @Transactional: public MyServiceImpl { @Resource private SessionContext context; @Transactional(rollbackFor = MyException.class) public void A() { ... } } Conclusion There's no denying that JavaEE has made giant steps in the right direction with its 6th version. And yet, small details keep pointing me toward Spring. If you know only about one - Spring or JavaEE 6, I encourage you to try "the other" and see for yourself which one you're more comfortable with. To go further: Transaction strategies: Understanding transaction pitfalls Spring's documentation in regard to transaction management Oracle's documentation for EJB 3.0 transaction management