martin fowler has written a really good article describing not only the disruptor , but also how it fits into the architecture at lmax . this gives some of the context that has been missing so far, but the most frequently asked question is still "what is the disruptor?". i'm working up to answering that. i'm currently on question number two: "why is it so fast?". these questions do go hand in hand, however, because i can't talk about why it's fast without saying what it does, and i can't talk about what it is without saying why it is that way. so i'm trapped in a circular dependency. a circular dependency of blogging. to break the dependency, i'm going to answer question one with the simplest answer, and with any luck i'll come back to it in a later post if it still needs explanation: the disruptor is a way to pass information between threads. as a developer, already my alarm bells are going off because the word "thread" was just mentioned, which means this is about concurrency, and concurrency is hard. concurrency 101 imagine two threads are trying to change the same value. case one: thread 1 gets there first: the value changes to "blah" then the value changes to "blahy" when thread 2 gets there. case two: thread 2 gets there first: the value changes to "fluffy" then the value changes to "blah" when thread 1 gets there. case three: thread 1 interrupts thread 2: thread 2 gets the value "fluff" and stores it as myvalue thread 1 goes in and updates value to "blah" then thread 2 wakes up and sets the value to "fluffy". case three is probably the only one which is definitely wrong, unless you think the naive approach to wiki editing is ok ( google code wiki, i'm looking at you...). in the other two cases it's all about intentions and predictability. thread 2 might not care what's in value, the intention might be to append "y" to whatever is in there regardless. in this circumstance, cases one and two are both correct. but if thread 2 only wanted to change "fluff" to "fluffy", then both cases two and three are incorrect. assuming that thread 2 wants to set the value to "fluffy", there are some different approaches to solving the problem. approach one: pessimistic locking (does the "no entry" sign make sense to people who don't drive in britain?) the terms pessimistic and optimistic locking seem to be more commonly used when talking about database reads and writes, but the principal applies to getting a lock on an object. thread 2 grabs a lock on entry as soon as it knows it needs it and stops anything from setting it. then it does its thing, sets the value, and lets everything else carry on. you can imagine this gets quite expensive, with threads hanging around all over the place trying to get hold of objects and being blocked. the more threads you have, the more chance that things are going to grind to a halt. approach two: optimistic locking in this case thread 2 will only lock entry when it needs to write to it. in order to make this work, it needs to check if entry has changed since it first looked at it. if thread 1 came in and changed the value to "blah" after thread 2 had read the value, thread 2 couldn't write "fluffy" to the entry and trample all over the change from thread 1. thread 2 could either re-try (go back, read the value, and append "y" onto the end of the new value), which you would do if thread 2 didn't care what the value it was changing was; or it could throw an exception or return some sort of failed update flag if it was expecting to change "fluff" to "fluffy". an example of this latter case might be if you have two users trying to update a wiki page, and you tell the user on the other end of thread 2 they'll need to load the new changes from thread 1 and then reapply their changes. potential problem: deadlock locking can lead to all sorts of issues, for example deadlock. imagine two threads that need access to two resources to do whatever they need to do: if you've used an over-zealous locking technique, both threads are going to sit there forever waiting for the other one to release its lock on the resource. that's when you reboot windows your computer. definite problem: locks are sloooow... the thing about locks is that they need the operating system to arbitrate the argument. the threads are like siblings squabbling over a toy, and the os kernel is the parent that decides which one gets it. it's like when you run to your dad to tell him your sister has nicked the transformer when you wanted to play with it - he's got bigger things to worry about than you two fighting, and he might finish off loading the dishwasher and putting on the laundry before settling the argument. if you draw attention to yourself with a lock, not only does it take time to get the operating system to arbitrate, the os might decide the cpu has better things to do than servicing your thread. the disruptor paper talks about an experiment we did. the test calls a function incrementing a 64-bit counter in a loop 500 million times. for a single thread with no locking, the test takes 300ms. if you add a lock (and this is for a single thread, no contention, and no additional complexity other than the lock) the test takes 10,000ms. that's, like, two orders of magnitude slower. even more astounding, if you add a second thread (which logic suggests should take maybe half the time of the single thread with a lock) it takes 224,000ms. incrementing a counter 500 million times takes nearly a thousand times longer when you split it over two threads instead of running it on one with no lock. concurrency is hard and locks are bad i'm just touching the surface of the problem, and obviously i'm using very simple examples. but the point is, if your code is meant to work in a multi-threaded environment, your job as a developer just got a lot more difficult: naive code can have unintended consequences. case three above is an example of how things can go horribly wrong if you don't realise you have multiple threads accessing and writing to the same data. selfish code is going to slow your system down. using locks to protect your code from the problem in case three can lead to things like deadlock or simply poor performance. this is why many organisations have some sort of concurrency problems in their interview process (certainly for java interviews). unfortunately it's very easy to learn how to answer the questions without really understanding the problem, or possible solutions to it. how does the disruptor address these issues? for a start, it doesn't use locks. at all. instead, where we need to make sure that operations are thread-safe (specifically, updating the next available sequence number in the case of multiple producers ), we use a cas (compare and swap/set) operation. this is a cpu-level instruction, and in my mind it works a bit like optimistic locking - the cpu goes to update a value, but if the value it's changing it from is not the one it expects, the operation fails because clearly something else got in there first. note this could be two different cores rather than two separate cpus. cas operations are much cheaper than locks because they don't involve the operating system, they go straight to the cpu. but they're not cost-free - in the experiment i mentioned above, where a lock-free thread takes 300ms and a thread with a lock takes 10,000ms, a single thread using cas takes 5,700ms. so it takes less time than using a lock, but more time than a single thread that doesn't worry about contention at all. back to the disruptor - i talked about the claimstrategy when i went over the producers . in the code you'll see two strategies, a singlethreadedstrategy and a multithreadedstrategy. you could argue, why not just use the multi-threaded one with only a single producer? surely it can handle that case? and it can. but the multi-threaded one uses an atomiclong (java's way of providing cas operations), and the single-threaded one uses a simple long with no locks and no cas. this means the single-threaded claim strategy is as fast as possible, given that it knows there is only one producer and therefore no contention on the sequence number. i know what you're thinking: turning one single number into an atomiclong can't possibly have been the only thing that is the secret to the disruptor's speed. and of course, it's not - otherwise this wouldn't be called "why it's so fast (part one )". but this is an important point - there's only one place in the code where multiple threads might be trying to update the same value. only one place in the whole of this complicated data-structure-slash-framework. and that's the secret. remember everything has its own sequence number? if you only have one producer then every sequence number in the system is only ever written to by one thread. that means there is no contention. no need for locks. no need even for cas. the only sequence number that is ever written to by more than one thread is the one on the claimstrategy if there is more than one producer. this is also why each variable in the entry can only be written to by one consumer . it ensures there's no write contention, therefore no need for locks or cas. back to why queues aren't up to the job so you start to see why queues, which may implemented as a ring buffer under the covers, still can't match the performance of the disruptor. the queue, and the basic ring buffer , only has two pointers - one to the front of the queue and one to the end: if more than one producer wants to place something on the queue, the tail pointer will be a point of contention as more than one thing wants to write to it. if there's more than one consumer, then the head pointer is contended, because this is not just a read operation but a write, as the pointer is updated when the item is consumed from the queue. but wait, i hear you cry foul! because we already knew this, so queues are usually single producer and single consumer (or at least they are in all the queue comparisons in our performance tests). there's another thing to bear in mind with queues/buffers. the whole point is to provide a place for things to hang out between producers and consumers, to help buffer bursts of messages from one to the other. this means the buffer is usually full (the producer is out-pacing the consumer) or empty (the consumer is out-pacing the producer). it's rare that the producer and consumer will be so evenly-matched that the buffer has items in it but the producers and consumers are keeping pace with each other. so this is how things really look. an empty queue: ...and a full queue: the queue needs a size so that it can tell the difference between empty and full. or, if it doesn't, it might determine that based on the contents of that entry, in which case reading an entry will require a write to erase it or mark it as consumed. whichever implementation is chosen, there's quite a bit of contention around the tail, head and size variables, or the entry itself if a consume operation also includes a write to remove it. on top of this, these three variables are often in the same cache line , leading to false sharing . so, not only do you have to worry about the producer and the consumer both causing a write to the size variable (or the entry), updating the tail pointer could lead to a cache-miss when the head pointer is updated because they're sat in the same place. i'm going to duck out of going into that in detail because this post is quite long enough as it is. so this is what we mean when we talk about "teasing apart the concerns" or a queue's "conflated concerns". by giving everything its own sequence number and by allowing only one consumer to write to each variable in the entry, the only case the disruptor needs to manage contention is where more than one producer is writing to the ring buffer. in summary the disruptor a number of advantages over traditional approaches: no contention = no locks = it's very fast. having everything track its own sequence number allows multiple producers and multiple consumers to use the same data structure. tracking sequence numbers at each individual place (ring buffer, claim strategy, producers and consumers), plus the magic cache line padding , means no false sharing and no unexpected contention. from http://mechanitis.blogspot.com/2011/07/dissecting-disruptor-why-its-so-fast.html

As you probably saw by now, JetBrains just announced that they are working on a brand new statically typed JVM language called Kotlin. I am planning to write a post to evaluate how Kotlin compares to the other existing languages, but first, I’d like to take a slightly different angle and try to answer a question I have already seen asked several times: what’s the point? We already have quite a few JVM languages, do we need any more? Here are a few reasons that come to mind. 1) Coolness New languages are exciting! They really are. I’m always looking forward to learning new languages. The more foreign they are, the more curious I am, but the languages I really look forward to discovering are the ones that are close to what I already know but not identical, just to find out what they did differently that I didn’t think of. Allow me to make a small digression in order to clarify my point. Some time ago, I started learning Japanese and it turned out to be the hardest and, more importantly, the most foreign natural language I ever studied. Everything is different from what I’m used to in Japanese. It’s not just that the grammar, the syntax and the alphabets are odd, it’s that they came up with things that I didn’t even think would make any sense. For example, in English (and many other languages), numbers are pretty straightforward and unique: one bag, two cars, three tickets, etc… Now, did it ever occur to you that a language could allow several words to mean “one”, and “two”, and “three”, etc…? And that these words are actually not arbitrary, their usage follows some very specific rules. What could these rules be? Well, in Japanese, what governs the word that you pick is… the shape of the object that you are counting. That’s right, you will use a different way to count if the object is long, flat, a liquid or a building. Mind-boggling, isn’t it? Here is another quick example: in Russian, each verb exists in two different forms, which I’ll call A and B to simplify. Russian doesn’t have a future tense, so when you want to speak at the present tense, you’ll conjugate verb A in the present, and when you want the future, you will use the B form… in the present tense. It’s not just that you need to learn two verbs per verb, you also need to know which one is which if you want to get your tenses right. Oh and these forms also have different meanings when you conjugate them in past tenses. End of digression. The reason why I am mentioning this is because this kind of construct bends your mind, and this goes for natural languages as much as programming languages. It’s tremendously exciting to read new syntaxes this way. For that reason alone, the arrival of new languages should be applauded and welcome. Kotlin comes with a few interesting syntactic innovations of its own, which I’ll try to cover in a separate post, but for now, I’d like to come back to my original point, which was to give you reasons why you should be excited about Kotlin, so let’s keep going down the list. 2) IDE support None of the existing JVM languages (Groovy, Scala, Fantom, Gosu, Ceylon) have really focused much on the tooling aspect. IDE plug-ins exist for each of them, all with varying quality, but they are all an afterthought, and they suffer from this oversight. The plug-ins are very slow to mature, they have to keep up with the internals of a compiler that’s always evolving and which, very often, doesn’t have much regards for the tools built on top of it. It’s a painful and frustrating process for tool creators and tool users alike. With Kotlin, we have good reasons to think that the IDE support will be top notch. JetBrains is basically announcing that they are building the compiler and the IDEA support in lockstep, which is a great way to guarantee that the language will work tremendously well inside IDEA, but also that other tools should integrate nicely with it as well (I’m rooting for a speedy Eclipse plug-in, obviously). 3) Reified generics This is a pretty big deal. Not so much for the functionality (I’ll get back to this in the next paragraph) but because this is probably the very first time that we see a JVM language with true support for reified generics. This innovation needs to be saluted. Correction from the comments: Gosu has reified generics Having said that, I don’t feel extremely excited by this feature because overall, I think that reified generics come at too high a price. I’ll try to dedicate a full blog post to this topic alone, because it deserves a more thorough treatment. 4) Commercial support JetBrains has a very clear financial interest in seeing Kotlin succeed. It’s not just that they are a commercial entity that can put money behind the development of the language, it’s also that the success of the language would most likely mean that they will sell more IDEA licenses, and this can also turn into an additional revenue stream derived from whatever other tools they might come up with that would be part of the Kotlin ecosystem. This kind of commercial support for a language was completely unheard of in the JVM world for fifteen years, and suddenly, we have two instances of it (Typesafe and now JetBrains). This is a good sign for the JVM community. 5) Still no Java successors Finally, the simple truth is that we still haven’t found any credible Java successor. Java still reigns supreme and is showing no sign of giving away any mindshare. Pulling numbers out of thin air, I would say that out of all the code currently running on the JVM today, maybe 94% of it is in Java, 3% is in Groovy and 1% is in Scala. This 94% figure needs to go down, but so far, no language has stepped up to whittle it down significantly. What will it take? Obviously, nothing that the current candidates are offering (closures, modularity, functional features, more concise syntax, etc…) has been enough to move that needle. Something is still missing. Could the missing piece be “stellar IDE support” or “reified generics”? We don’t know yet because no contender offers any of these features, but Kotlin will, so we will soon know. Either way, I am predicting that we will keep seeing new JVM languages pop up at a regular pace until one finally claims the prize. And this should be cause for rejoicing for everyone interested in the JVM ecosystem. So let’s cheer for Kotlin and wish JetBrains the best of luck with their endeavor. I can’t wait to see what will come out of this. Oh, and secretly, I am rooting for Eclipse to start working on their own JVM language too, obviously. From http://beust.com/weblog/2011/07/20/five-reasons-why-should-rejoice-about-kotlin/

this tutorial will walk you through out how to use the ext js 4 file upload field in the front end and spring mvc 3 in the back end. this tutorial is also an update for the tutorial ajax file upload with extjs and spring framework , implemented with ext js 3 and spring mvc 2.5. ext js file upload form first, we will need the ext js 4 file upload form. this one is the same as showed in ext js 4 docs . ext.onready(function(){ ext.create('ext.form.panel', { title: 'file uploader', width: 400, bodypadding: 10, frame: true, renderto: 'fi-form', items: [{ xtype: 'filefield', name: 'file', fieldlabel: 'file', labelwidth: 50, msgtarget: 'side', allowblank: false, anchor: '100%', buttontext: 'select a file...' }], buttons: [{ text: 'upload', handler: function() { var form = this.up('form').getform(); if(form.isvalid()){ form.submit({ url: 'upload.action', waitmsg: 'uploading your file...', success: function(fp, o) { ext.msg.alert('success', 'your file has been uploaded.'); } }); } } }] }); }); html page then in the html page, we will have a div where we are going to render the ext js form. this page also contains the required javascript imports extjs/resources/css/ext-all.css" /> click on "browse" button (image) to select a file and click on upload button fileupload bean we will also need a fileupload bean to represent the file as a multipart file: package com.loiane.model; import org.springframework.web.multipart.commons.commonsmultipartfile; /** * represents file uploaded from extjs form * * @author loiane groner * http://loiane.com * http://loianegroner.com */ public class fileuploadbean { private commonsmultipartfile file; public commonsmultipartfile getfile() { return file; } public void setfile(commonsmultipartfile file) { this.file = file; } } file upload controller then we will need a controller. this one is implemented with spring mvc 3. package com.loiane.controller; import org.springframework.stereotype.controller; import org.springframework.validation.bindingresult; import org.springframework.validation.objecterror; import org.springframework.web.bind.annotation.requestmapping; import org.springframework.web.bind.annotation.requestmethod; import org.springframework.web.bind.annotation.responsebody; import com.loiane.model.extjsformresult; import com.loiane.model.fileuploadbean; /** * controller - spring * * @author loiane groner * http://loiane.com * http://loianegroner.com */ @controller @requestmapping(value = "/upload.action") public class fileuploadcontroller { @requestmapping(method = requestmethod.post) public @responsebody string create(fileuploadbean uploaditem, bindingresult result){ extjsformresult extjsformresult = new extjsformresult(); if (result.haserrors()){ for(objecterror error : result.getallerrors()){ system.err.println("error: " + error.getcode() + " - " + error.getdefaultmessage()); } //set extjs return - error extjsformresult.setsuccess(false); return extjsformresult.tostring(); } // some type of file processing... system.err.println("-------------------------------------------"); system.err.println("test upload: " + uploaditem.getfile().getoriginalfilename()); system.err.println("-------------------------------------------"); //set extjs return - sucsess extjsformresult.setsuccess(true); return extjsformresult.tostring(); } ext js form return some people asked me how to return something to the form to display a message to the user. we can implement a pojo with a success property. the success property is the only thing ext js needs as a return: package com.loiane.model; /** * a simple return message for ext js * * @author loiane groner * http://loiane.com * http://loianegroner.com */ public class extjsformresult { private boolean success; public boolean issuccess() { return success; } public void setsuccess(boolean success) { this.success = success; } public string tostring(){ return "{success:"+this.success+"}"; } } spring config don’t forget to add the multipart file config in the spring config file: nullpointerexception i also got some questions about nullpointerexception. make sure the fileupload field name has the same name as the commonsmultipartfile property in the fileuploadbean class: extjs : { xtype: 'filefield', name: 'file', fieldlabel: 'file', labelwidth: 50, msgtarget: 'side', allowblank: false, anchor: '100%', buttontext: 'select a file...' } java: public class fileuploadbean { private commonsmultipartfile file; } these properties always have to match! you can still use the spring mvc 2.5 code with the ext js 4 code presented in this tutorial. download you can download the source code from my github repository (you can clone the project or you can click on the download button on the upper right corner of the project page): https://github.com/loiane/extjs4-file-upload-spring you can also download the source code form the google code repository: http://code.google.com/p/extjs4-file-upload-spring/ both repositories have the same source. google code is just an alternative. happy coding! from http://loianegroner.com/2011/07/extjs-4-file-upload-spring-mvc-3-example/

So far, in all my examples, I have been using the declarative way of creating an Android UI using XML. However, there could arise certain situations when you may have to create UI programmatically. Sincere advice would be to avoid such a design since android has a wonderful architecture where the UI and the program are well separated. However, for those few exceptional cases where we may need too… here is how we do it. Every single view or viewgroup element has an equivalent java class in the SDK. The structure and naming of the classes and methods is very similar to the XML vocabulary that we are used to so far. Let us start with a LinearLayout. How would we declare it in an XML? This just contains a TextView embedded in a LinearLayout. A very trivial example. But serves the purpose intended. Let me show how almost every single element here corresponds to a class or a method call in the class. So the equivalent code in the onCreate(…) method of an activity would be like this: super.onCreate(savedInstanceState); lLayout = new LinearLayout(this); lLayout.setOrientation(LinearLayout.VERTICAL); //-1(LayoutParams.MATCH_PARENT) is fill_parent or match_parent since API level 8 //-2(LayoutParams.WRAP_CONTENT) is wrap_content lLayout.setLayoutParams(new LayoutParams( LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT)); tView = new TextView(this); tView.setText("Hello, This is a view created programmatically! " + "You CANNOT change me that easily :-)"); tView.setLayoutParams(new LayoutParams( LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT)); lLayout.addView(tView); setContentView(lLayout); Like this any layout view can be created. But from this small example you can notice two outstanding things – very tedious to code for every attribute of the view. And any simple change in the view, you need to change the code, compile, deploy and only then you see the effect of the change – unlike in a layout editor. You can download the sample code here.

Sometimes there is too much logic to deal with in a single class. You tried extracting methods, but they are so many that the design is still complex to understand. The next step in the refactoring quest is Extract Class, the creation of a new class whose objects will be referenced from the original class. Fields and methods may be moved in the new class, in order for the original to get smaller and more manageable. Why class inflation happens? This refactoring is always caused by classes growing in responsibilities. My personal hypothesis is that we as developers have a bias for the add field|method operations preferring it to the add class. Usually, creating a new class also mean adding an entire new file (hopefully) and more design considerations like its namespace and name. The mental cost for the developer is heavier, but the results are often better than for smaller extractions, as classes can be reused independently; extracted methods are instead clustered together. Moreover, our libraries (e.g. ORMs such as Doctrine 2) reinforce this bias by making significatively more difficult to extract a Value Object (should you serialize it? write a custom DBAL type?) or even another Entity (should I link to it with a @OneToOne? @OneToMany? Which cascade options will work? Which constraints the relationships has?) By the way, this solution is a manifestation of composition over inheritance in the refactoring realm (while there are also other options where the class is made smaller by introducing superclasses instead of an unrelated type.) What are the signs it's time to extract a class? You may encounter a subset of methods and fields that cluster together: for example, they are identified by a prefix; or they have have a temporal coupling which makes them change together faster or slower than the other fields. They may be of similar types (scalar or superclass). Another option to target higher cohesion is to simply see which fields are used together by each method in the class. Fowler's suggestion is to try removing each field (conceptually), and think about which other fields become useless. Repeat this and you'll find the subsets of fields to extract if they exist. Steps Divide responsibilities in source class and extracted class: fields and methods should be classified to one or the two targets. This is true for public and private members, but the latter may change scope to public to be visible from the source class. Create a new class, and check the names of the source and the new one. In the extracted class, you decide the name on the fly; in the source class, you have to change the old name if it's no longer applicable (and later, also the name of references in the system to its objects). It may be the case that the extracted class steals the name of the source one. Place a field in the source class referencing an object (or more) of the extracted class. The field may be initialized in the constructor, and also injected with a constructor parameter if the change is not too invasive. Move Field iteratively from the source to the extracted class. Move Method iteratively. If you bundle steps 4 and 5, you'll be faster, but the point is you should be able to go in smaller steps when it is necessary. Likewise, TDD is taught with baby steps because it gives you the ability to make them when required: everyone is capable of cutting a giant piece of code and fiddling with it for hours until it works again. But that involves a rewriting part, not only refactoring. Reduce the interfaces that each class exposes. Often the extracted one only needs public methods for what the original class uses, while the original class maintains its old protocol to avoiding ripple effects towards the rest of the object graph. In fact, it's not even said that you should expose the extracted class. In many cases, you don't have to; but you'll be able to use it independently by creating other objects, while there is often no need to expose this particular object, composed by the source class (and the Law of Demeter says you shouldn't.) Tests should be executable after each movement of fields or methods. Example In the example, we pass from an initial state where formatting and HTML logic is crammed into the same class: assertEquals('10,000.00', $moneyAmount->toHtml()); } } class MoneyAmount { /** * @param int $amount */ public function __construct($amount) { $this->amount = $amount; } public function toHtml() { $amount = $this->amount; $formatted = ''; while (strlen($amount) > 3) { $cut = strlen($amount) % 3; $cut = $cut == 0 ? 3 : $cut; $formatted .= substr($amount, 0, $cut) . ','; $amount = substr($amount, $cut); } $formatted .= $amount . '.00'; $html = "$formatted"; return $html; } } To two separated classes, one modelling the logical amount and its formatting, one taking care of printing HTML tags. assertEquals('10,000.00', $moneyAmount->toHtml()); } } class MoneySpan { /** * @param int $amount */ public function __construct(MoneyAmount $amount) { $this->amount = $amount; } public function toHtml() { $html = '' . $this->amount->format() . ''; return $html; } } class MoneyAmount { private $amount; public function __construct($amount) { $this->amount = $amount; } public function format() { $amount = $this->amount; $formatted = ''; while (strlen($amount) > 3) { $cut = strlen($amount) % 3; $cut = $cut == 0 ? 3 : $cut; $formatted .= substr($amount, 0, $cut) . ','; $amount = substr($amount, $cut); } return $formatted . $amount . '.00'; } } You can see the four intermediate steps in the Github history of the file.

In Spring Roo in Action, Chapter 3, I discuss how Roo automatically executes the Bean Validators when persisting a live entity. However, when running unit tests, we don't have a live entity at all, nor do we have a Spring container - so how can we exercise the validation without actually hitting our Roo application and the database? The following post is ancillary material from the upcoming book Spring Roo in Action, by Ken Rimple and Srini Penchikala, with Gordon Dickens. You can purchase the MEAP edition of the book, and participate in the author forum, at www.manning.com/rimple. The answer is that we have to bootstrap the validation framework within the test ourselves. We can use the CourseDataOnDemand class's getNewTransientEntityName method to generate a valid, transient JPA entity. Then, we can: Mock static entity methods, such as findById, to bring back pre-fabricated class instances of your entity Initialize the validation engine, bootstrapping a JSR-303 bean validation framework engine, and perform validation on your entity Set any appropriate properties to apply to a particular test condition Initialize a test instance of the entity validator and assert the appropriate validation results are returned The concept in action... Given a Student entity with the following definition: @RooEntity @RooJavaBean @RooToString public class Student { @NotNull private String emergencyContactInfo; ... } The listing below shows a unit test method that ensures the NotNull validation fires against missing emergency contact information on the Student entity: @Test public void testStudentMissingEmergencyContactValidation() { // setup our test data StudentDataOnDemand dod = new StudentDataOnDemand(); // tell the mock to expect this call Student.findStudent(1L); // tell the mocking API to expect a return from the prior call in the form of // a new student from the test data generator, dod AnnotationDrivenStaticEntityMockingControl.expectReturn( dod.getNewTransientStudent(0)); // put our mock in playback mode AnnotationDrivenStaticEntityMockingControl.playback(); // Setup the validator API in our unit test LocalValidatorFactoryBean validator = new LocalValidatorFactoryBean(); validator.afterPropertiesSet(); // execute the call from the mock, set the emergency contact field // to an invalid value Student student = Student.findStudent(1L); student.setEmergencyContactInfo(null); // execute validation, check for violations Set> violations = validator.validate(student, Default.class); // do we have one? Assert.assertEquals(1, violations.size()); // now, check the constraint violations to check for our specific error ConstraintViolation violation = violations.iterator().next(); // contains the right message? Assert.assertEquals("{javax.validation.constraints.NotNull.message}", violation.getMessageTemplate()); // from the right field? Assert.assertEquals("emergencyContactInfo", violation.getPropertyPath().toString()); } Analysis The test starts with a declaration of a StudentOnDemand object, which we'll use to generate our test data. We'll get into the more advanced uses of the DataOnDemand Framework later in the chapter. For now, keep in mind that we can use this class to create an instance of an Entity, with randomly assigned, valid data. We then require that the test calls the Student.findStudent method, passing it a key of 1L. Next, we'll tell the entity mocking framework that the call should return a new transient Student instance. At this point, we've defined our static mocking behavior, so we'll put the mocking framework into playback mode. Next, we issue the actual Student.findById(1L) call, this time storing the result as the member variable student. This call will trip the mock, which will return a new transient instance. We then set the emergencyContactInfo field to null, so that it becomes invalid, as it is annotated with a @NotNull annotation. Now we are ready to set up our bean validation framework. We create a LocalValidatorFactoryBean instance, which will boot the Bean Validation Framework in the afterPropertiesSet() method, which is defined for any Spring Bean implementing InitializingBean. We must call this method ourselves, because Spring is not involved in our unit test. Now we're ready to run our validation and assert the proper behavior has occurred. We call our validator's validate method, passing it the student instance and the standard Default validation group, which will trigger validation. We'll then check that we only have one validation failure, and that the message template for the error is the same as the one for the @NotNull validation. We also check to ensure that the field that caused the validation was our emergencyContactInfo field. In our answer callback, we can launch the Bean Validation Framework, and execute the validate method against our entity. In this way, we can exercise our bean instance any way we want, and instead of persisting the entity, can perform the validation phase and exit gracefully. Caveats... There are a few things slightly wrong here. First of all, the Data on Demand classes actually use Spring to inject relationships to each other, which I've logged a bug against as ROO-2497. You can override the setup of the data on demand class and manually create the DoD of the referring one, which is fine. They have slated to work on this bug for Roo 1.2, so it should be fixed sometime in the next few months. Also, realize that this is NOT easy to do, compared to writing an integration test. However, this test runs markedly faster. If you have some sophisticated logic that you've attached to a @AssertTrue annotation, this is the way to test it in isolation. From http://www.rimple.com/tech/2011/7/17/testing-entity-validations-with-a-mock-entity-roo-in-action.html

The "WZL Gear Toolbox", by the Laboratory for Machine Tools and Production Engineering at RWTH Aachen University, represents a unified graphical user interface containing different simulation programs for gear applications. It enables the usage of the following simulations: manufacturing simulation "GearGenerator" process simulation "SPARTApro" process simulation "KegelSpan" tooth contact analysis "ZaKo3D" The uniform graphical user interface enables the user to realize the simulations and to analyze the calculation results in a comfortable way. Thus, the "WZL Gear Toolbox" enables the analysis of the running behavior of gears by means of tooth contact analysis of the manufacturing-related deviations from the generating grinding. The long-term goal of the uniform graphical user interface is to provide a software tool containing the whole production chain of gear manufacturing. The "WZL Gear Toolbox" is funded by the WZL Gear Research Circle. Screenshot

Learn more about Java collection performance in this post.

Recently I was asked by a customer how they could import VMs from an existing vSphere environment into a vCloud Director environment. This particular customer is aiming to pull VMs from an existing managed hosting business so they’ll use the third of three options that I’ve described below. For pulling VMs into vCD you basically have 3 options. 1) You can log into the vCD interface and upload VMs manually through the interface. This is done one by one. This is a good option if you’re an end user that doesn’t have vCC setup (option 2) or doesn’t want to go through the process of using the vCloud API (a more automated version of this option). The downside of this option for my customer is it would require additional storage since the VMs would first upload to a staging area (the vCloud Director transfer pool) and then copy to their destination. 2) You can setup vCloud Connector (vCC) in your existing vSphere environment and use that to move VMs from the vCenter Client interface into vCloud Director. This again is done one at a time. The benefit is you’re using your existing tool set (vCenter Client) to perform this operation. The down side is you’ll need to configure vCC for each user of the vCenter Client that wants to do this operation. This would be a viable option for my customer since they own the source vSphere environment. It still isn’t the most straightforward for them and it can’t be automated with an API call so this option is out. 3) If you own the vSphere environment like this customer does and you want to migrate that vSphere environment into a parallel vCD environment then you can just add the managing VC to the vCD environment and import VMs directly into vCD. This is the method that I’m going to detail below. If you’re ready to go about moving VMs into your vCD environment then the first thing you need to do is prepare to add your managing VC to your vCD environment. To do this you’ll need to deploy a vShield Manager instance. Even if you don’t use the security features of vSM you’ll still need to have one present so you can get through the add VC wizard in vCD. vSM comes as a virtual appliance so we’ll begin by just adding it through the deploy OVF wizard in VC. After vSM is deployed you can see it as a resource VM in our current vSphere environment. After we finish importing everything we can remove the vSM VM again. Now that we have vSM in place we can go and attach our managing VC to our vCD environment. Right now my test vCD environment already has one VC under its control as you can see here. Now we’re going to go through the wizard to add our second VC which is the one we want to import our VMs from. The wizard is pretty simple. First we give it our VC name and credentials. On the next screen we give the name and credentials to the vSM we deployed earlier. After the wizard completes you can see we have our old VC and our new VC under the control of our vCD environment. This does not preclude you from continuing to use your vSphere environment as you always have. It just now allows vCD to see and consume some of those resources such as the existing VMs. The next step is very important. We need to login to vCD as the system admin. This is because org admins don’t get to see the underlying vSphere resources and so they won’t be able to do the import straight from VC. Once we’re logged in as the system admin we can open up an organization to perform the import of VMs into that org. In my case the test organization is called MikeD. You can see the System tab in the screenshot below as well which indicates that we were initially logged in as the system admin and simply opened the organization in another tab. Now that we’re inside the organization of our choice we can click the import VC button on the top toolbar. The import vApp/VM wizard should appear and we can see a listing of all of the VMs in our vSphere environment. To import one it’s as simple as selecting the VM, giving it a name, telling vCD which orgVDC we would like to place it in and then hitting ok. There is one last choice to make and that’s whether or not to copy the VM to vCD which will put a new instance in the orgVDC but leave a copy on VC or move the VM which will delete the VM from the original VC environment and make a new copy in the vCD orgVDC resource destination. This choice is up to the person doing the migration and both choices have merit. A lot of people like copy because they have the original copy in place should a customer decide they need to continue using it or they just want it for backup. After you’re done with the wizard your VM will start to import. After a little while you’ll see the finished VM in the interface. Of course you’ll also see the in progress VM along with a task that’s running. As you can see the overall process for setting this up is not that labor intensive. It is a little labor intensive to go into the interface to do the import one VM at a time. Of course you could always use the vCloud API to do the imports once you have the vSphere environment added to vCD. All of this will allow current managed hosting providers to migrate their VMs to the new vCD based clouds they are creating. Hopefully this has been useful. As always if there are any comments or questions then just put them below and I’ll get back to you.

More than a few times recently we’ve wanted to use pattern matching on a collection of pairs/tuples and have run into trouble doing so. It’s easy enough if you don’t try and pattern match: > List(("Mark", 4), ("Charles", 5)).filter(pair => pair._2 == 4) res6: List[(java.lang.String, Int)] = List((Mark,4)) But if we try to use pattern matching: List(("Mark", 4), ("Charles", 5)).filter(case(name, number) => number == 4) We end up with this error: :1: error: illegal start of simple expression List(("Mark", 4), ("Charles", 5)).filter(case(name, number) => number == 4) It turns out that we can only use this if we pass the function to filter using {} instead of (): > List(("Mark", 4), ("Charles", 5)).filter { case(name, number) => number == 4 } res7: List[(java.lang.String, Int)] = List((Mark,4)) It was pointed out to me on the Scala IRC channel that the reason for the compilation failure has nothing to do with trying to do a pattern match inside a higher order function but that it’s not actually possible to use a case token without the {}. [23:16] mneedham: hey – trying to understand how pattern matching works inside higher order functions. Don’t quite get this code -> https://gist.github.com/1079110 any ideas? [23:17] dwins: mneedham: scala requires that “case” statements be inside curly braces. nothing to do with higher-order functions [23:17] mneedham: is there anywhere that’s documented or is that just a known thing? [23:18] mneedham: I expected it to work in normal parentheses [23:21] amacleod: mneedham, it’s documented. Whether it’s documented simply as “case statements need to be in curly braces” is another question The first line of Section 8.5 ‘Pattern Matching Anonymous Functions’ of the Scala language spec proves what I was told: Syntax: BlockExpr ::= ‘{’ CaseClauses ‘} It then goes into further detail about how the anonymous function gets converted into a pattern matching statement which is quite interesting reading. From http://www.markhneedham.com/blog/2011/07/12/scala-pattern-matching-a-pair-inside-mapfilter/

Most file/serialization formats can be broadly broking into two formats, Human Readable Text and Machine Readble Binary. The Human Readable formats have the advantage of being easily understood by a person reading them. Machine readable formats are easier/faster for a machine to encode/decode. There are formats which attempt to be a little of both. XML, JSon, CSV are examples of these. However these do not achieve close to the performance a binary format can achieve. Myth: Machine Readable Binary is always more compact than a Human Readable Binary can be more compact, however the obscurity of its format makes it difficult to ensure every byte counts. i.e. its usually hard enough getting something work. Making it compact as well is an added complication. However with Human Readable formats, determing how the format can be made more compact is more easily understood. As text: 38 bytes long, [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] As binary: 290 bytes long, ....sr..java.util.ArrayListx.....a....I..sizexp....w.....sr..java.lang.Long; .....#....J..valuexr..java.lang.Number...........xp........sq.~..........sq.~ ..........sq.~..........sq.~..........sq.~..........sq.~..........sq.~ ..........sq.~..........sq.~..........sq.~..........sq.~..........x Even though the first format is more compact, you can immedately see you could drop the [ ] and spaces after the ", " to make it more compact. With the binary formats, it is hard to know where to start. ComparingHumanReadableToBinaryMain.java List longs = new ArrayList(); for(long i=-1;i<=10;i++) longs.add(i); String asText = longs.toString(); byte[] bytes1 = asText.getBytes(); System.out.println("As text: "+ bytes1.length+" bytes long, "+asText); ByteArrayOutputStream baos = new ByteArrayOutputStream(); ObjectOutputStream oos = new ObjectOutputStream(baos); oos.writeObject(longs); oos.close(); byte[] bytes2 = baos.toByteArray(); System.out.println("As binary: "+bytes2.length+" bytes long, " +new String(bytes2, 0).replaceAll("[^\\p{Graph}]", ".")); Myth: Machine Readable Binary is always faster than a Human Readable Its assumed the cost of parsing data in a human readable format always makes it slower, however machine sreadbale formats have to deal with an issue human readbale formats takes for granted, that is byte endianness. For human readable formats the order of digits is fairly obvious, however for machine formats the byte endianess of the data might not match that the natrual byte order of the CPU, leading to a source of overhead (as it has to swap the bytes around) One example of this is using big-endian (e.g. TCP/Network byte order) on a little endian machine e.g. Windows/Linux Intel/AMD. A common class which has this issue is DataInputStream and DataOutputStream which re-arranges the byte order (even if the native byte order matches) For this reason, a fast human readable parse can be as fast or faster. In an earlier article I showed how a Human Readable format could be used to read/write integers 30% faster than using DataInput/DataOuput. Writing human readable data faster than binary. Myth: Using a Human Readable Format makes it easy to read Just using a human readable format doesn't mean it will be easier to read than a machine readable format. Reusing existing tools as much as possible makes human readable format preferrable. However, machine readable formats can come with tools which decode the data and make maintain it easier. If you have data which can only be managed with the use of specialist tools, being human readable is not much advantage. Images are a good example of where a machine readable format is the best option. It is hard to image editing or viewing an image without the need for a specialist tool. A practical human readable format would undoubtably lower the quality of the image. ;) ________/.- ,’_______`-. \ _________\ /`__________\’/ _________ /___’a___a`___\ _________|____,’(_)`.____ | _________\___( ._|_. )___ / __________\___ .__,’___ / __________.-`._______,’-.__ ________,’__,’___`-’___`.__`. _______/____/____V_____\___\_ _____,’____/_____o______\___`.__ ___,’_____|______o_______|_____`. __|_____,’|______o_______|`._____| ___`.__,’_.-\_____o______/-._`.__,’ __________/_`.___o____,’__\_ __.””-._,’_____`._:_,’_____`.,-””._ _/_,-._`_______)___(________’_,-.__\ (_(___`._____,’_____`.______,’___)_) _\_\____\__,’________`.____/.___/_/ On the other hand human readable formats can be almost as obscure. This is a piece of code written in a language I am not worthy of mentioning. ;) Its is descibed as "used to list all of the prime numbers between 1 and R" (!R)@&{&/x!/:2_!x}'!R Conclusion If you are designing a file format, start with a human readable one as its much easier to understand. If this is not compact enough, consider compressing it. If it is not fast enough concider making it a binary format, but make sure it really is faster to use such a format. If you are going to use a binary format make sure you have tools in place to supprot viewing (possibly editing) the data (which you would get for free with a text format) From http://vanillajava.blogspot.com/2011/07/human-readable-vs-machine-readble.html

I am Angelo D'Agnano and currently I work at the NATO Programming Centre as Software Architect.

In the old days... You would have a closet in your startup company with a rack of computers. Provisioning involved: Deciding on your architectural direction, what, where & how Ordering the new hardware Waiting weeks for the packages to arrive Setup the hardware, wire things together, power up Discover some component is missing, or failed and order replacement Wait longer... Finally get all the pieces setup Configure software components and go Along came some industrious folks who realized power and data to your physical location wasn't reliable. So datacenters sprang up. With data centers, most of the above steps didn't change except between steps 3 & 4 you would send your engineers out to the datacenter location. Trips back and forth ate up time and energy. Then along came managed hosting. Managed hosting saved companies a lot of headache, wasted man hours, and other resources. They allowed your company to do more of what it does well, run the business, and less on managing hardware and infrastructure. Provisioning now became: Decide on architecture direction Call hosting provider and talk to sales person Wait a day or two Setup & configure software components and go Obviously this new state of affairs improved infrastructure provisioning dramatically. It simplified the process and sped it up as well. What's more a managed hosting provider could keep spare parts and standard components on hand in much greater volume than a small firm. That's a big plus. This evolution continued because it was a win-win for everyone. The only downside was when engineers made mistakes, and finger pointing began. But despite all of that, a managed hosting provider which does only that, can do it better, and more reliably than you can yourself. So where are we in present day? We are all either doing, or looking out cloud provisioning of infrastructure. What's cloud provisioning? It is a complete paradigm shift, but along the same trajectory as what we've described above. Now you removed all the waiting. No waiting for sales team, or the ordering process. That's automatic. No waiting for engineers to setup the servers, they're already setup. They are allocated by your software and scripts. Even the setup and configuration of software components, Operating System and services to run on that server - all automatic. This is such a dramatic shift, that we are still feeling the affects of it. Traditional operations teams have little experience with this arrangement, and perhaps little trust in virtual servers. Business units are also not used to handing the trigger to infrastructure spending over to ops teams or to scripts and software. However the huge economic pressures continue to push firms to this new model, as well as new operational flexibility. Gartner predicts this trend will only continue. The advantages of cloud infrastructure provisioning include: Metered payment - no huge outlay of cash for new infrastructure Infrastructure as a service - scripted components automate & reduced manual processes Devops - Manage infrastructure like code with version control and reproduceability Take unused capacity offline easily & save on those costs Disaster Recovery is free - reuse scripts to build standard components Easily meet seasonal traffic requirements - spinup additional servers instantly On Quora Sean Hull asks - What is infrastructure provisioning and why is it important?

Lucene's near-real-time (NRT) search feature, available since 2.9, enables an application to make index changes visible to a new searcher with fast turnaround time. In some cases, such as modern social/news sites (e.g., LinkedIn, Twitter, Facebook, Stack Overflow, Hacker News, DZone, etc.), fast turnaround time is a hard requirement. Fortunately, it's trivial to use. Just open your initial NRT reader, like this: // w is your IndexWriter IndexReader r = IndexReader.open(w, true); (That's the 3.1+ API; prior to that use w.getReader() instead). The returned reader behaves just like one opened with IndexReader.open: it exposes the point-in-time snapshot of the index as of when it was opened. Wrap it in an IndexSearcher and search away! Once you've made changes to the index, call r.reopen() and you'll get another NRT reader; just be sure to close the old one. What's special about the NRT reader is that it searches uncommitted changes from IndexWriter, enabling your application to decouple fast turnaround time from index durability on crash (i.e., how often commit is called), something not previously possible. Under the hood, when an NRT reader is opened, Lucene flushes indexed documents as a new segment, applies any buffered deletions to in-memory bit-sets, and then opens a new reader showing the changes. The reopen time is in proportion to how many changes you made since last reopening that reader. Lucene's approach is a nice compromise between immediate consistency, where changes are visible after each index change, and eventual consistency, where changes are visible "later" but you don't usually know exactly when. With NRT, your application has controlled consistency: you decide exactly when changes must become visible. Recently there have been some good improvements related to NRT: New default merge policy, TieredMergePolicy, which is able to select more efficient non-contiguous merges, and favors segments with more deletions. NRTCachingDirectory takes load off the IO system by caching small segments in RAM (LUCENE-3092). When you open an NRT reader you can now optionally specify that deletions do not need to be applied, making reopen faster for those cases that can tolerate temporarily seeing deleted documents returned, or have some other means of filtering them out (LUCENE-2900). Segments that are 100% deleted are now dropped instead of inefficiently merged (LUCENE-2010). How fast is NRT search? I created a simple performance test to answer this. I first built a starting index by indexing all of Wikipedia's content (25 GB plain text), broken into 1 KB sized documents. Using this index, the test then reindexes all the documents again, this time at a fixed rate of 1 MB/second plain text. This is a very fast rate compared to the typical NRT application; for example, it's almost twice as fast as Twitter's recent peak during this year's superbowl (4,064 tweets/second), assuming every tweet is 140 bytes, and assuming Twitter indexed all tweets on a single shard. The test uses updateDocument, replacing documents by randomly selected ID, so that Lucene is forced to apply deletes across all segments. In addition, 8 search threads run a fixed TermQuery at the same time. Finally, the NRT reader is reopened once per second. I ran the test on modern hardware, a 24 core machine (dual x5680 Xeon CPUs) with an OCZ Vertex 3 240 GB SSD, using Oracle's 64 bit Java 1.6.0_21 and Linux Fedora 13. I gave Java a 2 GB max heap, and used MMapDirectory. The test ran for 6 hours 25 minutes, since that's how long it takes to re-index all of Wikipedia at a limited rate of 1 MB/sec; here's the resulting QPS and NRT reopen delay (milliseconds) over that time: The search QPS is green and the time to reopen each reader (NRT reopen delay in milliseconds) is blue; the graph is an interactive Dygraph, so if you click through above, you can then zoom in to any interesting region by clicking and dragging. You can also apply smoothing by entering the size of the window into the text box in the bottom left part of the graph. Search QPS dropped substantially with time. While annoying, this is expected, because of how deletions work in Lucene: documents are merely marked as deleted and thus are still visited but then filtered out, during searching. They are only truly deleted when the segments are merged. TermQuery is a worst-case query; harder queries, such as BooleanQuery, should see less slowdown from deleted, but not reclaimed, documents. Since the starting index had no deletions, and then picked up deletions over time, the QPS dropped. It looks like TieredMergePolicy should perhaps be even more aggressive in targeting segments with deletions; however, finally around 5:40 a very large merge (reclaiming many deletions) was kicked off. Once it finished the QPS recovered somewhat. Note that a real NRT application with deletions would see a more stable QPS since the index in "steady state" would always have some number of deletions in it; starting from a fresh index with no deletions is not typical. Reopen delay during merging The reopen delay is mostly around 55-60 milliseconds (mean is 57.0), which is very fast (i.e., only 5.7% "duty cycle" of the every 1.0 second reopen rate). There are random single spikes, which is caused by Java running a full GC cycle. However, large merges can slow down the reopen delay (once around 1:14, again at 3:34, and then the very large merge starting at 5:40). Many small merges (up to a few 100s of MB) were done but don't seem to impact reopen delay. Large merges have been a challenge in Lucene for some time, also causing trouble for ongoing searching. I'm not yet sure why large merges so adversely impact reopen time; there are several possibilities. It could be simple IO contention: a merge keeps the IO system very busy reading and writing many bytes, thus interfering with any IO required during reopen. However, if that were the case, NRTCachingDirectory (used by the test) should have prevented it, but didn't. It's also possible that the OS is [poorly] choosing to evict important process pages, such as the terms index, in favor of IO caching, causing the term lookups required when applying deletes to hit page faults; however, this also shouldn't be happening in my test since I've set Linux's swappiness to 0. Yet another possibility is Linux's write cache becomes temporarily too full, thus stalling all IO in the process until it clears; in this case perhaps tuning some of Linux's pdflush tunables could help, although I'd much rather find a Lucene-only solution so this problem can be fixed without users having to tweak such advanced OS tunables, even swappiness. Fortunately, we have an active Google Summer of Code student, Varun Thacker, working on enabling Directory implementations to pass appropriate flags to the OS when opening files for merging (LUCENE-2793 and LUCENE-2795). From past testing I know that passing O_DIRECT can prevent merges from evicting hot pages, so it's possible this will fix our slow reopen time as well since it bypasses the write cache. Finally, it's always possible other OSs do a better job managing the buffer cache, and wouldn't see such reopen delays during large merges. This issue is still a mystery, as there are many possibilities, but we'll eventually get to the bottom of it. It could be we should simply add our own IO throttling, so we can control net MB/sec read and written by merging activity. This would make a nice addition to Lucene! Except for the slowdown during merging, the performance of NRT is impressive. Most applications will have a required indexing rate far below 1 MB/sec per shard, and for most applications reopening once per second is fast enough. While there are exciting ideas to bring true real-time search to Lucene, by directly searching IndexWriter's RAM buffer as Michael Busch has implemented at Twitter with some cool custom extensions to Lucene, I doubt even the most demanding social apps actually truly need better performance than we see today with NRT. NIOFSDirectory vs MMapDirectory Out of curiosity, I ran the exact same test as above, but this time with NIOFSDirectory instead of MMapDirectory: There are some interesting differences. The search QPS is substantially slower -- starting at 107 QPS vs 151, though part of this could easily be from getting different compilation out of hotspot. For some reason TermQuery, in particular, has high variance from one JVM instance to another. The mean reopen time is slower: 67.7 milliseconds vs 57.0, and the reopen time seems more affected by the number of segments in the index (this is the saw-tooth pattern in the graph, matching when minor merges occur). The takeaway message seems clear: on Linux, use MMapDirectory not NIOFSDirectory! Optimizing your NRT turnaround time My test was just one datapoint, at a fixed fast reopen period (once per second) and at a high indexing rate (1 MB/sec plain text). You should test specifically for your use-case what reopen rate works best. Generally, the more frequently you reopen the faster the turnaround time will be, since fewer changes need to be applied; however, frequent reopening will reduce the maximum indexing rate. Most apps have relatively low required indexing rates compared to what Lucene can handle and can thus pick a reopen rate to suit the application's turnaround time requirements. There are also some simple steps you can take to reduce the turnaround time: Store the index on a fast IO system, ideally a modern SSD. Install a merged segment warmer (see IndexWriter.setMergedSegmentWarmer). This warmer is invoked by IndexWriter to warm up a newly merged segment without blocking the reopen of a new NRT reader. If your application uses Lucene's FieldCache or has its own caches, this is important as otherwise that warming cost will be spent on the first query to hit the new reader. Use only as many indexing threads as needed to achieve your required indexing rate; often 1 thread suffices. The fewer threads used for indexing, the faster the flushing, and the less merging (on trunk). If you are using Lucene's trunk, and your changes include deleting or updating prior documents, then use the Pulsing codec for your id field since this gives faster lookup performance which will make your reopen faster. Use the new NRTCachingDirectory, which buffers small segments in RAM to take load off the IO system (LUCENE-3092). Pass false for applyDeletes when opening an NRT reader, if your application can tolerate seeing deleted doccs from the returned reader. While it's not clear that thread priorities actually work correctly (see this Google Tech Talk), you should still set your thread priorities properly: the thread reopening your readers should be highest; next should be your indexing threads; and finally lowest should be all searching threads. If the machine becomes saturated, ideally only the search threads should take the hit. Happy near-real-time searching!

one of the first things you learn to do incorrectly as a programmer is commenting your code. my experience with student and recently graduated programmers tells me that college is a really good place to learn really bad code commenting techniques. this is just one of those areas where in-theory and in-practice don’t align well. there are two factors working against you learning good commenting technique in college. unlike the real world, you do a lot of small one-off projects as a solo developer. there’s no one out there fantasizing about dropping a boulder on you for making them decipher your coding atrocity. that commenting style you are emulating from your textbook is only a good practice when the comments are intended for a student learning to program. it is downright annoying to professional programmers. these tips are primarily intended for upstart programmers who are transitioning into the real world of programming, and hopefully will prevent a few from looking quite so n00bish during their first code review. code review? oh yeah, that’s something else they didn’t teach you in school, but that’s a whole other article, i’ll defer to jason cohen on that one. so let’s get started… (1) comments are not subtitles it’s easy to project your own worldview that code is a foreign language understood only by computers, and that you are doing the reader a service by explaining what each line does in some form of human language. or perhaps you are doing it for the benefit of that non-programmer manager who will certainly want to read your code (spoiler: he won’t). look, in the not too distant future, you will be able to read code almost as easily as your native language, and everyone else who will even glance at it almost certainly already can. by then you will realize how silly it is to write comments like these: // loop through all bananas in the bunch foreach(banana b in bunch) { monkey.eat(b); //make the monkey eat one banana } you may have been taught to program by first writing pseudo-code comments then writing the real code into that wire-frame. this is a perfectly reasonable approach for a novice programmer. just be sure to replace the comments with the code , and don’t leave them in there. computer: enemy is matching velocity. gwen demarco: the enemy is matching velocity! sir alexander dane: we heard it the first time! gwen demarco: gosh, i’m doing it. i’m repeating the darn computer! -galaxy quest exceptions: code examples used to teach a concept or new programming language. programming languages that aren’t remotely human readable (assembly, perl) (2) comments are not an art project this is a bad habit propagated by code samples in programing books and open source copyright notices that are desperate to make you pay attention to them. /* _ _ _ _ _ _ _ _ _ _ _ _ (c).-.(c) (c).-.(c) (c).-.(c) (c).-.(c) (c).-.(c) (c).-.(c) / ._. \ / ._. \ / ._. \ / ._. \ / ._. \ / ._. \ __\( y )/__ __\( y )/__ __\( y )/__ __\( y )/__ __\( y )/__ __\( y )/__ (_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._) || m || || o || || n || || k || || e || || y || _.' `-' '._ _.' `-' '._ _.' `-' '._ _.' `-' '._ _.' `-' '._ _.' `-' '._ (.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.) `-' `-' `-' `-' `-' `-' `-' `-' `-' `-' `-' `-' -it's monkey business time! (version 1.5) */ why, that’s silly. you’d never do something so silly in your comments. orly? does this look familiar? +------------------------------------------------------------+ | module name: classmonkey | | module purpose: emulate a monkey | | inputs: bananas | | outputs: grunts | | throws: poop | +------------------------------------------------------------+ programmers love to go “touch up” their code to make it look good when their brain hurts and they want something easy to do for a while. it may be a waste of time, but at least they are wasting it during periods where they wouldn’t be productive anyway. the trouble is that it creates a time-wasting maintenance tax imposed on anyone working with the code in the future just to keep the pretty little box intact when the text ruins the symmetry of it. even programmers who hate these header blocks tend to take the time to maintain them because they like consistency and every other method in the project has one. how much is it bugging you that the right border on that block is misaligned? yeah. that’s the point. (3) header blocks: nuisance or menace? this one is going to be controversial, but i’m holding my ground. i don’t like blocks of header comments at the top of every file, method or class. not in a boat, not with a goat. why? well let me tell you, george mcfly… they are enablers for badly named objects/methods – of course, header blocks aren’t the cause for badly named identifiers, but they are an easy excuse to not put in the work to come up with meaningful names, an often deceptively difficult task. it provides too much slack to just assume the consumer can just read the “inline documentation” to solve the mystery of what the dothemonkeything method is all about. johnfx’s commandment: the consumer of thy code should never have to see its source code to use it, not even the comments. they never get updated: we all know that methods are supposed to remain short and sweet, but real life gets in the way and before you know it you have a 4k line class and the header block is scrolled off of the screen in the ide 83% of the time. out of sight, out of mind, never updated. the bad news is that they are usually out of date. the good news is that people rarely read them so the opportunity for confusion is mitigated somewhat. either way, why waste your time on something that is more likely to hurt than help? johnfx’s maxim of plagiarized ideas : bad documentation is worse than no documentation. exception: some languages ( java / c# ) have tools that can digest specially formatted header block comments into documentation or intellisense/autocomplete hints. still, remember rule (2) and stick to the minimum required by the tool and draw the line at creating any form of ascii art . (4) comments are not source control this issue is so common that i have to assume that programmers (a) don’t know how to use source control; or (b) don’t trust it. archetype 1: “the historian” // method name: pitythefoo (included for the sake of redundancy) // created: feb 18, 2009 11:33pm // author: bob // revisions: sue (2/19/2009) - lengthened monkey's arms // bob (2/20/2009) - solved drooling issue void pitythefoo() { ... } the programmers involved in the evolution of this method probably checked this code into a source control system designed to track the change history of every file, but decided to clutter up the code anyway. these same programmers more than likely always leave the check-in comments box empty on their commits. i think i hate this type of comment worst of all, because it imposes a duty on other programmers to keep up the tradition of duplicating effort and wasting time maintaining this chaff. i almost always delete this mess from any code i touch without an ounce of guilt. i encourage you to do the same. archetype 2: “the code hoarder” void monkeyshines() { if (monkeysonbed(jumping).count > max) { monkeysonbed.breakhead(); // code removed, smoothie shop closed. // leaving it in case a new one opens. // monkeysonbed.drink(bananasmoothie); } } another feature of any tool that has any right to call itself a scm is the ability to recover old versions of code, including the parts you removed. if you want to be triple super extra sure, create a branch to help you with your trust issues . (5) comments are a code smell comments are little signposts in your code explaining it to future archaeologists that desperately need to understand how 21st century man sorted lists of purchase orders. unfortunately, as donald norman explained so brilliantly in the design of everyday things , things generally need signs because their affordances have failed. in plain english, when you add a comment you are admitting that you have written code that doesn’t communicate its purpose well. sign:"this is a mop sink." why would that be necces... oh. despite what your prof told you in college, a high comment to code ratio is not a good thing. i’m not saying to avoid them completely, but if you have a 1-1 or even a 5-1 ratio of loc to comments, you are probably overdoing it. the need for excessive comments is a good indicator that your code needs refactoring . whenever you think, “this code needs a comment” follow that thought with, “how could i modify the code so its purpose is obvious?” talk with your code, not your comments. technique 1 : use meaningful identifiers and constants (even if they are single use) // before // calculate monkey's arm length // using it's height and the magic monkey arm ratio double length = h * 1.845; //magic numbers are evil! // after - no comment required double armlength = height * monkey_arm_height_ratio; technique 2: use strongly typed input and output parameters // before // begin: flip the "hairy" bit on monkeys foreach(monkey m in thesemonkeys) { // 5-6 steps to flip bit. } // end: flip the "hairy" bit on monkeys // after no comment required fliphairybit(thesemonkeys); as an added bonus, technique 3 will tend to reduce the size of your methods and minimizing the nesting depth ( see also “flattening arrow code” ) all of which contribute to eliminating the need for commenting the closing tags of blocks like this: } // ... if see evil } // .. . while monkey do. } // ... if monkey see. } // ... class monkey } // ... namespace primate acknowledgments several of the ideas presented here, and a good deal of the fundamental things i know about programming as part of a team, and not as a lone-wolf working on a college project i learned from the book code complete by steve mcconnell. if you are a working programmer and have not read this book year, stop what you are doing and read it before you write another line of code. from http://improvingsoftware.com/2011/06/27/5-best-practices-for-commenting-your-code/

Tomcat 7 has been improved a lot and along with everything else that it brings, a very nice feature is provided - an API for embedding Tomcat into your application. The API was provided in earlier versions of Tomcat but it was quite cumbersome to use. To to start the embedded version of Tomcat one may need to build the required JARs. svn co https://svn.apache.org/repos/asf/tomcat/trunk tomcat cd tomcat ant embed-jars ls -l output/embed total 5092 -rw-r--r-- 1 anton None 56802 2011-03-06 17:09 LICENSE -rw-r--r-- 1 anton None 1194 2011-03-06 17:09 NOTICE -rw-r--r-- 1 anton None 1690519 2011-03-06 17:09 ecj-3.6.jar -rw-r--r-- 1 anton None 234625 2011-03-06 17:09 tomcat-dbcp.jar -rw-r--r-- 1 anton None 2402517 2011-03-06 17:09 tomcat-embed-core.jar -rw-r--r-- 1 anton None 781989 2011-03-06 17:09 tomcat-embed-jasper.jar -rw-r--r-- 1 anton None 34106 2011-03-06 17:09 tomcat-embed-logging-juli.jar The following snippet demonstrates the embedded Tomcat usage with a deployed servlet instance. import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.File; import java.io.IOException; import java.io.Writer; public class Main { public static void main(String[] args) throws LifecycleException, InterruptedException, ServletException { Tomcat tomcat = new Tomcat(); tomcat.setPort(8080); Context ctx = tomcat.addContext("/", new File(".").getAbsolutePath()); Tomcat.addServlet(ctx, "hello", new HttpServlet() { protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { Writer w = resp.getWriter(); w.write("Hello, World!"); w.flush(); } }); ctx.addServletMapping("/*", "hello"); tomcat.start(); tomcat.getServer().await(); } } The only two JARs required are tomcat-embed-core.jar and tomcat-embed-logging-juli.jar. It means that there will be no JSP support and pooling will also be disabled. But that's enough to start a servlet and in most cases that is what you probably need. From http://arhipov.blogspot.com/2011/03/embedded-tomcat-minimal-version.html

One of the things that changed from Solr 1.4.1 to 1.5+ was the introduction of a parameter to tell Solr / Lucene which kind of compability version its index files should be created and used in. Solr now refuses to start if you do not provide this setting (if you’re upgrading a previous installation from 1.4.1 or earlier). The fix isn’t really straight forward, and you’ll probably have to recreate your index files if you’re just arriving at the scene with Solr / Lucene 3.2 and 4.0. Solr 3.0 (1.5) might be able to upgrade the files from the 2.9 version, but if you’re jumping from Lucene 2.9 to 4.0, the easiest solution seems to be to delete the current index and reindex (set up replication, disable replication from the master, query the slave while reindexing the master, etc.. and you’ll have no downtime while doing this!). You’ll need to add a parameter to your solrconfig.xml file as well in the section. LUCENE_CURRENT Other valid values are LUCENE_30, LUCENE_31, LUCENE_32 and LUCENE_40. These values represent specific versions of the index structure, while LUCENE_CURRENT will use the version depending on which particular release of Lucene you’re using. The version format will be upgraded automagically between most releases, so you’ll probably be fine by using LUCENE_CURRENT. If you however are trying to load index files that are more than one version older, you may have to use one of the other values.

This tutorial will walk you through how to download, install and run Jetty – a 100 % Java HTTP and Servlet Container. If you do not know Jetty, the following is what Wikipedia says about it: Jetty is a pure Java-based HTTP server and servlet container (Application server) developed as a free and open source project as part of the Eclipse Foundation. It is currently used in products such as ActiveMQ,[1]Alfresco, [2]Apache Geronimo,[3]Apache Maven, Google App Engine,[4]Eclipse,[5]FUSE,[6]HP OpenView, JBoss,[7]Liferay,[8]Ubuntu, Twitter’s Streaming API[9] and Zimbra.[10] Jetty is also used as a standard Java application server by many open source projects such as Eucalyptus and Hadoop. I am doing some experiments, and I decided to use Jetty instead of TomCat. So let’s get this server up and running! 1 – Downloading You can download Jetty from two sources: Eclipse or Codehaus. http://jetty.codehaus.org/jetty/ http://www.eclipse.org/jetty/downloads.php The current stable version is 7, so I downloaded this one from Eclipse page. The compressed file is platform independent. So if you use Mac, Linux or Windows, that is ok, it will run on any OS. 2 – Installing Simply uncompress the file to a directory. You should have something like this: Installation is complete! Let’s get it running. 3 – Running Jetty Open a terminal. Go to the Jetty installation directory. Enter the following command: java -jar start.jar Now open a browser and go to localhost to check if Jetty was installed sucessfully: http://localhost:8080/ You should get a page like this: And it is done! From http://loianegroner.com/2011/07/installing-and-running-jetty/

The Data Mapper pattern is a mechanism for persistence where the application model and the data source have no dependencies between each other. For example, a group of PHP classes and a relational database may be used together without having the PHP classes extends a base Active Record class, thanks to a Data Mapper like Doctrine 2. But everytime we talk about the Data Mapper pattern, we assume there is a relational database on the other side of the persistence boundary. We always save objects; we always map them to MySQL or Postgres tables; but it's not mandatory. In fact, you can store objects also in NoSQL stores, with no more impedance mismatch that the one already existing with relational databases. Doctrine is expanding with two projects which have the goal of replicating the easy object-relational mapping of Doctrine 2 to other stores. These two projects are the MongoDB and CouchDB Object-Document mappers. Since the basic unit of persistence is the document in both cases, objects (or group of them) are stored as documents. The retrieval process depends on the features of the underlying database: it may consists of simple queries (MongoDB) or just of a unique identifier (CouchDB). How do an ODM work? If you use Doctrine 2, you'll find out that many of its concepts translates well to the other mappers. Instead of Entities, we have Documents in our model: id = "myuser-1234"; $user->username = "test"; $this->dm->persist($user); $this->dm->flush(); $this->dm->clear(); $userNew = $this->dm->find($this->type, $user->id); There is a lot of reuse between the various projects: Doctrine\Common is a dependency for them all, and you will be able to use the same ArrayCollection with any of the mappers. Features There's really all I would need to get started: storage of new documents, update, removal; pretty much what you expect; Unit Of Work pattern for tracking what has been modified and execute a single flush() to save changes; support for collections and both one-to-many or many-to-one associations; support for embedded documents (read Value Objects); both embedding of one or many objects as document fields is possible. This is a striking difference with respect to ORMs: Doctrine 2 does not support Value Objects mapping. cascade of persistence and removal of objects; specification of the mapping metadata via annotations, XML, YAML or PHP (like for Doctrine 2). Projects status The concept of Object-Document Mapper, at least in this name, is pretty unique to PHP and Ruby. Mandango is a PHP alternative and Mongoid a Ruby one, for the MongoDB case. Both projects are in the midst of releasing their first stable version: we are talking about bleeding edge technology here. The MongoDB mapper has been around for more than an year and is now in beta3. The CouchDB has an alpha release, but I found out that the current version shipped via PEAR is broken: if you want to play with it, checkout it from github and initialize its submodules (the standalone Symfony components and Doctrine\Common): git clone https://github.com/doctrine/couchdb-odm.git couchdb_odm cd couchdb_odm git submodule init git submodule update For the MongoDB mapper, the PEAR installation should suffice: pear install pear.doctrine-project.org/DoctrineMongoDBODM-1.0.0BETA3 Conclusions I think I'm going to explore more the CouchDB ODM, since it is a departure from the relational model. MongoDB is still similar to traditional databases in the way queries are satisfied: specifying a set of constraints like in SQL WHERE clauses. I'm more interested instead in approached that skip the Mapper for reporting (like Command-Query Responsibility Separation), and maybe CouchDB could be a good fit. By the way, object-document mapping is an alternative to explore: Data Mappers are all the rage today in PHP (they already were yesterday in other languages.) If we already accept that relational databases aren't always the solution, extending NoSQL with Data Mappers is a natural choice.

In my daily work, I use both an RDBMS and MarkLogic, an XML database. MarkLogic can be considered akin to the newer NoSQL databases, but it has the added structure of XML and standard languages in XQuery and XPath. The NoSQL databases are typically storing documents or key-value pairs, and some other things in between. Given that any datastore will be searched at some point, you will always care how the data is actually stored or whether there is some way to query it easily. Once you start thinking about the problem, you quickly generalize to the “how do I persist any type of data” question. However, my focus is not going to be the comparison of the various data stores, but the comparison of how data is stored. More specifically, I want to show the object serialization, mainly the Java built in method, as a data persistence format is evil. Given what you normally read on this blog, this may seem like an oddly timed post, but I have run into serialization issues lately in some production code and Mark Needham recently wrote an interesting post about this as well. Coincidentally, Mark is also working with MarkLogic, and there is an interesting item in his post: The advantage of doing things this way [using lightweight wrappers] is that it means we have less code to write than we would with the serialisation/deserialisation approach although it does mean that we’re strongly coupled to the data format that our storage mechanism uses. However, since this is one bit of the architecture which is not going to change it seems to makes sense to accept the leakage of that layer. The interesting part of this is that he has accepted using the data format of the storage mechanism, XML in MarkLogic in this case. Why is this interesting? First, it is a move away from the ORM technologies that try to hide the complexities of converting data into objects in the RDBMS world. Also, this is a glimpse into the types of issues that could arise from non-RDBMS storage choices as well as how to persist objects in general. So, an RDBMS is typically used to map object attributes to a table and columns. The mapping is mostly straightforward with some defined relationship for child objects and collections. This is a well-known area, called Object-Relational Mapping (ORM), and several open source and commercial options exist. In this scenario, object attributes are stored in a similar datatype, meaning a String is stored as a varchar and an int is stored as an integer. But, what happens when you move away from an RDBMS for data persistence? If you look at Java and its session objects, pure object serialization is used. Assuming that an application session is fairly short-lived, meaning at most a few hours, object serialization is simple, well supported and built into the Java concept of a session. However, when the data persistence is over a longer period of time, possibly days or weeks, and you have to worry about new releases of the application, serialization quickly becomes evil. As any good Java developer knows, if you plan to serialize an object, even in a session, you need a real serialization ID (serialVersionUID), not just a 1L, and you need to implement the Serializable interface. However, most developers do not know the real rules behind the Java deserialization process. If your object has changed, more than just adding simple fields to the object, it is possible that Java cannot deserialize the object correctly even if the serialization ID has not changed. Suddenly, you cannot retrieve your data any longer, which is inherently bad. Now, may developers reading this may say that they would never write code that would have this problem. That may be true, but what about a library that you use or some other developer no longer employed by your company? Can you guarantee that this problem will never happen? The only way to guarantee that is to use a different serialization method. What options do we have? Obviously, there are the NoSQL datastores but the actual object format is the relevant question not which solution to choose. Besides the obvious serialized object, some NoSQL datastores use JSON to store objects, MarkLogic uses XML and there are others that store just key-value pairs. Key-value pairs are typically a mapping of a text key to a value that is a serialized object, either a binary or textual format. So, that leaves us with XML, JSON and other textual formats. One of the benefits of a structured format like XML or JSON is that they can be made searchable and provide some level of context. I have talked about data formats before, so I won’t go into a comparison again. However, do these types of formats avoid the issues that native Java object serialization has? This is really dependent upon what library you are using for serialization. Some libraries will deserialize an object without any issues regardless of whether the object field list has changed. Other libraries could have problems depending upon whether a serialized field exists in the target object, or there might not be solid support for collections (though that is doubtful at this point). Given that even structured formats could have serialization issues, is the only safe path hand-coded mappings like those used by ORM tools? Some JSON and XML serialization tools use the same mapping methods as the ORM tools in order to avoid these problems. However, once you define these mappings, you are explicitly stating how an object gets translated. This explicit definition will require maintenance, but that is definitely cleaner than trying to trace down a serialization defect in some random stack trace. So is implicit object serialization really worth the potential headaches? Or should we just consider it evil and never speak of it again? From http://regulargeek.com/2011/07/06/is-object-serialization-evil/