A generally useful question posed by Charming asks how many queues one should use in RabbitMQ for the maximum message passing throughput/performance. I thought I'd distill the answers by Brian Kelly and RobotEyes here for anyone who's worried about their performance with RabbitMQ. And I'll bet that some of these pointers are relevant to other message queues as well: From the RabbitMQ blog: RabbitMQ's queues are fastest when they're empty. When a queue is empty, and it has consumers ready to receive messages, then as soon as a message is received by the queue, it goes straight out to the consumer. In the case of a persistent message in a durable queue, yes, it will also go to disk, but that's done in an asynchronous manner and is buffered heavily. The main point is that very little book-keeping needs to be done, very few data structures are modified, and very little additional memory needs allocating. From the rabbitmq-discuss mailing group: Use a larger prefetch count. Small values hurt performance. A topic exchange is slower than a direct or a fanout exchange. Make sure queues stay short. Longer queues impose more processing overhead. If you care about latency and message rates then use smaller messages. Use an efficient format (e.g. avoid XML) or compress the payload. Experiment with HiPE, which helps performance. Avoid transactions and persistence. Also avoid publishing in immediate or mandatory mode. Avoid HA. Clustering can also impact performance. You will achieve better throughput on a multi-core system if you have multiple queues and consumers. Use at least v2.8.1, which introduces flow control. Make sure the memory and disk space alarms never trigger. Virtualisation can impose a small performance penalty. Tune your OS and network stack. Make sure you provide more than enough RAM. Provide fast cores and RAM. Hope that gives you some ideas for improving your queueing performance.

this post was originally published as part of the java advent series . if you like it, please spread the word by sharing, tweeting, fb, g+ and so on! want to write for the java advent blog? we are looking for contributors to fill all 24 slot and would love to have your contribution! contact attila balazs to contribute! it is common knowledge that java optimizes the substring operation for the case where you generate a lot of substrings of the same source string. it does this by using the (value, offset, count) way of storing the information. see an example below: in the above diagram you see the strings “hello” and “world!” derived from “hello world!” and the way they are represented in the heap: there is one character array containing “hello world!” and two references to it. this method of storage is advantageous in some cases, for example for a compiler which tokenizes source files. in other instances it may lead you to an outofmemorerror (if you are routinely reading long strings and only keeping a small part of it – but the above mechanism prevents the gc from collecting the original string buffer). some even call it a bug . i wouldn’t go so far, but it’s certainly a leaky abstraction because you were forced to do the following to ensure that a copy was made: new string(str.substring(5, 6)) . this all changed in may of 2012 or java 7u6. the pendulum is swung back and now full copies are made by default. what does this mean for you? for most probably it is just a nice piece of java trivia if you are writing parsers and such, you can not rely any more on the implicit caching provided by string. you will need to implement a similar mechanism based on buffering and a custom implementation of charsequence if you were doing new string(str.substring) to force a copy of the character buffer, you can stop as soon as you update to the latest java 7 (and you need to do that quite soon since java 6 is being eold as we speak ). thankfully the development of java is an open process and such information is at the fingertips of everyone! a couple of more references (since we don’t say pointers in java ) related to strings: if you are storing the same string over and over again (maybe you’re parsing messages from a socket for example), you should read up on alternatives to string.intern() (and also consider reading chapter 50 from the second edition of effective java: avoid strings where other types are more appropriate) look into (and do benchmarks before using them!) options like usecompressedstrings (which seems to have been removed ), usestringcache and stringcache hope i didn’t strung you along too much and you found this useful! until next time - attila balazs meta: this post is part of the java advent calendar and is licensed under the creative commons 3.0 attribution license. if you like it, please spread the word by sharing, tweeting, fb, g+ and so on! want to write for the blog? we are looking for contributors to fill all 24 slot and would love to have your contribution! contact attila balazs to contribute!

From an original article on Wakaleo.com Conventional test reports, generated by tools such as JUnit or TestNG, naturally focus on what tests have been executed, and whether they passed or failed. While this is certainly useful from a testing perspective, these reports are far from telling the whole picture. BDD reporting tools like Cucumber and JBehave take things a step further, introducing the concept of "pending" tests. A pending test is one that has been specified (for example, as an acceptance criteria for a user story), but which has not been implemented yet. In BDD, we describe the expected behaviour of our application using concrete examples, that eventually form the basis of the "acceptance criteria" for the user stories we are implementing. BDD tools such as Cucumber and JBehave not only report on test results: they also report on the user stories that these tests validate. However this reporting is still limited for large projects, where the numbers of user stories can become unwieldy. User stories are not created in isolation: rather, user stories help describe features, which support capabilities that need to be implemented to achieve the business goals of the application. So it makes sense to be able to report on test results not only at the user story level, but also at higher levels, for example in terms of features and capabilities. This makes it easier to report on not only what stories have been implemented, but also what features and capabilities remain to be done. An example of such a report is shown in Figure 1 (or see the full report here). Figure 1: A test coverage report listing both tested and untested requirements. In agile projects, it is generally considered that a user story is not complete until all of its automated acceptance tests pass. Similarly, a feature cannot be considered ready to deliver until all of the acceptance criteria for the underlying user stories have been specified and implemented. However, sensible teams shy away from trying to define all of the acceptance criteria up-front, leaving this until the "last responsible moment", often shortly before the user story is scheduled to be implemented. For this reason, reports that relate project progress and status only in terms of test results are missing out on the big picture. To get a more accurate idea of what features have been delivered, which ones are in progress, and what work remains to be done, we must think not in terms of test results, but in terms of the requirements as we currently understand them, matching the currently implemented tests to these requirements, but also pointing out what requirements currently have no acceptance criteria defined. And when graphs and reports illustrate how much progress has been made, the requirements with no acceptance criteria must also be part of the picture. Requirements-level BDD reporting with Thucydides Thucydides is an open source tool that puts some of these concepts into practice. Building on top of BDD tools such as JBehave, or using just ordinary JUnit tests, Thucydides reports not only on how the tests did, but also fits them into the broader picture, showing what requirements have been tested and, just as importantly, what requirements haven't. You can learn more about Thucydides in this tutorial or on the Thucydides website. During the rest of this article, we will see how to report on both your requirements and your test results using Thucydides, using a very simple directory-based approach. You can follow along with this example by cloning the Github project at https://github.com/thucydides-webtests/thucydides-simple-demo Simple requirements in Thucydides - a directory-based approach Thucydides can integrate with many different requirement management systems, and it is easy to write your own plugin to tailor the integration to suite your particular environment. A popular approach, for example, is to store requirements in JIRA and to use Thucydides to read the requirements hierarcy directly from the JIRA cards. However the simplest approach, which uses a directory-based approach, is probably the easiest to use to get started, and it is that approach that we will be looking at here. Requirements can usually be organized in a hierarchial structure. By default, Thucydides uses a three-level hierarchy of requirements. At the top level, capabilities represent a high-level capacity that the application must provide to meet the application's business goals. At the next level down, features help deliver these capabilities. To make implementation easier, a feature can be broken up into user stories, each of which in turn can contain a number of acceptance criteria. Figure 2: JUnit test directories mirror the requirements hierarchy. Of course, you don't have to use this structure if it doesn't suit you. You can override the thucydides.capability.types system property to provide your own hierarchy. For example, if you wanted a hierarchy with modules,epics, and features, you would just set thucydides.capability.types to "module,epic,feature". When we use the default directory-based requirements strategy in Thucydides, the requirements are stored in a hierarchial directory structure that matches the requirements hierarchy. At the lowest level, a user story is represented by a JBehave *.story file, an easyb story, or a JUnit test. All of the other requirements are represented as directories (see Figure 2 for an example of such a structure). In each requirements directory, you can optionally place a file called narrative.txt, which contains a free-text summary of the requirement. This will appear in the reports, with the first line appearing as the requirement title. A typical narrative text is illustrated in the following example: Learn the meaning of a word In order to learn the meaning of a word that I don't know As an online reader I want to be able to find out the meaning of the word If you are implementing the acceptance criteria as JUnit tests, just place the JUnit tests in the package that matches the correspoinding requirement. You need to use the thucydides.test.root system property to specify the root package of your requirements. For the example in Figure 2, this value should be set to nz.govt.nzqa.lssu.stories. Figure 3: The narrative.txt file appears in the reports to describe a requirement. If you are using JBehave, just place the *.story files in the src/test/resources/stories directory, again respecting a directory structure that corresponds to your requirement hierarchy. The narrative.txt files also work for JBehave requirements. Progress is measured by the total number of passing, failing or pending acceptance criteria, either for the whole project (at the top level), or within a particular requirement as you drill down the requirements hierarchy. For the purposes of reporting, a requirement with no acceptance criteria is attributed an arbitrary number of "imaginary" pending acceptance criteria. Thucydides considers that you need 4 tests per requirement by default, but you can override this value using the thucydides.estimated.tests.per.requirement system property. Figure 3: For JBehave, everything goes under src/test/resources/stories. Conclusion BDD is an excellent approach for communicating with, and reporting back to, stakeholders. However, for accurate acceptance test reporting on real-world projects, you need to go beyond the story level, and cater for the whole requirements hierarchy. In particular, you need to not only report on tests that have been executed, but also allow for the tests that haven't been written yet. Thucydides puts these concepts into practice: using a simple directory-based convention, you can easily integrate your requirements hierarcy into your acceptance tests.

This is a continuation from the previous post. Specification Many games have a start screen or main menu of some sort. (Though I love games like Braid that bypass the whole notion.) Let’s begin by designing our start screen. We’ll have a solid color background. Perhaps the ever lovely cornflower blue. Then we’ll draw the name of our game and provide an instruction to the player. In order to make sure we have the player’s attention, we’ll animate the color of the instruction. It will morph from black to red and back again. Finally, when the player clicks the screen we’ll transition to the main game. Or at least we’ll stub out the transition. Here’s a demo based on the code we’ll cover later in this post (as well as that from the previous post.) Implementation Here’s the code to implement our start screen. // `input` will be defined elsewhere, it's a means // for us to capture the state of input from the player var startScreen = (function(input) { // the red component of rgb var hue = 0; // are we moving toward red or black? var direction = 1; var transitioning = false; // record the input state from last frame // because we need to compare it in the // current frame var wasButtonDown = false; // a helper function // used internally to draw the text in // in the center of the canvas (with respect // to the x coordinate) function centerText(ctx, text, y) { var measurement = ctx.measureText(text); var x = (ctx.canvas.width - measurement.width) / 2; ctx.fillText(text, x, y); } // draw the main menu to the canvas function draw(ctx, elapsed) { // let's draw the text in the middle of the canvas // note that it's ineffecient to calculate this // in every frame since it never changes // however, I leave it here for simplicity var y = ctx.canvas.height / 2; // create a css color from the `hue` var color = 'rgb(' + hue + ',0,0)'; // clear the entire canvas // (this is not strictly necessary since we are always // updating the same pixels for this screen, however it // is generally what you need to do.) ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height); // draw the title of the game // this is static and doesn't change ctx.fillStyle = 'white'; ctx.font = '48px monospace'; centerText(ctx, 'My Awesome Game', y); // draw instructions to the player // this animates the color based on the value of `hue` ctx.fillStyle = color; ctx.font = '24px monospace'; centerText(ctx, 'click to begin', y + 30); } // update the color we're drawing and // check for input from the user function update() { // we want `hue` to oscillate between 0 and 255 hue += 1 * direction; if (hue > 255) direction = -1; if (hue < 0) direction = 1; // note that this logic is dependent on the frame rate, // that means if the frame rate is slow then the animation // is slow. // we could make it indepedent on the frame rate, but we'll // come to that later. // here we magically capture the state of the mouse // notice that we are not dealing with events inside the game // loop. // we'll come back to this too. var isButtonDown = input.isButtonDown(); // we want to know if the input (mouse click) _just_ happened // that means we only want to transition away from the menu to the // game if there _was_ input on the last frame _but none_ on the // current one. var mouseJustClicked = !isButtonDown && wasButtonDown; // we also check the value of `transitioning` so that we don't // initiate the transition logic more the once (like if the player // clicked the mouse repeatedly before we finished transitioning) if (mouseJustClicked && !transitioning) { transitioning = true; // do something here to transition to the actual game } // record the state of input for use in the next frame wasButtonDown = isButtonDown; } // this is the object that will be `startScreen` return { draw: draw, update: update }; }()); Explanation Recall that our start screen is meant to be invoked by our game loop. The game loop doesn’t know about the specifics of the start screen, but it does expect it to have a certain shape. This enables us to swap out screen objects without having to modify the game loop itself. The shape that the game loop expects is this: { update: function(timeElapsedSinceLastFrame) { }, draw: function(drawingContext) { } } Update Let’s begin with the start screen’s update function. The first bit of logic is this: hue += 1 * direction; if (hue > 255) direction = -1; if (hue < 0) direction = 1; Perhaps hue is not the best choice of variable names. It represents the red component for an RGB color value. The range of values for this component is 0 (no red) to 255 (all the reds!). On each iteration of our loop we “move” the hue towards either the red or black. The variable direction can be either 1 or -1. A value of 1 means we are moving towards 255 and a value of -1 means we are moving towards 0. When we cross a boundary, we flip the direction. Keen observers will ask why we bother with 1 * direction. In our current logic, it’s an unnecessary step and unnecessary steps in game development are generally bad. In this case, I wanted to separate the rate of change from the direction. In order words, you could modify that expression to 2 * direction and the color would change twice as fast. This leads us to another important point. Our rate of change is tied to how quickly our loop iterates; most likely 60fps. However, it’s not guaranteed to be 60fps and that makes this approach a dangerous practice. Once way to detach ourselves from the loop’s speed would be to use the elapsed time that is being passed into our update function. Let’s say that we want to it to take 2 full seconds to go from red to black regardless of how often the update function is called. There’s a span of 256 discrete values between red and black. To make our calculations clear, let’s say there are 256 units and we’ll label these units R. Also, the elapsed time will be in milliseconds (ms). For a given frame, if were are given a slice of elapsed time in ms, we’ll want to calculate how many R units to increase (or decrease) hue by for that slice. Our rate of change can be defined as 256 **R** / 2000 **ms** or 0.128 R/ms. (You can read that as “0.128 units of red per millisecond”.) This rate of change is a constant for our start screen and as such we can define it once (as opposed to calculating it inside the update function). Now that we have the rate of change , we only need to multiply it by the elapsed time received in update to determine how many Rs we want. A revised version of the function would look like this: var rate = 0.128; // R/ms function update(elapsed) { var amount = rate * elapsed; hue += amount * direction; if (hue > 255) direction = -1; if (hue < 0) direction = 1; } One consequence of this change is that hue will no longer be integral values (as much as that can be said in JavaScript.) This means that we’d really want to have two values for the hue: an actual value and a rounded value. This is because the RBG model requires an integral value for each color component. function update(elapsed) { var amount = rate * elapsed; hue += amount * direction; if (hue > 255) direction = -1; if (hue < 0) direction = 1; rounded_hue = Math.round(hue); } Draw Let’s turn our attention to draw for a moment. One of the first things you generally do is to clear the entire screen. This is simple to do with the canvas API’s clearRect method. ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height); Notice that ctx is an instance of CanvasRenderingContext2D and not a HTMLCanvasElement. However, there is a handy back reference to the canvas element that we use to grab the actual width and height. There are other options other than clearing the entire canvas, but I’m not going to address this in this post. Also, there are some performance considerations. See the article listed under references. After clearing the screen, we want to draw something new. In this case, the game title and the instructions. In both cases I want to center the text horizontally. I created a helper function that I can provide with the text to render as well as the vertical position (y). function centerText(ctx, text, y) { var measurement = ctx.measureText(text); var x = (ctx.canvas.width - measurement.width) / 2; ctx.fillText(text, x, y); } measureText returns the width in pixels that the rendered text will take up. We use this in combination with the canvas element’s width to determine the x position for the text. fillText is responsible for actually drawing the text. The rendering context ctx is stateful. Meaning that, what happens when you call methods like measureText or fillText depends on the state of the rendering context. The state can be modified by setting its properties. var y = ctx.canvas.height / 2; ctx.fillStyle = 'white'; ctx.font = '48px monospace'; centerText(ctx, 'My Awesome Game', y); The properties fillStyle and font change the state of the rendering context and hence affect the methods calls inside of centerText. This state applies to all future methods calls. This means that all calls to fillText will use the color white until you can the fillStyle. Notice too that we are calculating the x and y values for the text on every frame. This is potentially wasteful since these values are unlikely to change. However, if we want to respond to changes in canvas size (or even changes to the text itself) then we’d want to continue calculating these on every frame. Otherwise, if we were confident that we didn’t need to do this, we could calculate these values once and cache them. Now let’s use the red component calculated in update to render the instructional text. var color = 'rgb(' + hue + ',0,0)'; ctx.fillStyle = color; ctx.font = '24px monospace'; centerText(ctx, 'click to begin', y + 30); fillStyle can be set in a number of ways. Earlier, we used the simple value white. Here were are using rgb() to set the individual components explicitly. Any CSS color should work with fillStyle. (I won’t be too surprised if some don’t though.) Now you might be wondering why we bothered calculating hue inside update since hue is all about what to draw on the screen. The reason is that draw is concerned with the mechanics of rendering. Anything that is modeling the game state should live in update. The tell in this example is that hue is dependent on elapsed time and the draw doesn’t know anything about that. Update (again) Moving back to update, the next bit deals with input from the player. In the sample code I’ve extracted the input logic away. The key thing here is that we are not relying on events to tell us about input from the player. Instead we have some helper, input in this case, that gives us the current state of the input. If event-driven logic says “tell me when this happens” then our game logic says “tell me if this is happening now”. The primary reason for this is to be deterministic. We can establish at the beginning of our update what the current input state is and that it won’t change before the next invocation of the function. In simple games this might be inconsequential, but in others it can be a subtle source of bugs. var isButtonDown = input.isButtonDown(); var mouseJustClicked = !isButtonDown && wasButtonDown; if (mouseJustClicked && !transitioning) { transitioning = true; // do something here to transition to the actual game } wasButtonDown = isButtonDown; We only want transition when the mouse button has been released. In this case, “released” is defined as “down on the last frame but up on this one”. Hence, we need to track what the mouse button’s state was on the last frame. That’s wasButtonDown and it lives outside of update. Secondly, we don’t want to trigger multiple transitions. That is, if our transition takes some time (perhaps due to animation) then we want to ignore subsequent clicks. We have our transitioning variable outside of update to track that for us. More to come… Update: I just realized that I didn't include a shim for requestAnimationFrame for the demo on jsfiddle. That means the demo will fail on many browsers. (Of course, it will also fail if there's no canvas support either.) If it doesn't work, check your console for errors.

Spring 3.1 introduced great caching abstraction layer. Finally we can abandon all home-grown aspects, decorators and code polluting our business logic related to caching. Since then we can simply annotate heavyweight methods and let Spring and AOP machinery do the work: @Cacheable("books") public Book findBook(ISBN isbn) {...} "books" is a cache name, isbn parameter becomes cache key and returned Book object will be placed under that key. The meaning of cache name is dependant on the underlying cache manager (EhCache, concurrent map, etc.) - Spring makes it easy to plug different caching providers. But this post won't be about caching feature in Spring... Some time ago my teammate was optimizing quite low-level code and discovered an opportunity for caching. He quickly applied @Cacheable just to discover that the code performed worse then it used to. He got rid of the annotation and implemented caching himself manually, using good old java.util.ConcurrentHashMap. The performance was much better. He blamed @Cacheable and Spring AOP overhead and complexity. I couldn't believe that a caching layer can perform so poorly until I had to debug Spring caching aspects few times myself (some nasty bug in my code, you know, cache invalidation is one of the two hardest things in CS). Well, the caching abstraction code is much more complex than one would expect (after all it's just get and put!), but it doesn't necessarily mean it must be that slow? In science we don't believe and trust, we measure and benchmark. So I wrote a benchmark to precisely measure the overhead of @Cacheable layer. Caching abstraction layer in Spring is implemented on top of Spring AOP, which can further be implemented on top of Java proxies, CGLIB generated subclasses or AspectJ instrumentation. Thus I'll test the following configurations: no caching at all - to measure how fast the code is with no intermediate layer manual cache handling using ConcurrentHashMap in business code @Cacheable with CGLIB implementing AOP @Cacheable with java.lang.reflect.Proxy implementing AOP @Cacheable with AspectJ compile time weaving (as similar benchmark shows, CTW is slightly faster than LTW) Home-grown AspectJ caching aspect - something between manual caching in business code and Spring abstraction Let me reiterate: we are not measuring the performance gain of caching and we are not comparing various cache providers. That's why our test method is as fast as it can be and I will be using simplest ConcurrentMapCacheManager from Spring. So here is a method in question: public interface Calculator { int identity(int x); } public class PlainCalculator implements Calculator { @Cacheable("identity") @Override public int identity(int x) { return x; } } I know, I know there is no point in caching such a method. But I want to measure the overhead of caching layer (during cache hit to be specific). Each caching configuration will have its own ApplicationContext as you can't mix different proxying modes in one context: public abstract class BaseConfig { @Bean public Calculator calculator() { return new PlainCalculator(); } } @Configuration class NoCachingConfig extends BaseConfig {} @Configuration class ManualCachingConfig extends BaseConfig { @Bean @Override public Calculator calculator() { return new CachingCalculatorDecorator(super.calculator()); } } @Configuration abstract class CacheManagerConfig extends BaseConfig { @Bean public CacheManager cacheManager() { return new ConcurrentMapCacheManager(); } } @Configuration @EnableCaching(proxyTargetClass = true) class CacheableCglibConfig extends CacheManagerConfig {} @Configuration @EnableCaching(proxyTargetClass = false) class CacheableJdkProxyConfig extends CacheManagerConfig {} @Configuration @EnableCaching(mode = AdviceMode.ASPECTJ) class CacheableAspectJWeaving extends CacheManagerConfig { @Bean @Override public Calculator calculator() { return new SpringInstrumentedCalculator(); } } @Configuration @EnableCaching(mode = AdviceMode.ASPECTJ) class AspectJCustomAspect extends CacheManagerConfig { @Bean @Override public Calculator calculator() { return new ManuallyInstrumentedCalculator(); } } Each @Configuration class represents one application context. CachingCalculatorDecorator is a decorator around real calculator that does the caching (welcome to the 1990s): public class CachingCalculatorDecorator implements Calculator { private final Map cache = new java.util.concurrent.ConcurrentHashMap(); private final Calculator target; public CachingCalculatorDecorator(Calculator target) { this.target = target; } @Override public int identity(int x) { final Integer existing = cache.get(x); if (existing != null) { return existing; } final int newValue = target.identity(x); cache.put(x, newValue); return newValue; } } SpringInstrumentedCalculator and ManuallyInstrumentedCalculator are exactly the same as PlainCalculator but they are instrumented by AspectJ compile-time weaver with Spring and custom aspect accordingly. My custom caching aspect looks like this: public aspect ManualCachingAspect { private final Map cache = new ConcurrentHashMap(); pointcut cacheMethodExecution(int x): execution(int com.blogspot.nurkiewicz.cacheable.calculator.ManuallyInstrumentedCalculator.identity(int)) && args(x); Object around(int x): cacheMethodExecution(x) { final Integer existing = cache.get(x); if (existing != null) { return existing; } final Object newValue = proceed(x); cache.put(x, (Integer)newValue); return newValue; } } After all this preparation we can finally write the benchmark itself. At the beginning I start all the application contexts and fetch Calculator instances. Each instance is different. For example noCaching is a PlainCalculator instance with no wrappers, cacheableCglib is a CGLIB generated subclass while aspectJCustom is an instance of ManuallyInstrumentedCalculator with my custom aspect woven. private final Calculator noCaching = fromSpringContext(NoCachingConfig.class); private final Calculator manualCaching = fromSpringContext(ManualCachingConfig.class); private final Calculator cacheableCglib = fromSpringContext(CacheableCglibConfig.class); private final Calculator cacheableJdkProxy = fromSpringContext(CacheableJdkProxyConfig.class); private final Calculator cacheableAspectJ = fromSpringContext(CacheableAspectJWeaving.class); private final Calculator aspectJCustom = fromSpringContext(AspectJCustomAspect.class); private static Calculator fromSpringContext(Class config) { return new AnnotationConfigApplicationContext(config).getBean(Calculator.class); } I'm going to exercise each Calculator instance with the following test. The additional accumulator is necessary, otherwise JVM might optimize away the whole loop (!): private int benchmarkWith(Calculator calculator, int reps) { int accum = 0; for (int i = 0; i < reps; ++i) { accum += calculator.identity(i % 16); } return accum; } Here is the full caliper test without parts already discussed: public class CacheableBenchmark extends SimpleBenchmark { //... public int timeNoCaching(int reps) { return benchmarkWith(noCaching, reps); } public int timeManualCaching(int reps) { return benchmarkWith(manualCaching, reps); } public int timeCacheableWithCglib(int reps) { return benchmarkWith(cacheableCglib, reps); } public int timeCacheableWithJdkProxy(int reps) { return benchmarkWith(cacheableJdkProxy, reps); } public int timeCacheableWithAspectJWeaving(int reps) { return benchmarkWith(cacheableAspectJ, reps); } public int timeAspectJCustom(int reps) { return benchmarkWith(aspectJCustom, reps); } } I hope you are still following our experiment. We are now going to execute Calculate.identity() millions of times and see which caching configuration performs best. Since we only call identity() with 16 different arguments, we hardly ever touch the method itself as we always get cache hit. Curious to see the results? benchmark ns linear runtime NoCaching 1.77 = ManualCaching 23.84 = CacheableWithCglib 1576.42 ============================== CacheableWithJdkProxy 1551.03 ============================= CacheableWithAspectJWeaving 1514.83 ============================ AspectJCustom 22.98 = Interpretation Let's go step by step. First of all calling a method in Java is pretty darn fast! 1.77 nanoseconds, we are talking here about 3 CPU cycles on my Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHz! If this doesn't convince you that Java is fast, I don't know what will. But back to our test. Hand-made caching decorator is also pretty fast. Of course it's slower by an order of magnitude compared to pure function call, but still blazingly fast compared to all @Scheduled benchmarks. We see a drop by 3 orders of magnitude, from 1.8 ns to 1.5 μs. I'm especially disappointed by the @Cacheable backed by AspectJ. After all caching aspect is precompiled directly into my Java .class file, I would expect it to be much faster compared to dynamic proxies and CGLIB. But that doesn't seem to be the case. All three Spring AOP techniques are similar. The greatest surprise is my custom AspectJ aspect. It's even faster than CachingCalculatorDecorator! maybe it's due to polymorphic call in the decorator? I strongly encourage you to clone this benchmark on GitHub and run it (mvn clean test, takes around 2 minutes) to compare your results. Conclusions You might be wondering why Spring abstraction layer is so slow? Well, first of all, check out the core implementation in CacheAspectSupport - it's actually quite complex. Secondly, is it really that slow? Do the math - you typically use Spring in business applications where database, network and external APIs are the bottleneck. What latencies do you typically see? Milliseconds? Tens or hundreds of milliseconds? Now add an overhead of 2 μs (worst case scenario). For caching database queries or REST calls this is completely negligible. It doesn't matter which technique you choose. But if you are caching very low-level methods close to the metal, like CPU-intensive, in-memory computations, Spring abstraction layer might be an overkill. The bottom line: measure! PS: both benchmark and contents of this article in Markdown format are freely available.

STOMP (Simple (or Streaming) Text-Oriented Messaging Protocol) is a simple text-oriented protocol, similar to HTTP. STOMP provides an interoperable wire format that allows clients to communicate with almost every available message broker. STOMP is easy to implement and gives you flexibility since it is language-agnostic, meaning clients and brokers developed in different languages can send and receive messages to and from each other. There are lots of server implementations that support STOMP (mostly compliant with the STOMP 1.0 specification). The following is a list of STOMP 1.0 compliant message servers: Apache ActiveMQ – http://activemq.apache.org Apache Apollo – http://activemq.apache.org/apollo CoilMQ – http://code.google.com/p/coilmq Gozirra - http://www.germane-software.com/software/Java/Gozirra HornetQ – http://www.jboss.com/hornetq MorbidQ – http://www.morbidq.com RabbitMQ - http://www.rabbitmq.com/plugins.html#rabbitmq-stomp Sprinkle - http://www.thuswise.org/sprinkle/index.html StompServer - http://stompserver.rubyforge.org/ On the client side, there are many implementations for a vast number of technologies. Below, you will find the libraries available for the most popular languages. Language Libraries C libstomp - http://stomp.codehaus.org/C C++ Apache CMS - http://activemq.apache.org/cms/ C# and .Net Apache NMS - http://activemq.apache.org/nms/ Flash as3-stomp - http://code.google.com/p/as3-stomp/ Java Gozirra - http://www.germane-software.com/software/Java/Gozirra/ Objective-C objc-stomp - https://github.com/juretta/objc-stomp Perl Net::Stomp::Client - http://search.cpan.org/dist/Net-STOMP-Client/ Net::Stomp - http://search.cpan.org/dist/Net-Stomp/ PHP stomp - http://www.php.net/manual/en/book.stomp.php stomp-php - http://stomp.fusesource.org/documentation/php/book.html Python stomper - https://github.com/oisinmulvihill/stomper stomp.py - http://code.google.com/p/stomppy/ Ruby on Rails stomp gem - https://rubygems.org/gems/stomp activemessaging - http://code.google.com/p/activemessaging/ STOMP is an alternative to other open messaging protocols such as AMQP (Advanced Message Queueing Protocol) and implementation specific wire protocols used in JMS (Java Message Service) brokers such as OpenWire. It distinguishes itself by covering a small subset of commonly used messaging operations (commands) rather than providing a comprehensive messaging API. Because STOMP is language-agnostic and easy to implement, it’s popular to application developers and technical architects alike. STOMP is also text-based. Since it does not use binary protocols like other message brokers, a wide range of client technologies work with STOMP, like Ruby, Python, and Perl. STOMP is simple to implement, but supports a wide range of core enterprise messaging features, such as authentication, messaging models (point to point and publish and subscribe), message acknowledgement, transactions, message headers and properties, etc.

The term security has many meanings based on the context and perspective in which it is used. Security from the perspective of software/system development is the continuous process of maintaining confidentiality, integrity, and availability of a system, sub-system, and system data. This definition at a very high level can be restated as the following: Computer security is a continuous process dealing with confidentiality, integrity, and availability on multiple layers of a system. Key Aspects of Software Security Integrity Confidentiality Availability Integrity within a system is the concept of ensuring only authorized users can only manipulate information through authorized methods and procedures. An example of this can be seen in a simple lead management application. If the business decided to allow each sales member to only update their own leads in the system and sales managers can update all leads in the system then an integrity violation would occur if a sales member attempted to update someone else’s leads. An integrity violation occurs when a team member attempts to update someone else’s lead because it was not entered by the sales member. This violates the business rule that leads can only be update by the originating sales member. Confidentiality within a system is the concept of preventing unauthorized access to specific information or tools. In a perfect world the knowledge of the existence of confidential information/tools would be unknown to all those who do not have access. When this this concept is applied within the context of an application only the authorized information/tools will be available. If we look at the sales lead management system again, leads can only be updated by originating sales members. If we look at this rule then we can say that all sales leads are confidential between the system and the sales person who entered the lead in to the system. The other sales team members would not need to know about the leads let alone need to access it. Availability within a system is the concept of authorized users being able to access the system. A real world example can be seen again from the lead management system. If that system was hosted on a web server then IP restriction can be put in place to limit access to the system based on the requesting IP address. If in this example all of the sales members where accessing the system from the 192.168.1.23 IP address then removing access from all other IPs would be need to ensure that improper access to the system is prevented while approved users can access the system from an authorized location. In essence if the requesting user is not coming from an authorized IP address then the system will appear unavailable to them. This is one way of controlling where a system is accessed. Through the years several design principles have been identified as being beneficial when integrating security aspects into a system. These principles in various combinations allow for a system to achieve the previously defined aspects of security based on generic architectural models. Security Design Principles Least Privilege Fail-Safe Defaults Economy of Mechanism Complete Mediation Open Design Separation Privilege Least Common Mechanism Psychological Acceptability Defense in Depth Least Privilege Design Principle The Least Privilege design principle requires a minimalistic approach to granting user access rights to specific information and tools. Additionally, access rights should be time based as to limit resources access bound to the time needed to complete necessary tasks. The implications of granting access beyond this scope will allow for unnecessary access and the potential for data to be updated out of the approved context. The assigning of access rights will limit system damaging attacks from users whether they are intentional or not. This principle attempts to limit data changes and prevents potential damage from occurring by accident or error by reducing the amount of potential interactions with a resource. Fail-Safe Defaults Design Principle The Fail-Safe Defaults design principle pertains to allowing access to resources based on granted access over access exclusion. This principle is a methodology for allowing resources to be accessed only if explicit access is granted to a user. By default users do not have access to any resources until access has been granted. This approach prevents unauthorized users from gaining access to resource until access is given. Economy of Mechanism Design Principle The Economy of mechanism design principle requires that systems should be designed as simple and small as possible. Design and implementation errors result in unauthorized access to resources that would not be noticed during normal use. Complete Mediation Design Principle The Complete Mediation design principle states that every access to every resource must be validated for authorization. Open Design Design Principle The Open Design Design Principle is a concept that the security of a system and its algorithms should not be dependent on secrecy of its design or implementation Separation Privilege Design Principle The separation privilege design principle requires that all resource approved resource access attempts be granted based on more than a single condition. For example a user should be validated for active status and has access to the specific resource. Least Common Mechanism Design Principle The Least Common Mechanism design principle declares that mechanisms used to access resources should not be shared. Psychological Acceptability Design Principle The Psychological Acceptability design principle refers to security mechanisms not make resources more difficult to access than if the security mechanisms were not present Defense in Depth Design Principle The Defense in Depth design principle is a concept of layering resource access authorization verification in a system reduces the chance of a successful attack. This layered approach to resource authorization requires unauthorized users to circumvent each authorization attempt to gain access to a resource. When designing a system that requires meeting a security quality attribute architects need consider the scope of security needs and the minimum required security qualities. Not every system will need to use all of the basic security design principles but will use one or more in combination based on a company’s and architect’s threshold for system security because the existence of security in an application adds an additional layer to the overall system and can affect performance. That is why the definition of minimum security acceptably is need when a system is design because this quality attributes needs to be factored in with the other system quality attributes so that the system in question adheres to all qualities based on the priorities of the qualities. Resources: Barnum, Sean. Gegick, Michael. (2005). Least Privilege. Retrieved on August 28, 2011 from https://buildsecurityin.us-cert.gov/bsi/articles/knowledge/principles/351-BSI.html Saltzer, Jerry. (2011). BASIC PRINCIPLES OF INFORMATION PROTECTION. Retrieved on August 28, 2011 from http://web.mit.edu/Saltzer/www/publications/protection/Basic.html Barnum, Sean. Gegick, Michael. (2005). Defense in Depth. Retrieved on August 28, 2011 from https://buildsecurityin.us-cert.gov/bsi/articles/knowledge/principles/347-BSI.html Bertino, Elisa. (2005). Design Principles for Security. Retrieved on August 28, 2011 from http://homes.cerias.purdue.edu/~bhargav/cs526/security-9.pdf

Optimization is a very common problem in data analytics. Given a set of variables (which one has control), how to pick the right value such that the benefit is maximized. More formally, optimization is about determining a set of variables x1, x2, … that maximize or minimize an objective function f(x1, x2, …). Unconstrained optimization In an unconstraint case, the variable can be freely selected within its full range. A typical solution is to compute the gradient vector of the objective function [∂f/∂x1, ∂f/∂x2, …] and set it to [0, 0, …]. Solve this equation and output the result x1, x2 … which will give the local maximum. In R, this can be done by a numerical analysis method. > f <- function(x){(x[1] - 5)^2 + (x[2] - 6)^2} > initial_x <- c(10, 11) > x_optimal <- optim(initial_x, f, method="CG") > x_min <- x_optimal$par > x_min [1] 5 6 Equality constraint optimization Moving onto the constrained case, lets say x1, x2 … are not independent and then have to related to each other in some particular way: g1(x1, x2, …) = 0, g2(x1, x2, …) = 0. The optimization problem can be expressed as … Maximize objective function: f(x1, x2, …) Subjected to equality constraints: g1(x1, x2, …) = 0 g2(x1, x2, …) = 0 A typical solution is to turn the constraint optimization problem into an unconstrained optimization problem using Lagrange multipliers. Define a new function F as follows ... F(x1, x2, …, λ1, λ2, …) = f(x1, x2, …) + λ1.g1(x1, x2, …) + λ2.g2(x1, x2, …) + … Then solve for ... [∂F/∂x1, ∂F/∂x2, …, ∂F/∂λ1, ∂F/∂λ2, …] = [0, 0, ….] Inequality constraint optimization In this case, the constraint is inequality. We cannot use the Lagrange multiplier technique because it requires equality constraint. There is no general solution for arbitrary inequality constraints. However, we can put some restriction in the form of constraint. In the following, we study two models where constraint is restricted to be a linear function of the variables: w1.x1 + w2.x2 + … >= 0 Linear Programming Linear programming is a model where both the objective function and constraint function is restricted as linear combination of variables. The Linear Programming problem can be defined as follows ... Maximize objective function: f(x1, x2) = c1.x1 + c2.x2 Subjected to inequality constraints: a11.x1 + a12.x2 <= b1 a21.x1 + a22.x2 <= b2 a31.x1 + a32.x2 <= b3 x1 >= 0, x2 >=0 As an illustrative example, lets walkthrough a portfolio investment problem. In the example, we want to find an optimal way to allocate the proportion of asset in our investment portfolio. StockA gives 5% return on average StockB gives 4% return on average StockC gives 6% return on average To set some constraints, lets say my investment in C must be less than sum of A, B. Also, investment in A cannot be more than twice of B. Finally, at least 10% of investment in each stock. To formulate this problem: Variable: x1 = % investment in A, x2 = % in B, x3 = % in C Maximize expected return: f(x1, x2, x3) = x1*5% + x2*4% + x3*6% Subjected to constraints: 10% < x1, x2, x3 < 100% x1 + x2 + x3 = 1 x3 < x1 + x2 x1 < 2 * x2 > library(lpSolve) > library(lpSolveAPI) > # Set the number of vars > model <- make.lp(0, 3) > # Define the object function: for Minimize, use -ve > set.objfn(model, c(-0.05, -0.04, -0.06)) > # Add the constraints > add.constraint(model, c(1, 1, 1), "=", 1) > add.constraint(model, c(1, 1, -1), ">", 0) > add.constraint(model, c(1, -2, 0), "<", 0) > # Set the upper and lower bounds > set.bounds(model, lower=c(0.1, 0.1, 0.1), upper=c(1, 1, 1)) > # Compute the optimized model > solve(model) [1] 0 > # Get the value of the optimized parameters > get.variables(model) [1] 0.3333333 0.1666667 0.5000000 > # Get the value of the objective function > get.objective(model) [1] -0.05333333 > # Get the value of the constraint > get.constraints(model) [1] 1 0 0 Quadratic Programming Quadratic programming is a model where both the objective function is a quadratic function (contains up to two term products) while constraint function is restricted as linear combination of variables. The Quadratic Programming problem can be defined as follows ... Minimize quadratic objective function: f(x1, x2) = c1.x12 + c2. x1x2 + c2.x22 - (d1. x1 + d2.x2) Subject to constraints a11.x1 + a12.x2 == b1 a21.x1 + a22.x2 == b2 a31.x1 + a32.x2 >= b3 a41.x1 + a42.x2 >= b4 a51.x1 + a52.x2 >= b5 Express the problem in Matrix form: Minimize objective ½(DTx) - dTx Subject to constraint ATx >= b First k columns of A is equality constraint As an illustrative example, lets continue on the portfolio investment problem. In the example, we want to find an optimal way to allocate the proportion of asset in our investment portfolio. StockA gives 5% return on average StockB gives 4% return on average StockC gives 6% return on average We also look into the variance of each stock (known as risk) as well as the covariance between stocks. For investment, we not only want to have a high expected return, but also a low variance. In other words, stocks with high return but also high variance is not very attractive. Therefore, maximize the expected return and minimize the variance is the typical investment strategy. One way to minimize variance is to pick multiple stocks (in a portfolio) to diversify the investment. Diversification happens when the stock components within the portfolio moves from their average in different direction (hence the variance is reduced). The Covariance matrix ∑ (between each pairs of stocks) is given as follows: 1%, 0.2%, 0.5% 0.2%, 0.8%, 0.6% 0.5%, 0.6%, 1.2% In this example, we want to achieve a overall return of at least 5.2% of return while minimizing the variance of return To formulate the problem: Variable: x1 = % investment in A, x2 = % in B, x3 = % in C Minimize variance: xT∑x Subjected to constraint: x1 + x2 + x3 == 1 X1*5% + x2*4% + x3*6% >= 5.2% 0% < x1, x2, x3 < 100% > library(quadprog) > mu_return_vector <- c(0.05, 0.04, 0.06) > sigma <- matrix(c(0.01, 0.002, 0.005, + 0.002, 0.008, 0.006, + 0.005, 0.006, 0.012), + nrow=3, ncol=3) > D.Matrix <- 2*sigma > d.Vector <- rep(0, 3) > A.Equality <- matrix(c(1,1,1), ncol=1) > A.Matrix <- cbind(A.Equality, mu_return_vector, diag(3)) > b.Vector <- c(1, 0.052, rep(0, 3)) > out <- solve.QP(Dmat=D.Matrix, dvec=d.Vector, Amat=A.Matrix, bvec=b.Vector, meq=1) > out$solution [1] 0.4 0.2 0.4 > out$value [1] 0.00672 > Integration with Predictive Analytics Optimization is usually integrated with predictive analytics, which is another important part of data analytics. For an overview of predictive analytics, here is my previous blog about it. The overall processing can be depicted as follows: In this diagram, we use machine learning to train a predictive model in batch mode. Once the predictive model is available, observation data is fed into it at real time and a set of output variables is predicted. Such output variable will be fed into the optimization model as the environment parameters (e.g. return of each stock, covariance ... etc.) from which a set of optimal decision variable is recommended.

In many Agile training programs and conferences, a common question that gets raised is, does Agile/Scrum work in maintenance projects? I always say "YES" and the team needs to tweak or invent the practices to suit their needs. Maintenance projects could be enhancement projects OR pure defect fixing projects. Enhancement projects involve new set of developments over existing one. Since the developers get a new set of requirements at the end of each iteration, one can apply the standard set of Scrum practices with little or no modification. Defect Fixing projects involve fixing defects on closed or current projects not in development. Sometimes these projects are boring, especially if a new team has been hired for defect fixing purposes only. The customer sends a set of defects on a daily basis or weekly basis with a deadline to deliver. The development team needs to fix them ASAP and send the patch for further testing. While coaching one of such a defect fixing projects, I found that the following Scrum practices can be applied without much modification: 1. Daily Scrum meetings 2. A Scrum of Scrum 3. Modeling days while solving complex defects 4. Information radiators displaying InProgress, completed, reopened, closed defects and other information 5. Usage of Wiki for collaborating with the customer 6. Requirement workshop while understanding complex defects 7. Review and Retrospective A common problem that I have found in defect fixing projects is setting the iteration length. Especially if the defects are given on a day to day basis without prior knowledge of what you are going to get, it makes the life of the development team bit difficult. This can be solved by collaborating with the customer and coming up with a plan to have 1 or 2 weeks of iteration length.

Gradle is a build tool like Ant and Maven only much, much better!

Apache Camel provides an event notifier support class which allows you to keep information about what happened on Exchange, Route and Endpoint. One of the benefits of this class is that you can easily audit messages created in Camel Routes, collect information and report that in log by example. When developing an application, it is very important to calculate/measure elapsed time on the platform to find which part of your code, processor or system integrated which is the bad duck and must be improved. In three steps, I would show you How to enable this mechanism to report : - Time elapsed to call an endpoint (could be another camel route, web service, ...) - Time elapsed on the route exchange STEP 1 - Create a Class implementing the EventNotifierSupport public class AuditEventNotifier extends EventNotifierSupport { public void notify(EventObject event) throws Exception { if (event instanceof ExchangeSentEvent) { ExchangeSentEvent sent = (ExchangeSentEvent) event; log.info(">>> Took " + sent.getTimeTaken() + " millis to send to external system : " + sent.getEndpoint()); } if (event instanceof ExchangeCompletedEvent) {; ExchangeCompletedEvent exchangeCompletedEvent = (ExchangeCompletedEvent) event; Exchange exchange = exchangeCompletedEvent.getExchange(); String routeId = exchange.getFromRouteId(); Date created = ((ExchangeCompletedEvent) event).getExchange().getProperty(Exchange.CREATED_TIMESTAMP, Date.class); // calculate elapsed time Date now = new Date(); long elapsed = now.getTime() - created.getTime(); log.info(">>> Took " + elapsed + " millis for the exchange on the route : " + routeId); } } public boolean isEnabled(EventObject event) { return true; } protected void doStart() throws Exception { // filter out unwanted events setIgnoreCamelContextEvents(true); setIgnoreServiceEvents(true); setIgnoreRouteEvents(true); setIgnoreExchangeCreatedEvent(true); setIgnoreExchangeCompletedEvent(false); setIgnoreExchangeFailedEvents(true); setIgnoreExchangeRedeliveryEvents(true); setIgnoreExchangeSentEvents(false); } protected void doStop() throws Exception { // noop } } Not really complicated and the code is explicit. Check the doStart() method to enable/disable the events for which you would like to gather information. This example uses only Exchange.CREATED_TIMESTAMP property but the next version of Camel 2.7.0 will provide you the property exchange.RECEIVED_TIMESTAMP and so you will be able to calculate more easily the time spend by the exchange to call the different processors till it arrives at the end of the route. This example collects Date information but you can imagine to use this mechanism to check if your route processes the message according to SLA, .... STEP 2 - Instantiate the bean in Camel Spring XML By adding this bean definition, Camel will automatically register it to the CamelContext created. STEP 3 - Collect info into the log 18:10:46,060 | INFO | tp1238469515-285 | AuditEventNotifier | ? ? | 68 - org.apache.camel.camel-core - 2.6.0.fuse-00-00 | >>> Took 3 millis for the exchange on the route : mock-HTTP-Server 18:10:46,062 | INFO | tp2056154542-293 | AuditEventNotifier | ? ? | 68 - org.apache.camel.camel-core - 2.6.0.fuse-00-00 | >>> Took 25 millis to send to external system : Endpoint[http://localhost:9191/sis] 18:10:46,077 | INFO | tp2056154542-293 | AuditEventNotifier | ? ? | 68 - org.apache.camel.camel-core - 2.6.0.fuse-00-00 | >>> Took 103 millis for the exchange on the route : ws-to-sis

This article is by Stephen Mouring Jr, appearing courtesy of Scott Leberknight. This is part two of a two part blog series on how to read/write Apache Hive data from MapReduce jos. Part one (Writing Hive Tables from MapReduce) is here. So just as sometimes you need to write data to Hive with a custom MapReduce job, sometimes you need to read that data back from Hive with a custom MapReduce job. As covered in part one, Hive is a layer that sits on HDFS and imposes a standard convention on the structure of the files so it can interpret them as columns and rows. Reading data out of Hive is just a matter of parsing the files correctly. Recall that files processed by MapReduce (and by extension, Hive) are output as key value pairs. Hive ignores the keys (read as a BytesWritable with a value of null) and reads/writes the values as Text objects. The value of the Text object for each row is the concatenation of all the column values delimited by the delimiter of the table (which Hive defaults to the "char 1" ASCII character). Seems like a simple problem, so my first thought was to just using String.split() in the map() method of the MapReduce job. String SEPARATOR_FIELD = new String(new char[] {1}); String[] rowColumns = new String (rowTextObject.getBytes()).split(SEPARATOR_FIELD); In theory this should have worked perfectly, but unfortunately I have found that String.split() actually consumes repeated delimiters. This is a problem if any of the values in the row are blank, since split() will shift the positions of your columns and you will be unable to match up what values belong with which columns. An alternative would be to create a String from the Text object and iterate through it using indexOf(). This approach however requires extra object creation and depending on the scale of your MapReduce job and the size of your rows, may slow you down needlessly. So an alternative is to use the Text object's find() method. String SEPARATOR_FIELD = new String(new char[] {1}); String[] rowColumns = new String[NUMBER_OF_COLUMNS_IN_YOUR_HIVE_TABLE]; int start = 0; int end = 0; for (int i = 0; i < rowColumns.length; ++i) { end = rowTextObject.find(SEPARATOR_FIELD, start); if (end == -1) { end = rowString.getLength(); } rowColumns[i] = new String(rowTextObject.getBytes(), start, end-start); start = end + 1; } This will parse out each value into the appropriately index of the rowColumns array. Blank values will also be handled correctly and result in blank strings being inserted into the rowColumns array.

This small tip about how to show the negative currency value in your application. For Example if you have currency value like -1234 and you want to display it like -$1,234 according to you culture. Problem In C# to one of the way to format currency value easily is use ToString("c") with value will do work for you. For example check below code //format -1234 as currency.. Console.WriteLine((-1234).ToString("c")); Output This will display value in output like this ($1,234.00). But the actual problem with the formatting is its not displaying - sign with currency value. Solution So to come out of this You need to create a custom NumberFormatInfo from your current locale. Then you can specify it's CurrencyNegativePattern, for example: CultureInfo currentCulture = Thread.CurrentThread.CurrentCulture; CultureInfo newCulture = new CultureInfo(currentCulture.Name); newCulture.NumberFormat.CurrencyNegativePattern = 1; Thread.CurrentThread.CurrentCulture = newCulture; Console.WriteLine((-1234).ToString("c")); Output Now output of the above code is -$1,234.00. That's what actually needed display negative sign for negative currency value. So in above code actual trick done by NumberFormatInfo.CurrencyNegativePattern Property, which is set to 1 which associated with pattern -$n. There number of different pattern supported which you can check on MSDN and assign value to property according to your need. This will not affect positive value of currency i.e. if you pas 1234 as value its dispaly as $1,234 only. And for this this example my current culture is "en-US" . Conclusion I Hope you all like this simple and easy example of formatting negative currency. you can explore more on MSDN on reference provided at end of post. NumberFormatInfo Class - http://msdn.microsoft.com/en-us/library/system.globalization.numberformatinfo.aspx NumberFormatInfo.CurrencyNegativePattern Property - http://msdn.microsoft.com/en-us/library/system.globalization.numberformatinfo.currencynegativepattern.aspx

Parsing script or function options and non-option arguments is easy in Bash with getopts but there are some catches, such as the need to reset OPTIND. We will se how to do it using getopts, shift, and case. The code below will parse the function arguments and remove them so that $1 will refer to the first non-option argument (i.e. not starting with -). You would invoke it f.ex. as latest_recur -x Hello -a '*.txt'. # Find the latest files under the current dir, recursively; options: # -a list all, not only 30 latest # - pattern passed to find's -name; ex.: "*.log.processed" function latest_recur { local show_all= OPTIND=1 while getopts "ax:" opt; do case $opt in a) show_all=yes ;; x) echo "You said: $OPTARG" ;; \?) echo "Invalid option: -$OPTARG" >&2; return 1;; esac done shift $((OPTIND-1)) if [ -z "$1" ]; then NAME_ARG=""; else NAME_ARG="-name $1"; fi find -type f $NAME_ARG | xargs --no-run-if-empty stat --format '%Y :%y %n' | sort -nr | if [ -z "$show_all" ]; then head -n 30 -; else cat -; fi } #5, #9 the variable used to store the flag must be defined/reset first #6 OPTIND is a global variable pointing to the next argument that getopts should parse; you must reset it manually (otherwise the next call to the function will ignore its arguments) #7 getopts parses one by one all the supported options (a, x here) and stores them into $opt; #14 we must manually shift away the processed option arguments so that the first non-option argument (‘*.txt’) will become argument number 1 as you can see at #16; OPTIND is set by getopts Getopts can do quite a lot. It supports short options with or without arguments such as “-lht”, “-l -h -t”, “-l -I ‘*.swp’”. It can also report/ignore unknown arguments etc., see its brief documentation and this small getopts tutorial. Briefly said, getopts takes opstring and varname; opstring is a list of letters optionally followed by ‘:’ to indicate that that flag requires a value; varname is the name of the variable to store the flag name into. If you put : in front of the opstring (“:ax:”) then it will not complain about unknown options or missing arguments for options that require them.

How to list the sizes of Hive tables in Hadoop in GBs.

In continuation with my earlier blog on Enabling SSL on Tomcat, in this blog I will go to next step and enable CLIENT-CERT based authorization on Tomcat. Again if you want to tryout the code go to my Github and download the code. For this sample, I assume that you have tried my earlier SSL example on Tomcat and have the setup. As per the SSL example I assume, You have setup Tomcat 6.0 version You have set the SSL Connector Configuration in Tomcat server.xml You have started the Tomcat server and run the SecureHttpClient0Test test In this blog, I will show you how to, Setup MemoryRealm In the server.xml comment the Realm tag and replace that with the code below, Setup user role setup In /conf/tomcat-users.xml Setup security-contraint Add access control in the individual application web.xml as below, Demo App /secure/* GET secureconn CLIENT-CERT Demo App secureconn Run JUnit test Open the class src/test/java/com/goSmarter/test/SecureHttpClient1Test.java file and change the below code to point to /conf folder public static final String path = "D:/apache-tomcat-6.0.36/conf/"; Start the Tomcat and run the JUnit test using “mvn test -Dtest=”com.goSmarter.test.SecureHttpClient1Test” If you want to debug the Realm, you need to increase the log level for Realm in /conf/logging.properties as below, org.apache.catalina.realm.level = ALL org.apache.catalina.realm.useParentHandlers = true org.apache.catalina.authenticator.level = ALL org.apache.catalina.authenticator.useParentHandlers = true If you notice there are 2 positive tests and 1 negative test, negative test will give a forbidden 403 return status when a wrong certificate is sent based on the security-constraint. I hope this blog helped you.

PhantomJS is a headless WebKit with JavaScript API. It can be used for headless website testing. PhantomJS has a lot of different uses. The interesting bit for me is to use PhantomJS as a lighter-weight replacement for a browser when running web acceptance tests. This enables faster testing, without a display or the overhead of full-browser startup/shutdown. I write my web automation using Selenium WebDriver, in Python. In future versions of PhantomJS, the GhostDriver component will be included. GhostDriver is a pure JavaScript implementation of the WebDriver Wire Protocol for PhantomJS. It's a Remote WebDriver that uses PhantomJS as back-end. So, Ghostdriver is the bridge we need to use Selenium WebDriver with Phantom.JS. Since it is not available in the current PhantomJS release, you can try it yourself by compiling a special version of PhantomJS: It wes pretty trvial to setup on Ubuntu (12.04): $ sudo apt-get install build-essential chrpath git-core libssl-dev libfontconfig1-dev $ git clone git://github.com/ariya/phantomjs.git $ cd phantomjs $ git checkout 1.8 $ ./build.sh $ git remote add detro https://github.com/detro/phantomjs.git $ git fetch detro && git checkout -b detro-ghostdriver-dev remotes/detro/ghostdriver-dev $ ./build.sh Then grab the `phantomjs` binary it produced (look inside `phantomjs/bin`). This is a self-contained executable, it can be moved to a different directory or another machine. Make sure it is located somewhere on your PATH, or declare it's location when creating your PhantomJS driver like the example below. for these examples, `phantomjs` binary is located in same directory as test script. Example: Python Using PhantomJS and Selenium WebDriver. #!/usr/bin/env python driver = webdriver.PhantomJS('./phantomjs') # do webdriver stuff here driver.quit() Example: Python Unit Test Using PhantomJS and Selenium WebDriver. #!/usr/bin/env python import unittest from selenium import webdriver class TestUbuntuHomepage(unittest.TestCase): def setUp(self): self.driver = webdriver.PhantomJS('./phantomjs') def testTitle(self): self.driver.get('http://www.ubuntu.com/') self.assertIn('Ubuntu', self.driver.title) def tearDown(self): self.driver.quit() if __name__ == '__main__': unittest.main(verbosity=2) resources: http://phantomjs.org/build.html https://github.com/detro/ghostdriver Selenium WebDriver Python API Documentation

This article is by Stephen Mouring, Jr. On my project we have a number of software components that run concurrently, some on a cron, and some as part of our build process. Many of these components need to mutate data in our data store and have the possibility of conflicting with one another. What is worse is that many of these processes run on separate machines making language level or even file system level synchronization impossible. Zookeeper is a natural solution to the problem. It is a distributed system for, among other things, managing coordination across a cluster of machines. Zookeeper manages information as a hierarchical system of "nodes" (much like a file system). Each node can contain data or can contain child nodes. Zookeeper supports several types of nodes. A node can be either "ephemeral" or "persistent" meaning it is either deleted when the process that created it ends or it remains until manually deleted. A node can also be "sequential" meaning each time a node is created with a given name, a sequence number is postfixed to that name. This allows you to create a series of nodes with the same name that are ordered in the same order they were created. To solved our problem we need to have a locking mechanism that works across processes and across machines that allows one holder of the lock to execute at a given time. Below is the Java code we wrote to solve the problem. I will go through it step by step. public class DistributedLock { private final ZooKeeper zk; private final String lockBasePath; private final String lockName; private String lockPath; public DistributedLock(ZooKeeper zk, String lockBasePath, String lockName) { this.zk = zk; this.lockBasePath = lockBasePath; this.lockName = lockName; } public void lock() throws IOException { try { // lockPath will be different than (lockBasePath + "/" + lockName) becuase of the sequence number ZooKeeper appends lockPath = zk.create(lockBasePath + "/" + lockName, null, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); final Object lock = new Object(); synchronized(lock) { while(true) { List nodes = zk.getChildren(lockBasePath, new Watch() { @Override public void process(WatchedEvent event) { synchronized (lock) { lock.notifyAll(); } } }); Collections.sort(nodes); // ZooKeeper node names can be sorted lexographically if (lockPath.endsWith(nodes.get(0)) { return; } else { lock.wait(); } } } } catch (KeeperException e) { throw new IOException (e); } catch (InterruptedException e) { throw new IOException (e); } } public void unlock() throws IOException { try { zk.delete(lockPath, -1); lockPath = null; } catch (KeeperException e) { throw new IOException (e); } catch (InterruptedException e) { throw new IOException (e); } } } (Disclaimer: Credit for this code goes to Aaron McCurry for developing the core mechanism of this lock as well as the design for using ZooKeeper. Kudos to Aaron!) Each process that wants to use the lock should instantiate an object of the DistributedLock class. The DistributedLock constructor takes three parameters. The first parameter is a reference to the ZooKeeper client. The second parameter is the "base path" where you want your lock nodes to reside in. Remember that ZooKeeper stores its nodes like a file system, so think of this base path as the directory you want your lock nodes created in. The third parameter is the name of the lock to use. Note you should use the same lock name for every process that you want to share the same lock. The lock name is the common reference that multiple processes lock on. Note: This class can support multiple locks if you use a different lock name for each lock you want to create. Say you have two data stores (A and B). You have several processes that need mutate A and B. You could use two different lock names (say LockA and LockB) to represent the locks for each data store. Any process that needs to mutate data store A could create a DistributedLock with a lockname of LockA. Likewise, any process that needs to mutate data store B could create a DistributedLock with a lockname of LockB. A proces that needs to mutate both datastores would create two DistributedLock objects (one with lock name of LockA and one with a lock name of LockB). Once your process has created a DistributedLock object it can then call the lock() method to attempt to acquire the lock. The lock() method will block until the lock is acquired. // lockPath will be different than (lockBasePath + "/" + lockName) becuase of the sequence number ZooKeeper appends lockPath = zk.create(lockBasePath + "/" + lockName, null, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); First of all, the lock() method creates a node in ZooKeeper to represent its "position in line" waiting for the lock. The node created is EPHEMERAL which means if our process dies for some reason, its lock or request for the lock with automatically disappear thanks to ZooKeeper's node management, so we do not have worry about timing out nodes or cleaning up stale nodes. final Object lock = new Object(); synchronized(lock) { while(true) { List nodes = zk.getChildren(lockBasePath, new Watch() { @Override public void process(WatchedEvent event) { synchronized (lock) { lock.notifyAll(); } } }); // Sequential ZooKeeper node names can be sorted lexographically! Collections.sort(nodes); // Are we the "topmost" node? (The node with the lowest sequence number that is.) if (lockPath.endsWith(nodes.get(0)) { return; } else { lock.wait(); } } } To understand the code above you need to understand how ZooKeeper works. ZooKeeper operates through a system of callbacks. When you call getChildren() you can pass in a "watcher" that will get called anytime the list of children changes. The gist of what we are doing here is this. We are creating an ordered list of nodes (sharing the same name). Whenever the list changes, every process that has registered a node is notified. Since the nodes are ordered, one node will be "on top" or in other words have the lowest sequence number. That node is the node that owns the lock. When a process detects that its node is the top most node, it proceeds to execute. When it is finished, it deletes its node, triggering a notification to all other processes who then determine who the next node is who has the lock. The tricky part of the code from a Java perspective is the use of nested synchronized blocks. The nested synchronization structure is used to ensure that the DistributedLock is able to process every update it gets from ZooKeeper and does not "lose" an update if two or more updates come from ZooKeeper in quick succession. The inner synchronized block in the Watcher method is called from an outside thread whenever ZooKeeper reports a change to its children. Since the Watcher callback is in a synchronized block keyed to the same Java lock object as the outer synchronized block, it means that the update from ZooKeeper cannot be processed until the contents of the outer synchronized block is finished. In other words, when an update comes in from ZooKeeper, it fires a notifyAll() which wakes up the loop in the lock() method. That lock method gets the updated children and sets a new Watcher. (Watchers have to be reset once they fire as they are not a perpetual callback. They fire once and then disappear.) If the newly reset Watcher fires before the rest of the loop executes, it will block because it is synchronized on the same Java lock object as the loop. The loop finishes its pass, and if it has not acquired the distrubted lock, it waits on the Java lock object. This frees the Watcher to execute whenever a new update comes, repeating the cycle. Once the lock() method returns, it means your process has the dsitributed lock and can continue to execute its business logic. Once it is complete it can release the lock by calling the unlock() method. public void unlock() throws IOException { try { zk.delete(lockPath, -1); lockPath = null; } catch (KeeperException e) { throw new IOException (e); } catch (InterruptedException e) { throw new IOException (e); } } All unlock() does is explictly delete this process's node which notifies all the other waiting processes and allows the next one in line to go. Because the nodes are EPHEMERAL, the process can exit without unlocking and ZooKeeper will eventually reap its node allowing the next process to execute. This is a good thing because it means if your process ends prematurely without you having a chance to call unlock() it will not block the remaining processes. Note that it is best to explicitly call unlock() if you can, because it is much faster than waiting for ZooKeeper to reap your node. You will delay the other processes less if you explicity unlock.

Domain Specific Software Engineering (DSSE) believes that creating every application from nothing is not advantageous when existing systems can be leveraged to create the same application in less time and with less cost. This belief is founded in the idea that forcing applications to recreate exiting functionality is unnecessary. Why would we build a better wheel when we already have four really good and proven wheels? DSSE suggest that we take an existing wheel and just modify it to fit an existing need of a system. This allows developers to leverage existing codebases so that more time and expense are focused on creating more usable functionality compared to just creating more functionality. As an example, how many functions do we need to create to send an email when one can be created and used by all other applications within the existing domain? Key Factors of DSSE Domain Technology Business A Domain in DSSE is used to control the problem space for a project. This control allows for applications to be developed within specific constrains that focus development is to a specific direction. Technology in DSSE offers a variety of technological solutions to be applied within a domain. Technology Examples: Tools Patterns Architectures & Styles Legacy Systems Business is the motivator for any originations to use DSSE in there software development process. Business reason to use DSSE: Minimize Costs Maximize market and Profits When these factors are used in combination additional factors and benefits can be found. Result of combining Key Factors of DSSE Domain + Business = Corporate Core Competencies Domain expertise improved by market and business expertise Domain + Technology = Application Family Architectures All possible technological solutions to problems in a domain without any business constraints. Business + Technology = Domain independent infrastructure Tools and techniques for building systems independent of all domains Domain + Business + Technology = Domain-specific software engineering Applies technology to domain related goals in the context of business and market expertise

Wow, I can’t believe it’s been 6 weeks since I last blogged. I still need to write up a review on Plumbr, after seeing it in action at Devoxx, but to ease me back into the writing game, here’s a small(-ish) but useful spot of Java. It’s always nice to borrow and steal concepts and ideas from other languages. Scala’s Option is one idea I really like, so I wrote an implementation in Java. It wraps an object which may or may not be null, and provides some methods to work with in a more kinda-sorta functional way. For example, the isDefined method adds an object-oriented way of checking if a value is null. It is then used in other places, such as the getOrElse method, which basically says “give me what you’re wrapping, or a fallback if it’s null”. public T getOrElse(T fallback) { return isDefined() ? get() : fallback; } In practice, this would replace tradional Java, such as public void foo() { String s = dao.getValue(); if (s == null) { s = "bar"; } System.out.println(s); } with the more concise and OO public void foo() { Option s = dao.getValue(); System.out.println(s.getOrElse("bar")); } However, what if I want to do something other than get a fallback value – say, throw an exception? More to the point, what if I want to throw a specific type of exception that is both specific in use and not hard-coded into Option? This requires a spot of cunning, and a splash of type inference. Because this is Java, we can start with a new factory – ExceptionFactory. This is a basic implementation that only creates exceptions constructed with a message, but you can of course expand the code as required. public interface ExceptionFactory { E create(String message); } Notice the – this is the key to how this works. Using the factory, we can now add a new method to Option: public T getOrThrow(ExceptionFactory exceptionFactory, String message) throws E { if (isDefined()) { return get(); } else { throw exceptionFactory.create(message); } } Again, notice the throws E – this is inferred from the exception factory. And that, believe it or not, is 90% of what it takes. The one irritation is the need to have exception factories. If you can stomach this, you’re all set. Let’s define a couple of custom exceptions to see this in action. public class ExceptionA extends Exception { public ExceptionA(String message) { super(message); } public static ExceptionFactory factory() { return new ExceptionFactory() { @Override public ExceptionA create(String message) { return new ExceptionA(message); } }; } } And the suspiciously similar ExceptionB public class ExceptionB extends Exception { public ExceptionB(String message) { super(message); } public static ExceptionFactory factory() { return new ExceptionFactory() { @Override public ExceptionB create(String message) { return new ExceptionB(message); } }; } } And finally, throw it all together: public class GenericExceptionTest { @Test(expected = ExceptionA.class) public void exceptionA_throw() throws ExceptionA { Option.option(null).getOrThrow(ExceptionA.factory(), "Some message pertinent to the situation"); } @Test public void exceptionA_noThrow() throws ExceptionA { String s = Option.option("foo").getOrThrow(ExceptionA.factory(), "Some message pertinent to the situation"); Assert.assertEquals("foo", s); } @Test(expected = ExceptionB.class) public void exceptionB_throw() throws ExceptionB { Option.option(null).getOrThrow(ExceptionB.factory(), "Some message pertinent to the situation"); } @Test public void exceptionB_noThrow() throws ExceptionB { String s = Option.option("foo").getOrThrow(ExceptionB.factory(), "Some message pertinent to the situation"); Assert.assertEquals("foo", s); } } The important thing to notice, as highlighted in bold above, is the exception declared in the method signature is specific – it’s not a common ancestor (Exception or Throwable). This means you can now use Options in your DAO layer, your service layer, wherever, and throw specific exceptions where and how you need. Download source: You can get the source code and tests from here – genex Sidenote One other interesting thing that came out of writing this was the observation that it’s possible to do this: public void foo() { throw null; } public void bar() { try { foo(); } catch (NullPointerException e) { ... } } It goes without saying that this is not a good idea