Spring for Apache Hadoop is a Spring project to support writing applications that can benefit of the integration of Spring Framework and Hadoop. This post describes how to use Spring Data Apache Hadoop in an Amazon EC2 environment using the “Hello World” equivalent of Hadoop programming – a Wordcount application. 1./ Launch an Amazon Web Services EC2 instance. - Navigate to AWS EC2 Console (“https://console.aws.amazon.com/ec2/home”): - Select Launch Instance then Classic Wizzard and click on Continue. My test environment was a “Basic Amazon Linux AMI 2011.09″ 32-bit., Instant type: Micro (t1.micro , 613 MB), Security group quick-start-1 that enables ssh to be used for login. Select your existing key pair (or create a new one). Obviously you can select another AMI and instance types depending on your favourite flavour. (Should you vote for Windows 2008 based instance, you also need to have cygwin installed as an additional Hadoop prerequisite beside Java JDK and ssh, see “Install Apache Hadoop” section) 2./ Download Apache Hadoop - as of writing this article, 1.0.0 is the latest stable version of Apache Hadoop, that is what was used for testing purposes. I downloaded hadoop-1.0.0.tar.gz and copied it into /home/ec2-user directory using pscp command from my PC running Windows: c:\downloads>pscp -i mykey.ppk hadoop-1.0.0.tar.gz [email protected]:/home/ec2-user (the computer name above – ec2-ipaddress-region-compute.amazonaws.com – can be found on AWS EC2 console, Instance Description, public DNS field) 3./ Install Apache Hadoop: As prerequisites, you need to have Java JDK 1.6 and ssh installed, see Apache Single-Node Setup Guide. (ssh is automatically installed with Basic Amazon AMI). Then install hadoop itself: $ cd ~ # change directory to ec2-user home (/home/ec2-user) $ tar xvzf hadoop-1.0.0.tar.gz $ ln -s hadoop-1.0.0 hadoop $ cd hadoop/conf $ vi hadoop-env.sh # edit as below export JAVA_HOME=/opt/jdk1.6.0_29 $ vi core-site.xml # edit as below – this defines the namenode to be running on localhost and listeing to port 9000. fs.default.name hdfs://localhost:9000 $ vi hdsf-site.xml # edit as below this defines that file system replicate is 1 (in production environment it is supposed to be 3 by default) dfs.replication 1 $ vi mapred-site.xml # edit as below – this defines the jobtracker to be running on localhost and listeing to port 9001. mapred.job.tracker localhost:9001 $ cd ~/hadoop $ bin/hadoop namenode -format $ bin/start-all.sh At this stage all hadoop jobs are running in pseudo distributed mode, you can verify it by running: $ ps -ef | grep java You should see 5 java processes: namenode, secondarynamenode, datanode, jobtracker and tasktracker. 4./ Install Spring Data Hadoop Download Spring Data Hadoop package from SpringSource community download site. As of writing this article, the latest stable version is spring-data-hadoop-1.0.0.M1.zip. $ cd ~ $ tar xzvf spring-data-hadoop-1.0.0.M1.zip $ ln -s spring-data-hadoop-1.0.0.M1 spring-data-hadoop 5./ Build and Run Spring Data Hadoop Wordcount example $ cd spring-data-hadoop/spring-data-hadoop-1.0.0.M1/samples/wordcount Spring Data Hadoop is using gradle as build tool. Check build.grandle build file. The original version packaged in the tar.gz file does not compile, it complains about thrift, version 0.2.0 and jdo2-api, version2.3-ec. Add datanucleus.org maven repository to the build.gradle file to support jdo2-api (http://www.datanucleus.org/downloads/maven2/) . Unfortunatelly, there seems to be no maven repo for thrift 0.2.0 . You should download thrift 0.2.0.jar and thrift.0.2.0.pom file e.g. from this repo: “http://people.apache.org/~rawson/repo“ and then add it to local maven repo. $ mvn install:install-file -DgroupId=org.apache.thrift -DartifactId=thrift -Dversion=0.2.0 -Dfile=thrift-0.2.0.jar -Dpackaging=jar $ vi build.grandle # modify the build file to refer to datanucleus maven repo for jdo2-api and the local repo for thrift repositories { // Public Spring artefacts mavenCentral() maven { url “http://repo.springsource.org/libs-release” } maven { url “http://repo.springsource.org/libs-milestone” } maven { url “http://repo.springsource.org/libs-snapshot” } maven { url “http://www.datanucleus.org/downloads/maven2/” } maven { url “file:///home/ec2-user/.m2/repository” } } I also modified the META-INF/spring/context.xml file in order to run hadoop file system commands manually: $ cd /home/ec2-user/spring-data-hadoop/spring-data-hadoop-1.0.0.M1/samples/wordcount/src/main/resources $vi META-INF/spring/context.xml # remove clean-script and also the dependency on it for JobRunner. xmlns=”http://www.springframework.org/schema/beans” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xmlns:context=”http://www.springframework.org/schema/context” xmlns:hdp=”http://www.springframework.org/schema/hadoop” xmlns:p=”http://www.springframework.org/schema/p” xsi:schemaLocation=”http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd”> fs.default.name=${hd.fs} Copy the sample file – nietzsche-chapter-1.txt – to Hadoop file system (/user/ec2-user-/input directory) $ cd src/main/resources/data $ hadoop fs -mkdir /user/ec2-user/input $ hadoop fs -put nietzsche-chapter-1.txt /user/ec2-user/input/data $ cd ../../../.. # go back to samples/wordcount directory $ ../gradlew Verify the result: $ hadoop fs -cat /user/ec2-user/output/part-r-00000 | more “AWAY 1 “BY 1 “Beyond 1 “By 2 “Cheers 1 “DE 1 “Everywhere 1 “FROM” 1 “Flatterers 1 “Freedom 1

If you have already used the ASP.NET ListView and DataPager controls, you know how easily you can display your data using custom templates and provide pagination functionality. You can do that in only few minutes. The ListView and DataPager controls work perfectly fine in combination with DataSource controls (SqlDataSource, LinqDataSource, ObjectDataSource etc.), however if you use them with custom data collection, without using Data Source controls, you may find some unexpected behavior and have to add some little more code to make it work. Note: I saw question related to this issue asked multiple times in different asp.net forums, so I thought it would be nice to document it here. Lets create a working demo together… 1. Create sample ASP.NET Web Application project, add new ASPX page. 2. Add ListView control and modify the markup so that you will end up having this: ( ) No data The ListView control has three templates defined: LayoutTemplate – where we define the way we want to represent our data. We have PlaceHolder where data from ItemTemplate will be placed. ListView recognizes this automatically. ItemTemplate – where the ListView control will show the items from the data source. It makes automatically iterating over each item in the collection (same as any other data source control) EmptyDataTemplate – If the collection has no data, this template will be displayed. 3. Add DataPager control and modify the markup in the following way: We add the DataPager control, associate it with the ListView1 control and add the PageSize property. After that, we need to define where we put the field type and button type. If we were binding from Data Source Control (SqlDataSource, LinqDataSource or any other…) this would be it and the ListView and DataPager would work perfectly fine. However, if we bind custom data collection to the ListView without using Data Source controls, we will have problems with the pagination. Lets add custom data in our C# code and bind it to the ListView. 4. C# code adding custom data collection - Define Product class public class Product { public string Name { get; set; } public decimal Price { get; set; } public string Currency { get; set; } } - Define method that will create List of products (sample data) List SampleData() { List p = new List(); p.Add(new Product() { Name = "Microsoft Windows 7", Price = 70, Currency = "USD" }); p.Add(new Product() { Name = "HP ProBook", Price = 320, Currency = "USD" }); p.Add(new Product() { Name = "Microsoft Office Home", Price = 60, Currency = "USD" }); p.Add(new Product() { Name = "NOKIA N900", Price = 350, Currency = "USD" }); p.Add(new Product() { Name = "BlackBerry Storm", Price = 100, Currency = "USD" }); p.Add(new Product() { Name = "Apple iPhone", Price = 400, Currency = "USD" }); p.Add(new Product() { Name = "HTC myTouch", Price = 200, Currency = "USD" }); return p; } This method should be part of the ASPX page class (e.g. inside _Default page class if your page is Default.aspx) - Bind the sample data to ListView on Page_Load protected void Page_Load(object sender, EventArgs e) { if (!IsPostBack) { BindListView(); } } void BindListView() { ListView1.DataSource = SampleData(); ListView1.DataBind(); } Now, run the project and you should see the data displayed where only the first five items will be shown. The data pager should have two pages (1 2) and you will be able to click the second page to navigate to the last two items in the `collection. Now, if you click on page 2, you will see it won’t display the last two items automatically, instead you will have to click again on page 2 to see the last two items. After, if you click on page 1, you will encounter another problem where the five items are displayed, but the data for the last two items in the ListView are shown (see print screen bellow) If you notice in the previous two pictures, the behavior doesn’t seem to work properly. The problem here is that DataPager doesn’t know about current ListView page changing property. Therefore, we should explicitly set the DataPager Page Properties in the ListView’s PagePropertiesChanging event. Here is what we need to do: 1. Add OnPagePropertiesChanging event to ListView control 2. Implement the ListView1_PagePropertiesChanging method protected void ListView1_PagePropertiesChanging(object sender, PagePropertiesChangingEventArgs e) { //set current page startindex, max rows and rebind to false lvDataPager1.SetPageProperties(e.StartRowIndex, e.MaximumRows, false); //rebind List View BindListView(); } Now, if you test the functionality, it should work properly. Hope this was helpful. Regards, Hajan

the question everybody always asks when they learn about activiti, is as old as software development itself: “how does it perform?”. up till now, when you would ask me that same question, i would tell you about how activiti minimizes database access in every way possible, how we break down the process structure into an ‘execution tree’ which allows for fast queries or how we leverage ten years of workflow framework development knowledge. you know, trying to get around the question without answering it. we knew it is fast, because of the theoretical foundation upon which we have built it. but now we have proof: real numbers …. yes, it’s going to be a lengthy post. but trust me, it’ll be worth your time! disclaimer: performance benchmarks are hard. really hard. different machines, slight different test setup … very small things can change the results seriously. the numbers here are only to prove that the activiti engine has a very minimal overhead, while also integrating very easily into the java eco-system and offering bpmn 2.0 process execution. the activiti benchmark project to test process execution overhead of the activiti engine, i created a little side project on github: https://github.com/jbarrez/activiti-benchmark the project contains currently 9 test processes, which we’ll analyse below. the logic in the project is pretty straightforward: a process engine is created for each test run each of the processes are sequentially executed on this process engine, using a threadpool from 1 up to 10 threads. all the processes are thrown into a bag, of which a number of random executions are drawn. all the results are collected and a html report with some nice charts are generated to run the benchmark, simply follow the instructions on the github page to build and execute the jar. benchmark results the test machine i used for the results is my (fairly old) desktop machine: amd phenom ii x4 940 3.0ghz, 8 gb 800mhz ram and an old-skool 7200 rpm hd running ubuntu 11.10. the database used for the test runs on the same machine on which the tests also run. so keep in mind that in a ‘real’ server environment the results could even be better! the benchmark project i mentioned above, was executed on a default ubuntu mysql 5 database. i just switched to the ‘large.cnf’ setting (which throws more ram at the db and stuff like that) instead the default config. each of the test processes ran for 2500 times, using a threadpool going from one to ten threads . in simpleton language: 2500 process executions using just one thread, 2500 threads using two threads, 2500 process executions using three … yeah, you get it. each benchmark run was done using a ‘default’ activiti process engine. this basically means a ‘regular’ standalone activiti engine, created in plain java. each benchmark run was also done in a ‘spring’ config. here, the process engine was constructed by wrapping it in the factory bean, the datasource is a spring datasource and also the transactions and connection pool is managed by spring (i’m actually using a tweaked bonecp threadpool) each benchmark run was executed with history on the default history level (ie. ‘audit’) and without history enabled (ie. history level ‘none’) . the processes are in detail analyzed in the sections below, but here are the integral results of the test runs already: activiti 5.9 – mysql – default – history enabled activiti 5.9 – mysql – default – history disabled activiti 5.9 – mysql – spring – history enabled activiti 5.9 – mysql – spring – history disabled i ran all the tests using the latest public release of activiti, being activiti 5.9. however, my test runs brought some potential performance fixes to the surface (i also ran the benchmark project through a profiler). it was quickly clear that most of the process execution time was done actually cleaning up when a process ended. basically, more than often queries were fired which were not necessary if we would save some more state in our execution tree. i sat together with daniel meyer from camunda and my colleague frederik heremans, and they’ve managed to commit fixes for this! as such, the current trunk of activiti, being activiti 5.10-snapshot at the moment, is significantly faster than 5.9 . activiti 5.10 – mysql – default – history enabled activiti 5.10 – mysql – default – history disabled activiti 5.10 – mysql – spring – history enabled activiti 5.10 – mysql – spring – history disabled from a high-level perspective (scroll down for detailed analysis), there are a few things to note: i had expected some difference between the default and spring config, due to the more ‘professional’ connection pool being used. however, the results for both environments are quite alike. sometimes the default is faster, sometimes spring. it’s hard to really find a pattern. as such, i omitted the spring results in the detailed analyses below. the best average timings are most of the times found when using four threads to execute the processes . this is probably due to having a quad-core machine. the best throughput numbers are most of the times found when using eight threads to execute the processes. i can only assume that is also has something to do with having a quad-core machine. when the number of threads in the threadpool go up, the throughput (processes executed / second) goes up, both it has a negative effect on the average time. certainly with more than six or seven threads, you see this effect very clear. this basically means that while the processes on itself take a little longer to execute, but due to the multiple threads you can execute more of these ‘slower’ processes in the same amount of time. enabling history does have an impact. often, enabling history will double execution time. this is logical, given that many extra records are inserted when history is on the default level (ie. ‘audit’). there was one last test i ran, just out of curiosity: running the best performing setting on an oracle xe 11.2 database. the oracle xe is a free version of the ‘real’ oracle database. no matter how hard, i tried, i couldn’t get it decently running on ubuntu. as such, i used an old windows xp install on that same machine. however, the os is 32 bit, wich means the system only has 3.2 of the 8gb of ram available. here are the results: activiti 5.10 – oracle on windows – default – history disabled the results speak for itself. oracle blows away any of the (single-threaded) results on mysql (and they are already very fast!). however, when going multi-threaded it is far worse than any of the mysql results. my guess is that these are due to the limitations of the xe version : only one cpu is used, only 1 gb of ram, etc. i would really like to run these test on a real oracle-managed-by-a-real-dba … feel free to contact me if you are interested ! in the next sections, we will take a detailed look into the performance numbers of each of the test processes. an excel sheet containing all the the numbers and charts below can be downloaded for yourself . process 1: the bare micromum (one transaction) the first process is not a very interesting one, business-wise at least. after starting the process, the end is immediately reached. not very useful on itself, but its numbers learn us one essential thing: the bare overhead of the activiti engine. here are the average timings: this process runs in a single transaction, which means that nothing is saved to the database when the history is disabled due to activiti’s optimizations. with history enabled, you’ll basically get the cost for inserting one row into the historical process instance table, which is around 4.44 ms here. it is also clear that our fix for activiti 5.10 has an enormous impact here. in the previous version, 99% of the time was spent in the cleanup check of the process. take a look at the best result here: 0.47 ms when using 4 threads to execute 2500 runs of this process. that’s only half a millisecond ! it’s fair to say that the activiti engine overhead is extremely small. the throughput numbers are equally impressive: in the best case here, 8741 processes are executed. per second. by the time you arrive here reading the post, you could have executed a few millions of this process . you can also see that there is little difference between 4 or 8 threads here. most of the execution time here is cpu time, and no potential collisions such as waiting for a database lock happens here. in these numbers, you can also easily see that the oracle xe doesn’t scale well with multiple threads (which is explained above). you will see the same behavior in the following results. process 2: the same, but a bit longer (one transaction) this process is pretty similar to the previous one. we have again only one transaction. after the process is started, we pass through seven no-op passthrough activities before reaching the end. some things to note here: the best result (again 4 threads, with history disabled) is actually better than the simpler previous process. but also note that the single threaded execution is a tad slower. this means that the process on itself is a bit slower, which is logical as is has more activities. but using more threads and having more activities in the process does allow for more potential interleaving. in the previous case, the thread was barely born before it was killed again. the difference between history enabled/disabled is bigger than the previous process. this is logical, as more history is written here (for each activity one record in the database). again, activiti 5.10 is far more superior to activiti 5.9. the throughput numbers follow these observations: there is more opportunity to use threading here. the best result lingers around 12000 process execution per second . again, it demonstrates the very lightweight execution of the activiti engine. process 3: parallelism in one transaction this process executes a parallel gateway that forks and one that joins in the same transaction. you would expect something along the lines of the previous results, but you’d be surprised: comparing these numbers with the previous process, you see that execution is slower. so why is this process slower, even if it has less activities? the reason lies with how the parallel gateway is implemented, especially the join behavior. the hard part, implementation-wise, is that you need to cope with the situation when multiple executions arrive at the join. to make sure that the behavior is atomic, we internally do some locking and fetch all child executions in the execution tree to find out whether the join activates or not. so it is quite a ‘costly’ operation, compared to the ‘regular’ activities. do mind, we’re talking here about only 5 ms single threaded and 3.59 ms in the best case for mysql . given the functionality that is required for implementing the parallel gateway functionality, this is peanuts if you’d ask me. the throughput numbers: this is the first process which actually contains some ‘logic’. in the best case above, it means 1112 processes can be executed in a second. pretty impressive, if you’d ask me! . process 4: now we’re getting somewhere (one transaction) this process already looks like something you’d see when modeling real business processes. we’re still running it in one database transaction though, as all the activities are automatic passthroughs. here we also have two forks and two joins. take a look at the lowest number: 6.88 ms on oracle when running with one thread. that’s freaking fast , taking in account all that is happening here. the history numbers are at least doubled here (activiti 5.10), which makes sense because there is quite a bit of activity audit logging going on here. you can also see that this causes to have a higher average time for four threads here, which is probably due to the implementation of the joining. if you know a bit about activiti internals, you’ll understand this means there are quite a bit of executions in the execution tree. we have one big concurrent root, but also multiple children which are sometimes also concurrent roots. but while the average time rises, the throughput definitely benefits: running this process with eight threads, allows you to do 411 runs of this process in a single second. there is also something peculiar here: the oracle database performs better with more thread concurrency. this is completely contrary with all other measurements, where oracle is always slower in that environment (see above for explanation). i assume it has something to do with the internal locking and forced update we are applying when forking/joining, which is better handled by oracle it seems. process 5: adding some java logic (single transaction) i added this process to see the influence of adding a java service task in a process. in this process, the first activity generates a random value, stores it as a process variable and then goes up or down in the process depending on the random value. the chance is about 50/50 to go up or down. the average timings are very very good. actually, the results are in the same range as those of process 1 and 2 above (which had no activities or only automatic passthroughs). this means that the overhead of integrating java logic into your process is nearly non-existant (nothing is of course for free). of course, you can still write slow code in that logic, but you can’t blame the activiti engine for that throughput numbers are comparable to those of process 1 and 2: very, very high. in the best case here, more than 9000 processes are executed per second . that indeed also means 9000 invocations of your own java logic. process 6, 7 and 8: adding wait states and transactions the previous processes demonstrated us the bare overhead of the activiti engine. here, we’ll take a look at how wait states and multiple transactions have influence on performance. for this, i added three test processes which contain user tasks. for each user task, the engine commits the current transaction and returns the thread to the client. since the results are pretty much compatible for these processes, we’re grouping them here. these are the processes: here are the average timings results, in order of the processes above. for the first process, containing just one user task: it is clear that having wait states and multiple transaction does have influence on the performance. this is also logical: before, the engine could optimize by not inserting the runtime state into the database, because the process was finished in one transaction. now, the whole state, meaning the pointers to where you are currently, need to be saved into the database. the process could be ‘sleeping’ like this for many days, months, years now …. the activiti engine doesn’t hold it into memory now anymore, and it is freed to give its full attention to other processes. if you check the results of the process with only one user task, you can see that in the best case (oracle, single thread – the 4 threads on mysql is pretty close) this is done in 6.27ms . this is really fast, if you take in account we have a few inserts (the execution tree, the task), a few updates (the execution tree) and deletes (cleaning up) going on here. the second process here, with 7 user tasks: the second chart learns us that logically, more transactions means more time. in the best case here the process is done in 32.12 ms . that is for seven transactions, which gives 4.6 ms for each transactions. so it is clear that average time scales in a linearly way when adding wait states. this makes of course sense, because transactions aren’t free. also note that enabling history does add quite some overhead here. this is due to having the history level set to ‘audit’, which stores all the user task information in the history tables. this is also noticeable from the difference between activiti 5.9 with history disabled and activiti 5.10 with history enabled: this is a rare case where activiti 5.10 with history enabled is slower than 5.9 with history disabled. but it is logical, given the volume of history stored here. and the third process learns us how user tasks and parallel gateways interact: the third chart learns us not much new. we have two user tasks now, and the more ‘expensive’ fork/join (see above). the average timings are how we expected them. the throughput charts are as you would expect given the average timings. between 70 and 250 processes per second. aw yeah! to save some space, you’ll need to click them to enlarge: process 9: so what about scopes? for the last process, we’ll take a look at ‘scopes’. a ‘scope’ is how we call it internally in the engine, and it has to do with variable visibility, relationships between the pointers indicating process state, event catching, etc. bpmn 2.0 has quite some cases for those scopes, for example with embedded subprocesses as shown in the process here. basically, every subprocess can have boundary events (catching an error, a message, etc) that only are applied on its internal activities when it’s scope is active. without going into too much technical details: to get scopes implemented in the correct way, you need some not so trivial logic. the example process here has 4 subprocesses, nested in each other. the inner process is using concurrency, which is a scope on itself again for the activiti engine. there are also two user tasks here, so that means two transactions. so let’s see how it performs: you can clearly see the big difference between activiti 5.9 and 5.10. scopes are indeed an area where the fixes around the ‘process cleanup’ at the end have a huge benefit, as many execution objects are created and persisted to represent the many different scopes. single threaded performance is not so good on activiti 5.9. luckily, as you can see from the gap between the blue and the red bars, those scopes do allow for high concurrency. the numbers of oracle, combined with the multi-threaded results of the 5.10 tests, do prove that scopes are now efficiently handled by the engine. the throughput charts prove that the process nicely scales with more threads, as you can see by the big gap between the red and green line in the second last block. in the best case, 64 processes of this more complex process are handled by the engine. random execution if you have already clicked on the full reports at the beginning of the post, you probably have noticed also random execution is tested for each environment. in this setting, 2500 process executions were done, both the process was randomly chosen. as shown in those reports this meant that over 2500 executions, each process was executed almost the same number of times (normal distribution). this last chart shows the best setting (activiti 5.10, history disabled) and how the throughput of those random process executions goes when adding more threads: as we’ve seen in many of the test above, once passed four threads things don’t change that much anymore. the numbers (167 processes/second) prove that in a realistic situation (ie. multiple processes executing at the same time), the activiti engine nicely scales up. conclusion the average timing charts show two things clearly: the activiti engine is fast and overhead is minimal ! the difference between history enabled or disabled is definitely noticeably. sometimes it comes even down to half the time needed. all history tests were done using the ‘audit’ level, but there is a simpler history level (‘activity’) which might be good enough for the use case. activiti is very flexible in history configuration, and you can tweak the history level for each process specifically. so do think about the level your process needs to have, if it needs to have history at all ! the throughput charts prove that the engine scales very well when more threads are available (ie. any modern application server). activiti is well designed to be used in high-throughput and availability (clustered) architectures . as i said in the introduction, the numbers are what they are: just numbers. my main point which i want to conclude here, is that the activiti engine is extremely lightweight. the overhead of using activiti for automating your business processes is small. in general, if you need to automate your business processes or workflows, you want top-notch integration with any java system and you like all of that fast and scalable … look no further!

Here are the most important subjects for software engineering, with brief explanations: 1.Object oriented analysis & design: For better maintainability, reusability and faster development, the most well accepted approach, shortly OOAD and its SOLID principals are very important for software engineering. 2.Software quality factors: Software engineering depends on some very important quality factors. Understanding and applying them is crucial. 3.Data structures & algorithms: Basic data structures like array, list, stack, tree, map, set etc. and useful algorithms are vital for software development. Their logical structure should be known. 4. Big-O notation: Big-O notation indicates the performance of an algorithm/code section. Understanding it is very important for comparing performances. 5.UML notation: UML is the universal and complete language for software design & analysis. If there is lack of UML in a development process, it feels there is no engineering. 6.Software processes and metrics: Software enginnering is not a random process. It requires a high level of systematic and some numbers to monitor those techniques. So, processes and metrics are essential. 7.Design patterns: Design patterns are standard and most effective solutions for specific problems. If you don't want to reinvent the wheel, you should learn them. 8.Operating systems basics: Learning OS basics is very important because all applications runs on it. By learning it, we can have better vision, viewpoints and performance for our applications. 9.Computer organization basics: All applications including OS requires a hardware for physical interaction. So, learning computer organization basics is vital again for better vision, viewpoints and performance. 10.Network basics: Network is related with computer organization, OS and the whole information transfer process. In any case we will face it while software development. So, it is important to learn network basics. 11.Requirement analysis: Requirement analysis is the starting point and one of the most important parts of software engineering. Performing it correctly and practically needs experience but it is very essential. 12.Software testing: Testing is another important part of software engineering. Unit testing, its best practices and techniques like black box, white box, mocking, TDD, integration testing etc. are subjects which must be known. 13.Dependency management: Library (JAR, DLL etc.) management, and widely known tools (Maven, Ant, Ivy etc.) are essential for large projects. Otherwise, antipatterns like Jar Hell are inevitable. 14.Continuous integration: Continuous integration brings easiness and automaticity for testing large modules, components and also performs auto-versioning. Its aim and tools (like Hudson etc.) should be known. 15.ORM (Object relational mapping): ORM and its widely known implementation Hibernate framework is an important technique for mapping objects into database tables. It reduces code length and maintenance time. 16.DI (Dependency Injection): DI or IoC (Inversion of Control) and its widely known implementation Spring framework makes life easy for object creation and lifetime management on big enterprise applications. 17.Version controlling systems: VCS tools (SVN, TFS, CVS etc.) are very important by saving so much time for collaborative works and versioning. Their logical viewpoint and standard cammands should be known. 18.Internationalization (i18n): i18n by extracting strings into external files is the best way of supporting multiple languages in our applications. Its practices on different IDEs and technologies must be known. 19.Architectural patterns: Understanding architectural design patterns (like MVC, MVP, MVVM etc.) is essential for producing a maintainable, clean, extendable and testable source code. 20.Writing clean code: Working code is not enough, it must be readable and maintainable also. So, code formatting and readable code development techniques are needed to be known and applied.

I've seen a lot of discussion about how to monitor Camel based applications. Most people are looking for the following features: ability to view services (contexts, endpoints, routes), to view performance statistics (route throughput, etc) and to perform basic operations (start/stop routes, send messages, etc). This post will breakdown the options (that I know of) that are available today (as of Camel 2.8). If you have used other approaches or know of other ongoing development in this area, please let me know. JMX APIs Camel uses JMX to provide a standardized way to access metadata about contexts/routes/endpoints defined in a given application. Also, you can use JMX to interact with these components (start/stop routes, etc) in some interesting ways. I recently had some very specific Camel/ActiveMQ monitoring requests from a client. After looking at the options, we ended up building a standalone Tomcat web app that used JSPs, jQuery, Ajax and JMX APIs to view route/endpoint statistics, manage Camel routes (stop, start, etc) and monitor/manipulate ActiveMQ queues. It provided some much needed visibility and management features for our Camel/ActiveMQ based message processing application... CamelContext If you have a handle to the CamelContext, there are various APIs that can help describe and manage routes and endpoints. These are used by the existing Camel Web Console and can be used to build custom interface to retrieve and use this information in various ways... here are some of the notable APIs... getRouteDefinitions() getEndpoints() getEndpointsMap() getRouteStatus(routeId) startRoute(routeId) stopRoute(routeId) removeRoute(routeId) addRoutes(routeBuilder) suspendRoute(routeId) resumeRoute(routeId) With a little creativity, you can use these APIs to manage/monitor and re-wire a Camel application dynamically. Camel Web Console This console provides web and REST interfaces to Camel contexts/routes/endpoints and allows you to view/manage endpoints/routes, send messages to endpoints, viewing route statistics, etc. That being said, using this web console with an existing Camel application is tricky at the moment. It's currently deployed as a war file that only has access to the CamelContext defined in its embedded spring XML file. Though the entire camel-web project can be embedded and customized in your application if you desire (and know Scalate). Given my recent client requirements, I opted to build my own basic app using JSPs/JMX as described above. There has been some recent support for deploying this console in OSGI, where it should be able to view any CamelContexts deployed in the container, etc. However, I'm yet to see this work...more on this later. Using Camel APIs There are also a number of Camel technologies/patterns that can be used to add monitoring to existing routes. wire tap - can add message logging (to a file or JMS queue/topic, etc) or other inline processing advicewith - can be used to modify existing routes to apply before/after operations or add/remove operations in a route intercept - can be used to intercept Exchanges while they are in route, can apply to all endpoints, certain endpoints or just starting endpoints BrowsableEndpoint - is an interface which Endpoints may implement to support the browsing of the exchanges which are pending or have been sent on it. That being said, it takes some creativity to use these effectively and caution to not adversely affect the routes you are trying to monitor. Hyperic HQ You can use this tool to monitor Servicemix (or any process), but it more geared towards system monitoring and JVM stats. I didn't find it useful for any Camel specific monitoring. jConsole/VisualVM these are standard JMX based consoles. They aren't web based and can't be customized (easily anyways) to provide anything more than a tree-like view of JMX MBeans. If you know where to look though, you can do a lot with it. Summary These are just some quick notes at this point. As I learn about other ways of monitoring Camel, I'll update this list and give some more detailed comparison. Any comments are welcome...

Some web applications may need a "Remember Me" functionality. This means that, after a user login, user will have access from same machine to all its data even after session expired. This access will be possible until user does a logout. If you are using Spring and its login form, then you should use "Remember Me" functionality already implemented inside the framework. Some web frameworks also offer a type of SignIn panel which already has "remember me" built-in. But in case you have to implement "Remember Me" functionality by your own, this can be easily achieved using Cookies. Java has a Cookie class named javax.servlet.http.Cookie. Algorithm is straight-forward: your login panel must contain a "Remember Me" check after a succesfull login with "Remember Me" check selected, you can create two cookies: one to keep the value for rememberMe and one to keep a token which has to identify the logged user. For sake of security, this token must never contain user name or user password. The ideea is to generate a random id as token value. And token value aside with user id must be saved in your storage (database) whenever a login is needed, you have to look if there is any cookie saved by you, and if so and your "rememberMe" value is true, you can take the user from storage based on your token and do an automatic login. when a logout is done, you have to delete the cookie that keeps the token To add a cookie, you have to specify the maximum age of the cookie in seconds : HttpServletResponse servletResponse = ...; Cookie c = new Cookie(COOKIE_NAME, encodeString(uuid)); c.setMaxAge(365 * 24 * 60 * 60); // one year servletResponse.addCookie(c); To delete a cookie, you have to find cookie by name and set its maximum age to 0, before adding it to servlet response: HttpServletRequest servletRequest = ...; HttpServletResponse servletResponse = ... ; Cookie[] cookies = servletRequest.getCookies(); for (int i = 0; i < cookies.length; i++) { Cookie c = cookies[i]; if (c.getName().equals(COOKIE_NAME)) { c.setMaxAge(0); c.setValue(null); servletResponse.addCookie(c); } }

Manipulating JAXWS header on the client Side like adding WSS username token or logging saop message.

In Spring 3.1 there is a nice new feature: "@Valid On @RequestBody Controller Method Arguments" From the documentation: "An @RequestBody method argument can be annotated with @Valid to invoke automatic validation similar to the support for @ModelAttribute method arguments. A resulting MethodArgumentNotValidException is handled in the DefaultHandlerExceptionResolver and results in a 400 response code." http://static.springsource.org/spring/docs/current/spring-framework-reference/html/new-in-3.1.html#d0e1654 Based on this new feature I am able to send objects in JSON format, validate them and check for possible errors. Before this feature was introduced my controller looked like this: @Controller @RequestMapping("/user") public class UserController { @RequestMapping(method=RequestMethod.POST) public ResponseEntity create(@Valid User user, BindingResult bindingResult) { if (bindingResult.hasErrors()) { //parse errors and send back to user } ... } But data wasn't sent as a JSON object. It came from simple HTML form. Let's play with Spring 3.1. I've updated my maven depedency to use version 3.1.1.RELEASE and added the Jackson dependency to be able to use the MappingJacksonHttpMessageConverter. cglib cglib-nodep 2.2 org.springframework spring-webmvc 3.1.1.RELEASE org.hibernate hibernate-validator 4.3.0.Final org.codehaus.jackson jackson-mapper-asl 1.9.7 All other configuration is placed in java class instead of xml: @Configuration @EnableWebMvc public class WebappConfig { @Bean public UserController userController() { return new UserController(); } @Bean public MappingJacksonJsonView mappingJacksonJsonView() { return new MappingJacksonJsonView(); } @Bean public ContentNegotiatingViewResolver contentNegotiatingViewResolver() { final ContentNegotiatingViewResolver contentNegotiatingViewResolver = new ContentNegotiatingViewResolver(); contentNegotiatingViewResolver.setDefaultContentType(MediaType.APPLICATION_JSON); final ArrayList defaultViews = new ArrayList(); defaultViews.add(mappingJacksonJsonView()); contentNegotiatingViewResolver.setDefaultViews(defaultViews); return contentNegotiatingViewResolver; } } After that I've changed my create method to: @RequestMapping(method=RequestMethod.POST, consumes = "application/json") public ResponseEntity create(@Valid @RequestBody User user, BindingResult bindingResult) { if (bindingResult.hasErrors()) { //parse errors and send back to user } ... Now I've sent a JSON object but received the following error: "An Errors/BindingResult argument is expected to be immediately after the model attribute argument in the controller method signature" According to the documentation "BindingResult is supported only after @ModelAttribute arguments" After removing BindingResult from my method's arguments it worked. But how to get the errors now ? According to the documentation a MethodArgumentNotValidException will be thrown. So let's declare the necessary ExceptionHandler. @ExceptionHandler(MethodArgumentNotValidException.class) public ResponseEntity handleMethodArgumentNotValidException( MethodArgumentNotValidException error ) { return parseErrors(error.getBindingResult()); } ... And that's all, I'm attaching the full source code as a working example.

While preparing my next exercise I had to first find a way how to connect to a remote SLSB from Spring. It turned out to be childishly simple. See how I did it. Writing SLSB component First I needed a remote SLSB. Using M2Eclipse create a Maven 2 project. From the archetypes selection window I typed mojo in filter field (mojo archetypes are codehaus archetypes, I wrote about them in: Don't use org.apache.maven.archetypes!) and selected ejb3 archetype. I wrote a simple SLSB component: package org.xh.studies.openejb; import javax.ejb.Remote; @Remote public interface Greeter { String sayHello(String to); } package org.xh.studies.openejb; import javax.ejb.Stateless; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @Stateless public class GreeterImpl implements Greeter { private static final Log log = LogFactory.getLog(GreeterImpl.class); public String sayHello(String to) { log.info("Saying hi to: " + to); return "Hello " + to + "! How are you?"; } } I built it with: mvn package Deploying to OpenEJB I downloaded OpenEJB archive from here: http://openejb.apache.org/download.html and unzipped it. I was done with installation :) I copied built in previous step jar to apps directory, then went to bin directory and ran openejb(.bat). Writing JUnit integration test First I wrote a simple integration test (using failsafe plugin, there are many post on my blog about it, the key one is: Maven2 integration testing with failsafe and jetty plugins) to verify if the SLSB is actually working: public class GreeterIT { private static Context ctx; @BeforeClass public static void setUp() throws NamingException { Properties env = new Properties(); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory"); env.put(Context.PROVIDER_URL, "ejbd://localhost:4201"); ctx = new InitialContext(env); } @AfterClass public static void tearDown() throws NamingException { ctx.close(); } @Test public void sayHelloRemoteEJBTest() throws NamingException { Greeter greeter = (Greeter) ctx.lookup("GreeterImplRemote"); String result = greeter.sayHello("Łukasz"); String expected = "Hello Łukasz! How are you?"; Assert.assertEquals(expected, result); } } I ran the test and it passed. Connecting to EJB from Spring I found out a Spring component called org.springframework.ejb.access.SimpleRemoteStatelessSessionProxyFactoryBean. It uses AOP behinde the scenes to create a proxy for a remote SLSB. All you need is just an interface. First I had to add Spring dependencies to my pom.xml. These were: spring-beans spring-context spring-aop Then I created a file called META-INF/spring/beans.xml and wrote: org.apache.openejb.client.RemoteInitialContextFactory ejbd://localhost:4201 In the above file I created a greeterBean which was a dynamic proxy and, thanks to AOP, would behave like a Greeter object. Writing Spring test I added one more field to my test: private static ApplicationContext springCtx; Then in @BeforeClass class I added ApplicationContext initialisation code: springCtx = new ClassPathXmlApplicationContext("META-INF/spring/beans.xml"); Finally I wrote a new method to test what I intended to do first, connecting to a remote SLSB from Spring: @Test public void sayHelloSpringProxyTest() throws NamingException { Greeter greeter = (Greeter) springCtx.getBean("greeterBean"); String result = greeter.sayHello("Łukasz"); String expected = "Hello Łukasz! How are you?"; Assert.assertEquals(expected, result); } It worked! Source code download A complete source code download can be found here: Connecting-to-EJB-from-Spring.zip. Summary Isn't it beautiful? You get an instance of a remote SLSB as a Spring bean. I definitely like it! thanks, Łukasz

If you are using Spring to access/configure resources (DAOs/services), then you might as well add some basic performance monitoring while you are at it. This is a trivial task with Spring AOP and doesn't require any changes to existing code, just some simple configuration. First, you need to include the spring-aop, aspectj and cglib libraries. If you are using Maven, simply include the following dependencies... org.aspectj aspectjweaver 1.5.4 cglib cglib-nodep 2.2 org.springframework spring-aop 2.5.6 Next, identify what needs monitoring and put the AOP hooks in place. Generally, this just requires adding a pointcut and advisor configuration in your existing Spring XML configuration file. This configuration will add method response time logging to all methods in the "com.mycompany.services" package. Note: these classes must be instantiated with the Spring context...otherwise, the AOP hooks will not be executed. Next, you need to setup your logging (log4j, etc) to enable TRACE on the interceptor class. That's it, now when you run your application, you will see the following logging... TRACE PerformanceMonitorInterceptor - StopWatch 'PerfTestService.processRequest': running time (millis) = 1322 TRACE PerformanceMonitorInterceptor - StopWatch 'PerfTestService.processRequest': running time (millis) = 98 TRACE PerformanceMonitorInterceptor - StopWatch 'PerfTestService.processRequest': running time (millis) = 1764 This is a some great raw data, but unfortunately is not very useful on its own. Its for every method call and doesn't provide any other stats. This quickly clutters up the log and without some way to process/aggregate the log entries, its hard to make sense out of it. So, unless you plan of writing some log parsing or using 3rd party software (like Splunk or Cati), then you really should do some processing of the data before writing it to the log file. One easy way to do this is to just write a simple interceptor class to use instead of the Spring default one (PerformanceMonitorInterceptor). Below is an example of this that provides periodic stats (last, average and greatest response time) as well as warning whenever a method response time exceeds a configured threshold. By default, it will log stats every 10 method calls and log a warning message anytime a method response time exceeds 1000ms. public class PerfInterceptor implements MethodInterceptor { Logger logger = LoggerFactory.getLogger(PerfInterceptor.class.getName()); private static ConcurrentHashMap methodStats = new ConcurrentHashMap(); private static long statLogFrequency = 10; private static long methodWarningThreshold = 1000; public Object invoke(MethodInvocation method) throws Throwable { long start = System.currentTimeMillis(); try { return method.proceed(); } finally { updateStats(method.getMethod().getName(),(System.currentTimeMillis() - start)); } } private void updateStats(String methodName, long elapsedTime) { MethodStats stats = methodStats.get(methodName); if(stats == null) { stats = new MethodStats(methodName); methodStats.put(methodName,stats); } stats.count++; stats.totalTime += elapsedTime; if(elapsedTime > stats.maxTime) { stats.maxTime = elapsedTime; } if(elapsedTime > methodWarningThreshold) { logger.warn("method warning: " + methodName + "(), cnt = " + stats.count + ", lastTime = " + elapsedTime + ", maxTime = " + stats.maxTime); } if(stats.count % statLogFrequency == 0) { long avgTime = stats.totalTime / stats.count; long runningAvg = (stats.totalTime-stats.lastTotalTime) / statLogFrequency; logger.debug("method: " + methodName + "(), cnt = " + stats.count + ", lastTime = " + elapsedTime + ", avgTime = " + avgTime + ", runningAvg = " + runningAvg + ", maxTime = " + stats.maxTime); //reset the last total time stats.lastTotalTime = stats.totalTime; } } class MethodStats { public String methodName; public long count; public long totalTime; public long lastTotalTime; public long maxTime; public MethodStats(String methodName) { this.methodName = methodName; } } } Now, you just need to wire this into your app by referencing this class in your Spring xml and logging config. When you run your app, you will see stats like this... WARN PerfInterceptor - method warning: processRequest(), cnt = 10, lastTime = 1072, maxTime = 1937 TRACE PerfInterceptor - method: processRequest(), cnt = 10, lastTime = 1072, avgTime = 1243, runningAvg = 1243, maxTime = 1937 WARN PerfInterceptor - method warning: processRequest(), cnt = 20, lastTime = 1466, maxTime = 1937 TRACE PerfInterceptor - method: processRequest(), cnt = 20, lastTime = 1466, avgTime = 1067, runningAvg = 892, maxTime = 1937 As you can see, these stats can provide valuable feedback about class/method performance with very little effort and without modifying any existing Java code. This information can easily be used to find bottlenecks in your application (generally database or threading related, etc)...good luck

Introduction: Building modern HTTP/RESTful/RPC services has become very easy with the new ASP.NET Web API framework. Using ASP.NET Web API framework, you can create HTTP services which can be accessed from browsers, machines, mobile devices and other clients. Developing HTTP services is now quite easy for ASP.NET MVC developer becasue ASP.NET Web API is now included in ASP.NET MVC. In addition to developing HTTP services, it is also important to return meaningful response to client if a resource(uri) not found(HTTP 404) for a reason(for example, mistyped resource uri). It is also important to make this response centralized so you can configure all of 'HTTP 404 Not Found' resource at one place. In this article, I will show you how to handle 'HTTP 404 Not Found' at one place. Description: Let's say that you are developing a HTTP RESTful application using ASP.NET Web API framework. In this application you need to handle HTTP 404 errors in a centralized location. From ASP.NET Web API point of you, you need to handle these situations, No route matched. Route is matched but no {controller} has been found on route. No type with {controller} name has been found. No matching action method found in the selected controller due to no action method start with the request HTTP method verb or no action method with IActionHttpMethodProviderRoute implemented attribute found or no method with {action} name found or no method with the matching {action} name found. Now, let create a ErrorController with Handle404 action method. This action method will be used in all of the above cases for sending HTTP 404 response message to the client. public class ErrorController : ApiController { [HttpGet, HttpPost, HttpPut, HttpDelete, HttpHead, HttpOptions, AcceptVerbs("PATCH")] public HttpResponseMessage Handle404() { var responseMessage = new HttpResponseMessage(HttpStatusCode.NotFound); responseMessage.ReasonPhrase = "The requested resource is not found"; return responseMessage; } } You can easily change the above action method to send some other specific HTTP 404 error response. If a client of your HTTP service send a request to a resource(uri) and no route matched with this uri on server then you can route the request to the above Handle404 method using a custom route. Put this route at the very bottom of route configuration, routes.MapHttpRoute( name: "Error404", routeTemplate: "{*url}", defaults: new { controller = "Error", action = "Handle404" } ); Now you need handle the case when there is no {controller} in the matching route or when there is no type with {controller} name found. You can easily handle this case and route the request to the above Handle404 method using a custom IHttpControllerSelector. Here is the definition of a custom IHttpControllerSelector, public class HttpNotFoundAwareDefaultHttpControllerSelector : DefaultHttpControllerSelector { public HttpNotFoundAwareDefaultHttpControllerSelector(HttpConfiguration configuration) : base(configuration) { } public override HttpControllerDescriptor SelectController(HttpRequestMessage request) { HttpControllerDescriptor decriptor = null; try { decriptor = base.SelectController(request); } catch (HttpResponseException ex) { var code = ex.Response.StatusCode; if (code != HttpStatusCode.NotFound) throw; var routeValues = request.GetRouteData().Values; routeValues["controller"] = "Error"; routeValues["action"] = "Handle404"; decriptor = base.SelectController(request); } return decriptor; } } Next, it is also required to pass the request to the above Handle404 method if no matching action method found in the selected controller due to the reason discussed above. This situation can also be easily handled through a custom IHttpActionSelector. Here is the source of custom IHttpActionSelector, public class HttpNotFoundAwareControllerActionSelector : ApiControllerActionSelector { public HttpNotFoundAwareControllerActionSelector() { } public override HttpActionDescriptor SelectAction(HttpControllerContext controllerContext) { HttpActionDescriptor decriptor = null; try { decriptor = base.SelectAction(controllerContext); } catch (HttpResponseException ex) { var code = ex.Response.StatusCode; if (code != HttpStatusCode.NotFound && code != HttpStatusCode.MethodNotAllowed) throw; var routeData = controllerContext.RouteData; routeData.Values["action"] = "Handle404"; IHttpController httpController = new ErrorController(); controllerContext.Controller = httpController; controllerContext.ControllerDescriptor = new HttpControllerDescriptor(controllerContext.Configuration, "Error", httpController.GetType()); decriptor = base.SelectAction(controllerContext); } return decriptor; } } Finally, we need to register the custom IHttpControllerSelector and IHttpActionSelector. Open global.asax.cs file and add these lines, configuration.Services.Replace(typeof(IHttpControllerSelector), new HttpNotFoundAwareDefaultHttpControllerSelector(configuration)); configuration.Services.Replace(typeof(IHttpActionSelector), new HttpNotFoundAwareControllerActionSelector()); Summary: In addition to building an application for HTTP services, it is also important to send meaningful centralized information in response when something goes wrong, for example 'HTTP 404 Not Found' error. In this article, I showed you how to handle 'HTTP 404 Not Found' error in a centralized location. Hopefully you will enjoy this article too.

Spring 3.1 now includes support for the long awaited environment aware feature called profiles. Now we can activate profiles in our application, which allows us to define beans by deployment regions, such as “dev”, “qa”, “production”, “cloud”, etc. We also can use this feature for other purposes: defining profiles for performance testing scenarios such as “cached” or “lazyload”. Essential Tokens Spring profiles are enabled using the case insensitive tokens spring.profiles.active or spring_profiles_active. This token can be set as: an Environment Variable a JVM Property Web Parameter Programmatic Spring also looks for the token, spring.profiles.default, which can be used to set the default profile(s) if none are specified with spring.profiles.active. Grouping Beans by Profile Spring 3.1 provides nested bean definitions, providing the ability to define beans for various environments: Nested must appear last in the file. Beans that are used in all profiles are declared in the outer as we always have, such as Service classes. If we put a single declaration at below any nested tags we will get the exception org.xml.sax.SAXParseException: cvc-complex-type.2.4.a: Invalid content was found starting with element 'bean'. Multiple beans can now share the same XML “id” In a typical scenario, we would want the DataSource bean to be called dataSource in both all profiles. Spring now allow us to create multiple beans within an XML file with the same ID providing they are defined in different sets. In other words, ID uniqueness is only enforced within each set. Automatic Profile Discovery (Programmatic) We can configure a class to set our profile(s) during application startup by implementing the appropriate interface. For example, we may configure an application to set different profiles based on where the application is deployed – in CloudFoundry or running as a local web application. In the web.xml file we can include an Servlet context parameter, contextInitializerClasses, to bootstrap this class: contextInitializerClasses com.gordondickens.springthreeone.services.CloudApplicationContextInitializer The Initializer class package com.gordondickens.springthreeone.services; import org.cloudfoundry.runtime.env.CloudEnvironment; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.context.ApplicationContextInitializer; import org.springframework.context.ConfigurableApplicationContext; public class CloudApplicationContextInitializer implements ApplicationContextInitializer { private static final Logger logger = LoggerFactory .getLogger(CloudApplicationContextInitializer.class); @Override public void initialize(ConfigurableApplicationContext applicationContext) { CloudEnvironment env = new CloudEnvironment(); if (env.getInstanceInfo() != null) { logger.info("Application running in cloud. API '{}'", env.getCloudApiUri()); applicationContext.getEnvironment().setActiveProfiles("cloud"); applicationContext.refresh(); } else { logger.info("Application running local"); applicationContext.getEnvironment().setActiveProfiles("dev"); } } } Annotation Support for JavaConfig If we are are using JavaConfig to define our beans, Spring 3.1 includes the @Profile annotation for enabling bean config files by profile(s). package com.gordondickens.springthreeone.configuration; import com.gordondickens.springthreeone.SimpleBean; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Profile; @Configuration @Profile("dev") public class AppConfig { @Bean public SimpleBean simpleBean() { SimpleBean simpleBean = new SimpleBean(); simpleBean.setMyString("Ripped Pants"); return simpleBean; } } Testing with XML Configuration With XML configuration we can simply add the annotation @ActiveProfiles to the JUnit test class. To include multiple profiles, use the format @ActiveProfiles(profiles = {"dev", "prod"}) package com.gordondickens.springthreeone; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.NoSuchBeanDefinitionException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.ApplicationContext; import org.springframework.test.context.ActiveProfiles; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import static junit.framework.Assert.assertNotNull; import static junit.framework.Assert.assertNull; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration @ActiveProfiles(profiles = "dev") public class DevBeansTest { @Autowired ApplicationContext applicationContext; @Test public void testDevBeans() { SimpleBean simpleBean = applicationContext.getBean("constructorBean", SimpleBean.class); assertNotNull(simpleBean); } @Test(expected = NoSuchBeanDefinitionException.class) public void testProdBean() { SimpleBean prodBean = applicationContext.getBean("prodBean", SimpleBean.class); assertNull(prodBean); } } Testing with JavaConfig JavaConfig allows us to configure Spring with or without XML configuration. If we want to test beans that are defined in a Configuration class we configure our test with the loader and classes arguments of the @ContextConfiguration annotation. package com.gordondickens.springthreeone.configuration; import com.gordondickens.springthreeone.SimpleBean; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ActiveProfiles; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.support.AnnotationConfigContextLoader; import static org.junit.Assert.assertNotNull; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(classes = AppConfig.class, loader = AnnotationConfigContextLoader.class) @ActiveProfiles(profiles = "dev") public class BeanConfigTest { @Autowired SimpleBean simpleBean; @Test public void testBeanAvailablity() { assertNotNull(simpleBean); } } Declarative Configuration in WEB.XML If we desire to set the configuration in WEB.XML, this can be done with parameters on ContextLoaderListener. Application Context contextConfigLocation /WEB-INF/app-config.xml spring.profiles.active DOUBLEUPMINT Log Results DEBUG PropertySourcesPropertyResolver - Found key 'spring.profiles.active' in [servletContextInitParams] with type [String] and value 'DOUBLEUPMINT' Environment Variable/JVM Parameter Setting an environment variable can be done with either spring_profiles_default or spring_profiles_active. In Unix/Mac it would be export SPRING_PROFILES_DEFAULT=DEVELOPMENT for my local system. We can also use the JVM “-D” parameter which also works with Maven when using Tomcat or Jetty plugins. Note: Remember the tokens are NOT case sensitive and can use periods or underscores as separators. For Unix systems, you need to use the underscore, as above. Logging of system level properties DEBUG PropertySourcesPropertyResolver - Found key 'spring.profiles.default' in [systemProperties] with type [String] and value 'dev,default' Summary Now we are equipped to activate various Spring bean sets, based on profiles we define. We can use traditional, XML based configuration, or the features added to support JavaConfig originally introduced in Spring 3.0.

Usually we have some required fields on our forms and it would be nice if ASP.NET MVC views can detect those fields automatically and display nice red asterisk after field label. As this functionality is not built in I built my own solution based on data annotations. In this posting I will show you how to show red asterisk after label of required fields. Here are the main information sources I used when working out my own solution: How can I modify LabelFor to display an asterisk on required fields? (stackoverflow) ASP.NET MVC – Display visual hints for the required fields in your model (Radu Enucă) Although my code was first written for completely different situation I needed it later and I modified it to work with models that use data annotations. If data member of model has Required attribute set then asterisk is rendered after field. If Required attribute is missing then there will be no asterisk. Here’s my code. You can take just LabelForRequired() methods and paste them to your own HTML extension class. public static class HtmlExtensions { [SuppressMessage("Microsoft.Design", "CA1006:DoNotNestGenericTypesInMemberSignatures", Justification = "This is an appropriate nesting of generic types")] public static MvcHtmlString LabelForRequired(this HtmlHelper html, Expression> expression, string labelText = "") { return LabelHelper(html, ModelMetadata.FromLambdaExpression(expression, html.ViewData), ExpressionHelper.GetExpressionText(expression), labelText); } private static MvcHtmlString LabelHelper(HtmlHelper html, ModelMetadata metadata, string htmlFieldName, string labelText) { if (string.IsNullOrEmpty(labelText)) { labelText = metadata.DisplayName ?? metadata.PropertyName ?? htmlFieldName.Split('.').Last(); } if (string.IsNullOrEmpty(labelText)) { return MvcHtmlString.Empty; } bool isRequired = false; if (metadata.ContainerType != null) { isRequired = metadata.ContainerType.GetProperty(metadata.PropertyName) .GetCustomAttributes(typeof(RequiredAttribute), false) .Length == 1; } TagBuilder tag = new TagBuilder("label"); tag.Attributes.Add( "for", TagBuilder.CreateSanitizedId( html.ViewContext.ViewData.TemplateInfo.GetFullHtmlFieldName(htmlFieldName) ) ); if (isRequired) tag.Attributes.Add("class", "label-required"); tag.SetInnerText(labelText); var output = tag.ToString(TagRenderMode.Normal); if (isRequired) { var asteriskTag = new TagBuilder("span"); asteriskTag.Attributes.Add("class", "required"); asteriskTag.SetInnerText("*"); output += asteriskTag.ToString(TagRenderMode.Normal); } return MvcHtmlString.Create(output); } } And here’s how to use LabelForRequired extension method in your view: @Html.LabelForRequired(m => m.Name) @Html.TextBoxFor(m => m.Name) @Html.ValidationMessageFor(m => m.Name) After playing with CSS style called .required my example form looks like this: These red asterisks are not part of original view mark-up. LabelForRequired method detected that these properties have Required attribute set and rendered out asterisks after field names. NB! By default asterisks are not red. You have to define CSS class called “required” to modify how asterisk looks like and how it is positioned.

Apache Camel has a very powerful bean injection framework which allows developers to focus only on solving business problems. However there are situations when you need to do a little bit more. Read below to see how easy it is to setup aspects (AspectJ) in Apache Camel. Use case In Qualitas I have an installation route which consists of 10 mandatory and 2 optional processors. Some processors like property resolvers or validators don't modify contents of message's body so I have to always copy body from the in message to out message. Also, all my processors require some headers to function properly. Finally, I would like to get a status updated after each of my processors either finishes processing successfully or fails. Setting up AspectJ This is just a plain Spring configuration frankly. All you have to do is: Apache Camel processor aspect To define aspect in AspectJ I used AspectJ-specific @Aspect and @Around annotations: @Aspect @Component @Order(Ordered.LOWEST) public class HeadersAndBodyCopierAspect { @Around("execution(* com.googlecode.qualitas.internal.installation..*.process(org.apache.camel.Exchange)) && args(exchange) && target(org.apache.camel.Processor)") public Object copyHeadersAndBody(ProceedingJoinPoint pjp, Exchange exchange) throws Throwable { Object retValue = pjp.proceed(); Message in = exchange.getIn(); Message out = exchange.getOut(); // always copy headers out.setHeaders(in.getHeaders()); // if output body is empty copy it from input if (out.getBody() == null) { out.setBody(in.getBody()); } return retValue; } } I also used @Order Spring-specific annotation to control the order of execution of my aspects and @Component for automatic context scanning. Now, the join point is defined as execution(* com.googlecode.qualitas.internal.installation..*.process(org.apache.camel.Exchange)) && args(exchange) && target(org.apache.camel.Processor) which basically means: apply this aspect to all process methods which are defined in all classes in com.googlecode.qualitas.internal.installation or subpackages and which take Exchange object as an argument there can be many custom methods whose names may be process and whose argument may be Exchange, so I added one more constraint, this class has to be an instance of Processor args(exchange) allows me to add Exchange object as an argument to my aspect More complex aspects and source code Of course in Spring you can inject other beans directly into your aspects. I used it in my ProcessStatusUpdaterAspect aspect which you can find in Qualitas repo on GoogleCode or GitHub. If you are interested in trying out the whole Qualitas system take a look at the following two links: BuildingTheProject and RunningTheProject. cheers, Łukasz

Here are the 10 important differences between WCF Services and ASP.NET Web Services.

I really like Spring, so I tend to use its features to the fullest. However, in some dark corners of its philosophy, I tend to disagree with some of its assumptions. One such assumption is the way database testing should work. In this article, I will explain how to configure your projects to make Spring Test and DBUnit play nice together in a multi-developers environment. Context My basic need is to be able to test some complex queries: before integration tests, I've to validate those queries get me the right results. These are not unit tests per se but let's assilimate them as such. In order to achieve this, I use since a while a framework named DBUnit. Although not maintained since late 2010, I haven't found yet a replacement (be my guest for proposals). I also have some constraints: I want to use TestNG for all my test classes, so that new developers wouldn't think about which test framework to use I want to be able to use Spring Test, so that I can inject my test dependencies directly into the test class I want to be able to see for myself the database state at the end of any of my test, so that if something goes wrong, I can execute my own queries to discover why I want every developer to have its own isolated database instance/schema Considering the last point, our organization let us benefit from a single Oracle schema per developer for those "unit-tests". Basic set up Spring provides the AbstractTestNGSpringContextTests class out-of-the-box. In turn, this means we can apply TestNG annotations as well as @Autowired on children classes. It also means we have access to the underlying applicationContext, but I prefer not to (and don't need to in any case). The structure of such a test would look like this: @ContextConfiguration(location = "classpath:persistence-beans.xml") public class MyDaoTest extends AbstractTestNGSpringContextTests { @Autowired private MyDao myDao; @Test public void whenXYZThenTUV() { ... } } Readers familiar with Spring and TestNG shouldn't be surprised here. Bringing in DBunit DbUnit is a JUnit extension targeted at database-driven projects that, among other things, puts your database into a known state between test runs. [...] DbUnit has the ability to export and import your database data to and from XML datasets. Since version 2.0, DbUnit can also work with very large datasets when used in streaming mode. DbUnit can also help you to verify that your database data match an expected set of values. DBunit being a JUnit extension, it's expected to extend the provided parent class org.dbunit.DBTestCase. In my context, I have to redefine some setup and teardown operation to use Spring inheritance hierarchy. Luckily, DBUnit developers thought about that and offer relevant documentation. Among the different strategies available, my tastes tend toward the CLEAN_INSERT and NONE operations respectively on setup and teardown. This way, I can check the database state directly if my test fails. This updates my test class like so: @ContextConfiguration(locations = {"classpath:persistence-beans.xml", "classpath:test-beans.xml"}) public class MyDaoTest extends AbstractTestNGSpringContextTests { @Autowired private MyDao myDao; @Autowired private IDatabaseTester databaseTester; @BeforeMethod protected void setUp() throws Exception { // Get the XML and set it on the databaseTester // Optional: get the DTD and set it on the databaseTester databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT); databaseTester.setTearDownOperation(DatabaseOperation.NONE); databaseTester.onSetup(); } @Test public void whenXYZThenTUV() { ... } } Per-user configuration with Spring Of course, we need to have a specific Spring configuration file to inject the databaseTester. As an example, here is one: However, there's more than meets the eye. Notice the databaseTester has to be fed a datasource. Since a requirement is to have a database per developer, there are basically two options: either use a in-memory database or use the same database as in production and provide one such database schema per developer. I tend toward the latter solution (when possible) since it tends to decrease differences between the testing environment and the production environment. Thus, in order for each developer to use its own schema, I use Spring's ability to replace Java system properties at runtime: each developer is characterized by a different user.name. Then, I configure a PlaceholderConfigurer that looks for {user.name}.database.properties file, that will look like so: db.username=myusername1 db.password=mypassword1 db.schema=myschema1 This let me achieve my goal of each developer using its own instance of Oracle. If you want to use this strategy, do not forget to provide a specific database.properties for the Continuous Integration server. Huh oh? Finally, the whole testing chain is configured up to the database tier. Yet, when the previous test is run, everything is fine (or not), but when checking the database, it looks untouched. Strangely enough, if you did load some XML dataset and assert it during the test, it does behaves accordingly: this bears all symptoms of a transaction issue. In fact, when you closely look at Spring's documentation, everything becomes clear. Spring's vision is that the database should be left untouched by running tests, in complete contradiction to DBUnit's. It's achieved by simply rollbacking all changes at the end of the test by default. In order to change this behavior, the only thing to do is annotate the test class with @TransactionConfiguration(defaultRollback=false). Note this doesn't prevent us from specifying specific methods that shouldn't affect the database state on a case-by-case basis with the @Rollback annotation. The test class becomes: @ContextConfiguration(locations = {classpath:persistence-beans.xml", "classpath:test-beans.xml"}) @TransactionConfiguration(defaultRollback=false) public class MyDaoTest extends AbstractTestNGSpringContextTests { @Autowired private MyDao myDao; @Autowired private IDatabaseTester databaseTester; @BeforeMethod protected void setUp() throws Exception { // Get the XML and set it on the databaseTester // Optional: get the DTD and set it on the databaseTester databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT); databaseTester.setTearDownOperation(DatabaseOperation.NONE); databaseTester.onSetup(); } @Test public void whenXYZThenTUV() { ... } } Conclusion Though Spring and DBUnit views on database testing are opposed, Spring's configuration versatility let us make it fit our needs (and benefits from DI). Of course, other improvements are possible: pushing up common code in a parent test class, etc. To go further: Spring Test documentation DBUnit site Database data verification Database testing best practices Generating DTD from your database schema

Dynamic and static properties are really interesting. Learn about managing dynamic property in Spring.

A Robust Splitter Aggregator Design Strategy - Messaging Gateway Adapter Pattern What do we mean by robust? In the context of this article, robustness refers to an ability to manage exception conditions within a flow without immediately returning to the caller. In some processing scenarios n of m responses is good enough to proceed to conclusion. Example processing scenarios that typically have these tendencies are: Quotations for finance, insurance and booking systems. Fan-out publishing systems. Why do we need Robust Splitter Aggregator Designs? First and foremost an introduction to a typical Splitter Aggregator pattern maybe necessary. The Splitter is an EIP pattern that describes a mechanism for breaking composite messages into parts in order that they can be processed individually. A Router is an EIP pattern that describes routing messages into channels - aiming them at specific messaging endpoints. The Aggregator is an EIP pattern that collates and stores a set of messages that belong to a group, and releases them when that group is complete. Together, those three EIP constructs form a powerful mechanism for dividing processing into distinct units of work. Spring Integration (SI) uses the same pattern terminology as EIP and so readers of that methodology will be quite comfortable with Spring Integration Framework constructs. The SI Framework allows significant customisations of all three of those constructs and furthermore, by simply using asynchronous channels as you would in any other multi-threaded configuration, allows those units of work to be executed in parallel. An interesting challenge working with SI Splitter Aggregator designs is building appropriately robust flows that operate predictably in a number of invocation scenarios. A simple splitter aggregator design can be used in many circumstances and operate without heavy customisation of the SI constructs. However, some service requirements demand a more robust processing strategy and therefore more complex configuration. The following sections describe and show what a Simple Splitter Aggregator design actually looks like, the type of processing your design must be able to deal with and then goes on to suggest candidate solutions for more robust processing. A Simple Splitter Aggregator Design The following Splitter Aggregator design shows a simple flow that receives document request messages into messaging gateway, splits the message into two processing routes and then aggregates the response. Note that the diagram has been built from EIP constructs in OmniGraffle rather than being an Integration Graph view from within STS; the channels are missing from the diagram for the sake of brevity. SI Constructs in detail: Messaging Gateways - there are three messaging gateways. A number of configurations are available for gateway specifications but significantly can return business objects, exceptions and nulls (following a timeout). The gateway to the far left is the service gateway for which we are defining the flow. The other two gateways, between the Router and Aggregator, are external systems that will be providing responses to business questions that our flow generates. The Splitter - a single splitter exists and is responsible for consuming the document message and producing a collection of messages for onward processing. The Java signature for the, most often, custom Splitter specifies a single object argument and a collection for return. The Recipient List Router - a single router exists, any appropriate router can be used, chose the one that closely matches your requirements - you can easily route by expression or payload type. The primary purpose of the router is route a collection of messages supplied by the splitter. This is a pretty typical Splitter Aggregator configuration. Aggregator - a single construct that is responsible for collecting messages together in a group in order that further processing can take place on the gateway responses. Although the Aggregator can be configured with attributes and bean definitions to provide alternative grouping and release strategies, most often the default aggregation strategy suffices. Interesting Aspects of Splitter Aggregator Operation Gateway - the inbound gateway, the one on the far left, may or may not have an error handling bean reference defined on it. If it does then that bean will have an opportunity to handle an exceptions thrown within the flow to the right of that gateway. If not, any exception will be thrown straight out of the gateway. Gateway - an optional default-reply-timeout can be set on each of the gateways, there are significant implications for setting this value, ensure that they're well understood. An expired timeout will result in a null being returned from the gateway. This is the very same condition that can lead to a thread getting parked if an upstream gateway also has no default-reply-timeout set. Splitter Input Channel - this can be a simple direct channel or a direct channel with a dispatcher defined on it. If the channel has a dispatcher specified the flow downstream of this point will be asynchronous, multi-threaded. This also changes the upstream gateway semantics as it usually means that an otherwise impotent default-reply-timeout becomes active. Splitter - the splitter must return a single object. The single object returned by the splitter is a collection, a java.util.List. The SI framework will take each member of that list and feed it into the output-channel of the Splitter - as with this example, usually straight into a router. The contract for Splitter List returns is as its use in Java - it may contain zero, one or more elements. If the splitter returns an empty list it's unlikely that the router will have any work to do and so the flow invocation will be complete. However, if the List contains one item, the SI framework will extract that item from the list and push it into the router, if this gets routed successfully, the flow will continue. Router - the router will simply route messages into one of two gateways in this example. Gateways - the two gateways that are used between the Splitter and Aggregator are interesting. In this example I have used the generic gateway EIP pattern to represent a message sub-system but not defined it explicitly - we could use an HTTP outbound gateway, another SI flow or any other external system. Of course, for each of those sub-systems, a number of responses is possible. Depending on the protocol and external system, the message request may fail to send, the response fail to arrive, a long running process invoked, a network error or timeout or a general processing exception. Aggregator - the single aggregator will wait for a number of responses depending on what's been created by the Splitter. In the case where the splitter return list is empty the Aggregator will not get invoked. In the case where the Splitter return list has one entry, the aggregator will be waiting for one gateway response to complete the group. In the case where the Splitter list has n entries the Aggregator will be waiting for n entries to complete the group. Custom correlation strategies, release strategies and message stores can be injected amongst a set of rich configuration aspects. Interesting Aspects of Simple Splitter Aggregator Operation The primary deciding factor for establishing whether this type of simple gateway is adequate for requirements is to understand what happens in the event of failure. If any exception occurring in your SI flow results in the flow invocation being abandoned and that suits your requirements, there's no need to read any further. If, however, you need to continue processing following failure in one of the gateways the remainder of this article may be of more interest. Exceptions, from any source, generated between the splitter and aggregator, will result in an empty or partial group being discarded by the Aggregator. The exception will propagate back to the closest upstream gateway for either handling by a custom bean or re-throwing by the gateway. Note that a custom release strategy on the Aggregator is difficult to use and especially so alongside timeouts but would not help in this case as the exception will propagate back to the leftmost gateway before the aggregator is invoked. It's also possible to configure exception handlers on the innermost gateways, the exception message could be caught but how do you route messages from a custom exception handler into the aggregator to complete the group, inject the aggregator channel definition into the custom exception handler? This is a poor approach and would involve unpacking an exception message payload, copying the original message headers into a new SI message and then adding the original payload - only four or five lines of code, but dirty it is. Following exception generation, exception messages (without modification) cannot be routed into an Aggregator to complete the group. The original message, the one that contains the correlation and sequence ids for the group and group position are buried inside the SI messages exception payload. If processing needs to continue following exception generation, it should be clear that in order to continue processing, the following must take place: the aggregation group needs to be completed, any exceptions must be caught and handled before getting back to the closet upstream gateway, the correlation and sequence identifiers that allow group completion in the aggregator are buried within the exception message payload and will require extraction and setting on the message that's bound for the aggregator A More Robust Solution - Messaging Gateway Adapter Pattern Dealing with exceptions and null returns from gateways naturally leads to a design that implements a wrapper around the messaging gateway. This affords a level of control that would otherwise be very difficult to establish. This adapter technique allows all returns from messaging gateways to be caught and processed as the messaging gateway is injected into the Service Activator and called directly from that. The messaging gateway no longer responds to the aggregator directly, it responds to a custom Java code Spring bean configured in the Service Activator namespace definition. As expected, processing that does not undergo exception will continue as normal. Those flows that experience exception conditions or unexpected or missing responses from messaging gateways need to process messages in such as way that message groups bound for aggregation can be completed. If the Service Activator were to allow the exception to be propagated outside of it's backing bean, the group would not complete. The same applies not just for exceptions but any return object that does not carry the prerequisite group correlation id and sequence headers - this is where the adaptation is applied. Exception messages or null responses from messaging gateways are caught and handled as shown in the following example code: import com.l8mdv.sample.*; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.integration.Message; import org.springframework.integration.MessageHeaders; import org.springframework.integration.support.MessageBuilder; import org.springframework.util.Assert; public class AvsServiceImpl implements AvsService { private static final Logger logger = LoggerFactory.getLogger(AvsServiceImpl.class); public static final String MISSING_MANDATORY_ARG = "Mandatory argument is missing."; private AvsGateway avsGateway; public AvsServiceImpl(final AvsGateway avsGateway) { this.avsGateway = avsGateway; } public Message service(Message message) { Assert.notNull(message, MISSING_MANDATORY_ARG); Assert.notNull(message.getPayload(), MISSING_MANDATORY_ARG); MessageHeaders requestMessageHeaders = message.getHeaders(); Message responseMessage = null; try { logger.debug("Entering AVS Gateway"); responseMessage = avsGateway.send(message); if (responseMessage == null) responseMessage = buildNewResponse(requestMessageHeaders, AvsResponseType.NULL_RESULT); logger.debug("Exited AVS Gateway"); return responseMessage; } catch (Exception e) { return buildNewResponse(responseMessage, requestMessageHeaders, AvsResponseType.EXCEPTION_RESULT, e); } } private Message buildNewResponse(MessageHeaders requestMessageHeaders, AvsResponseType avsResponseType) { Assert.notNull(requestMessageHeaders, MISSING_MANDATORY_ARG); Assert.notNull(avsResponseType, MISSING_MANDATORY_ARG); AvsResponse avsResponse = new AvsResponse(); avsResponse.setError(avsResponseType); return MessageBuilder.withPayload(avsResponse) .copyHeadersIfAbsent(requestMessageHeaders).build(); } private Message buildNewResponse(Message responseMessage, MessageHeaders requestMessageHeaders, AvsResponseType avsResponseType, Exception e) { Assert.notNull(responseMessage, MISSING_MANDATORY_ARG); Assert.notNull(responseMessage.getPayload(), MISSING_MANDATORY_ARG); Assert.notNull(requestMessageHeaders, MISSING_MANDATORY_ARG); Assert.notNull(avsResponseType, MISSING_MANDATORY_ARG); Assert.notNull(e, MISSING_MANDATORY_ARG); AvsResponse avsResponse = new AvsResponse(); avsResponse.setError(avsResponseType, responseMessage.getPayload(), e); return MessageBuilder.withPayload(avsResponse) .copyHeadersIfAbsent(requestMessageHeaders).build(); } } Notice the last line of the catch clause of the exception handling block. This line of code copies the correlation and sequence headers into the response message, this is mandatory if the aggregation group is going to be allowed to complete and will always be necessary following an exception as shown here. Consequences of using this technique There's no doubt that introducing a Messaging Gateway Adapter into SI config makes the configuration more complex to read and follow. The key factor here is that there is no longer a linear progression through the configuration file. This because the Service Activator must forward reference a Gateway or a Gateway defined before it's adapting Service Activator - in both cases the result is the same. Resources Note:- The design for the software that drove creation of this meta-pattern was based on a requirement that a number of external risk assessment services would be accessed by a single, central Risk Assessment Service. In order to satisfy clients of the service, invocation had to take place in parallel and continue despite failure in any one of those external services. This requirement lead to the design of the Messaging Gateway Adapter Pattern for the project. Spring Integration Reference Manual The solution approach for this problem was discussed directly with Mark Fisher (SpringSource) in the context of building Risk Assessment flows for a large US financial institution. Although the configuration and code is protected by NDA and copyright, it's acceptable to express the design intention and similar code in this article.

eclipse comes with a large set of different views: they allow the developer to represent the information in various forms and with different angles. most of these views are navigation oriented: a perfect example for this is the projects view or the outline view . but over time i add more projects, more resources to my project, and at a certain time things get overwhelming. i have a lot of projects, and i do not want to switch between workspace too often. yes, i can open and close projects, but this gets cumbersome too. thankfully, there is a solution in eclipse: working sets . working sets allow me to group elements for display in views. with that, i can do operations on a set of elements in that working set. especially as i’m using many projects the same time, working sets are a big help to focus on the right set of things at a time. i can define a set of things i want to look at, work with, or whatever: it allows me to get be productive in the universe of my environment. building/compiling a working set a nice feature of using working sets is to build a set of projects. instead of selecting a set of projects and then to compile them together, i use a working set. i use the menu project > build working set > select working set… to create or change working sets: menu to select working set if i have no working set defined, then this will show the following dialog where i can press new… to create a new one: creating new working set to create a working set of c/c++ projects, i select c/c++ and press next : new c c++ working set next i give a name and select the project(s) which shall be in my working set, and press finish : defining working set to build my set of projects, i can select the working set and press ok : selected working set search in a working set it is possible to limit the search to a working set. for this i can choose a working set as scope in the search dialogs: search in a working set managing working sets to manage working sets, i press ctrl+3 (see quick access ) and choose manage working sets… : ctrl+3 with manage working sets… note: i can add extra tool-bars and menus for working sets too (this is explained later). then i can manage my working sets: manage working set configuration project view filtering with working sets i can filter the projects shown in the project view based on working sets. for this i select the small triangle and select/define a working set or choose one from the most recently used sets: working sets for project view with this i can easily filter and focus on a subset of projects: project view with applied working set really cool tip: i’m using working sets as well to avoid too many workspaces. instead of having projects spread over different workspaces, i can keep them in one workspace and use working sets instead. there is an added benefit of using working sets: having too many projects open at the same time in eclipse can slow down the ide: using working sets allows me just to switch quickly between the set of projects i’m working on. but it does not stop at filtering by projects: you can filter even things inside the project structure. i simply deselect things i don’t want to see and can focus on what is important for me: filtering project files export and import of working sets note: import and export of working sets is not part of the standard codewarrior eclipse distribution. you get the import/export feature installed with the mqx plugins (www.freescale.com/mqx) or with the anyedit plugins (http://andrei.gmxhome.de/anyedit/index.html). to export a working set, i use the menu file > export > other > export working set : export working sets with mqx plugins note: the anyedit plugins come as well with an import/expert working set wizard. the file format is different, and the mqx plugin allows drag&drop of the file into eclipse. this gives the following dialog where i can specify the file name and the root of projects: export working set dialog this will store the settings in an xml file. importing the working set is done with file > import > other > import working set . tip: i’m using *.wsd extension for working sets. that way i can simply drag&drop the file into eclipse to import it. other kinds of working sets working sets do not stop at projects and files: select working set type i can create working sets of breakpoints or analysis/trace points. or i can create working sets of any resource files or tasks. the possibilities are nearly endless and depend as well on the extra plugins installed. window working sets and now back to the really cool part. one question remains: what are window working sets? window working sets the thing is that every view and dialog has its own working set setting. in my example below i use a working set ‘coldfire’ for the projects view, but my search dialog has a ‘kinetis’ working set configured: two different working sets sometimes i want this, but not always. what i need is a ‘global’ working set. and here the window working sets comes to rescue me. for this i’m going to add some menus and toolbars to make it really easy… for this i choose the menu window > customize perspecti ve. in the command groups i enable ‘window working set’. additionally it is a good idea to enable ‘working set manipulation’ as well: window working set commands the same way i can enable the toolbar and menu visibility. this gives me added tool-bars to switch between working sets and to add/remove things from a working set quickly: working set toolbars in a similar way, it gives me menu access as well: working sets menu and here is the trick: using ‘ window working set ‘ really means ‘ using the global workbench working set ‘. to select the global workbench working set, i use the toolbar icon to switch between window (or workbench) working sets: selecting global window or workbench working set in the individual views i choose to use the window working sets instead a selection of working sets: selected global window or workbench working set now my working set settings are shared and common for all views: if i switch the working set, it will switch for all views where i have set it to ‘window working set’: window working set applied to multiple views that way my working set is the same across views, and switching between different project settings is done with a simple mouse click. summary working sets are an extremely powerful feature to get focus on a subset of things inside eclipse, based on my workflow. as with many great eclipse features, i need to know about it until you really appreciate the power of it. who knows how many other hidden treasures are buried in eclipse i hope this article helps to save you a few mouse clicks. happy work-setting but over time i add more projects, more resources to my project, and at a certain time things get overwhelming. i have a lot of projects, and i do not want to switch between workspace too often.

Spring Integration (SI) Gateways Spring Integration Gateways () provide a semantically rich interface to message sub-systems. Gateways are specified using namespace constructs, these reference a specific Java interface () that is backed by an object dynamically implemented at run-time by the Spring Integration framework. Furthermore, these Java interfaces can, if you so wish, be defined entirely independent of any Spring artefacts - that's both code and configuration. One of the primary advantages of using the SI gateway as an interface to message sub-systems is that it's possible to automatically adopt the benefit of rich, default and customisable, gateway configuration. One such configuration attribute deserves further scrutiny and discussion primarily because it's easy to misunderstand and misconfigure around - default-reply-timeout. Primary Motivator for Gateway Analysis During recent consulting engagements, I've encountered a number of deployments that use Spring Integration Gateway specifications that may, in some circumstances, lead to production operational instability. This has often been in high-pressure environments or those where technology support is not backed by adequate training, testing, review or technology mentoring. How do gateways behave in Spring Integration (R2.0.5) One of the key sections, regarding gateways, in the Spring Integration manual clearly explains gateway semantics. Below is a 2-dimensional table of possible non-standard gateway returns for each of the scenarios that the SI Manual (r2.0.5) refers to. Gateway Non-standard Responses Runtime Events default-reply-timeout=x Single-threaded default-reply-timeout=x Multi-threaded default-reply-timeout=null Single-threaded default-reply-timeout=null Multi-threaded 1. Long Running Process Thread Parked null returned Thread Parked Thread Parked 2. Null Returned Downstream null returned null returned Thread Parked Thread Parked 3. void method Downstream null returned null returned Thread Parked Thread Parked 4. Runtime Exception Error handler invoked or exception thrown. Error handler invoked or exception thrown. Error handler invoked or exception thrown. Error handler invoked or exception thrown. The key parts of this table are the conditions that lead to invoking threads being parked (noted in red), nulls returned (noted in orange) and exceptions (noted in green). Each contributor consists of configuration that is under the developers control, deployed code that is under developers control and conditions that are usually not under developers control. Clearly, the column headings in the table above are divided into two sections; two gateway configuration attributes. The default-reply-timeout is set by the SI configured and is the amount of time that a client call is wiling to wait for a response from the gateway. Secondly, synchronous flows are represented by Single-threaded flows, asynchronous by Multi-threaded flows. A synchronous, or single-threaded flow, is one such as the following: The implicit input channel (gateway-request-channel) has no associated dispatcher configured. An asynchronous, or multi-threaded flow, is one such as the following: The explicit input channel has a dispatcher configured ("taskExecutor"). This task executor specifies a thread pool that supplies threads for execution and whose configuration as above marks a thread boundary. Note: This is not the only way of making channels asynchronous The other configuration attribute referenced is default-reply-timeout, this is set on the gateway namespace configuration such as the example above. Note that both of these runtime aspects are set by the configurer during SI flow design and implementation. They are entirely under developer control. The 'Runtime Events' column indicates gateway relevant runtime events that have to be considered during gateway configuration - these are obviously not under developer control. Trigger conditions for these events are not as unusual as one may hope. 1. Long Running Processes It's not uncommon for thread pools to become exhausted because all pooled threads are waiting for an external resource accessed through a socket, this may be a long running database query, a firewall keeping a connection open despite the server terminating etc. There is significant potential for these types of trigger. Some long-running processes terminate naturally, sometimes they never completed - an application restart is required. 2. Null returned downstream A null may be returned from a downstream SI construct such as a Transformer, Service Activator or Gateway. A Gateway may return null in some circumstances such as following a gateway timeout event. 3. Void method downstream Any custom code invoked during an SI flow may use a void method signature. This can also be caused by configuration in circumstances where flows are determined dynamically at runtime. 4. Runtime Exception RuntimeException's can be triggered during normal operation and are generally handled by catching them at the gateway or allowing them to propagate through. The reason that they are coloured green in the table above is that they are generally much easier to handle than timeouts. Gateway Timeout Handling Strategies There are four possible outcomes from invoking a gateway with a request message, all of these as a result of specific runtime events: a) an ordinary message response, b) an exception message, c) a null or d) no-response. Ordinary business responses and exceptions are straight forward to understand and will not be covered further in this article. The two significant outcomes that will be explored further are strategies for dealing with nulls and no-response. Generally speaking, long running processes either terminate or not. Long running processes that terminate may eventually return a message through the invoked gateway or timeout depending on timeout configuration, in which case a null may be returned. The severity of this as a problem depends on throughput volume, length of long running process and system resources (thread-pool size). Configuration exists for default-reply-timeout In the case where a long running process event is underway and a default-reply-timeout has been set, as long as the long running process completes before the default-reply-timeout expires, there is no problem to deal with. However, if the long running process does not complete before that timeout expires one of three outcomes will apply. Firstly, if the long running process terminates subsequent to the reply timeout expiry, the gateway will have already returned null to the invoker so the null response needs handling by the invoker. The thread handling the long-running process will be returned to the pool. Secondly, if the long running process does not terminate and a reply timeout has been set, the gateway will return null to the gateway invoker but the thread executing the long-running process will not get returned to the pool. Thirdly, and most significantly, if a default-reply-timeout has been configured but the long running process is running on the same thread as the invoker, i.e. synchronous channels supply messages to that process, the thread will not return, the default-reply-timeout has no affect. Assuming the most common processing scenario, a long running process completes either before or after the reply timeout expiry. When a null is returned by the gateway, the invoker is forced to deal with a null response. It's often unacceptable to force gateway consumers to deal with null responses and is not necessary as with a little additional configuration, this can be avoided. Absent Configuration for default-reply-timeout The most significant danger exists around gateways that have no default-reply-timeout configuration set. A long running process or a null returned from downstream will mean that the invoking thread is parked. This is true for both synchronous and asynchronous flows and may ultimately force an application to be restarted because the invoker thread pool is likely to start on a depletion course if this continues to occur. Spring Integration Timeout Handling Design Strategies For those Spring Integration configuration designers that are comfortable with gateway invokers dealing with null responses, exceptions and set default-reply-timeouts on gateways, there's no need to read further. However, if you wish to provide clients of your gateway a more predictable response, a couple of strategies exist for handling null responses from gateways in order that invokers are protected from having to deal with them. Firstly, the simpliest solution is to wrap the gateway with a service activator. The gateway must have the default-reply-timeout attribute value set in order to avoid unnecessary parking of threads. In order to avoid the consequence of long-running threads it's also very prudent to use a dispatcher soon after entry to the gateway - this breaks the thread boundary. Whilst this is a valid technical approach, the impact is that we have forced a different entry point to our message sub-system. Entry is now via a Service Activator rather than a Gateway. A side affect of this change is that the testing entry point changes. Integration tests that would normally reference a gateway to send a message now have to locate the backing implementation for the Service Activator, not ideal. An alternative approach toward solving this problem would be to configure two gateways with a Service Activator between them. Only one of the gateways would be exposed to invokers, the outer one. Both Gateways would reference the same service interface. The outer gateway specification would not specify the default-reply-timeout but would specify the input and output channels in the same way that a single gateway would. The Service Activator between the Gateways would handle null gateway responses and possibly any exceptions if preferred to the gateway error handler approach. An example is as follows: The Service Activator bean (enrollmentServiceGatewayHandler) deals with both null and exception responses from the adapted gateway (enrollmentServiceAdaptedGateway), in the situation where these are generated a business response detailing the error is generated. Spring Integration R2.1 Changes async-executor on gateway spec