have been working on a fairly simple JMeter load script that I can run a series of 4 sequential pages against an Apache server, but the goal was to have the server support 2,000 concurrent requests for 5 minutes without error. Most of my issues in this exercise have been with JMeter and the client machine used to test the Apache server. To begin, I must state I was originally configuring Apache with a prefork MPM: StartServers 100 MinSpareServers 75 MaxSpareServers 100 ServerLimit 2000 MaxClients 2000 MaxRequestsPerChild 0 At approximately line 72 of Jmeter.bat, there are several entries the manage the JVM for running Jmeter. set HEAP=-Xms512m -Xmx512m set NEW=-XX:NewSize=128m -XX:MaxNewSize=128m set SURVIVOR=-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=50% set TENURING=-XX:MaxTenuringThreshold=2 set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m I decided to start with 1,000 concurrent requests for 5 minutes just to see how the test would fair. With the above settings I started getting OOM errors almost immediately so I decided to increase the HEAP and NEW memory to eliminate the issue and wanted to add more GC settings to increase the JVM’s ability to clean up: set HEAP=-Xms1024m -Xmx1024m -Xss128k set NEW=-XX:NewSize=256m -XX:MaxNewSize=256m set SURVIVOR=-XX:SurvivorRatio=14 -XX:TargetSurvivorRatio=50% set "TENURING=-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxTenuringThreshold=4" set "EVACUATION=-XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+UseCompressedStrings -XX:+OptimizeStringConcat" set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 set PERM=-XX:PermSize=64m -XX:MaxPermSize=64m This did resolve the JMeter OOM issues, but now started getting Apache errors. During the ramp-up phase, I started getting connection refused errors: Response code: Non HTTP response code: org.apache.http.conn.HttpHostConnectException Response message: Non HTTP response message: Connection to http://pasundtastgprt2:8001 refused I started looking at the Apache server and noticed that the number of httpd threads was at 1,000 and it appeared that JMeter was running out of memory because the requests where starting to back up. This is why we load test right! So I decided to run a worker MQM and recompiled Apache to support the new MPM ServerLimit 80 StartServers 25 MaxClients 2000 MinSpareThreads 75 MaxSpareThreads 125 ThreadsPerChild 5 MaxRequestsPerChild 0 I started testing this configuration and while monitoring the server running 1,000 concurrent requests and the server looked like Apache was handling 1,000 requests just fine. I was running a simple command to output the sockets and httpd processes on the server during the load test: while true do echo -----`date '+%r'` -----: netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n echo httpd processes: [`ps aux | grep httpd | wc -l`] echo . sleep 30 done Then when I was monitoring the load test, I was concerned about seeing 82 httpd processes running which was the ServerLimit I had set. -----08:02:37 AM -----: 1 established) 1 Foreign 4 CLOSE_WAIT 17 LISTEN 32 FIN_WAIT2 41 ESTABLISHED 69 FIN_WAIT1 630 SYN_RECV 45386 TIME_WAIT [82] httpd processes . I now increased the load to my target of 2,000 concurrent requests and restarted the JMeter test and was able to get to around 1,800 concurrent request and started getting connection refused errors again. I suspected that my Servers where maxed out and was not able to create anymore threads for those servers where having: 80 server * 5 threads each server == 400 requests processed concurrently So I increased the number of threads to 25 80 server * 25 threads each server == 2,000 requests processed concurrently To end up with this worker setting: ServerLimit 80 StartServers 25 MaxClients 2000 MinSpareThreads 75 MaxSpareThreads 125 ThreadsPerChild 25 MaxRequestsPerChild 0 I was then able to turn the load up to 2,000 concurrent requests. -----08:53:27 AM -----: 1 established) 1 FIN_WAIT2 1 Foreign 4 CLOSE_WAIT 12 CLOSING 17 LISTEN 129 ESTABLISHED 621 FIN_WAIT1 1203 SYN_RECV 55556 TIME_WAIT [53] httpd processes At this point we are only using 53 Servers and 2,000 clients. The tests sustained zero errors for 5 minutes during the test. As a test I increased the ThreadsPerChild to 40 and run the load test against 3,000 concurrent requests. I was able to get to around 2,800 concurrent requests then I started getting connection refused errors: Error Count: 1 Response code: Non HTTP response code: org.apache.http.conn.HttpHostConnectException Response message: Non HTTP response message: Connection to http://pasundtastgprt2:8001 refused and I also started getting JMeter errors: Response code: Non HTTP response code: java.net.BindException Response message: Non HTTP response message: Address already in use: connect So in the furure I would like to see how much further I can push the server and I think I can get to 3,000 concurrent and most likely far more than that on my current installation. From http://www.baselogic.com/blog/development/adding-memory-jmeter/

Even a basic planning problem, such as bin packing, can be notoriously hard to solve and scale. One might consider the brute force algorithm. Let's take a look at how that algorithm works out on the cloud balance example of Drools Planner: Given a set of servers with different hardware (CPU, memory and network bandwidth) and given a set of processes with different hardware requirements, assign each process to 1 server and minimize the total cost of the active servers. The brute force algorithm is simple: try every combination between processes where each process is assigned to each server. For example, if we have 6 processes (P0, P1, P2, P3, P4, P5) and 2 servers (S0, S1), we'd try these solutions: P0->S0, P1->S0, P2->S0, P3->S0, P4->S0, P5->S0 P0->S0, P1->S0, P2->S0, P3->S0, P4->S0, P5->S1 P0->S0, P1->S0, P2->S0, P3->S0, P4->S1, P5->S0 P0->S0, P1->S0, P2->S0, P3->S0, P4->S1, P5->S1 ... P0->S1, P1->S1, P2->S1, P3->S1, P4->S1, P5->S1 On my machine, it takes 15ms to calculate the score of these 2^6 combinations. When I scale out to 9 processes and 3 servers, which are 3^9 combinations, it becomes 1582ms. So it scales like this: Notice that despite that the number of processes has not even doubled, the running time multiplied by 100! For comparison, I 've added the running time of the First Fit algorithm. And it gets worse: for 12 processes and 4 servers, which are 4^12 combinations, it take more than 17 minutes: What if we want to distribute 3000 processes over 1000 servers? With this kind of scalability, it will take too long. In fact, the brute force algorithm is useless. Luckily, Drools Planner implements several other optimization algorithms, which can handle such loads. If you want to know more about them, take a look at the Drools Planner manual or come to my talk at JUDCon London (31 Oct - 1 Nov). This article was originally posted on the Drools & jBPM blog.

Below is a list of some tools that can help you examine your Java source code for potential problems: 1. PMD from http://pmd.sourceforge.net/ License: PMD is licensed under a “BSD-style” license PMD scans Java source code and looks for potential problems like: * Possible bugs – empty try/catch/finally/switch statements * Dead code – unused local variables, parameters and private methods * Suboptimal code – wasteful String/StringBuffer usage * Overcomplicated expressions – unnecessary if statements, for loops that could be while loops * Duplicate code – copied/pasted code means copied/pasted bugs You can download everything from here, and you can get an overview of all the rules at the rulesets index page. PMD is integrated with JDeveloper, Eclipse, JEdit, JBuilder, BlueJ, CodeGuide, NetBeans/Sun Java Studio Enterprise/Creator, IntelliJ IDEA, TextPad, Maven, Ant, Gel, JCreator, and Emacs. 2. FindBug from http://findbugs.sourceforge.net License: L-GPL FindBugs, a program which uses static analysis to look for bugs in Java code. And since this is a project from my alumni university (IEEE – University of Maryland, College Park – Bill Pugh) , I have to definitely add this contribution to this list. 3. Clover from http://www.cenqua.com/clover/ License: Free for Open Source (more like a GPL) Measures statement, method, and branch coverage and has XML, HTML, and GUI reporting. and comprehensive plug-ins for major IDEs. * Improve Test Quality * Increase Testing Productivity * Keep Team on Track Fully integrated plugins for NetBeans, Eclipse , IntelliJ IDEA, JBuilder and JDeveloper. These plugins allow you to measure and inspect coverage results without leaving the IDE. Seamless Integration with projects using Apache Ant and Maven. * Easy integration into legacy build systems with command line interface and API. Fast, accurate, configurable, detailed coverage reporting of Method, Statement, and Branch coverage. Rich reporting in HTML, PDF, XML or a Swing GUI Precise control over the coverage gathering with source-level filtering. Historical charting of code coverage and other metrics. Fully compatible with JUnit 3.x & 4.x, TestNG, JTiger and other testing frameworks. Can also be used with manual, functional or integration testing. 4. Macker from http://innig.net/macker/ License: GPL Macker is a build-time architectural rule checking utility for Java developers. It’s meant to model the architectural ideals programmers always dream up for their projects, and then break — it helps keep code clean and consistent. You can tailor a rules file to suit a specific project’s structure, or write some general “good practice” rules for your code. Macker doesn’t try to shove anybody else’s rules down your throat; it’s flexible, and writing a rules file is part of the development process for each unique project. 5 EMMA from http://emma.sourceforge.net/ License: EMMA is distributed under the terms of Common Public License v1.0 and is thus free for both open-source and commercial development. Reports on class, method, basic block, and line coverage (text, HTML, and XML). EMMA can instrument classes for coverage either offline (before they are loaded) or on the fly (using an instrumenting application classloader). Supported coverage types: class, method, line, basic block. EMMA can detect when a single source code line is covered only partially. Coverage stats are aggregated at method, class, package, and “all classes” levels. Output report types: plain text, HTML, XML. All report types support drill-down, to a user-controlled detail depth. The HTML report supports source code linking. Output reports can highlight items with coverage levels below user-provided thresholds. Coverage data obtained in different instrumentation or test runs can be merged together. EMMA does not require access to the source code and degrades gracefully with decreasing amount of debug information available in the input classes. EMMA can instrument individial .class files or entire .jars (in place, if desired). Efficient coverage subset filtering is possible, too. Makefile and ANT build integration are supported on equal footing. EMMA is quite fast: the runtime overhead of added instrumentation is small (5-20%) and the bytecode instrumentor itself is very fast (mostly limited by file I/O speed). Memory overhead is a few hundred bytes per Java class. EMMA is 100% pure Java, has no external library dependencies, and works in any Java 2 JVM (even 1.2.x). 6. XRadar from http://xradar.sourceforge.net/ License: BSD (me thinks) The XRadar is an open extensible code report tool currently supporting all Java based systems. The batch-processing framework produces HTML/SVG reports of the systems current state and the development over time – all presented in sexy tables and graphs. The XRadar gives measurements on standard software metrics such as package metrics and dependencies, code size and complexity, code duplications, coding violations and code-style violations. 7. Hammurapi from Hammurapi Group License: (if anyone knows the license for this email me Venkatt.Guhesan at Y! dot com) Hammurapi is a tool for execution of automated inspection of Java program code. Following the example of 282 rules of Hammurabi’s code, we are offered over 120 Java classes, the so-called inspectors, which can, at three levels (source code, packages, repository of Java files), state whether the analysed source code contains violations of commonly accepted standards of coding. Relevant Links: http://en.sdjournal.org/products/articleInfo/93 http://wiki.hammurapi.biz/index.php?title=Hammurapi_4_Quick_Start 8. Relief from http://www.workingfrog.org/ License: GPL Relief is a design tool providing a new look on Java projects. Relying on our ability to deal with real objects by examining their shape, size or relative place in space it gives a “physical” view on java packages, types and fields and their relationships, making them easier to handle. 9. Hudson from http://hudson-ci.org/ License: MIT Hudson is a continuous integration (CI) tool written in Java, which runs in a servlet container, such as Apache Tomcat or the GlassFish application server. It supports SCM tools including CVS, Subversion, Git and Clearcase and can execute Apache Ant and Apache Maven based projects, as well as arbitrary shell scripts and Windows batch commands. 10. Cobertura from http://cobertura.sourceforge.net/ License: GNU GPL Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage. 11. SonarSource from http://www.sonarsource.org/ (recommended by Vishwanath Krishnamurthi – thanks) License: LGPL Sonar is an open platform to manage code quality. As such, it covers the 7 axes of code quality: Architecture & Design, Duplications, Unit Tests, Complexity, Potential bugs, Coding rules, Comments. From http://mythinkpond.wordpress.com/2011/07/14/java-tools-for-source-code-optimization-and-analysis/

I am an experienced Java developer who has used various IDEs and prefer NetBeans IDE over all others by a long shot. I am also very fond of Maven as the tool to simplify and automate nearly every aspect of the development of my Java project throughout its lifecycle. Recently, I started developing Android applications and naturally I looked for a Maven plugin that would manage my Android projects. Luckily I found the maven-android-plugin which worked like a charm and allowed me to use Maven for developing my Android projects. The Android Emulator from the Android SDK seemed unusably slow. Lucklily, I found a way to use an Android Virtual Machine for VirtualBox that worked nearly as fast as my native computer! This page documents my experiences. Tested Environment Dev machine: Ubuntu 11.04 Linux IDE: NetBeans VirtualBox: 4.0.8 r71778 Android SDK Revision 11, Add on XML Schema #1, Repository XML Schema #3 (from About in SDK and AVD Manager) Android Version: 2.2 Overview of Steps Download and install the Android SDK on your dev machine Attach an Android Device to dev machine Configure and load your device for development and other use Create an initial Android maven project Connect Android Device to Android SDK Debug Android app using NetBeans Graphical Debuger Download and Install Android SDK Download and install the Android SDK on your dev machine as described here. Make sure to set the following in dev machine ~/.bashrc file: export ANDROID_HOME=$HOME/android-sdk-linux_x86 #Change as needed export PATH="$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$PATH" Attaching an Android Device to Dev Machine If you have an actual device that is usually always best. If not, you must use a virtual Android device which usually has various limitations (e.g. no GPS, Camera etc.). The Android SDK makes it easy to create a new Virtual Device but the resulting device is painfully slow in my experience and not usable. Do not bother with this. Instead, create a virtual Android device using VirtualBox as described in the following steps: Install virtual box and initial Android VM as described here: http://androidspin.com/2011/01/24/howto-install-android-x86-2-2-in-virtualbox/ http://geeknizer.com/how-to-run-google-android-in-virtualbox-vmware-on-netbooks/ Configure Android VM so it is connected bidirectionally with your dev machine over TCP as described here: http://stackoverflow.com/questions/61156/virtualbox-host-guest-network-setup I used the approach of configuring a HOST ONLY network adapater and a second NAT adapter on the Android VM within virtual box. Configuring your Android Device This section describes various things I did to setup a dev environment for my Android device: Root the device. I used Universal AndRoot Install ConnectBot so you have ssh and related network utilities Creating Initial Android Maven Application Create initial project using instructions here. I found it best to create stub project structure using the maven-archtype-plugin and the archtypes at https://github.com/akquinet/android-archetypes/wiki Connecting Android VM Device to Android SDK In order for your code to be deployed from NetBeans IDE to Android Device and in order for you to monitor your deployed app from the Dalvik Debug Monitor (ddms) you need to connect your android VM device to the android sdk over TCP as described in the following steps. On Android Device open the Terminal Emulator Type su to become root (your device must be rooted for this Type following commands in root shell: setprop service.adb.tcp.port 5555 stop adbd start adbd Type the following commands on dev machine shell. TODO: Note that IP address below is whatever is the ip address associated with the device (see ifconfig on linux for device vboxnet0) adb tcpip 5555 adb connect 192.168.0.101:5555 For details on above steps see: http://stackoverflow.com/questions/2604727/how-can-i-connect-to-android-with-adb-over-tcp Set up port forwarding as described here http://redkrieg.com/2010/10/11/adb-over-ssh-fun-with-port-forwards/ (this is where I am most fuzzy) Build your maven android project using Right-Click / Clean and Build Now for the acid test whether you can deploy your app to the device from NetBeans IDE! Right-click / Custom / Goal to show Run Maven dialog. Enter android:deploy in Goals field. Select Remember As button and enter android:deploy for its text field. If all is well, the app will deploy to the device and will show up in its "Applications" screen. Debugging Android App Using NetBeans Graphical Debugger Once you can build and deploy your app to the real or virtual Android device, here are the steps to debug the app using NetBeans debugger: On Device: Start the app (TODO: determine how to start app on device with JVM options so it can wait for debugger connection. This should be easy) On Dev Machine run Dalvik Debug Monitor (ddms) in background: $ANDROID_HOME/tools/ddms & Lookup your app in ddms and get its debug port. This is described here but does not address NetBeans specifically In NetBeans do: Debug / Attach Debugger and specify the port looked up in ddms in previous step. You may leave rest of the fields with defaults. Click OK

Five easy steps to reset MySQL root password. Stop the MySQL server. Start the MySQL server with the --skip-grant-tables option. (it will not prompt for password) Connect to MySQL server as the root user. Setup new MySQL root password. Exit and restart the MySQL server. ### Shell Commands ### /etc/init.d/mysql stop mysqld_safe --skip-grant-tables & mysql -u root ### SQL Commands ### use mysql; UPDATE user SET password=PASSWORD("new-password") WHERE user='root'; flush privileges; exit ### Shell Commands ### /etc/init.d/mysql stop /etc/init.d/mysql start mysql -u root -pnew-password

There has been a lot of buzz lately on high-availability and clustering. Most developers don't care and why should they? These features should be transparent to the application architecture and not something of concern to the developers of that application. But knowledge never hurts, so I emerged myself into the world of load balancing, heartbeats and virtual IP addresses. And you know what? Next time we need a infrastructure like this, I can at least sit down with the guys from the infrastructure department and at least know what the hell they are talking about. So what exactly is a high-availability clustered infrastructure (HACI, as I'll call it from now on) ? In essence, it should be a zero-downtime infrastructure (or at least perceived as one by the end user, which means never ever returning a default browser 404 page), capable of horizontal scaling when the need for it arises and without a single point of failure. It's the SLA writer's dream. A basic HACI setup looks like this: The users enters through a virtual IP address, assigned to one of the two load balancers. Only one of the load-balancers is active (the active master, LB1), the other one is there in the event LB1 fails ((LB2, a passive slave). The two load balancers are redundant, ie. having the exact same configuration. The load balancers redirect all traffic to the real servers. This can be done through round-robin assignment or through other means like sticky sessions, where the same user is redirected to the same server each and every time within a session. Servers can be added at any moment and configured on the load balancers. Ideally, the load balancer configuration is aware of the hardware specification and balances the load accordingly, but that's beyond the scope of this article (it involves adding weights). If all servers balanced by the load balancer fail, a backup server should be used to redirect all traffic coming from the load balancer. This can be a very lightweight server, which purpose is only to provide a sensible error page to the user (something like 'Sorry, we are performing maintenance'). Again, perception and immediate feedback to the user is key. You don't want to show the user a plain 404 page. Off course, if the backup server goes down too, you're in trouble (off course, by that time, warning bells should have gone off on every level in the hierarchy). So how to achieve this with as little effort as possible? If you want to try this out, I suggest you start by installing a virtual machine like VirtualBox or VMWare. This way you can try out the configuration yourself. In this example, I'll be load-balancing 3 Tomcat servers using sticky sessions using 2 load balancers in active-passive mode. I'm assuming all 3 Tomcat servers share the same hardware configuration, so they are all able to handle the same amount of traffic each. I'm also throwing in a backup server, in case all 3 Tomcat servers go down (serving a custom 503 page kindly informing the user of a catastrophic failure, instead of dropping the standard 404 bomb). You want to start off by assigning IP addresses to the servers. This will make your life a bit easier. We'll need 7 addresses: 3 for the tomcat server, 1 for the backup server, 2 for the loadbalancers and 1 virtual IP address to be shared between the load balancers (and which will be the entry point for your users). So our assignment will be: Virtual IP 10.0.5.99 www.haci.local LB1 10.0.5.100 lb1.haci.local #MASTER LB2 10.0.5.101 lb2.haci.local #SLAVE WEB1 10.0.5.102 web1.haci.local WEB2 10.0.5.103 web2.haci.local WEB3 10.0.5.104 web3.haci.local BACKUP 10.0.5.105 backup.haci.local Setting up the web servers is easy. You just install Tomcat on each server and create a simple JSP file to be served to users (make a small change, like the background color, on each server to distinguish the servers). I won't be covering session replication between the Tomcat servers, as it'll take me too far. If you want, you can configure the appropriate session replication and storage (using multicast or JDBC for example). The backup server I'm using is a basic LAMP server that returns a simple 503 page on every request it gets. The 503 error code is important, because it reflects the current state of the system: currently unavailable. For the loadbalancers I'll be using 2 applications: HAProxy and keepalived. HAProxy is going to handle load balancing, while keepalived will handle the failover between the two load balancers. First, we're going to configure HAProxy for both LB1 and LB2. Installing HAProxy is quite easy on an ubuntu system. Just do a sudo apt-get install haproxy and you're off. After the install, backup the current HAProxy config and start editing away. cp /etc/haproxy.cfg /etc/haproxy.cfg_orig cat /dev/null > /etc/haproxy.cfg vi /etc/haproxy.cfg The content of the config to reflect our setup should become something like this (same config on LB1 and LB2): global log 127.0.0.1 local0 log 127.0.0.1 local1 notice #log loghost local0 info maxconn 4096 #debug #quiet user haproxy group haproxy defaults log global mode http option httplog option dontlognull retries 3 redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000 frontend http-in bind 10.0.5.99:80 default_backend servers backend servers mode http stats enable stats auth someuser:somepassword balance roundrobin cookie JSESSIONID prefix option httpclose option forwardfor option httpchk HEAD /check.txt HTTP/1.0 server web1 10.0.5.102:80 cookie haci_web1 check server web2 10.0.5.103:80 cookie haci_web2 check server web3 10.0.5.104:80 cookie haci_web3 check server webbackup 10.0.5.105:80 backup After this, enable HAProxy on both LB1 and LB2 by editing /etc/defaults/haproxy # Set ENABLED to 1 if you want the init script to start haproxy. ENABLED=1 # Add extra flags here. #EXTRAOPTS="-de -m 16" So far for the HAProxy configuration. We can't start it up yet, as LB1 and LB2 aren't listening yet on the virtual IP address. Next we'll configure the failover of the loadbalancers using keepalived. Installing it on Ubuntu is as easy as it was for HAProxy: sudo apt-get install keepalived. But its configuration is slightly different on both load balancers. First, we need to configure the both servers to be able to listen to the shared IP address. Add the following line to /etc/sysctl.conf: net.ipv4.ip_nonlocal_bind=1 And run sysctl -p Now, we configure keepalived so that LB1 is configured as the main load balancer and binds to the shared IP address, while LB2 is on standby, ready to take over whenever LB1 goes down. The configuration for LB1 looks like this (edit /etc/keepalived/keepalived.conf): vrrp_script chk_haproxy { # Requires keepalived-1.1.13 script "killall -0 haproxy" # cheaper than pidof interval 2 # check every 2 seconds weight 2 # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 101 # 101 on master, 100 on backup virtual_ipaddress { 10.0.5.99 } track_script { chk_haproxy } } Start up keepalived and check whether it is listening to the virtual IP address. /etc/init.d/keepalived start ip addr sh eth0 It should return something like this, indicating it is listening to the virtual IP address 2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:a5:5b:93 brd ff:ff:ff:ff:ff:ff inet 10.0.5.100/24 brd 10.0.5.255 scope global eth0 inet 10.0.5.99/32 scope global eth0 inet6 fe80::20c:29ff:fea5:5b93/64 scope link valid_lft forever preferred_lft forever Next, we configure LB2. The configuration is almost the same, exception for the priority. vrrp_script chk_haproxy { # Requires keepalived-1.1.13 script "killall -0 haproxy" # cheaper than pidof interval 2 # check every 2 seconds weight 2 # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 100 # 101 on master, 100 on backup virtual_ipaddress { 10.0.5.99 } track_script { chk_haproxy } } Start up keepalived and check the network interface. /etc/init.d/keepalived start ip addr sh eth0 It should return something like this, indicating it is not listening to the virtual IP address 2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:a5:5b:93 brd ff:ff:ff:ff:ff:ff inet 10.0.5.101/24 brd 10.0.5.255 scope global eth0 inet6 fe80::20c:29ff:fea5:5b93/64 scope link valid_lft forever preferred_lft forever Now, start up HAProxy on both LB1 and LB2. /etc/init.d/haproxy start Now you can issue requests to 10.0.5.99 (or www.haci.local), which will go to LB1, which in turn will load-balance the request to either WEB1, WEB2 and WEB3. You can test the load balancing by turning off WEB1 (or the server you're currently on). You can also the backup server by turning all main webservers (WEB1, WEB2 and WEB3). And you can test the loadbalancer failover by turning off LB1. At that point LB2 will kick in and act as the master, loadbalancing all requests. When you turn LB1 back on, it'll take over the master role once again. HAProxy allows you to add extra servers very easily, reloading the configuration without breaking existing sessions. See the HAProxy documentation for more info or on ServerFault. (http://serverfault.com/questions/165883/is-there-a-way-to-add-more-backend-server-to-haproxy-without-restarting-haproxy). Cheap and effective. While most enterprise shops have hardware load balancers, which also have these possibilities and more, if you're on a tight budget or need to simulate a HACI environment for development purposes (a lesson here: always simulate your production environment when you're testing during development), this might be the sane option. To finish, I'll quickly explain how to set up the backup server (a simple LAMP server). Create a vhost configuration on the apache for www.haci.local or any other domain pointing to the virtual IP address and set up mod_rewrite for it: RewriteEngine On RewriteCond %{REQUEST_URI} !\.(css|gif|ico|jpg|js|png|swf|txt)$ [NC] RewriteConf %{REQUEST_URI} !/503.php RewriteRule .* /503.php [L] Then create the 503.php file and add this to the top of it: Sorry, our servers are currently undergoing maintenance. Please check back with us in a while. Thank you for your patience. You can decorate the 503.php file any way you like. You can even use CSS, JavaScript and image files in the php file. Now, back to my IDE. I'm getting withdrawal symptoms.

I've blogged a few times now about Git (which I pronounce with a hard 'g' a la "get", as it's supposed to be named for Linus Torvalds, a self-described git, but which I've also heard called pronounced with a soft 'g' like "jet"). Either way, I'm finding it way more efficient and less painful than either CVS or SVN combined. So, to continue this series ([1], [2], [3]), here is how (and why) to pull an SVN repo down as a Git repo, but with the omission of old (irrelevant) revisions and branches. Using SVN for SVN repos In days of yore when working with the JBoss Tools and JBoss Developer Studio SVN repos, I would keep a copy of everything in trunk on disk, plus the current active branch (most recent milestone or stable branch maintenance). With all the SVN metadata, this would eat up substantial amounts of disk space but still require network access to pull any old history of files. The two repos were about 2G of space on disk, for each branch. Sure, there's tooling to be able to diff and merge between branches w/o having both branches physically checked out, but nothing beats the ability to place two folders side by side OFFLINE for deep comparisons. So, at times, I would burn as much as 6-8G of disk simply to have a few branches of source for comparison and merging. With my painfullly slow IDE drive, this would grind my machine to a halt, especially when doing any SVN operation or counting files / disk usage. Using Git for SVN repos naively Recently, I started using git-svn to pull the whole JBDS repo into a local Git repo, but it was slow to create and still unwieldy. And the JBoss Tools repo was too large to even create as a Git repo - the operation would run out of memory while processing old revisions of code to play forward. At this point, I was stuck having individual Git repos for each JBoss Tools component (major source folder) in SVN: archives, as, birt, bpel, build, etc. It worked, but replicating it when I needed to create a matching repo-collection for a branch was painful and time-consuming. As well, all the old revision information was eating even more disk than before: jbosstools' trunk as multiple git-svn clones: 6.1G devstudio's trunk as single git-svn clone: 1.3G So, now, instead of a couple Gb per branch, I was at nearly 4x as much disk usage. But at least I could work offline and not deal w/ network-intense activity just to check history or commit a change. Still, far from ideal. Cloning SVN with standard layout & partial history This past week, I discovered two ways to make the git-svn experience at least an order of magnitude better: Standard layout (-s) - this allows your generated Git repo to contain the usual trunk, branches/* and tags/* layout that's present in the source SVN repo. This is a win because it means your repo will contain the branch information so you can easily switch between branches within the same repo on disk. No more remote network access needed! Revision filter (-r) - this allows your generated Git repo to start from a known revision number instead of starting at its birth. Now instead of taking hours to generate, you can get a repo in minutes by excluding irrelevant (ancient) revisions. So, why is this cool? Because now, instead of having 2G of source+metadata to copy when I want to do a local comparison between branches, the size on disk is merely: jbosstools' trunk as single git-svn clone w/ trunk and single branch: 1.3G devstudio's trunk as single git-svn clone w/ trunk and single branch: 0.13G So, not only is the footprint smaller, but the performance is better and I need never do a full clone (or svn checkout) again - instead, I can just copy the existing Git repo, and rebase it to a different branch. Instead of hours, this operation takes seconds (or minutes) and happens without the need for a network connection. Okay, enough blather. Show me the code! Check out the repo, including only the trunk & most recent branch # Figure out the revision number based on when a branch was created, then # from r28571, returns -r28571:HEAD rev=$(svn log --stop-on-copy \ http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \ | egrep "r[0-9]+" | tail -1 | sed -e "s#\(r[0-9]\+\).\+#-\1:HEAD#") # now, fetch repo starting from the branch's initial commit git svn clone -s $rev http://svn.jboss.org/repos/jbosstools jbosstools_GIT Now you have a repo which contains trunk & a single branch git branch -a # list local (Git) and remote (SVN) branches * master remotes/jbosstools-3.2.x remotes/trunk Switch to the branch git checkout -b local/jbosstools-3.2.x jbosstools-3.2.x # connect a new local branch to remote one Checking out files: 100% (609/609), done. Switched to a new branch 'local/jbosstools-3.2.x' git svn info # verify now working in branch URL: http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x Repository Root: http://svn.jboss.org/repos/jbosstools Switch back to trunk git checkout -b local/trunk trunk # connect a new local branch to remote trunk Switched to a new branch 'local/trunk' git svn info # verify now working in branch URL: http://svn.jboss.org/repos/jbosstools/trunk Repository Root: http://svn.jboss.org/repos/jbosstools Rewind your changes, pull updates from SVN repo, apply your changes; won't work if you have local uncommitted changes git svn rebase Fetch updates from SVN repo (ignoring local changes?) git svn fetch Create a new branch (remotely with SVN) svn copy \ http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \ http://svn.jboss.org/repos/jbosstools/branches/some-new-branch From http://divby0.blogspot.com/2011/01/howto-partially-clone-svn-repo-to-git.html

we've all seen them on various websites. crappy search utilities. they are a constant reminder that search is not something you should take lightly when building a website or application. search is not just google's game anymore. when a java library called lucene was introduced into the apache ecosystem, and then solr was built on top of that, open source developers began to wield some serious power when it came to customizing search features. in this article you'll be introduced to apache solr and a wealth of applications that have been built with it. the content is divided as follows: introduction setup solr applications summary 1. introduction apache solr is an open source search server. it is based on the full text search engine called apache lucene . so basically solr is an http wrapper around an inverted index provided by lucene. an inverted index could be seen as a list of words where each word-entry links to the documents it is contained in. that way getting all documents for the search query "dzone" is a simple 'get' operation. one advantage of solr in enterprise projects is that you don't need any java code, although java itself has to be installed. if you are unsure when to use solr and when lucene, these answers could help. if you need to build your solr index from websites, you should take a look into the open source crawler called apache nutch before creating your own solution. to be convinced that solr is actually used in a lot of enterprise projects, take a look at this amazing list of public projects powered by solr . if you encounter problems then the mailing list or stackoverflow will help you. to make the introduction complete i would like to mention my personal link list and the resources page which lists books, articles and more interesting material. 2. setup solr 2.1. installation as the very first step, you should follow the official tutorial which covers the basic aspects of any search use case: indexing - get the data of any form into solr. examples: json, xml, csv and sql-database. this step creates the inverted index - i.e. it links every term to its documents. querying - ask solr to return the most relevant documents for the users' query to follow the official tutorial you'll have to download java and the latest version of solr here . more information about installation is available at the official description . next you'll have to decide which web server you choose for solr. in the official tutorial, jetty is used, but you can also use tomcat. when you choose tomcat be sure you are setting the utf-8 encoding in the server.xml . i would also research the different versions of solr, which can be quite confusing for beginners: the current stable version is 1.4.1. use this if you need a stable search and don't need one of the latest features. the next stable version of solr will be 3.x the versions 1.5 and 2.x will be skipped in order to reach the same versioning as lucene. version 4.x is the latest development branch. solr 4.x handles advanced features like language detection via tika, spatial search , results grouping (group by field / collapsing), a new "user-facing" query parser ( edismax handler ), near real time indexing, huge fuzzy search performance improvements, sql join-a like feature and more. 2.2. indexing if you've followed the official tutorial you have pushed some xml files into the solr index. this process is called indexing or feeding. there are a lot more possibilities to get data into solr: using the data import handler (dih) is a really powerful language neutral option. it allows you to read from a sql database, from csv, xml files, rss feeds, emails, etc. without any java knowledge. dih handles full-imports and delta-imports. this is necessary when only a small amount of documents were added, updated or deleted. the http interface is used from the post tool, which you have already used in the official tutorial to index xml files. client libraries in different languages also exist. (e.g. for java (solrj) or python ). before indexing you'll have to decide which data fields should be searchable and how the fields should get indexed. for example, when you have a field with html in it, then you can strip irrelevant characters , tokenize the text into 'searchable terms', lower case the terms and finally stem the terms . in contrast, if you would have a field with text in it that should not be interpreted (e.g. urls) you shouldn't tokenize it and use the default field type string. please refer to the official documentation about field and field type definitions in the schema.xml file. when designing an index keep in mind the advice from mauricio : "the document is what you will search for. " for example, if you have tweets and you want to search for similar users, you'll need to setup a user index - created from the tweets. then every document is a user. if you want to search for tweets, then setup a tweet index; then every document is a tweet. of course, you can setup both indices with the multi index options of solr. please also note that there is a project called solr cell which lets you extract the relevant information out of several different document types with the help of tika. 2.3. querying for debugging it is very convenient to use the http interface with a browser to query solr and get back xml. use firefox and the xml will be displayed nicely: you can also use the velocity contribution , a cross-browser tool, which will be covered in more detail in the section about 'search application prototyping' . to query the index you can use the dismax handler or standard query handler . you can filter and sort the results: q=superman&fq=type:book&sort=price asc you can also do a lot more ; one other concept is boosting. in solr you can boost while indexing and while querying. to prefer the terms in the title write: q=title:superman^2 subject:superman when using the dismax request handler write: q=superman&qf=title^2 subject check out all the various query options like fuzzy search , spellcheck query input , facets , collapsing and suffix query support . 3. applications now i will list some interesting use cases for solr - in no particular order. to see how powerful and flexible this open source search server is. 3.1. drupal integration the drupal integration can be seen as generic use case to integrate solr into php projects. for the php integration you have the choice to either use the http interface for querying and retrieving xml or json. or to use the php solr client library . here is a screenshot of a typical faceted search in drupal : for more information about faceted search look into the wiki of solr . more php projects which integrates solr: open source typo3- solr module magento enterprise - solr module . the open source integration is out dated. oxid - solr module . no open source integration available. 3.2. hathi trust the hathi trust project is a nice example that proves solr's ability to search big digital libraries. to quote directly from the article : "... the index for our one million book index is over 200 gigabytes ... so we expect to end up with a two terabyte index for 10 million books" other examples for libraries: vufind - aims to replace opac internet archive national library of australia 3.3. auto suggestions mainly, there are two approaches to implement auto-suggestions (also called auto-completion) with solr: via facets or via ngramfilterfactory . to push it to the extreme you can use a lucene index entirely in ram. this approach is used in a large music shop in germany. live examples for auto suggestions: kaufda.de 3.4. spatial search applications when mentioning spatial search, people have geographical based applications in mind. with solr, this ordinary use case is attainable . some examples for this are : city search - city guides yellow pages kaufda.de spatial search can be useful in many different ways : for bioinformatics, fingerprints search, facial search, etc. (getting the fingerprint of a document is important for duplicate detection). the simplest approach is implemented in jetwick to reduce duplicate tweets, but this yields a performance of o(n) where n is the number of queried terms. this is okay for 10 or less terms, but it can get even better at o(1)! the idea is to use a special hash set to get all similar documents. this technique is called local sensitive hashing . read this nice paper about 'near similarity search and plagiarism analysis' for more information. 3.5. duckduckgo duckduckgo is made with open source and its "zero click" information is done with the help of solr using the dismax query handler: the index for that feature contains 18m documents and has a size of ~12gb. for this case had to tune solr: " i have two requirements that differ a bit from most sites with respect to solr: i generally only show one result, with sometimes a couple below if you click on them. therefore, it was really important that the first result is what people expected. false positives are really bad in 0-click, so i needed a way to not show anything if a match wasn't too relevant. i got around these by a) tweaking dismax and schema and b) adding my own relevancy filter on top that would re-order and not show anything in various situations. " all the rest is done with tuned open source products. to quote gabriel again: "the main results are a hybrid of a lot of things, including external apis, e.g. bing, wolframalpha, yahoo, my own indexes and negative indexes (spam removal), etc. there are a bunch of different types of data i'm working with. " check out the other cool features such as privacy or bang searches . 3.6. clustering support with carrot2 carrot2 is one of the "contributed plugins" of solr. with carrot2 you can support clustering : " clustering is the assignment of a set of observations into subsets (called clusters) so that observations in the same cluster are similar in some sense. " see some research papers regarding clustering here . here is one visual example when applying clustering on the search "pannous" - our company : 3.7. near real time search solr isn't real time yet, but you can tune solr to the point where it becomes near real time, which means that the time ('real time latency') that a document takes to be searchable after it gets indexed is less than 60 seconds even if you need to update frequently. to make this work, you can setup two indices. one write-only index "w" for the indexer and one read-only index "r" for your application. index r refers to the same data directory of w, which has to be defined in the solrconfig.xml of r via: /pathto/indexw/data/ to make sure your users and the r index see the indexed documents of w, you have to trigger an empty commit every 60 seconds: wget -q http://localhost:port/solr/update?stream.body=%3ccommit/%3e -o /dev/null everytime such a commit is triggered a new searcher without any cache entries is created. this can harm performance for visitors hitting the empty cache directly after this commit, but you can fill the cache with static searches with the help of the newsearcher entry in your solrconfig.xml. additionally, the autowarmcount property needs to be tuned, which fills the cache with a newsearcher from old entries. also, take a look at the article 'scaling lucene and solr' , where experts explain in detail what to do with large indices (=> 'sharding') and what to do for high query volume (=> 'replicating'). 3.8. loggly = full text search in logs feeding log files into solr and searching them at near real-time shows that solr can handle massive amounts of data and queries the data quickly. i've setup a simple project where i'm doing similar things , but loggly has done a lot more to make the same task real-time and distributed. you'll need to keep the write index as small as possible otherwise commit time will increase too great. loggly creates a new solr index every 5 minutes and includes this when searching using the distributed capabilities of solr ! they are merging the cores to keep the number of indices small, but this is not as simple as it sounds. watch this video to get some details about their work. 3.9. solandra = solr + cassandra solandra combines solr and the distributed database cassandra , which was created by facebook for its inbox search and then open sourced. at the moment solandra is not intended for production use. there are still some bugs and the distributed limitations of solr apply to solandra too. tthe developers are working very hard to make solandra better. jetwick can now run via solandra just by changing the solrconfig.xml. solandra also has the advantages of being real-time (no optimize, no commit!) and distributed without any major setup involved. the same is true for solr cloud. 3.10. category browsing via facets solr provides facets , which make it easy to show the user some useful filter options like those shown in the "drupal integration" example. like i described earlier , it is even possible to browse through a deep category tree. the main advantage here is that the categories depend on the query. this way the user can further filter the search results with this category tree provided by you. here is an example where this feature is implemented for one of the biggest second hand stores in germany. a click on 'schauspieler' shows its sub-items: other shops: game-change 3.11. jetwick - open twitter search you may have noticed that twitter is using lucene under the hood . twitter has a very extreme use case: over 1,000 tweets per second, over 12,000 queries per second, but the real-time latency is under 10 seconds! however, the relevancy at that volume is often not that good in my opinion. twitter search often contains a lot of duplicates and noise. reducing this was one reason i created jetwick in my spare time. i'm mentioning jetwick here because it makes extreme use of facets which provides all the filters to the user. facets are used for the rss-alike feature (saved searches), the various filters like language and retweet-count on the left, and to get trending terms and links on the right: to make jetwick more scalable i'll need to decide which of the following distribution options to choose: use solr cloud with zookeeper use solandra move from solr to elasticsearch which is also based on apache lucene other examples with a lot of facets: cnet reviews - product reviews. electronics reviews, computer reviews & more. shopper.com - compare prices and shop for computers, cell phones, digital cameras & more. zappos - shoes and clothing. manta.com - find companies. connect with customers. 3.12. plaxo - online address management plaxo.com , which is now owned by comcast, hosts web addresses for more than 40 million people and offers smart search through the addresses - with the help of solr. plaxo is trying to get the latest 'social' information of your contacts through blog posts, tweets, etc. plaxo also tries to reduce duplicates . 3.13. replace fast or google search several users report that they have migrated from a commercial search solution like fast or google search appliance (gsa) to solr (or lucene). the reasons for that migration are different: fast drops linux support and google can make integration problems. the main reason for me is that solr isn't a black box —you can tweak the source code, maintain old versions and fix your bugs more quickly! 3.14. search application prototyping with the help of the already integrated velocity plugin and the data import handler it is possible to create an application prototype for your search within a few hours. the next version of solr makes the use of velocity easier. the gui is available via http://localhost:port/solr/browse if you are a ruby on rails user, you can take a look into flare. to learn more about search application prototyping, check out this video introduction and take a look at these slides. 3.15. solr as a whitelist imagine you are the new google and you have a lot of different types of data to display e.g. 'news', 'video', 'music', 'maps', 'shopping' and much more. some of those types can only be retrieved from some legacy systems and you only want to show the most appropriated types based on your business logic . e.g. a query which contains 'new york' should result in the selection of results from 'maps', but 'new yorker' should prefer results from the 'shopping' type. with solr you can set up such a whitelist-index that will help to decide which type is more important for the search query. for example if you get more or more relevant results for the 'shopping' type then you should prefer results from this type. without the whitelist-index - i.e. having all data in separate indices or systems, would make it nearly impossible to compare the relevancy. the whitelist-index can be used as illustrated in the next steps. 1. query the whitelist-index, 2. decide which data types to display, 3. query the sub-systems and 4. display results from the selected types only. 3.16. future solr is also useful for scientific applications, such as a dna search systems. i believe solr can also be used for completely different alphabets so that you can query nucleotide sequences - instead of words - to get the matching genes and determine which organism the sequence occurs in, something similar to blast . another idea you could harness would be to build a very personalized search. every user can drag and drop their websites of choice and query them afterwards. for example, often i only need stackoverflow, some wikis and some mailing lists with the expected results, but normal web search engines (google, bing, etc.) give me results that are too cluttered. my final idea for a future solr-based app could be a lucene/solr implementation of desktop search. solr's facets would be especially handy to quickly filter different sources (files, folders, bookmarks, man pages, ...). it would be a great way to wade through those extra messy desktops. 4. summary the next time you think about a problem, think about solr! even if you don't know java and even if you know nothing about search: solr should be in your toolbox. solr doesn't only offer professional full text search, it could also add valuable features to your application. some of them i covered in this article, but i'm sure there are still some exciting possibilities waiting for you!

I have been using JBoss 4.2.3 for over a year right now and really like it (although the clustering / JMS stuff seems WAY overcomplicated - and needing 80 config files - not fun!). But anyway it is time to upgrade to JBoss 5 and the path has been marred by many stops and starts and has been surprisingly difficult. In any event I found the following links to be a life saver and thought I would share them http://community.jboss.org/wiki/MigrationfromJBoss4.pdf http://venugopaal.wordpress.com/2009/02/02/jboss405-to-jboss-5ga/ http://www.tikalk.com/java/migrating-your-application-jboss-4x-jboss-5x The real kickers have to be 1) Increased adherence to the Java spec that cause WARs / JARs to no longer deploy 2) Changed location of XML files 3) Changed XML filenames 4) Changed XML file contents Man they don't make it easy do they? From http://softarc.blogspot.com/2011/01/migrating-from-jboss-4-to-jboss-5.html

It's been almost 4 years since I wrote a blog post called "EasyMock - Pros, Cons, and Best Practices, and a lot has happened since. You don't hear about EasyMock much any more, and Mockito seems to have replaced it in mindshare. And for good reason: it is better. A Good Humane Interface for Stubbing Just like EasyMock, Mockito allows you to chain method calls together to produce less imperative looking code. Here's how you can make a Stub for the canonical Warehouse object: Warehouse mock = Mockito.mock(Warehouse.class); Mockito.when(mock.hasInventory(TALISKER, 50)). thenReturn(true); I know, I like a crazy formatting. Regardless, giving your System Under Test (SUT) indirect input couldn't be easier. There is no big advantage over EasyMock for stubbing behavior and passing a stub off to the SUT. Giving indirect input with mocks and then using standard JUnit asserts afterwards is simple with both tools, and both support the standard Hamcrest matchers. Class (not just Interface) Mocks Mockito allows you to mock out classes as well as interfaces. I know the EasyMock ClassExtensions allowed you to do this as well, but it is a little nicer to have it all in one package with Mockito. Supports Test Spies, not just Mocks There is a difference between spies and mocks. Stubs allow you to give indirect input to a test (the values are read but never written), Spies allow you to gather indirect output from a test (the mock is written to and verified, but does not give the test input), and Mocks are both (your object gives indirect input to your test through Stubbing and gathers indirect output through spying). The difference is illustrated between two code examples. In EasyMock, you only have mocks. You must set all input and output expectations before running the test, then verify afterwards. // arrange Warehouse mock = EasyMock.createMock(Warehouse.class); EasyMock.expect( mock.hasInventory(TALISKER, 50)). andReturn(true).once(); EasyMock.expect( mock.remove(TALISKER, 50)). andReturn(true).once(); EasyMock.replay(mock); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert EasyMock.verify(mock); That's a lot of code, and not all of it is needed. The arrange section is setting up a stub (the warehouse has inventory) and setting up a mock expectation (the remove method will be called later). The assertion in all this is actually the little verify() method at the end. The main point of this test is that remove() was called, but that information is buried in a nest of expectations. Mockito improves on this by throwing out both the record/playback mode and a generic verify() method. It is shorter and clearer this way: // arrange Warehouse mock = Mockito.mock(Warehouse.class); Mockito.when(mock.hasInventory(TALISKER, 50)). thenReturn(true); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert Mockito.verify(warehouse).remove(TALISKER, 50); The verify step with Mockito is spying on the results of the test, not recording and verifying. Less code and a clearer picture of what really is expected. Update: There is a separate Spy API you can use in Mockito as well: http://mockito.googlecode.com/svn/branches/1.8.3/javadoc/org/mockito/Mockito.html#13 Better Void Method Handling Mockito handles void methods better than EasyMock. The fluent API works fine with a void method, but in EasyMock there were some special methods you had to write. First, the Mockito code is fairly simple to read: // arrange Warehouse mock = Mockito.mock(Warehouse.class); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert Mockito.verify(warehouse).remove(TALISKER, 50); Here is the same in EasyMock. Not as good: // arrange Warehouse mock = EasyMock.createMock(Warehouse.class); mock.remove(TALISKER, 50); EasyMock.expectLastMethodCall().once(); EasyMock.replay(mock); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert EasyMock.verify(mock); Mock Object Organization Patterns Both Mockito and EasyMock suffer from difficult maintenance. What I said in my original EasyMock post holds true for Mockito: The method chaining style interface is easy to write, but I find it difficult to read. When a test other than the one I'm working on fails, it's often very difficult to determine what exactly is going on. I end up having to examine the production code and the test expectation code to diagnose the issue. Hand-rolled mock objects are much easier to diagnose when something breaks... This problem is especially nasty after refactoring expectation code to reduce duplication. For the life of me, I cannot follow expectation code that has been refactored into shared methods. Now, four years later, I have a solution that works well for me. With a little care you can make your mocks reusable, maintainable, and readable. This approach was battle tested over many months in an Enterprise Environment(tm). Create a private static method the first time you need a mock. Any important data needs to be passed in as a parameter. Using constants or "magic" fields hides important information and obfuscates tests. For example: User user = createMockUser("userID", "name"); ... assertEquals("userID", result.id()); assertEquals("name", result.name(); Everything important is visible and in the test, nothing important is hidden. You need to completely hide the replay state behind this factory method if you're still on EasyMock. The Mock framework in use is an implementation detail and try not to let it leak. Next, as your dependencies grow, be sure to always pass them in as factory method parameters. If you need a User and a Role object, then don't create one method that creates both mocks. One method instantiates one object, otherwise it is a parameter and compose your mock objects in the test method: User user = createMockUser( "userID", "name", createMockRole("role1"), createMockRole("role2") ); When each object type has a factory method, then it makes it much easier to compose the different types of objects together. Reuse. But you can only reuse the methods when they are simple and with few dependencies, otherwise they become too specific and difficult to understand. The first time you need to reuse one of these methods, then move the method to a utility class called "*Mocker", like UserMocker or RoleMocker. Follow a naming convention so that they are always easy to find. If you remembered to make the private factory methods static then moving them should be very simple. Your client code ends up looking like this, but you can use static imports to fix that: User user = UserMocker.createMockUser( "userID", "name", RoleMocker.createMockRole("role1"), RoleMocker.createMockRole("role2") ); User overloaded methods liberally. Don't create one giant method with every possible parameter in the parameter list. There are good reasons to avoid overloading in production, but this is test. Use overloading so that the test methods only display data relevant to that test and nothing more. Using Varargs can also help keep a clean test. Lastly, don't use constants. Constants hide the important information out of sight, at the top of the file where you can't see it or in a Mocker class. It's OK to use constants within the test case, but don't define constants in the Mockers, it just hides relevant information and makes the test harder to read later. Avoid Abstract Test Cases Managing mock objects within abstract test cases has been very difficult for me, especially when managing replay and record states. I've given up mixing mock objects and abstract TestCase objects. When something breaks it simply takes too long to diagnose. An alternative is to create custom assertion methods that can be reused. Beyond that, I've given up on Abstract TestCase objects anyway, on the grounds of preferring composition of inheritance. Don't Replace Asserts with Verify My original comments about EasyMock are still relevant for Mockito: The easiest methods to understand and test are methods that perform some sort of work. You run the method and then use asserts to make sure everything worked. In contrast, mock objects make it easy to test delegation, which is when some object other than the SUT is doing work. Delegation means the method's purpose is to produce a side-effect, not actually perform work. Side-effect code is sometimes needed, but often more difficult to understand and debug. In fact, some languages don't even allow it! If you're test code contains assert methods then you have a good test. If you're code doesn't contain asserts, and instead contains a long list of verify() calls, then you're relying on side effects. This is a unit-test bad smell, especially if there are several objects than need to be verified. Verifying several objects at the end of a unit test is like saying, "My test method needs to do several things: x, y, and z." The charter and responsibility of the method is no longer clear. This is a candidate for refactoring. No More All or Nothing Testing Mockito's verify() methods are much more flexible than EasyMock's. You can verify that only one or two methods on the mock were called, while EasyMock had just one coarse verify() method. With EasyMock I ended up littering the code with meaningless expectations, but not so in Mockito. This alone is reason enough to switch. Failure: Expected X received X For the most part, Mockito error messages are better than EasyMock's. However, you still sometimes see a failure that reads "Failure. Got X Expected X." Basically, this means that your toString() methods produce the same results but equals() does not. Every user who starts out gets confused by this message at some point. Be Warned. Don't Stop Handrolling Mocks Don't throw out hand-rolled mock objects. They have their place. Subclass and Override is a very useful technique for creating a testing seam, use it. Learn to Write an ArgumentMatcher Learn to write an ArgumentMatcher. There is a learning curve but it's over quickly. This post is long enough, so I won't give an example. That's it. See you again in 4 years when the next framework comes out! From http://hamletdarcy.blogspot.com/2010/09/mockito-pros-cons-and-best-practices.html

It's amazing how the JBoss Team put together an easy way to do JMS Clustering, out of the box!!. I'll start with an easy example, creating a Queue named "MyClusteredQueue". In this example I'm using JBoss AS 5.1. and two computers connected on the same network, with these IP's: - Computer A: 192.168.0.143 - Computer B: 192.168.0.210 So, here are the steps: 1) Install the JBoss on both computers. We are going to use the "all" configuration for both computers. 2) We create our Queue on both servers. Go to $JBOSS_HOME/server/all/deploy/messaging/ and edit the destinations-service.xml file. Add the MyClusteredQueue before the last server tag. It looks like this: jboss.messaging:service=ServerPeer jboss.messaging:service=PostOffice true This is how you add a Queue to the JBoss, and the people how are familiar with this, the only new thing is to add the attribute "Clustered". This step must be set on both computers. At the end of the article you can find the files. 3) Write the MDB to consume the messages, and deploy it on the two computers. (I'm using an EJB 3 - MDB style). import java.net.InetAddress; import javax.ejb.ActivationConfigProperty; import javax.ejb.MessageDriven; import javax.jms.Message; import javax.jms.MessageListener; import javax.jms.ObjectMessage; import org.apache.log4j.Logger; /** * @author felipeg * */ @MessageDriven(activationConfig = { @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"), @ActivationConfigProperty(propertyName="destination", propertyValue="queue/MyClusteredQueue") }) public class JMSClusterClientHandler implements MessageListener { Logger log = Logger.getLogger(JMSClusterClientHandler.class); @Override public void onMessage(Message message) { try{ if (message instanceof ObjectMessage) { InetAddress addr = InetAddress.getLocalHost(); log.info("########## Processing Host: " + addr.getHostName() + " ##########" ); ObjectMessage objMessage = (ObjectMessage) message; Object obj = objMessage.getObject(); log.info("Object received:" + obj.toString()); } } catch (Exception e) { e.printStackTrace(); } } } 4) Start the jboss with the following options: Computer A: $ cd $JBOSS_HOME/bin $ ./run.sh -c all -b 192.168.0.143 -Djboss.messaging.ServerPeerID=1 Computer B: $ cd $JBOSS_HOME/bin $ ./run.sh -c all -b 192.168.0.210 -Djboss.messaging.ServerPeerID=2 It is necesary to give an ID to each server and this is accomplished with this directive: -Djboss.messaging.ServerPeerID When you start the jboss on computer A, you should see the logs (server.log) telling you that there is one node ready and listening, and once you start the jboss on computer B, on the log will appear the two nodes, the two IP's ready to consume messages. 5) Now it's time to send a Message to the Queue. To accomplish this it's necessary to change the connection factory to "ClusteredConnectionFactory" (JMSDispatcher.java - See the code below). Also on the jndi.properties (if you are using the default InitialContext) file it's necessary to add the two computers ip's separated by comma to the java.naming.provider.url property. (In my case a create a Properties variable and I set all the necessary properties, JMSDispatcher.java - see the code below). java.naming.provider.url=192.168.0.143:1099,192.168.0.210:1099 The client that I wrote is a web application, that consist in one index.jsp page, which contains a form that prompts you for the name of the queue, the type of messaging (Queue or Topic), the server ip and port, how many times it will send the message and the actual message to be sent; also the web application has a Servlet (JMSClusteredClient.java - see code below) that receives the postback and helper class (JMSDispatcher.java - see code below) that sends the message to the jboss servers. You can to deploy it in any computer. In my case I deployed it on the Computer A. And you can access it through this URL: http://192.168.0.143:8080/JMSWeb/ (just modify the IP where the client war was deployed). If you notice (on the index.jsp - code below) I've already put some default values that reflects the name of the Queue, and the IP's of my two computers. Now, If you increment the number of times that the message will be sent (maybe a 10) and fill out the message box, and click "Send" you should see on the two servers some of the messages being consumed by the MDB. Here are the Files to create the client: index.jsp JMS Clustered - Test Client Server: QueueTopic Times:Message: Servlet: JMSClusteredClient.java public class JMSClusteredClient extends HttpServlet { private static final long serialVersionUID = 1L; /** * @see HttpServlet#service(HttpServletRequest request, HttpServletResponse response) */ protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { PrintWriter out = response.getWriter(); String topicqueue = request.getParameter("topicqueue"); String message = request.getParameter("message"); String server = request.getParameter("server"); String messageType = request.getParameter("messageType"); String times = request.getParameter("times"); int intTimes = Integer.parseInt(times); JMSDispatcher dispatcher = new JMSDispatcher(); dispatcher.setTopicQueueName(topicqueue); dispatcher.setServer(server); dispatcher.setMessageType(messageType); try { for(int count =1; count <= intTimes;count++){ dispatcher.sendMessage( count + " of " + times + " " + message); } out.println("Message [" + message + "] sent successfully to [" + topic + "] to the [" + server + "] server " + times + " times."); } catch (JMSException e) { e.printStackTrace(); out.println("Error:" + e.getMessage()); } catch (NamingException e) { out.println("Error:" + e.getMessage()); e.printStackTrace(); } finally{ out.close(); } } } A utility to send the messages: JMSDispatcher.java public class JMSDispatcher { /** * */ private static final long serialVersionUID = 7105145023422143880L; private static Logger log = Logger.getLogger(JMSDispatcher.class); private final String CONNECTION_FACTORY_CLUSTERED = "ClusteredConnectionFactory"; private final String CONNECTION_FACTORY = "ConnectionFactory"; private final String TOPIC = "TOPIC"; private final String QUEUE = "QUEUE"; private String topicQueueName; private String server; private String messageType; public void setTopicQueueName(String value){ this.topicQueueName = value; } public void setServer(String value){ this.server = value; } public void setMessageType(String value){ this.messageType = value; } public void sendMessage(Object objectMessage) throws JMSException, NamingException{ log.debug("##### Setting up a Queue/Topic Message: #####"); if (TOPIC.equals(messageType)){ sendTopicMessage(objectMessage); } else if (QUEUE.equals(messageType)){ sendQueueMessage(objectMessage); } log.debug("##### Publishing Message: Done #####"); } private void sendQueueMessage(Object objectMessage) throws JMSException, NamingException{ try{ InitialContext initialContext = getInitialContext(); QueueConnectionFactory qcf = (QueueConnectionFactory) initialContext.lookup(CONNECTION_FACTORY_CLUSTERED); QueueConnection queueConn = qcf.createQueueConnection(); Queue queue = (Queue) initialContext.lookup(topicQueueName); QueueSession queueSession = queueConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE); queueConn.start(); QueueSender send = queueSession.createSender(queue); ObjectMessage om = queueSession.createObjectMessage((Serializable)objectMessage); setMessageProperties(om); log.debug("##### Publishing Message to a Queue: " + queueName + "#####"); send.send(om); send.close(); queueConn.stop(); queueSession.close(); queueConn.close(); }catch(MessageFormatException ex){ log.error("##### The MESSAGE is not Serializable ####"); throw ex; }catch(MessageNotWriteableException ex){ log.error("##### The MESSAGE is not Readable ####"); throw ex; }catch(JMSException ex){ log.error("##### JMS provider fails to set the object due to some internal error. ####"); throw ex; } } private void sendTopicMessage(Object objectMessage) throws JMSException, NamingException{ try{ InitialContext initialContext = getInitialContext(); TopicConnectionFactory tcf = (TopicConnectionFactory)initialContext.lookup(CONNECTION_FACTORY_CLUSTERED); TopicConnection topicConn = tcf.createTopicConnection(); Topic topic = (Topic) initialContext.lookup(topicQueueName); TopicSession topicSession = topicConn.createTopicSession(false,TopicSession.AUTO_ACKNOWLEDGE); topicConn.start(); TopicPublisher send = topicSession.createPublisher(topic); ObjectMessage om = topicSession.createObjectMessage(); om.setObject((Serializable)objectMessage); setMessageProperties(om); log.debug("##### Publishing Message to a Topic: " + topicName + "#####"); send.publish(om); send.close(); topicConn.stop(); topicSession.close(); topicConn.close(); }catch(MessageFormatException ex){ log.error("##### The MESSAGE is not Serializable ####"); throw ex; }catch(MessageNotWriteableException ex){ log.error("##### The MESSAGE is not Readable ####"); throw ex; }catch(JMSException ex){ log.error("##### JMS provider fails to set the object due to some internal error. ####"); throw ex; } } private InitialContext getInitialContext() throws NamingException{ Properties jboss = new Properties(); jboss.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory"); jboss.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces"); jboss.put("java.naming.provider.url", server); return new InitialContext(jboss); } } And the web.xml JMSWeb index.jsp JMSClusteredClient JMSClusteredClient com.blogspot.felipeg48.jms.web.JMSClusteredClient JMSClusteredClient /JMSClusteredClient Happy Clustering!!

Today's JEE doesn't require configuration files any more. Most of the configuration, if not all, is done over properly annotated classes. As such, it's the responsibility of the underlying containers to find these annotations and act accordingly. At a first glance it appears that there is no other way for a container to implement that than to fully scan a given deployment. And we all know this can be very time consuming, especially if there are multiple container components that require this information, and have no integration hooks available to get to a container's already gathered information. From requirements collected here, I've introduced a new MC Scanning sub-project. The main goal or idea behind this lib is very simple: unify all of JBossAS component scanning into a single-pass scan. Instead of doing the resource scanning for every component, we just do it once, properly delegating the work to various container components. Another goal was to also enable usage of pre-indexed information, so that there would actually be no need for the scanning itself - e.g. one could pre-index all of jar's annotations during the build. Read the other parts in DZone's exclusive JBoss Microcontainer Series: Part 1 -- Component Models Part 2 –- Advanced Dependency Injection and IoC Part 3 -- the Virtual File System Project structure scanning-spi - Contains a simple scanner, metadata SPI, and initial helpers to help you extend / use a simple version of this lib. scanning-impl - Provides component agnostic scanning API. It also includes generic metadata implementation and its usage. plugins - This module holds custom component-scanning implementations. Current implementations are: Annotations Hibernate Hierarchy JSF Web Weld deployers - Integration with VDF; new custom deployers. indexer - This module contains utils for creating pre-indexed handles, and merging them into existing jars. It includes Ant task and Maven plugin. testsuite - Tests for all other modules. Basic building blocks The org.jboss.scanning.spi.Scanner class is the most abstract - most basic interface to interact with any scanner implementation. It only has scan() method. For any really useful operation one will have to use some concrete implementation's constructors, properties ... and then use scan() to trigger the scan operation. The main interface of interest for us is org.jboss.scanning.spi.ScanningPlugin: package org.jboss.scanning.spi; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; import org.jboss.classloading.spi.visitor.ResourceFilter; import org.jboss.classloading.spi.visitor.ResourceVisitor; /** * Scanning plugin. * Defines what to do with a resource. * * @param exact handle type * @param exact handle interface * @author Ales Justin */ public interface ScanningPlugin extends ResourceFilter, ResourceVisitor { /** * Create plugins handle/utility. * e.g. AnnotationRepository for annotations scanning * * @return new handle instance */ T createHandle(); /** * Read serialized handle. * * @param is the serialized handle's input stream. * @return de-serialized handle * @throws Exception for any error */ ScanningHandle readHandle(InputStream is) throws Exception; /** * Write / serialize handle. * * @param os the output stream to serialize handle. * @param handle the handle * @throws IOException for any IO error */ void writeHandle(OutputStream os, T handle) throws IOException; /** * Cleanup handle. * * @param handle the handle to cleanup */ void cleanupHandle(U handle); /** * Get handle interface. * * @return the handle interface */ Class getHandleInterface(); /** * Get handle's key. * Used to attach handle to map/attachments. * * @return the handle's key */ String getAttachmentKey(); /** * Get handle's file name. * Used to attach handle to jar and/or get pre-indexed. * * @return the handle's file name */ String getFileName(); /*** * Get recurse filter. * * @return the recurse filter */ ResourceFilter getRecurseFilter(); } Most of the functionality is already implemented in its abstract form (AbstractScanningPlugin) so you only need to provide the custom logic. As you can already see from the plugin's signature, the plugin introduces a handle. A handle is what will hold the scanning information for particular component; e.g. an annotation repository. We can see handle's implementation defined as parameter T, where handle's interface is parameter U. /** * Scanning handle. * * Represents a simple interface resource scanning results must implement * in order to be able to merge pre-existing results. * * @param exact handle type * @author Ales Justin */ public interface ScanningHandle { /** * Merge existing handle with sub-handle / pre-existing handle. * * @param subHandle the sub handle */ void merge(T subHandle); } The main purpose of handle introduction is to allow for pre-existing handle's merging in type safe manner. How to make usage of plugins as easy as possible in MC? As we can see Scanner (or its actual implementations) takes a set of plugins to handle artifacts. But since plugins are mostly mutable, we need some sort of factory to help use create these plugins. For VDF based usage this is how our factory looks like: import org.jboss.deployers.structure.spi.DeploymentUnit; import org.jboss.scanning.spi.ScanningHandle; import org.jboss.scanning.spi.ScanningPlugin; /** * Deployment based scanning plugin factory. * Used for incallback automatching. * * @param exact handle type * @author Ales Justin */ public interface DeploymentScanningPluginFactory { /** * Is this plugin relevant to unit. * * @param unit the unit to check against * @return true if it's relevant, false otherwise */ boolean isRelevant(DeploymentUnit unit); /** * Create scanning plugin from deployment unit. * * @param unit the deployment unit * @return new scanning plugin */ ScanningPlugin create(DeploymentUnit unit); } Also, as the javadoc says, this interface is nicely used for MC's incallback usage (incallback is a kind of dependency injection where one component can insert itself into another via a method call, as explained in one of the previous articles on JBoss Microcontainer). Usage example Let's see what we need to implement in order to get annotation scanning into the repository. public class AnnotationsScanningPluginFactory implements DeploymentScanningPluginFactory { public boolean isRelevant(DeploymentUnit unit) { // any better check? -- metadata complete is already done elsewhere // see JBossMetaDataDeploymentUnitFilter in JBossAS return true; } public ScanningPlugin create(DeploymentUnit unit) { ReflectProvider provider = DeploymentUtilsFactory.getProvider(unit); ResourceOwnerFinder finder = DeploymentUtilsFactory.getFinder(unit); return new AnnotationsScanningPlugin(provider, finder, unit.getClassLoader()); } } public class AnnotationsScanningPlugin extends AbstractClassLoadingScanningPlugin { /** The repository */ private final DefaultAnnotationRepository repository; /** The visitor */ private final ResourceVisitor visitor; public AnnotationsScanningPlugin(ClassLoader cl) { this(IntrospectionReflectProvider.INSTANCE, ClassResourceOwnerFinder.INSTANCE, cl); } public AnnotationsScanningPlugin(ReflectProvider provider, ResourceOwnerFinder finder, ClassLoader cl) { repository = new DefaultAnnotationRepository(cl); visitor = new GenericAnnotationVisitor(provider, finder, repository); } protected DefaultAnnotationRepository doCreateHandle() { return repository; } protected ClassLoader getClassLoader() { return repository.getClassLoader(); } @Override public void cleanupHandle(AnnotationIndex handle) { if (handle instanceof DefaultAnnotationRepository) DefaultAnnotationRepository.class.cast(handle).cleanup(); } public Class getHandleInterface() { return AnnotationIndex.class; } public ResourceFilter getFilter() { return visitor.getFilter(); } public void visit(ResourceContext resource) { visitor.visit(resource); } } public class GenericAnnotationVisitor extends ClassHierarchyResourceVisitor { /** The mutable repository */ private MutableAnnotationRepository repository; public GenericAnnotationVisitor(ReflectProvider provider, ResourceOwnerFinder finder, MutableAnnotationRepository repository) { super(provider, finder); if (repository == null) throw new IllegalArgumentException("Null repository"); this.repository = repository; } protected boolean isRelevant(ClassInfo classInfo) { return repository.isAlreadyChecked(classInfo.getName()) == false; } public ResourceFilter getFilter() { return ClassFilter.INSTANCE; } @Override protected void handleAnnotations(ElementType type, Signature signature, Annotation[] annotations, String className, URL ownerURL) { if (annotations != null && annotations.length > 0) { for (Annotation annotation : annotations) { repository.putAnnotation(annotation, type, className, signature, ownerURL); } } } } While the repository is just a-bit-smarter Map. Integration with VDF Using the Indexer public class Main { private static final Logger log = Logger.getLogger(Main.class.getName()); /** * Usage */ private static void usage() { System.out.println("Usage: Indexer "); } /** * Main. * The output is file named .jar.mcs. * * @param args the program arguments */ public static void main(String[] args) { try { int offset = 2; if (args.length < offset) { File input = new File(args[0]); String[] providers = args[1].split(","); URL[] urls = new URL[args.length - offset]; // add the rest of classpath for (int i = 0; i < urls.length; i++) urls[i] = new File(args[i + offset]).toURI().toURL(); ScanUtils.scan(input, Constants.applyAliases(providers), urls); } else { usage(); } } catch (Throwable t) { log.log(Level.SEVERE, t.getMessage(), t); } } } Pre-existing or pre-indexed information For each scanning plugin we look for artifact's META-INF/ entry. String fileName = plugin.getFileName(); for (URL root : roots) { InputStream is = getInputStream(root, Scanner.META_INF + fileName); if (is != null) { ScanningHandle pre = plugin.readHandle(is); handle.merge(pre); It's plugin's responsibility to know how to read pre-existing handle. By default we use plain Java serialization together with gzip. public ScanningHandle readHandle(InputStream is) throws Exception { try { GZIPInputStream gis = new GZIPInputStream(is); ObjectInputStream ois = createObjectInputStream(gis); return (ScanningHandle) ois.readObject(); } finally { is.close(); } } public void writeHandle(OutputStream os, T handle) throws IOException { GZIPOutputStream gos = new GZIPOutputStream(os); ObjectOutputStream oos = new ObjectOutputStream(gos); try { oos.writeObject(handle); oos.flush(); } finally { oos.close(); } } How to limit scanning? There already was a jboss-scanning.xml, I've just enhanced it a bit. The recurse filter is now a bit smarter, and consequently faster, than it used to be in previous version. package org.jboss.scanning.plugins.filter; import java.net.URL; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Set; import org.jboss.classloading.spi.visitor.ResourceContext; import org.jboss.classloading.spi.visitor.ResourceFilter; import org.jboss.scanning.spi.metadata.PathEntryMetaData; import org.jboss.scanning.spi.metadata.PathMetaData; import org.jboss.scanning.spi.metadata.ScanningMetaData; import org.jboss.vfs.util.PathTokenizer; /** * Simple recurse filter. * * It searches for path substring in url string, * and tries to match the tree structure as far as it goes. */ public class ScanningMetaDataRecurseFilter implements ResourceFilter { /** Path tree roots */ private Map roots; public ScanningMetaDataRecurseFilter(ScanningMetaData smd) { if (smd == null) throw new IllegalArgumentException("Null metadata"); List paths = smd.getPaths(); if (paths != null && paths.isEmpty() == false) { roots = new HashMap(); for (PathMetaData pmd : paths) { RootNode pathNode = new RootNode(); roots.put(pmd.getPathName(), pathNode); Set includes = pmd.getIncludes(); if (includes != null && includes.isEmpty() == false) { pathNode.explicitInclude = true; for (PathEntryMetaData pemd : includes) { String name = pemd.getName(); String[] tokens = name.split("\\."); Node current = pathNode; for (String token : tokens) current = current.addChild(token); if (pemd.isRecurse()) current.recurse = true; // mark last one as recurse } } } } } public boolean accepts(ResourceContext resource) { if (roots == null) return false; URL url = resource.getUrl(); String urlString = url.toExternalForm(); for (Map.Entry root : roots.entrySet()) { if (urlString.contains(root.getKey())) { RootNode rootNode = root.getValue(); if (rootNode.explicitInclude) // we have explicit includes in path, try tree path { String resourceName = resource.getResourceName(); List tokens = PathTokenizer.getTokens(resourceName); Node current = rootNode; // let's try to walk some tree path for (String token : tokens) { // if we're here, the rest is recursively matched if (current.recurse) break; current = current.getChild(token); // no fwd path if (current == null) return false; } } return true; } } return false; } private static class Node { private Map children; private boolean recurse; public Node addChild(String value) { if (children == null) children = new HashMap(); Node child = children.get(value); if (child == null) { child = new Node(); children.put(value, child); } return child; } public Node getChild(String child) { return children != null ? children.get(child) : null; } } private static class RootNode extends Node { private boolean explicitInclude; } } Javassist based JBoss Reflect In order to avoid loading the actual resource's underlying class, we use Javassist under the hood - via JBoss Refect project. DeploymentUnit unit = assertDeploy(jar); try { TIFScanningPlugin plugin = unit.getAttachment(TIFScanningPlugin.class); assertNotNull(plugin); Kernel kernel = assertBean("Kernel", Kernel.class); KernelConfigurator configurator = kernel.getConfigurator(); ClassLoader cl = unit.getClassLoader(); String name = JarMarkOnClass.class.getName(); TypeInfo ti = configurator.getTypeInfo(name, cl); TypeInfo visited = plugin.getResources().get(name); assertSame(ti, visited); // let's check if the cache is working Method findLoadedClass = ClassLoader.class.getDeclaredMethod("findLoadedClass", String.class); findLoadedClass.setAccessible(true); Object clazz = findLoadedClass.invoke(cl, name); assertNull(clazz); // should not be loaded } finally { undeploy(unit); } But the overall usage of helper utils is pluggable: /** * Find the util for deployment. * Newly created utils are grouped per module. * * @author Ales Justin */ public class DeploymentUtilsFactory { /** The default impls */ private static Map, UtilFactory> defaults = new WeakHashMap, UtilFactory>(); static { addImplementation(ReflectProvider.class, new ReflectProviderUtilFactory()); addImplementation(ResourceOwnerFinder.class, new ResourceOwnerFinderUtilFactory()); } /** * Add the util impl. * * @param iface the interface * @param factory the util factory */ public static void addImplementation(Class iface, UtilFactory factory) { defaults.put(iface, factory); } /** * Remove the util impl. * * @param iface the interface */ public static void removeImplementation(Class iface) { defaults.remove(iface); } /** * Get util. * * @param unit the deployment unit * @param utilType the util type * @return util instance */ public static T getUtil(DeploymentUnit unit, Class utilType) { if (utilType == null) throw new IllegalArgumentException("Null util type"); DeploymentUnit moduleUnit = getModuleUnit(unit); T util = moduleUnit.getAttachment(utilType); if (util == null) { UtilFactory factory = defaults.get(utilType); if (factory == null) throw new IllegalArgumentException("No util factory defined for " + utilType); Object instance = factory.create(moduleUnit); util = utilType.cast(instance); moduleUnit.addAttachment(utilType, util); } return util; } /** * Get module unit. * * @param unit the current unit * @return unit containing Module, or exception if no such unit exists */ public static DeploymentUnit getModuleUnit(DeploymentUnit unit) { if (unit == null) throw new IllegalArgumentException("Null unit"); // group util per module DeploymentUnit moduleUnit = unit; while(moduleUnit != null && moduleUnit.isAttachmentPresent(Module.class) == false) moduleUnit = moduleUnit.getParent(); if (moduleUnit == null) throw new IllegalArgumentException("No module in unit: " + unit); return moduleUnit; } /** * Wrap util lookup in lazy lookup. * * @param unit the deployment unit * @param utilType the util type * @return lazy util proxy */ public static T getLazyUtilProxy(DeploymentUnit unit, Class utilType) { // null check is in handler LazyUtilsProxyHandler handler = new LazyUtilsProxyHandler(unit, utilType); Object proxy = Proxy.newProxyInstance(unit.getClassLoader(), new Class[]{utilType}, handler); return utilType.cast(proxy); } /** * Get reflect provider. * * @param unit the depoyment unit * @return the provider */ public static ReflectProvider getProvider(DeploymentUnit unit) { return getUtil(unit, ReflectProvider.class); } /** * Get finder. * * @param unit the depoyment unit * @return the finder */ public static ResourceOwnerFinder getFinder(DeploymentUnit unit) { return getUtil(unit, ResourceOwnerFinder.class); } /** * Cleanup the util. * * @param util the util to cleanup */ public static void cleanup(Object util) { if (util instanceof CachingResourceOwnerFinder) CachingResourceOwnerFinder.class.cast(util).cleanup(); } } Meaning it's easy to swap utils behavior for particular deployment unit. e.g. different ResourceOwnerFinder or ReflectProvider Reporting issues As usual, use the forums: MC user forum MC dev forum In my previous article I've actually promised an article on our current native OSGi framework work, but since I was heavily into writing this new scanning lib I though I could share my thoughts / ideas on it while they were still hot. Specially with scanning being a constantly present topic in today's enterprise usage. One thing to note here - all of this is still at prototype stage, with no real release yet, although the main concepts have been in the making for a while, from initial support in VDF, to custom Papaki library, Scannotations, ... hence they've grown from experience. But this doesn't mean feedback is not welcome. :-) I'll be trying to fulfill my promise next time with an OSGi article, unless our Reflect gets the best of me. ;-) P.S.: As usual, again thanks to Marko for doing the editing of this article. About the Author Ales Justin was born in Ljubljana, Slovenia and graduated with a degree in mathematics from the University of Ljubljana. He fell in love with Java eight years ago and has spent most of his time developing information systems, ranging from customer service to energy management. He joined JBoss in 2006 to work full time on the Microcontainer project, currently serving as its lead. He also contributes to JBoss AS and is Seam, Weld and Spring integration specialist. He represent JBoss on 'OSGi' expert groups.

The last domain logic pattern we will treat in this series is the Service Layer one. In its simplest form, a Service Layer is a set of service classes that deal with application logic, and that are characterized from being used from different front-ends. Source code, at every level of abstraction, is the representation of data entities and their related behavior, particularly in an object-oriented paradigm. There are different types of logic which this behavior can be partitioned into: business logic is encapsulated in a Domain Model, and it is specific to the particular domain the application works in. The added value of an enterprise application ensues primarily from its business logic. application logic is in the scope of a Service Layer, and it is not strictly domain-specific, although its implementation may be. For example, translating objects into an XML or Json representation is part of application logic, even if it is executed with application-specific classes which depends on an underlying Domain Model. The task of representing data in a particular format is oblivious to the domain, as it does not belong to forums or social networks platforms only, or to an electronic or chemistry domain. presentation logic finds in an user interface its quintessence, and it can be considered as the subset of application logic which governs the end user view of the system. I would not consider a difference of format in the whole representation of an object as presentation logic, though, as it falls into the realm of reusability I would want to keep in a Service Layer. Commonly this different concerns of an application are organized in different layers, where each layer resides on the top of the previous one. In classic approaches, there is an infrastructure layer which the business logic layer depends on, and which deals for example with persistence issues. In more evolved approaches, however, keeping the Domain Model as the very core of the application is the most sound choice, moving infrastructure in a Service Layer which can plug in the Domain Model via implementing certain adapter interfaces (like a Repository or a Factory). Thus a Service Layer is particularly useful for example when there are different front-ends to a common Domain Model. These front-ends, such as user interfaces or REST Apis, delegate the application logic to a Service Layer, which encapsulates it. Pushing this logic into the Domain Model would clutter the core of the application, since is is not strictly necessary to work with it. Responsibilities of a Service Layer include, for starters, CRUD functions over objects of the respective Domain Model. Some of these responsibilities may be already included in the Domain Model (Factories for Entities and Value Objects), while other ones are usually only specified as interfaces with the implementations left to an infrastructure Service Layer (Repositories). Example of the latter components are the bread and butter of a Service Layer. Persistence-related actions such as saving objects in a database, data translation (to and from Json, XML, yaml), integration of mail and every service which do not reside in the same PHP execution environment of the Domain Model is a candidate for a class or component in a Service Layer, whose implementations can be stubbed out in acceptance testing. Note that services are also implemented in a Domain Model when they do not contain knowledge which spans outside of the domain and of basic language structures. Generally speaking, services are a point of connection between different Entities and objects, which accomodate operations that would couple the other objects together if implemented directly on them. Only when there are more concerns involved than the execution of logic on domain objects - persistence or orthogonal operations - these services shall be moved to an upper level component like a Service Layer. In some cases, the Service Layer becomes overly generic and orthogonal to the underlying Domain Model, to the point that it is recycled in different applications. Object-relational mappers are part of the infrastructure, and a common example of reusable services. Though, the composition of libraries objects (has-a relationship) is preferable to the direct usage of them, or to their subclassing. Generic frameworks and libraries have a catch-all Api, while a specific Service Layer defines only the use cases actually employed by the front-ends - for example removing the unnecessary update of crucial entities that should be immutable by modelling. Finally, a Service Layer can work without an underlying Domain Model, by interfacing to external web (and non-web) services. If there is local state involved, however, it is difficult to avoid having a basic Domain Model just to define data structures that the Services pass around. The client code, which resides in front-end, has no knowledge of the difference between methods that act over a local Domain Model or an external entity: the Service Layer effectively isolates the upper layer from changes in the lower one. The code sample builds on the sample presented in the Domain Model article, continuing with the idea of an upper level layer. The sample features two service classes, one as a stubbed implementation of an interface of the Domain Model and one that it is employed only at an higher level. setSender('[email protected]'); $mail->setRecipient('[email protected]'); $mail->setSubject('Important stuff'); $mail->setText('I wanted to talk you about...'); return array( $mail ); } } /** * Service that belongs only to the very Service Layer. * It may have an interface, but it will be kept in this layer. */ class EmailTransferService { /** * Transforming an object to XML is a common task for a Service Layer. * However, dependencies towards the Domain Model are well-accepted * but they don't mean that this layer's classes should be * included in the lower layer. * Keeping the XML transformation in one place aids different front-ends * in sharing code. */ public function toXml(Email $mail) { return "\n" . " " . $mail->getSender() . "\n" . " " . $mail->getRecipient() . "\n" . " " . $this->_encode($mail->getSubject()) . "\n" . " " . $this->_encode($mail->getText()) . "\n" . "\n"; } protected function _encode($text) { return htmlentities($text); } } // client code $repository = new DumbEmailRepository(); $transferService = new EmailTransferService(); foreach ($repository->getEmailsFor('[email protected]') as $mail) { echo $transferService->toXml($mail); }

There are at least four methods that can be used in different combinations to make the process of setting up a complete development environment a lot less painful.

Fault injection testing[1] is a very useful element of a comprehensive test strategy in that it enables you to concentrate on an area that can be difficult to test; the manner in which the application under test is able to handle exceptions. It's always possible to perform exception testing in a black box mode, where you set up external conditions that will cause the application to fail, and then observe those application failures. Setting and automating (and reproducing) these such as these can, however, be time consuming. (And a pain in the neck, too!) JBoss Byteman I recently found a bytecode injection tool that makes it possible to automate fault injection tests. JBoss Byteman[2] is an open-source project that lets you write scripts in a Java-like syntax to insert events, exceptions, etc. into application code. Byteman version 1.1.0 is available for download from: http://www.jboss.org/byteman - the download includes a programmer's guide. There's also a user forum for asking questions here: http://www.jboss.org/index.html?module=bb&op=viewforum&f=310, and a jboss.org JIRA project for submitted issues and feature requests here: https://jira.jboss.org/jira/browse/BYTEMAN A Simple Example The remainder of this post describes a simple example, on the scale of the classic "hello world" example, of using Byteman to insert an exception into a running application. Let's start by defining the exception that we will inject into our application: package sample.byteman.test; /** * Simple exception class to demonstrate fault injection with byteman */ public class ApplicationException extends Exception { private static final long serialVersionUID = 1L; private int intError; private String theMessage = "hello exception - default string"; public ApplicationException(int intErrNo, String exString) { intError = intErrNo; theMessage = exString; } public String toString() { return "**********ApplicationException[" + intError + " " + theMessage + "]**********"; } } /* class */ There's nothing complicated here, but note the string that is passed to the exception constructor at line 13. Now, let's define our application class: package sample.byteman.test; /** * Simple class to demonstrate fault injection with byteman */ public class ExceptionTest { public void doSomething(int counter) throws ApplicationException { System.out.println("called doSomething(" + counter + ")"); if (counter > 10) { throw new ApplicationException(counter, "bye!"); } System.out.println("Exiting method normally..."); } /* doSomething() */ public static void main(String[] args) { ExceptionTest theTest = new ExceptionTest(); try { for (int i = 0; i < 12; i ++) { theTest.doSomething (i); } } catch (ApplicationException e) { System.out.println("caught ApplicationException: " + e); } } } /* class*/ The application instantiates an instance of ExceptionTest at line 18, then runs the doSomething method in a loop until a counter is greater then 10. Then it raises the exception that we defined earlier. When we run the application, we see this output: java -classpath bytemanTest.jar sample.byteman.test.ExceptionTest called doSomething(0) Exiting method normally... called doSomething(1) Exiting method normally... called doSomething(2) Exiting method normally... called doSomething(3) Exiting method normally... called doSomething(4) Exiting method normally... called doSomething(5) Exiting method normally... called doSomething(6) Exiting method normally... called doSomething(7) Exiting method normally... called doSomething(8) Exiting method normally... called doSomething(9) Exiting method normally... called doSomething(10) Exiting method normally... called doSomething(11) caught ApplicationException: **********ApplicationException[11 bye!]********** OK. Nothing too exciting so far. Let's make things more interesting by scripting a Byteman rule to inject an exception before the doSomething method has a chance to print any output. Our Byteman script looks like this: # # A simple script to demonstrate fault injection with byteman # RULE Simple byteman example - throw an exception CLASS sample.byteman.test.ExceptionTest METHOD doSomething(int) AT INVOKE PrintStream.println BIND buffer = 0 IF TRUE DO throw sample.byteman.test.ApplicationException(1,"ha! byteman was here!") ENDRULE Line 4 - RULE defines the start of the RULE. The following text on this line is not executed Line 5 - Reference to the class of the application to receive the injection Line 6 - And the method in that class. Note that since if we had written this line as "METHOD doSomething", the rule would have matched any signature of the soSomething method Line 7 - Our rule will fire when the PrintStream.println method is invoked Line 8 - BIND determince values for variables which can be referenced in the rule body - in our example, the recipient of the doSomething method call that triggered the rule, is identified by the parameter reference $0 Line 9 - A rule has to include an IF clause - in our example, it's always true Line 10 - When the rule is triggered, we throw an exception - note that we supply a string to the exception constructor Now, before we try to run this run, we should check the its syntax. To do this, we build our application into a .jar (bytemanTest.jar in our case) and use bytemancheck.sh sh bytemancheck.sh -cp bytemanTest.jar byteman.txt checking rules in sample_byteman.txt TestScript: parsed rule Simple byteman example - throw an exception RULE Simple byteman example - throw an exception CLASS sample.byteman.test.ExceptionTest METHOD doSomething(int) AT INVOKE PrintStream.println BIND buffer : int = 0 IF TRUE DO throw (1"ha! byteman was here!") TestScript: checking rule Simple byteman example - throw an exception TestScript: type checked rule Simple byteman example - throw an exception TestScript: no errors Once we get a clean result, we can run the application with Byteman. To do this, we run the application and specify an extra argument to the java command. Note that Byteman requires JDK 1.6 or newer. java -javaagent:/opt/Byteman_1_1_0/build/lib/byteman.jar=script:sample_byteman.txt -classpath bytemanTest.jar sample.byteman.test.ExceptionTest And the result is: caught ApplicationException: **********ApplicationException[1 ha! byteman was here!]********** Now that the Script Works, Let's Improve it! Let's take a closer look and how we BIND to a method parameter. If we change the script to read as follows: # # A simple script to demonstrate fault injection with byteman # RULE Simple byteman example - throw an exception CLASS sample.byteman.test.ExceptionTest METHOD doSomething(int) AT INVOKE PrintStream.println BIND counter = $1 IF TRUE DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!") ENDRULE In line 8, the BIND clause now refers to the int method parameter by index using the syntax $1. This change makes the value available inside the rule body by enabling us to use the name "counter." The value of counter is then supplied as the argument to the constructor for the ApplicationException class. This new version of the rule demonstrates shows how we can use local state as derived from the trigger method to construct our exception object. But wait there's more! Let's use the "counter" value as a counter. It's useful to be able to force an exception the first time a method is called. But, it's even more useful to be able to force an exception at a selected invocation of a method. Let's add a test for that counter value to the script: # # A simple script to demonstrate fault injection with byteman # RULE Simple byteman example 2 - throw an exception at 3rd call CLASS sample.byteman.test.ExceptionTest METHOD doSomething(int) AT INVOKE PrintStream.println BIND counter = $1 IF counter == 3 DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!") ENDRULE In line 9, we've changed the IF clause to make use of the counter value. When we run the test with this script, the first 2 calls to doSomething succeed, but the third one fails. One Last Thing - Changing the Script for a Running Process So far, so good. We've been able to inject a fault/exception into our running application, and even specify which iteration of a loop in which it happens. Suppose, however, we want to change a value in a byteman script, while the application is running? No problem! Here's how. First, we need to alter our application so that it can run for a long enough time for us to alter the byteman script. Here's a modified version of the doSomething method that waits for user input: public void doSomething(int counter) throws ApplicationException { BufferedReader lineOfText = new BufferedReader(new InputStreamReader(System.in)); try { System.out.println("Press "); String textLine = lineOfText.readLine(); } catch (IOException e) { e.printStackTrace(); } System.out.println("called doSomething(" + counter + ")"); if (counter > 10) { throw new ApplicationException(counter, "bye!"); } System.out.println("Exiting method normally..."); } If we run this version of the application, we'll see output like this: Press called doSomething(0) Exiting method normally... Press called doSomething(1) Exiting method normally... Press called doSomething(2) Exiting method normally... caught ApplicationException: **********ApplicationException[3 ha! byteman was here!]********** Let's run the application again, but this time, don't press . While the application is waiting for input, create a copy of the byteman script. In this copy, change the IF clause to have a loop counter set to a different value, say '5.' Then, open up a second command shell window and enter this command: Byteman_1_1_0/bin/submit.sh sample_byteman_changed.txt Then, return to the first command shell window and start pressing return, and you'll see this output: Press redefining rule Simple byteman example - throw an exception called doSomething(0) Exiting method normally... Press called doSomething(1) Exiting method normally... Press called doSomething(2) Exiting method normally... Press called doSomething(3) Exiting method normally... Press called doSomething(4) Exiting method normally... caught ApplicationException: **********ApplicationException[5 ha! byteman was here!]********** So, we were able to alter the value in the original byteman script, without stopping the application under test! Pitfalls Along the Way Some of the newbee mistakes that I made along the way were: Each RULE needs an IF clause - even if you want the rule to always fire The methods referenced in a RULE cannot be static - if they are static, then there is no $0 (aka this) to reference Yes, I had several errors and some typos the first few times I tried this. A syntax checker is always my best friend. ;-) Closing Thoughts With this simple example, we're able to inject injections into a running application in an easily automated/scripted manner. But, We've only scratched the surface with Byteman. In subsequent posts, I'm hoping to explore using Byteman to cause more widespread havoc in software testing. References [1] http://en.wikipedia.org/wiki/Fault_injection [2] http://www.jboss.org/byteman (Special thanks to Andrew Dinn for his help! ;-)

We're finally back with our next article in the Microcontainer series. In the first two articles we demonstrated how Microcontainer supports , and showed its powerful . In this article, we'll explain Classloading and Deployers, but first we must familiarize ourselves with VFS. VFS stands, as expected, for Virtual File System. What does VFS solve for us, or why is it useful? Here, at JBoss, we saw that a lot of similar resource handling code was scattered/duplicated all over the place. In most cases it was code that was trying to determine what type of resource a particular resource was, e.g. is it a file, a directory, or a jar loading resources through URLs. Processing of nested archives was also reimplemented again, and again in different libraries. Read the other parts in DZone's exclusive JBoss Microcontainer Series: Part 4 -- ClassLoading Layer Example: public static URL[] search(ClassLoader cl, String prefix, String suffix) throws IOException { Enumeration[] e = new Enumeration[]{ cl.getResources(prefix), cl.getResources(prefix + "MANIFEST.MF") }; Set all = new LinkedHashSet(); URL url; URLConnection conn; JarFile jarFile; for (int i = 0, s = e.length; i < s; ++i) { while (e[i].hasMoreElements()) { url = (URL)e[i].nextElement(); conn = url.openConnection(); conn.setUseCaches(false); conn.setDefaultUseCaches(false); if (conn instanceof JarURLConnection) { jarFile = ((JarURLConnection)conn).getJarFile(); } else { jarFile = getAlternativeJarFile(url); } if (jarFile != null) { searchJar(cl, all, jarFile, prefix, suffix); } else { boolean searchDone = searchDir(all, new File(URLDecoder.decode(url.getFile(), "UTF-8")), suffix); if (searchDone == false) { searchFromURL(all, prefix, suffix, url); } } } } return (URL[])all.toArray(new URL[all.size()]); } private static boolean searchDir(Set result, File file, String suffix) throws IOException { if (file.exists() && file.isDirectory()) { File[] fc = file.listFiles(); String path; for (int i = 0; i < fc.length; i++) { path = fc[i].getAbsolutePath(); if (fc[i].isDirectory()) { searchDir(result, fc[i], suffix); } else if (path.endsWith(suffix)) { result.add(fc[i].toURL()); } } return true; } return false; } There were also many problems with file locking on Windows systems, which forced us to copy all hot-deployable archives to another location to prevent locking those in deploy folders (which would prevent their deletion and filesystem based undeploy). File locking was a major problem that could only be addressed by centralizing all the resource loading code in one place. Recognizing a need to deal with all of these issues in one place, wrapping it all into a simple and useful API, we created the VFS project. VFS public API Basic usage in VFS can be split in two pieces: simple resource navigation visitor pattern API As mentioned, in plain JDK resource handling navigation over resources is far from trivial. You must always check what kind of resource you're currently handling, and this is very cumbersome. With VFS we wanted to limit this to a single resource type - VirtualFile. public class VirtualFile implements Serializable { /** * Get certificates. * * @return the certificates associated with this virtual file */ Certificate[] getCertificates() /** * Get the simple VF name (X.java) * * @return the simple file name * @throws IllegalStateException if the file is closed */ String getName() /** * Get the VFS relative path name (org/jboss/X.java) * * @return the VFS relative path name * @throws IllegalStateException if the file is closed */ String getPathName() /** * Get the VF URL (file://root/org/jboss/X.java) * * @return the full URL to the VF in the VFS. * @throws MalformedURLException if a url cannot be parsed * @throws URISyntaxException if a uri cannot be parsed * @throws IllegalStateException if the file is closed */ URL toURL() throws MalformedURLException, URISyntaxException /** * Get the VF URI (file://root/org/jboss/X.java) * * @return the full URI to the VF in the VFS. * @throws URISyntaxException if a uri cannot be parsed * @throws IllegalStateException if the file is closed * @throws MalformedURLException for a bad url */ URI toURI() throws MalformedURLException, URISyntaxException /** * When the file was last modified * * @return the last modified time * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ long getLastModified() throws IOException /** * Returns true if the file has been modified since this method was last called * Last modified time is initialized at handler instantiation. * * @return true if modifed, false otherwise * @throws IOException for any error */ boolean hasBeenModified() throws IOException /** * Get the size * * @return the size * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ long getSize() throws IOException /** * Tests whether the underlying implementation file still exists. * @return true if the file exists, false otherwise. * @throws IOException - thrown on failure to detect existence. */ boolean exists() throws IOException /** * Whether it is a simple leaf of the VFS, * i.e. whether it can contain other files * * @return true if a simple file. * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ boolean isLeaf() throws IOException /** * Is the file archive. * * @return true if archive, false otherwise * @throws IOException for any error */ boolean isArchive() throws IOException /** * Whether it is hidden * * @return true when hidden * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ boolean isHidden() throws IOException /** * Access the file contents. * * @return an InputStream for the file contents. * @throws IOException for any error accessing the file system * @throws IllegalStateException if the file is closed */ InputStream openStream() throws IOException /** * Do file cleanup. * * e.g. delete temp files */ void cleanup() /** * Close the file resources (stream, etc.) */ void close() /** * Delete this virtual file * * @return true if file was deleted * @throws IOException if an error occurs */ boolean delete() throws IOException /** * Delete this virtual file * * @param gracePeriod max time to wait for any locks (in milliseconds) * @return true if file was deleted * @throws IOException if an error occurs */ boolean delete(int gracePeriod) throws IOException /** * Get the VFS instance for this virtual file * * @return the VFS * @throws IllegalStateException if the file is closed */ VFS getVFS() /** * Get the parent * * @return the parent or null if there is no parent * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ VirtualFile getParent() throws IOException /** * Get a child * * @param path the path * @return the child or null if not found * @throws IOException for any problem accessing the VFS * @throws IllegalArgumentException if the path is null * @throws IllegalStateException if the file is closed or it is a leaf node */ VirtualFile getChild(String path) throws IOException /** * Get the children * * @return the children * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ List getChildren() throws IOException /** * Get the children * * @param filter to filter the children * @return the children * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed or it is a leaf node */ List getChildren(VirtualFileFilter filter) throws IOException /** * Get all the children recursively * * This always uses {@link VisitorAttributes#RECURSE} * * @return the children * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed */ List getChildrenRecursively() throws IOException /** * Get all the children recursively * * This always uses {@link VisitorAttributes#RECURSE} * * @param filter to filter the children * @return the children * @throws IOException for any problem accessing the virtual file system * @throws IllegalStateException if the file is closed or it is a leaf node */ List getChildrenRecursively(VirtualFileFilter filter) throws IOException /** * Visit the virtual file system * * @param visitor the visitor * @throws IOException for any problem accessing the virtual file system * @throws IllegalArgumentException if the visitor is null * @throws IllegalStateException if the file is closed */ void visit(VirtualFileVisitor visitor) throws IOException } As you can see you have all of the usual read-only File System operations, plus a few options to cleanup or delete the resource. Cleanup or deletion handling is needed when we're dealing with some internal temporary files; e.g. from nested jars handling. To switch from JDK's File or URL resource handling to new VirtualFile we need a root. It is the VFS class that knows how to create one with the help of URL or URI parameter. public class VFS { /** * Get the virtual file system for a root uri * * @param rootURI the root URI * @return the virtual file system * @throws IOException if there is a problem accessing the VFS * @throws IllegalArgumentException if the rootURL is null */ static VFS getVFS(URI rootURI) throws IOException /** * Create new root * * @param rootURI the root url * @return the virtual file * @throws IOException if there is a problem accessing the VFS * @throws IllegalArgumentException if the rootURL */ static VirtualFile createNewRoot(URI rootURI) throws IOException /** * Get the root virtual file * * @param rootURI the root uri * @return the virtual file * @throws IOException if there is a problem accessing the VFS * @throws IllegalArgumentException if the rootURL is null */ static VirtualFile getRoot(URI rootURI) throws IOException /** * Get the virtual file system for a root url * * @param rootURL the root url * @return the virtual file system * @throws IOException if there is a problem accessing the VFS * @throws IllegalArgumentException if the rootURL is null */ static VFS getVFS(URL rootURL) throws IOException /** * Create new root * * @param rootURL the root url * @return the virtual file * @throws IOException if there is a problem accessing the VFS * @throws IllegalArgumentException if the rootURL */ static VirtualFile createNewRoot(URL rootURL) throws IOException /** * Get the root virtual file * * @param rootURL the root url * @return the virtual file * @throws IOException if there is a problem accessing the VFS * @throws IllegalArgumentException if the rootURL */ static VirtualFile getRoot(URL rootURL) throws IOException /** * Get the root file of this VFS * * @return the root * @throws IOException for any problem accessing the VFS */ VirtualFile getRoot() throws IOException } You can see three different methods that look a lot alike - getVFS, createNewRoot and getRoot. Method getVFS returns a VFS instance, and what's important, it doesn't yet create a VirtualFile instance. Why is this important? Because there are methods which help us configure a VFS instance (see VFS class API javadocs), before telling it to create a VirtualFile root. The other two methods, on the other hand, use default settings for root creation. The difference between createNewRoot and getRoot is in caching details, which we'll delve in later on. URL rootURL = ...; // get root url VFS vfs = VFS.getVFS(rootURL); // configure vfs instance VirtualFile root1 = vfs.getRoot(); // or you can get root directly VirtualFile root2 = VFS.crateNewRoot(rootURL); VirtualFile root3 = VFS.getRoot(rootURL); The other useful thing about VFS API is its implementation of a proper visitor pattern. This way it's very simple to recursively gather different resources, something quite impossible to do with plain JDK resource loading. public interface VirtualFileVisitor { /** * Get the search attribues for this visitor * * @return the attributes */ VisitorAttributes getAttributes(); /** * Visit a virtual file * * @param virtualFile the virtual file being visited */ void visit(VirtualFile virtualFile); } VirtualFile root = ...; // get root VirtualFileVisitor visitor = new SuffixVisitor(".class"); // get all classes root.visit(visitor); VFS Architecture While public API is quite intuitive, real implementation details are a bit more complex. We'll try to explain the concepts in a quick pass. Each time you create a VFS instance, its matching VFSContext instance is created. This creation is done via VFSContextFactory. Different protocols map to different VFSContextFactory instances - e.g. file/vfsfile map to FileSystemContextFactory, zip/vfszip map to ZipEntryContextFactory. Also, each time a VirtualFile instance is created, its matching VirtualFileHandler is created. It's this VirtualFileHandler instance that knows how to handle different resource types properly - VirtualFile API just delegates invocations to its VirtualFileHandler reference. As one could expect, VFSContext instance is the one that knows how to create VirtualFileHandler instances accordingly to a resource type - e.g. ZipEntryContextFactory creates ZipEntryContext, which then creates ZipEntryHandler. Existing implementations Apart from files, directories (FileHandler) and zip archives (ZipEntryHandler) we also support other more exotic usages. The first one is Assembled, which is similar to what Eclipse calls Linked Resources. Its idea is to take existing resources from different trees, and "mock" them into single resource tree. AssembledDirectory sar = AssembledContextFactory.getInstance().create("assembled.sar"); URL url = getResource("/vfs/test/jar1.jar"); VirtualFile jar1 = VFS.getRoot(url); sar.addChild(jar1); url = getResource("/tmp/app/ext.jar"); VirtualFile ext1 = VFS.getRoot(url); sar.addChild(ext); AssembledDirectory metainf = sar.mkdir("META-INF"); url = getResource("/config/jboss-service.xml"); VirtualFile serviceVF = VFS.getRoot(url); metainf.addChild(serviceVF); AssembledDirectory app = sar.mkdir("app.jar"); url = getResource("/app/someapp/classes"); VirtualFile appVF = VFS.getRoot(url); app.addPath(appVF, new SuffixFilter(".class")); Another implementation is in-memory files. In our case this came out of a need to easily handle AOP generated bytes. Instead of mucking around with temporary files, we simply drop bytes into in-memory VirtualFileHandlers. URL url = new URL("vfsmemory://aopdomain/org/acme/test/Test.class"); byte[] bytes = ...; // some AOP generated class bytes MemoryFileFactory.putFile(url, bytes); VirtualFile classFile = VFS.getVirtualFile(new URL("vfsmemory://aopdomain"), "org/acme/test/Test.class"); InputStream bis = classFile.openStream(); // e.g. load class from input stream Extension hooks It's quite easy to extend VFS with a new protocol, similar to what we've done with Assembled and Memory. All you need is a combination of VFSContexFactory, VFSContext, VirtualFileHandler, FileHandlerPlugin and URLStreamHandler implementations. The first one is trivial, while the others depend on the complexity of your task - e.g. you could implement rar, tar, gzip or even remote access. In the end you simply register this new VFSContextFactory with VFSContextFactoryLocator. See this article's demo for a simple gzip example Features One of the first major problems we stumbled upon was proper usage of nested resources, more exactly nested jar files. e.g. normal ear deployments: gema.ear/ui.war/WEB-INF/lib/struts.jar In order to read contents of struts.jar we have two options: handle resources in memory create top level temporary copies of nested jars, recursively The first option is easier to implement, but it's very memory-consuming--just imagine huge apps in memory. The other approach leaves a bunch of temporary files, which should be invisible to plain user. Hence expecting them to disappear once the deployment is undeployed. Now imagine the following scenario: A user gets a hold of VFS's URL instance, which points to some nested resource. The way plain VFS would handle this is to re-create the whole path from scratch, meaning it would unpack nested resources over and over again. This would (and it did) lead to a huge pile of temporary files. How to avoid this? The way we approached this is by using VFSRegistry, VFSCache and TempInfo. When you ask for VirtualFile over VFS (getRoot, not createNewRoot), VFS asks VFSRegistry implementation to provide the file. Existing DefaultVFSRegistry first checks if matching root VFSContext for provided URI exists. If it does, it first tries to navigate to existing TempInfo (link to temporary files), falling back to regular navigation if no such temporary file exists. This way we completely re-use any already unpacked temporary files, saving time and disk space. If no matching VFSContext is found in cache, we create a new VFSCache entry, and continue with default navigation. It's then up to VFSCache implementation used, how it handles cached VFSContext entries. VFSCache is configurable via VFSCacheFactory - by default we don't cache anything, but there are a few useful existing VFSCache implementations, ranging from LRU to timed cache. API Use case There is a class called VFSUtils which is part of a public API, and it is sort of a dumping ground of useful functionality. It contains a bunch of helpful methods and configuration settings (system property keys, actually). Check the API javadocs for more details. Existing issues / workarounds Another issue that came up - expectedly - was inability of some frameworks to properly work on top of VFS. The problem lied in custom VFS urls like: vfsfile, vfszip, vfsmemory. In most cases you could still work around it with plain URL or URLConnection usage, but a lot of frameworks do a strict match on file or jar protocol, which of course fails. We were able to patch some frameworks (e.g. Facelets) and provide extensions to others (e.g. Spring). If you are a library developer, and your library has a simple pluggable resource loading mechanism, then we suggest you simply extend it with VFS based implementation. If there are no hooks, try to limit your assumptions to more general usage based on URL or URLConnection. Conclusion While VFS is very nice to use, it comes at a price. It adds additional layer on top of JDK's resource handling, meaning extra invocations are always present when you're dealing with resources. We also keep some of the jar handling info in memory to make it easy to get hold of a specific resource, but at the expense of some extra memory consumption. Overall VFS proved to be a very useful library as it hides away many use cases that are painful with plain JDK, and provides a comprehensive API for working with resources - i.e. visitor pattern implementation. We're constantly following user feedback to VFS issues they encounter, making each version a bit better. Now, that we got to know VFS, it's time we move on to MC's new Classloading layer! About the Author Ales Justin was born in Ljubljana, Slovenia and graduated with a degree in mathematics from the University of Ljubljana. He fell in love with Java seven years ago and has spent most of his time developing information systems, ranging from customer service to energy management. He joined JBoss in 2006 to work full time on the Microcontainer project, currently serving as its lead. He also contributes to JBoss AS and is Seam and Spring integration specialist. He represent JBoss on 'JSR-291 Dynamic Component Support for Java SE' and 'OSGi' expert groups.

Get a perspective on the aspects of JVM internals, controls, and switches that can be used to optimize your Java application.

Optimistic locking lets concurrent transactions process simultaneously, but detects and prevent collisions, this works best for applications where most concurrent transactions do not conflict. JPA Optimistic locking allows anyone to read and update an entity, however a version check is made upon commit and an exception is thrown if the version was updated in the database since the entity was read. In JPA for Optimistic locking you annotate an attribute with @Version as shown below: public class Employee { @ID int id; @Version int version; The Version attribute will be incremented with a successful commit. The Version attribute can be an int, short, long, or timestamp. This results in SQL like the following: “UPDATE Employee SET ..., version = version + 1 WHERE id = ? AND version = readVersion” The advantages of optimistic locking are that no database locks are held which can give better scalability. The disadvantages are that the user or application must refresh and retry failed updates. Optimistic Locking Example In the optimistic locking example below, 2 concurrent transactions are updating employee e1. The transaction on the left commits first causing the e1 version attribute to be incremented with the update. The transaction on the right throws an OptimisticLockException because the e1 version attribute is higher than when e1 was read, causing the transaction to roll back. Additional Locking with JPA Entity Locking APIs With JPA it is possible to lock an entity, this allows you to control when, where and which kind of locking to use. JPA 1.0 only supported Optimistic read or Optimistic write locking. JPA 2.0 supports Optimistic and Pessimistic locking, this is layered on top of @Version checking described above. JPA 2.0 LockMode values : OPTIMISTIC (JPA 1.0 READ): perform a version check on locked Entity before commit, throw an OptimisticLockException if Entity version mismatch. OPTIMISTIC_FORCE_INCREMENT (JPA 1.0 WRITE) perform a version check on locked Entity before commit, throw an OptimisticLockException if Entity version mismatch, force an increment to the version at the end of the transaction, even if the entity is not modified. PESSIMISTIC: lock the database row when reading PESSIMISTIC_FORCE_INCREMENT lock the database row when reading, force an increment to the version at the end of the transaction, even if the entity is not modified. There are multiple APIs to specify locking an Entity: EntityManager methods: lock, find, refresh Query methods: setLockMode NamedQuery annotation: lockMode element OPTIMISTIC (READ) LockMode Example In the optimistic locking example below, transaction1 on the left updates the department name for dep , which causes dep's version attribute to be incremented. Transaction2 on the right gives an employee a raise if he's in the "Eng" department. Version checking on the employee attribute would not throw an exception in this example since it was the dep Version attribute that was updated in transaction1. In this example the employee change should not commit if the department was changed after reading, so an OPTIMISTIC lock is used : em.lock(dep, OPTIMISTIC). This will cause a version check on the dep Entity before committing transaction2 which will throw an OptimisticLockException because the dep version attribute is higher than when dep was read, causing the transaction to roll back. OPTIMISTIC_FORCE_INCREMENT (write) LockMode Example In the OPTIMISTIC_FORCE_INCREMENT locking example below, transaction2 on the right wants to be sure that the dep name does not change during the transaction, so transaction2 locks the dep Entity em.lock(dep, OPTIMISTIC_FORCE_INCREMENT) and then calls em.flush() which causes dep's version attribute to be incremented in the database. This will cause any parallel updates to dep to throw an OptimisticLockException and roll back. In transaction1 on the left at commit time when the dep version attribute is checked and found to be stale, an OptimisticLockException is thrown Pessimistic Concurrency Pessimistic concurrency locks the database row when data is read, this is the equivalent of a (SELECT . . . FOR UPDATE [NOWAIT]) . Pessimistic locking ensures that transactions do not update the same entity at the same time, which can simplify application code, but it limits concurrent access to the data which can cause bad scalability and may cause deadlocks. Pessimistic locking is better for applications with a higher risk of contention among concurrent transactions. The examples below show: reading an entity and then locking it later reading an entity with a lock reading an entity, then later refreshing it with a lock The Trade-offs are the longer you hold the lock the greater the risks of bad scalability and deadlocks. The later you lock the greater the risk of stale data, which can then cause an optimistic lock exception, if the entity was updated after reading but before locking. The right locking approach depends on your application: what is the risk of risk of contention among concurrent transactions? What are the requirements for scalability? What are the requirements for user re-trying on failure? For More Information: Preventing Non-Repeatable Reads in JPA Using EclipseLink Java Persistence API 2.0: What's New ? What's New and Exciting in JPA 2.0 Beginning Java™ EE 6 Platform with GlassFish™ 3 Pro EJB 3: Java Persistence API (JPA 1.0)

The industry is recognizing the fact that performance testing & engineering should be part of the project execution road map starting from the requirements gathering phase. At many times during project executions, performance engineering related activities are executed based on customer need or slow response time of application after development phase gets completed. Glassbox can be leveraged (by developers/testers/business users) during and after the development cycle to monitor the response times of requests with-out being aware of underlying application structure and code details. Analysis generated by Glassbox gives direct pointers on where is the bottleneck which causes slow response time for that particular request/page/URL. About Glassbox Glassbox is an open source web application which aid in performance monitoring and troubleshooting of multiple web applications deployed in container. Troubleshooting It contains the built-in knowledge repository of common problems which are used to pinpoint the issues and suggestions on causes as Java code executes. Performance Monitoring It monitors the requests as Java code executes and provides details about response times. Glassbox web client (AJAX GUI) provides nice summary dashboard view which contains various attributes like (server-name, application name, operation/request-URL, average time, no. of executions, status (slow / OK) and analysis details). By default, an operation that takes more than 1 sec execution time is marked as SLOW status. Such SLA can be modified using Glassbox properties file. Analysis part describes the problem precisely and very clearly in plain English words, rather than displaying large code/exception trace. This definitely increases developer productivity by reducing developer’s time spent in log files and using IDE debuggers. Internals The two main components of Glassbox are Monitor and Agent. Monitor uses Aspect-Oriented Programming (AOP) to monitor the JVM activity. Agent diagnoses and presents the monitoring results and uses knowledge repository to cross reference the problem with suggestions/solutions. Glassbox agent supports viewing of the analysis results using JMX (eg. Java 5 JConsole) Consoles. Glassbox extensively uses the AOP approach internally to monitor the Java code. This gives the benefit of not making any changes to source code or build-process and hence can work with any legacy web application/jar file as well. Technologies Glassbox should work on any application server that supports Servlet 2.3 or later. The servers where Glassbox is tested and installation process is automated are Apache Tomcat, weblogic, websphere, Resin, Oracle OC4J, websphere, Resin, Jetty & GlassFish. Overhead Having Glassbox application running on same container would generate a performance overhead. Typically this would affect the response time and memory overhead. Hence it is recommended to start the Glassbox application only when it’s required for performance monitoring. Licensing Glassbox is an open source project, it is free to download and run. Glassbox uses the GNU Lesser General Public License to distribute software and documentation. Demo Application Development & Deployment to Tomcat To test the capabilities of Glassbox, a sample application is developed which has a TestServlet class. This servlet calls DelayGenerator class’s generateDelay() method. This method calls Thread class’s sleep() method which suspends the execution of servlet. A counter is being initialized in DelayGenerator class which determines the time interval till which servlet is needed to be suspended. TestServlet.java /** * File: TestServlet.java * @author Viral Thakkar */ package com.infosys.star.glassbox; import java.io.IOException; import java.io.PrintWriter; import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; public class TestServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { DelayGenerator delayObj = new DelayGenerator(); int delay = delayObj.generateDelay(); response.setContentType("text/html"); PrintWriter out = response.getWriter(); out.println(""); out.println(" Hello World from Test Servlet : "+delay+" milliseconds "); out.println(""); out.flush(); } } DelayGenerator.java /** * File: DelayGenerator.java * @author Viral Thakkar */ package com.infosys.star.glassbox; public class DelayGenerator { private static int counter = 1; public int generateDelay() { try { Thread.sleep(counter * 100); counter++; } catch (InterruptedException e) { e.printStackTrace(); } return counter*100; } } Glassbox Installation & Integration to Apache Tomcat 6.0 Glassbox installation is very straightforward for non-clustered environment for the server where it’s automated. Simply drop the glassbox.war file at the appropriate folder inside server folder or perform the server specific steps/configuration to deploy the war file. Browse to server url with context root as glassbox – http://<>:/glassbox. Follow the instructions available on this page. According to specific server, this page would suggest the configuration changes for a server. Please refer to Glassbox User Guide document for details on how to install Glassbox for clustered application server environment. For Apache Tomcat 6.0- Add following command line arguments to Tomcat’s Java options: -Dglassbox.install.dir=C:\Tomcat6.0\lib\glassbox -Djava.rmi.server.useCodebaseOnly=true -javaagent:C:\Tomcat6.0\lib\aspectjweaver.jar Monitoring & Technical Analysis Glassbox web client (URL- http://<>:<>/glassbox ) shows the summary and detailed view of all the requests/operations that container/JVM has executed. Summary Section View Different attributes (columns) which gets displayed in this table are as below - Attribute / Column Name Comments Status This indicates whether operation/request is performing OK, SLOW or FAILING Analysis For SLOW/FAILING status, this value provides the small summary of the cause of the problem. Operation This is name of the operation/request of an application Server Name of the server where monitoring is being done. In a clustered environment, this allows to distinguish operations on different servers. Executions This value indicates how many times this operation has run since the application server was started or Glassbox’s statistics were last reset. Click the request in above summary table to view its detailed analysis in below detailed section. Detailed Section View The details area provides information relating to operations selected in the summary table. Different sub-sections which gets displayed in this view are as below - Sub-section Name Comments Executive Summary High level summary view of the selected operation gets displayed in a table format. This is neat view to senior stake holders who are not interested in technical details. Technical Summary This section contains more technical details in paragraph and table representation formats to provide insight into root cause of the problem if any, like which operation, query is slow and statistics of same. Details like stack trace, thread lock name are provided to find and fix the problem. “Common solutions” sub section shows pointers to resolve the identified problem/s. “Glassbox has ruled out other potential problems” sub section saves time to know what problems have already been ruled out. Executive Summary View Technical Summary -> Technical Details Views Above two snapshots are parts of the Technical Details section and provide minute details at code level with line number so as to pinpoint where the problem is. Here cause is identified at Class com.infosys.star.glassbox.DelayGenerator inside Method generateDelay at line number 12 where Thread.sleep is invoked. Perform Load Testing Using JMeter and Monitor Using Glassbox Apache JMeter is used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. It can be used to make a graphical analysis of performance or to test server/script/object behavior under heavy concurrent load. Using JMeter, create a test plan that simulates 10 users requesting for 1 page 5 times. i.e. 10 x 1 x 5 = 50 HTTP requests. First step is to add a Thread Group element. The Thread Group tells JMeter the number of users to simulate, how often the users should send requests, and the how many requests they should send. Next step is to add HTTP Request element to added Thread Group. In parallel, have the Glassbox up and running to monitor response time statistics of the load generated by JMeter application. Below is the Executive summary view of above test in Glassbox web UI interface. Section “Monitoring & Technical Analysis” contains the details to understand the Glassbox generated analysis. Conclusion Glassbox is not the replacement for performance testing tool like load runner. Glassbox aids in the project to various stakeholders in finding, conveying and fixing the performance problems at all phases starting build (development) to post deployment. Glassbox application to be started/installed only during monitoring time so as to avoid the performance overhead for other applications due to CPU & memory footprint occupied by Glassbox application on the container. During load testing of the application, Glassbox turns out to be good option to figure out the root causes inside an application code. References Glassbox web site - http://www.glassbox.com/glassbox/Home.html Glassbox User Guide - http://nchc.dl.sourceforge.net/sourceforge/glassbox/Glassboxv2.0UserGuide.pdf Apache JMeter - http://jakarta.apache.org/jmeter/ Download & Support Glassbox Download Link - http://www.glassbox.com/glassbox/Downloads.html Glassbox forum Link - http://sourceforge.net/forum/forum.php?forum_id=575670 About Author Viral Thakkar is a Technical Architect with the Banking and Capital Markets vertical at Infosys. He has 9.5 years of technology consulting experience mainly on Java/JEE technologies and frameworks with large banks and financial institutions across the globe. He has been part of many small and large-scale initiatives related to application development, architecture creation and strategy definition. From http://viralpatel.net/blogs

This article is going to show you how to build a RichFaces application with Spring.