People tell me A and B, They tell me how I have to see, Things that I have seen already clear, So they push me then from side to side (I Want Out - Helloween) Developing an application that uses RESTful web API may imply developing server and client side. Writing integration tests for the server side can be as easy as using Arquillian to start up server and REST-assured to test that the services works as expected. The problem is how to test the client side. In this post, we are going to see how to test the client side apart from using mocks. As a brief description, to test the client side, what we need is a local server which can return recorded JSON responses. The rest-client-driver is a library which simulates a RESTful service. You can set expectations on the HTTP requests you want to receive during a test. So it is exactly what we need for our java client side. Note that this project is really helpful to write tests when we are developing RESTful web clients for connecting to services developed by third parties like Flickr Rest API, Jira Rest API, Github ... First thing to do is adding rest-client-driver dependency: com.github.rest-driver rest-client-driver 1.1.27 test Next step we are going to create a very simple Jersey application which simply invokes a get method to required URI. public class GithubClient { private static final int HTTP_STATUS_CODE_OK = 200; private String githubBaseUri; public GithubClient(String githubBaseUri) { this.githubBaseUri = githubBaseUri; } public String invokeGetMethod(String resourceName) { Client client = Client.create(); WebResource webResource = client.resource(githubBaseUri+resourceName); ClientResponse response = webResource.type("application/json") .accept("application/json").get(ClientResponse.class); int statusCode = response.getStatus(); if(statusCode != HTTP_STATUS_CODE_OK) { throw new IllegalStateException("Error code "+statusCode); } return response.getEntity(String.class); } } And now we want to test that invokeGetMethod really gets the required resource. Let's suppose that this method in production code will be responsible of getting all issues name from a project registered on github. Now we can start to write the test: @Rule public ClientDriverRule driver = new ClientDriverRule(); @Test public void issues_from_project_should_be_retrieved() { driver.addExpectation( onRequestTo("/repos/lordofthejars/nosqlunit/issues"). withMethod(Method.GET), giveResponse(GET_RESPONSE)); GithubClient githubClient = new GithubClient(driver.getBaseUrl()); String issues = githubClient.invokeGetMethod("/repos/lordofthejars/nosqlunit/issues"); assertThat(issues, is(GET_RESPONSE)); } We use ClientDriverRule @Rule annotation to add the client-driver to a test. And then using methods provided by RestClientDriver class, expectations are recorded. See how we are setting the base URL using driver.getBaseUrl() With rest-client-driver we can also record http status response using giveEmptyResponse method: @Test(expected=IllegalStateException.class) public void http_errors_should_throw_an_exception() { driver.addExpectation( onRequestTo("/repos/lordofthejars/nosqlunit/issues") .withMethod(Method.GET), giveEmptyResponse().withStatus(401)); GithubClient githubClient = new GithubClient(driver.getBaseUrl()); githubClient.invokeGetMethod("/repos/lordofthejars/nosqlunit/issues"); } And obviously we can record a put action: Note that in this example, we are setting that our request should contain given message body to response a 204 status code. This is a very simple example, but keep in mind that also works with libraries like gson or jackson. Also rest-driver project comes with a module that can be used to assert server responses (like REST-assured project) but this topic will be addressed into another post. I wish you have found this post useful. We keep learning, Alex.

I would consider this post as best practice for using Spring in enterprise application development.

when writing testable code, your first port of call is often to abstract any dependencies and make them easy to mock. this is the same for any of your codebase that talks to ftp servers. testing the way your code behaves under real world conditions makes integration tests important regardless of abstraction, though. here’s a simple trick to test ftp code in the wild. a recent project of mine has involved writing code that talks to ftp servers with the goal of adding additional continuous integration automation to a project. although all of my main methods are easily abstracted and injectable, my project still needs to actually talk to ftp servers at the end of the day, and i need to test that these very methods do the right thing when they are met with different conditions; be they bad credentials, lack of read/write permissions etc. the challenge integration tests can be brittle at the best of times, so ensuring that they are repeatable and can be setup and torn down can often be almost as much of a challenge as writing your actual code itself. an ftp server is usually a static service that is installed on a server. you might think that running one and ensuring it stays up and doesn’t get hacked just so that all your integration tests work is a necessary evil, but there is an easier way. run local. run often. i was running an ftp server on my build server just so that it was “always around” for my tests until i stumbled across an interesting project over on github to do just this . the approach i'm about to show you doesn’t need you to go to the effort of running a dedicated server at all. all you need to do is add a single executable to your unit test project and wrap your unit test in a using statement. the ftp server executable is a single file ftp server called ftpdmin which offers a read/write ftp server that can be fired up from the command line with a minimum feature set and only a few command line parameters to make it all tick. by implementing idisposable the helper class that wraps around this command line exe allows you to take advantage of the using() pattern to take care of your executable’s lifetime and have it die when your code is done testing. steps to make it happen download ftpdmin from here . add the exe to the root of your test project (you can put this anywhere, but you’ll have to update the helper class below). now add the exe to your project (i.e “view all items” in your test project’s solution explorer, and add the exe). set the exe to “copy always” in it’s solution properties. add the following code to a helper class in your test project: public class ftptestserver: idisposable { private readonly process ftpprocess; public ftptestserver(string rootdirectory, int port = 21, bool allowuploads = true) { var psinfo = new processstartinfo { filename = appdomain.currentdomain.basedirectory + "\\ftpdmin.exe", arguments = string.format("-p {0} -ha 127.0.0.1 \"{1}\" {2}", port, rootdirectory, allowuploads ? string.empty : "-g"), windowstyle = processwindowstyle.hidden }; ftpprocess = process.start(psinfo); } public void dispose() { if (ftpprocess.hasexited) return; ftpprocess.kill(); ftpprocess.waitforexit(); } } now you can enjoy being able to write really clean integration testing code that starts and ftp server every time you run your tests and then tear it down when your test is done. an example integration test showing connecting to “127.0.0.1”: [testmethod] public void ftpcode_upload_canconnect() { try { // fire up a new ftp server instance using (new ftptestserver(rootdirectory: "./")) { // code that talks to an ftp server on 127.0.0.1 } } catch (webexception e) { assert.fail("failed to connect to our ftp server"); } } how awesome is that? the power of using ftpdmin is that it can be told to deny write permissions to simulate bad user permissions as well: [testmethod] public void ftpcode_upload_throwswebexception() { try { // fire up a new ftp server instance using (new ftptestserver(rootdirectory: "./", allowuploads: false)) { // code that talks to an ftp server on 127.0.0.1 } } catch (webexception e) { assert.fail("our code failed to upload a file because of invalid permissions"); } } all in all, the above has been a complete lifesaver when it comes to making my integration test projects portable – if a new developer joins my project, they instantly get access to my ftp test harness just by pulling down my project’s source code.

This article aims to show you how to quickly fix the most common java security code violations. It assumes that you are familiar with the concept of code rules and violations and how Sonar reports on them. However, if you haven’t heard these terms before then you might take a look at Sonar Concepts or the forthcoming book about Sonar for a more detailed explanation. To get an idea, during Sonar analysis, your project is scanned by many tools to ensure that the source code conforms with the rules you’ve created in your quality profile. Whenever a rule is violated… well a violation is raised. With Sonar you can track these violations with violations drilldown view or in the source code editor. There are hundreds of rules, categorized based on their importance. Ill try, in future posts, to cover as many as I can but for now let’s take a look at some common security rules / violations. There are two pairs of rules (all of them are ranked as critical in Sonar ) we are going to examine right now. 1. Array is Stored Directly ( PMD ) and Method returns internal array ( PMD ) These violations appear in the cases when an internal Array is stored or returned directly from a method. The following example illustrates a simple class that violates these rules. public class CalendarYear { private String[] months; public String[] getMonths() { return months; } public void setMonths(String[] months) { this.months = months; } } To eliminate them you have to clone the Array before storing / returning it as shown in the following class implementation, so noone can modify or get the original data of your class but only a copy of them. public class CalendarYear { private String[] months; public String[] getMonths() { return months.clone(); } public void setMonths(String[] months) { this.months = months.clone(); } } 2. Nonconstant string passed to execute method on an SQL statement (findbugs) and A prepared statement is generated from a nonconstant String (findbugs) Both rules are related to database access when using JDBC libraries. Generally there are two ways to execute an SQL Commants via JDBC connection : Statement and PreparedStatement. There is a lot of discussion about pros and cons but it’s out of the scope of this post. Let’s see how the first violation is raised based on the following source code snippet. Statement stmt = conn.createStatement(); String sqlCommand = "Select * FROM customers WHERE name = '" + custName + "'"; stmt.execute(sqlCommand); You’ve already noticed that the sqlcommand parameter passed to execute method is dynamically created during run-time which is not acceptable by this rule. Similar situations causes the second violation. String sqlCommand = "insert into customers (id, name) values (?, ?)"; Statement stmt = conn.prepareStatement(sqlCommand); You can overcome this problems with three different ways. You can either use StringBuilder or String.format method to create the values of the string variables. If applicable you can define the SQL Commands as Constant in class declaration, but it’s only for the case where the SQL command is not required to be changed in runtime. Let’s re-write the first code snippet using StringBuilder Statement stmt = conn.createStatement(); stmt.execute(new StringBuilder("Select FROM customers WHERE name = '"). append(custName). append("'").toString()); and using String.format Statement stmt = conn.createStatement(); String sqlCommand = String.format("Select * from customers where name = '%s'", custName); stmt.execute(sqlCommand); For the second example you can just declare the sqlCommand as following private static final SQLCOMMAND = insert into customers (id, name) values (?, ?)"; There are more security rules such as the blocker Hardcoded constant database password but I assume that nobody is still hardcodes passwords in source code files… In following articles I’m going to show you how to adhere to performance and bad practice rules. Until then I’m waiting for your comments or suggestions.

Date and time handling in Java is a somewhat tricky part when you are new to the language. Time can be accessed via the static method System.currentTimeMillis() which returns the current time in milliseconds from January 1st 1970. If you prefer to work with Objects instead you can use java.util.Date, a class whose methods are mostly deprecated in recent versions of Java. To work with time offsets, say add one month to a date, there is java.util.GregorianCalendar. All in all, those methods described here are not very convenient to work with. Java 7 and below are lacking a good date and time API. The Joda Time library is a common drop-in if you need to work with date/time. With JSR 310 (Java Specification Request) this is about to change. JSR 310 adds a new date, time and calendar API to Java 8. The ThreeTen project provides a reference implementation to this new API and can already be utilized in current Java projects (I however recommend not to do this for production). As the README states: The API is currently considered usable and accurate, yet incomplete and subject to change. If you use this API you must be able to handle incompatible changes in later versions. Building ThreeTen Building the ThreeTen project is relatively easy. It requires both Git and Ant to be installed on your system. git clone git://github.com/ThreeTen/threeten.git cd threeten ant This will first fetch the most recent version of ThreeTen and then start the build process using ant. Note that building the library also requires either OpenJDK 1.6 or Oracle JDK 1.6. JSR 310 The new API specifies a number of new classes which are divided into the categories of continuous and human time. Continuous time is based on Unix time and is represented as a single incrementing number. Class Description Instant A point in time in nanoseconds from January 1st 1970 Duration An amount of time measured in nanoseconds Human time is based on fields that we use in our daily lifes such as day, hour, minute and second. It is represented by a group of classes, some of which we will discuss in this article. Class Description LocalDate a date, without time of day, offset or zone LocalTime the time of day, without date, offset or zone LocalDateTime the date and time, without offset or zone OffsetDate a date with an offset such as +02:00, without time of day or zone OffsetTime the time of day with an offset such as +02:00, without date or zone OffsetDateTime the date and time with an offset such as +02:00, without a zone ZonedDateTime the date and time with a time zone and offset YearMonth a year and month MonthDay month and day Year/MonthOfDay/DayOfWeek/... classes for the important fields DateTimeFields stores a map of field-value pairs which may be invalid Calendrical access to the low-level API Period a descriptive amount of time, such as "2 months and 3 days" In addition to the above classes three support classes have been implemented. The Clock class wraps the current time and date, ZoneOffset is a time offset from UTC and ZoneId defines a time zone such as 'Australia/Brisbane'. Using the API Getting the current time The current time is represented by the Clock class. The class is abstract, so you can not create instances of it. The systemUTC() static method will return the current time based on your system clock and set to UTC. import javax.time.Clock; Clock clock = Clock.systemUTC(); To use the default time zone on your system there also is systemDefaultZone(). Clock clock = Clock.systemDefaultZone(); The millis() method can then be used to access the current time in milliseconds from January 1st, 1970. This shows, that the Clock class and all subclasses are wrapped around System.currentTimeMillis(). Clock clock = Clock.systemDefaultZone(); long time = clock.millis(); Working with time zones To work with time zones you need to import the ZoneId class. The class provides a method to get the default system time zone: import javax.time.ZoneId; import javax.time.Clock; ZoneId zone = ZoneId.systemDefault(); Clock clock = Clock.system(zone); As seen above, the ZoneId can then be used to get an instance of a Clock with that time zone. Other time zones can be accessed by their name, e.g.: ZoneId zone = ZoneId.of("Europe/Berlin"); Clock clock = Clock.system(zone); Getting human date and time Working with a time represented in a single long variable is not what we wanted. We want to work with objects that represent human readable time. The LocalDate, LocalTime and LocalDateTime classes do just that. import javax.time.LocalDate; // The now() method returns the current DateTime LocalDate date = LocalDate.now(); System.out.printf("%s-%s-%s", date.getYear(), date.getMonthValue(), date.getDayOfMonth() ); Using LocalDate to print the current date Doing calculations with times and dates One of the most important functionalities of JSR-310 is that you can do calculations with dates and times. The API makes it very easy to do that. import javax.time.LocalTime; import javax.time.Period; import static javax.time.calendrical.LocalPeriodUnit.HOURS; Period p = Period.of(5, HOURS); LocalTime time = LocalTime.now(); LocalTime newTime; newTime = time.plus(5, HOURS); // or newTime = time.plusHours(5); // or newTime = time.plus(p); Three ways of adding 5 hours to the current time Each class that represents human time implements the AdjustableDateTime interface. The interface requires the plus and the minus method that take a value and a PeriodUnit as argument. Conclusion This article gave a (very) brief introduction into the new date and time API that will ship with Java 8. The API seems to be very consistent and well thought through and provides many ways to interact with dates and times. Upon release of Java 8 the API will be moved from the javax.time package over to java.time, so there will be no conflict if you start using the current implementation.

Today we're going to build a simple TCP proxy server. The scenario: we've got one host (the client) that establishes a TCP connection to another one (the remote). client —> remote We want to set up a proxy server in the middle, so the client will establish the connection with the proxy and the proxy will forward it to the remote, keeping in mind the remote response also. With node.js is really simple to perform those kind of network operations. client —> proxy -> remote var net = require('net'); var LOCAL_PORT = 6512; var REMOTE_PORT = 6512; var REMOTE_ADDR = "192.168.1.25"; var server = net.createServer(function (socket) { socket.on('data', function (msg) { console.log(' ** START **'); console.log('<< From client to proxy ', msg.toString()); var serviceSocket = new net.Socket(); serviceSocket.connect(parseInt(REMOTE_PORT), REMOTE_ADDR, function () { console.log('>> From proxy to remote', msg.toString()); serviceSocket.write(msg); }); serviceSocket.on("data", function (data) { console.log('<< From remote to proxy', data.toString()); socket.write(data); console.log('>> From proxy to client', data.toString()); }); }); }); server.listen(LOCAL_PORT); console.log("TCP server accepting connection on port: " + LOCAL_PORT); Simple, isn’t it? Source code in github

First of all we should define EIPs and why we should use them. As the name implies, these are tested solutions for specific design problems encountered during many years in the development of IT systems. And what is all the more important is that they are technology-agnostic which means it does not matter what programming language or operating system you use. Patterns are divided into seven sections: 1. Messaging Systems, 2. Messaging Channels, 3. Message Constructions, 4. Message Routing, 5. Message Transformation, 6. Messaging endpoints, 7. System management. The purpose of this article is to encourage you to use the patterns so I will discuss briefly only one or two such patterns from each of above sections. If you want to explore then further, visit http://www.eaipatterns.com/ or read Gregor Hohpe’s book mentioned in the introduction of this series. Message Channel (from Messaging Systems) A message channel is a logical channel which is used to connect the applications. One application writes messages to the channel and the other one (or others) reads that message from the channel. Message queue and message topic are examples of message channels. Message Translator (from Messaging Systems) Message translator transforms messages in one format to another. For example one application sends a message in XML format, but the other accepts only JSON messages so one of the parties (or mediator) has to transform XML data to JSON. This is probably the most widely used integration pattern. Publish-Subscribe Channel (from Messaging Channels) This type of channel broadcasts an event or notification to all subscribed receivers. This is in contrast with a point-to-point channel . Each subscriber receive the message once and next copy of this message is deleted from channel. The most common implementation of this patter is messaging topic. Dead Letter Channel (from Messaging Channels) The Dead Letter Channel describe scenario, what to do if the messaging system determines that it cannot deliver a message to the specified recipient. This may be caused for example by connection problems or other exception like overflowed memory or disc space. Usually, before sending the message to the Dead Letter Channel, multiple attempts to redeliver message are taken. Correlation Identifier (from Message Construction) Correlation Identifier gives the possibility to match request and reply message when asynchronous messaging system is used. This is usually accomplished in the following way: Producer: Generate unique correlation identifier. Producer: Send message with attached generated correlation identifier. Consumer: Process messages and send reply with attached correlation identifier given in request message. Producer: Correlate request and reply message based on correlation identifier. Content-Based Router (from Message Routing) Content-Based Router examines message contents and route messages based on data contained in the message. Content Enricher (from Message Transformation) Content Enricher as the name suggests enrich message with missing information. Usually external data source like database or web service is used. Event-Driven Consumer (from Messaging Endpoints) Event-Driver Consumer enables you to provide a action that is called automatically by the messaging channel or transport layer. It is asynchronous type of pattern because receiver does not have a running thread until a callback thread delivers a message. Polling Consumer (from Messaging Endpoints) Polling Consumer is used when we want receiver to poll for a message, process it and next poll for another. What is very important is that this pattern is synchronous because it blocks thread until a message is received. This is in contrast with a event-driven consumer. An example of using this pattern is file polling. Wire Tap (from System Management) Wire Tap copy a message and route it to a separate channel, while the original message is forwarded to the destination channel. Usually Wire Tap is used to inspect message or for analysis purposes.

If you have ever used a WordPress premium theme then you would of seen the custom theme options page that is available. The theme options page that is found under the appearance menu which allows the admin of the WordPress site to change some of the settings on the theme. Most premium themes will come with options to change the colors of fonts, backgrounds, change images or font types...anything that allows you to style the WordPress theme. Some of the most common fields to change are: Theme Options - To edit the theme logo, change the stylesheet, upload a new favicon, Add Google analytics code, enter your feedburner URL and add custom CSS. Styling Options - Change the background colour or change the background image. Fonts - Change the font on all your header tags or the main content text. Social - Providing you theme with your social media profiles will make it easier to link to them in parts of your theme or display your latest tweets. Option pages can also be used on plugins to change settings and to customize the plugin. Examples Of Theme Options Pages Here is what some of the theme options page from premium themes look like. How To Build A Theme Option Page When creating an option page there are a few things you need to setup. Add Menu - If you want to display the menu under the appearance menu or if you want to give the options page it's own menu. Add Sections - These are sections of settings you are adding to the options page. Register Settings - Settings are the different fields you are adding to the options page, they need to be registered with the settings API. Display Settings - The settings API will be used to call a function to display the setting. Validate Setting - When the user saves the settings field the input will need to be validated before stored in the options table. Feedback Messages - When the settings are saved you need to be able to feedback to the user if the settings were saved successfully or if there was an error during validation. To help us perform all these tasks there is a WordPress API called the Settings API. This API allows admin pages to handle setting forms semi-automatically. With the API you can define pages for the settings, sections for the settings and fields for the settings. This works by registering setting fields to be displayed within sections and page will display these sections. WordPress uses the Settings API by default on existing admin pages, this means that by using the Settings API you can add to existing pages by registering new settings. All validation must be performed by the developer of the settings pages but the Settings API will control the creation of the form and storing the values in the form in the options table. Add Menu To WordPress Admin When adding a menu to the WordPress admin screen you have loads of flexibility you have the option of adding brand new menu items or adding the menu as a sub menu. To add a top level menu just use the following function add_menu_page(). $page_title - The title used on the settings page. $menu_title - The title used on the menu. $capability - Only displays the menu if the user matches this capability. $menu_slug - The unique name of the menu slug. $function - This is the callback function to run to display the page. $icon_url - Display a icon just for the menu. $position - This allows you to choose when the menu item appears in the list. If you prefer to have the menu under the appearance parent menu you can use the following code snippet. Or you can use the function add_theme_page() which will add a sub-menu under the appearance menu. add_theme_page( $page_title, $menu_title, $capability, $menu_slug, $function); Registering The Settings To start off we need to register the settings group we are going to store the settings page values. This will use the Settings API to define the group of settings, we will then add the settings to a group. When you store the settings in this group they are stored in the wp_options database table so you can get these values out at a later date. The wp_options table is a key value pairing stored in the database. This is what you should use when storing long term data on your WordPress site. If you are storing a lot of data it's best practice to turn the data into an array and store it under one key, instead of storing all the values over multiple keys. This means that if you have a settings page to change the site logo, background color, font, font size etc, you won't have an option for each of these but you will group them into an option group. The reason you do this is to increase on database efficiency by not adding too many rows to the options database. To register settings on the Settings API you need to use the function register_setting(). The parameters you pass into this are: Option Group - The name of the group of settings you are going to store. This must match the group name used in the settings_field() function. Option name - The name of the option which will be saved, this is the key that is used in the options table. Sanitize Callback - This is the function that is used to validate the settings for this option group. Add Sections To Settings Once the settings are registered we can add section groups to the Settings API. This will allow us to organise the settings on the page, so that you can add styles to display these differently on the page. The benefit of adding sections on your Settings API is so that we can call the function do_settings_sections() as this will display all the settings under this one section. To create you own settings all you have to do is use the function add_settings_section(). The parameters you need to use on this function are: Id - String to use for the ID of the section. Title - The title to use on the section. Callback - This is the function that will display the settings on the page. Page - This is the page that is displaying the section, should match the menu slug of the page. Add Fields To The Sections The last important function we need to use to add settings to the page is the add_settings_field() function, this is used as part of the Settings API to define fields to a section. The function will need to know the page slug and the section Id before you can define the settings to use. All the settings which you setup here will be stored in the options table under the key used in the register_settings() function. To use this function you need to add the following parameters. ID - ID of the field Title - Title of the field. Callback - Function used to display the setting. This is very important as it is used to display the input field you want. Page - Page which is going to display the field should be the same as the menu slug on the section. Section - Section Id which the field will be added to. $args - Additional arguments which are passed to the callback function. Example Of Using The Settings API There is a lot of information to take in above to create this settings page so it can seem a bit complicated but once you get your head around the structure the Settings API uses it's actually quite easy to understand. The best way to understand how this all works is to show you with an example. Create A Theme Option Page With A Textbox Field In this example we will create a theme option page and add a textbox on the page to add additional text to the index.php. Just add the following to your functions.php file to create a theme options page. First we start off by creating the menu item under the appearance menu by using the add_theme_page() function on the admin_menu action. /** * Theme Option Page Example */ function pu_theme_menu() { add_theme_page( 'Theme Option', 'Theme Options', 'manage_options', 'pu_theme_options.php', 'pu_theme_page'); } add_action('admin_menu', 'pu_theme_menu'); As you can see above we set the callback function to the theme options page to be pu_theme_page so we need to create this function to display our page. Here we create a form to submit to the options.php so that we can save in the options table, we call settings_fields() to the get the settings in register_settings() and use the do_settings_sections() function to display our settings. /** * Callback function to the add_theme_page * Will display the theme options page */ function pu_theme_page() { ?> Custom Theme Options Created by Paulund. 'text', 'id' => 'pu_textbox', 'name' => 'pu_textbox', 'desc' => 'Example of textbox description', 'std' => '', 'label_for' => 'pu_textbox', 'class' => 'css_class' ); add_settings_field( 'example_textbox', 'Example Textbox', 'pu_display_setting', 'pu_theme_options.php', 'pu_text_section', $field_args ); } The callback function on creating sections can be used to add addition information that will appear above every section, on this example we are just leaving it blank. /** * Function to add extra text to display on each section */ function pu_display_section($section){ } The callback function on the add_settings_field() function is pu_display_setting, this is the function that is going to echo the display of any input's on the page. The parameter to this function is the $args value on the add_settings_field() we can use this to add things like id, name, default value etc. We want to get any existing values from the wp_option table to display any values which previously typed in by the user, do to this we get the values from the table by using the get_option() function. /** * Function to display the settings on the page * This is setup to be expandable by using a switch on the type variable. * In future you can add multiple types to be display from this function, * Such as checkboxes, select boxes, file upload boxes etc. */ function pu_display_setting($args) { extract( $args ); $option_name = 'pu_theme_options'; $options = get_option( $option_name ); switch ( $type ) { case 'text': $options[$id] = stripslashes($options[$id]); $options[$id] = esc_attr( $options[$id]); echo ""; echo ($desc != '') ? "$desc" : ""; break; } } Finally we can validate the values added to the form by creating the validation callback function pu_validate_settings. All this does at the moment is loop through the inputs passed to it and checks if it's a letter or a number. The return of this function is what will be added to the database. /** * Callback function to the register_settings function will pass through an input variable * You can then validate the values and the return variable will be the values stored in the database. */ function pu_validate_settings($input) { foreach($input as $k => $v) { $newinput[$k] = trim($v); // Check the input is a letter or a number if(!preg_match('/^[A-Z0-9 _]*$/i', $v)) { $newinput[$k] = ''; } } return $newinput; } If you copy all the snippets above into your functions.php file you will see this options form under the appearance menu. Using Theme Options Within Your Theme Now that you understand how to create a theme options page you need to be able to use this value in your theme so you can change the settings. All the settings are stored in the wp_options table with WordPress it's very easy to get these values out all you have to do is use the get_option() function. The option name is the name you put on the register_settings() function. So in our example above you will use this code. The $options variable will now store an array of the values from the theme options, which you can display the value of the textbox we put on the page by using this snippet. Conclusion That's the basics that you need to understand to use the Settings API, now you can take this information and create your own theme options page. Experiment with different input types you can add to the form, experiment with different validation methods you want to use. In future tutorials I will post how you can use some of the inbuilt WordPress third party applications to create a better user experience on your theme options panel. This will include things like color pickers, date pickers, jQuery UI features etc. As you can see we have created a settings option page in just over 100 lines of code, so it's not a hard thing to do but that are a few steps to it and the features can be expanded on. For this reason people have created theme option frameworks to allow you to easily create a theme option page with much higher level of complexity with the options. But like many other frameworks I always recommend you learn the basics before using a framework, this is why it's important to understand how the Settings API works before using or creating a settings page framework.

While exploring HDFS, I came across these two syntaxes for querying HDFS: > hadoop dfs > hadoop fs Initally I couldn't differentiate between the two, and kept wondering why we have two different syntaxes for a common purpose. I found a number of people online with the same question -- their thoughts are below: Per Chris's explanation: it seems like there's no difference between the two syntaxes. If we look at the definitions of the two commands (hadoop fs and hadoop dfs) in $HADOOP_HOME/bin/hadoop ... elif [ "$COMMAND" = "datanode" ] ; then CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_DATANODE_OPTS" elif [ "$COMMAND" = "fs" ] ; then CLASS=org.apache.hadoop.fs.FsShell HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "dfs" ] ; then CLASS=org.apache.hadoop.fs.FsShell HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "dfsadmin" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DFSAdmin HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" ... That was his reasoning. Unconvinced, I kept looking for a more persuasive answer, and these excerpts made more sense to me: FS relates to a generic file system which can point to any file systems like local, HDFS etc. But dfs is very specific to HDFS. So when we use FS it can perform operation with from/to local or hadoop distributed file system to destination. But specifying DFS operation relates to HDFS. Below are two excerpts from the Hadoop documentation that describe these two as different shells. FS Shell The FileSystem (FS) shell is invoked by bin/hadoop fs. All the FS shell commands take path URIs as arguments. The URI format is scheme://autority/path. For HDFS the scheme is hdfs, and for the local filesystem the scheme is file. The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://namenodehost/parent/child or simply as /parent/child (given that your configuration is set to point to hdfs://namenodehost). Most of the commands in FS shell behave like corresponding Unix commands. DFShell The HDFS shell is invoked by bin/hadoop dfs. All the HDFS shell commands take path URIs as arguments. The URI format is scheme://autority/path. For HDFS the scheme is hdfs, and for the local filesystem the scheme is file. The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://namenode:namenodeport/parent/child or simply as /parent/child (given that your configuration is set to point to namenode:namenodeport). Most of the commands in HDFS shell behave like corresponding Unix commands. So, based on the above, we can conclude that it all depends on the scheme configuration. When using these two commands with absolute URI (i.e. scheme://a/b) the behavior shall be identical. Only it's the default configured scheme value for file and hdfs for fs and dfs respectively, which is the cause for difference in behavior.

Hadoop MapReduce is a YARN-based system for parallel processing of large data sets. In this article, learn to quickly start writing the simplest MapReduce job.

For the last two weeks I’ve been working on some interesting use cases for the good ol’ failover transport. I finally have some time at my hands, so here’s a brief recap of what’s coming in 5.6 release in this area. First there’s a new feature, called Priority Backup. It’s described in details here, but in a nutshell it provides you with the mechanism of prioritizing your failover urls and keep your clients connected to them as soon as they are available. The most obvious use case for this is to keep your clients connected to the broker in local data center whenever you can. By doing this, you can both have better performances and stability of your clients, but also save on your bandwidth bills. Another improvement is coming for automatic broker cluster feature. Although this feature is not new, I spent some time hardening it and thought to share some more insight in how (and when) to use it in your projects. In search of high availability, people often default to master-slave architecture. This makes sense in most use cases, but if your flow is purely non-persistent you can probably come up with more optimal architecture. Instead of having one broker at the time handling all your load, and other one just waiting for it to fail, you’ll get more efficient system with some kind of active-active configuration where (possibly multiple) brokers share the load all the time. Ideally clients would be evenly distributed and would rebalance if anything changes. Brokers don’t need to share any messages as clients are distributed and messages are non-persistent so they will be lost if broker fails. So can you achieve this kind of architecture with ActiveMQ? Sure you do. That’s where automatic rebalance and clustering shines. First of all, brokers should be networked but only so they can exchange information on their availability. They shouldn’t exchange the messages (but of course can if your use case needs it). In 5.6 you do that with pure static networks, using configuration like So now imagine three brokers A,B and C forming a full mesh. In addition every broker uses rebalance options on their transport connectors All that is left for the client to do is connect to one of the brokers it knows like failover:(brokerA) and the broker will fill it with all information on other brokers in the cluster and whether it should reconnect to one of them or not. So having a large number of clients connecting like this, very soon they’ll rebalance over available brokers. You can stop one of the brokers in the cluster for updates and clients will rebalance over remaining ones. You can even add a new broker to the cluster and everything will get rebalanced without any need for you to touch your clients. So, basically in this way you have both load balancing and high availability for your non-persistent messages. Additionally, your clients are automatically updated with all information they need, and no manual intervention is needed. Although the basic support for clustering was there since 5.4, I did some more hardening and better rebalancing, so it’s coming in the Apache ActiveMQ 5.6 (and the next Fuse 5.5.1) release. Also, there are some more great stuff regarding broker clustering coming soon, so stay tuned and happy messaging.

From their site, it says the Apache Camel is a versatile open-source integration framework based on known Enterprise Integration Patterns. It might seem like a vague definition, but I want to tell you that this is a very productive Java library that can solve many of typical IT problems! You can think of it as a very light weight ESB framework with "batteries" included. In every jobs I've been to so far, folks are writing their own solutions in one way or another to solve many common problems (or they would buy some very expensive enterprisy ESB servers that takes months and months to learn, config, and maintain). Things that we commonly solve are integration (glue) code of existing business services together, process data in a certain workflow manner, or move and transform data from one place to another etc. These are very typical need in many IT environments. The Apache Camel can be used in cases like these; not only that, but also in a very productive and effective way! In this article, I will show you how to get started with Apache Camel along with just few lines of Groovy script. You can certainly also start off with a full Java project to try out Camel, but I find Groovy will give you the shortest working example and learning curve. Getting started with Apache Camel using Groovy So let's begin. First let's see a hello world demo with Camel + Groovy. @Grab('org.apache.camel:camel-core:2.10.0') @Grab('org.slf4j:slf4j-simple:1.6.6') import org.apache.camel.* import org.apache.camel.impl.* import org.apache.camel.builder.* def camelContext = new DefaultCamelContext() camelContext.addRoutes(new RouteBuilder() { def void configure() { from("timer://jdkTimer?period=3000") .to("log://camelLogger?level=INFO") } }) camelContext.start() addShutdownHook{ camelContext.stop() } synchronized(this){ this.wait() } Save above into a file named helloCamel.groovy and then run it like this: $ groovy helloCamel.groovy 388 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.10.0 (CamelContext: camel-1) is starting 445 [main] INFO org.apache.camel.management.ManagementStrategyFactory - JMX enabled. 447 [main] INFO org.apache.camel.management.DefaultManagementLifecycleStrategy - StatisticsLevel at All so enabling load performance statistics 678 [main] INFO org.apache.camel.impl.converter.DefaultTypeConverter - Loaded 170 type converters 882 [main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route1 started and consuming from: Endpoint[timer://jdkTimer?period=3000] 883 [main] INFO org.apache.camel.impl.DefaultCamelContext - Total 1 routes, of which 1 is started. 887 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.10.0 (CamelContext: camel-1) started in 0.496 seconds 898 [Camel (camel-1) thread #1 - timer://jdkTimer] INFO camelLogger - Exchange[ExchangePattern:InOnly, BodyType:null, Body:[Body is null]] 3884 [Camel (camel-1) thread #1 - timer://jdkTimer] INFO camelLogger - Exchange[ExchangePattern:InOnly, BodyType:null, Body:[Body is null]] 6884 [Camel (camel-1) thread #1 - timer://jdkTimer] INFO camelLogger - Exchange[ExchangePattern:InOnly, BodyType:null, Body:[Body is null]] ... The little script above is simple but it presented few key features of Camel Groovyness. The first and last section of the helloCamel.groovy script are just Groovy featuers. The @Grab annotation will automatically download the dependency jars you specify. We import Java packages to use its classes later. At the end we ensure to shutdown Camel before exiting JVM through the Java Shutdown Hook mechanism. The program will sit and wait until user press CTRL+C, just as a typical server process behavior. The middle section is where the Camel action is. You would always create a Camel context to begin (think of it as the server or manager for the process.) And then you would add a Camel route (think of it as a workflow or pipeflow) that you like to process data (Camel likes to call these data "messages"). The route consists of a "from" starting point (where data generated), and one or more "to" points (where data going to be processed). Camel calls these destination 'points' as 'Endpoints'. These endpoints can be expressed in simple URI string format such as "timer://jdkTimer?period=3000". Here we are generating timer message in every 3 secs into the pipeflow, and then process by a logger URI, which will simply print to console output. After Camel context started, it will start processing data through the workflow, as you can observe from the output example above. Now try pressing CTRL+C to end its process. Notice how the Camel will shutdown everything very gracefully. 7312 [Thread-2] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.10.0 (CamelContext: camel-1) is shutting down 7312 [Thread-2] INFO org.apache.camel.impl.DefaultShutdownStrategy - Starting to graceful shutdown 1 routes (timeout 300 seconds) 7317 [Camel (camel-1) thread #2 - ShutdownTask] INFO org.apache.camel.impl.DefaultShutdownStrategy - Route: route1 shutdown complete, was consuming from: Endpoint[timer://jdkTimer?period=3000] 7317 [Thread-2] INFO org.apache.camel.impl.DefaultShutdownStrategy - Graceful shutdown of 1 routes completed in 0 seconds 7321 [Thread-2] INFO org.apache.camel.impl.converter.DefaultTypeConverter - TypeConverterRegistry utilization[attempts=2, hits=2, misses=0, failures=0] mappings[total=170, misses=0] 7322 [Thread-2] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.10.0 (CamelContext: camel-1) is shutdown in 0.010 seconds. Uptime 7.053 seconds. So that's our first taste of Camel ride! However, we titled this section as "Hello World!" demo, and yet we haven't seen any. But you might have also noticed that above script are mostly boiler plate code that we setup. No user logic has been added yet. Not even the logging the message part! We simply configuring the route. Now let's modify the script little bit so we will actually add our user logic to process the timer message. @Grab('org.apache.camel:camel-core:2.10.0') @Grab('org.slf4j:slf4j-simple:1.6.6') import org.apache.camel.* import org.apache.camel.impl.* import org.apache.camel.builder.* def camelContext = new DefaultCamelContext() camelContext.addRoutes(new RouteBuilder() { def void configure() { from("timer://jdkTimer?period=3000") .to("log://camelLogger?level=INFO") .process(new Processor() { def void process(Exchange exchange) { println("Hello World!") } }) } }) camelContext.start() addShutdownHook{ camelContext.stop() } synchronized(this){ this.wait() } Notice how I can simply append the process code part right after the to("log...") line. I have added a "processor" code block to process the timer message. The logic is simple: we greet the world on each tick. Making Camel route more concise and practical Now, do I have you at Hello yet? If not, then I hope you will be patient and continue to follow along for few more practical features of Camel. First, if you were to put Camel in real use, I would recommend you setup your business logic separately from the workflow route definition. This is so that you can clearly express and see your entire pipeflow of route at a glance. To do this, you want to move the "processor", into a service bean. @Grab('org.apache.camel:camel-core:2.10.0') @Grab('org.slf4j:slf4j-simple:1.6.6') import org.apache.camel.* import org.apache.camel.impl.* import org.apache.camel.builder.* import org.apache.camel.util.jndi.* class SystemInfoService { def void run() { println("Hello World!") } } def jndiContext = new JndiContext(); jndiContext.bind("systemInfoPoller", new SystemInfoService()) def camelContext = new DefaultCamelContext(jndiContext) camelContext.addRoutes(new RouteBuilder() { def void configure() { from("timer://jdkTimer?period=3000") .to("log://camelLogger?level=INFO") .to("bean://systemInfoPoller?method=run") } }) camelContext.start() addShutdownHook{ camelContext.stop() } synchronized(this){ this.wait() } Now, see how compact this workflow route has become? The Camel's Java DSL such as "from().to().to()" for defining route are so clean and simple to use. You can even show this code snip to your Business Analysts, and they would likely be able to verify your business flow easily! Wouldn't that alone worth a million dollars? How about another demo: FilePoller Processing File polling processing is a very common and effective way to solve many business problems. If you work for commercial companies long enough, you might have written one before. A typical file poller would process incoming files from a directory and then process the content, and then move the file into a output directory. Let's make a Camel route to do just that. @Grab('org.apache.camel:camel-core:2.10.0') @Grab('org.slf4j:slf4j-simple:1.6.6') import org.apache.camel.* import org.apache.camel.impl.* import org.apache.camel.builder.* import org.apache.camel.util.jndi.* class UpperCaseTextService { def String transform(String text) { return text.toUpperCase() } } def jndiContext = new JndiContext(); jndiContext.bind("upperCaseTextService", new UpperCaseTextService()) def dataDir = "/${System.properties['user.home']}/test/file-poller-demo" def camelContext = new DefaultCamelContext(jndiContext) camelContext.addRoutes(new RouteBuilder() { def void configure() { from("file://${dataDir}/in") .to("log://camelLogger") .to("bean://upperCaseTextService?method=transform") .to("file://${dataDir}/out") } }) camelContext.start() addShutdownHook{ camelContext.stop() } synchronized(this){ this.wait() } Here you see I defined a route to poll a $HOME/test/file-poller-demo/in directory for text files. Once it's found it will log it to console, and then process by a service that transform the content text into upper case. After this, it will send the file into $HOME/test/file-poller-demo/out directory. My goodness, reading the Camel route above probably express what I wrote down just as effective. Do you see the benefits here? What's the "batteries" included part. If you've used Python programming before, you might have heard the pharase that they claim often: Python has "batteries" included. This means their interpreter comes with a rich of libaries for most of the common programming need. You can often write python program without have to download separated external libraries. I am making similar analogies here with Apache Camel. The Camel project comes with so many ready to use components that you can find just about any transport protocals that can carry data. These Camel "components" are ones that support different 'Endpoint URI' that we have seen in our demos above. We have simply shown you timer, log, bean, and file components, but there are over 120 more. You will find jms, http, ftp, cfx, or tcp just to name a few. The Camel project also has an option for you to define route in declarative xml format. The xml is just an extension of a Spring xml config with Camel's namespace handler added on top. Spring is optional in Camel, but you can use it together in a very powerful way.

Before trying this out make sure you have installed GlassFish in your machine. Installation is just easy as downloading the zip archive from here and unzipping it to a desired location. Creating a new domain Open up a terminal, change directory to the GlassFish installation directory and run following. bin/asadmin This will enable you to use asadmin tool. Now execute the following command to create a new domain. after running this command you will probably have to give admin password and master password. create-domain --adminport 5000 --profile developer --user admin domain2 Now open up another terminal and change directory to [glassfish-installation]/domains/ and you will see the newly created domain2 has appeared there. Open up a browser and go to http://localhost:5000/. This will bring you the GlassFish GUI admin console. This is one place you can deploy your war file. But in this post I'm not much focusing on it, instead I will show you how to do this using the command line. But GUI lovers, for any consolation I have put some screenshots which you can follow if you prefer the GUI way. In case you want to delete a domain use the following command. delete-domain domain2 Starting the domain To start domain2 run following command. start-domain domain2 Deploying a war file Use the following command to deploy your war file. deploy --port 5000 --host localhost /home/pavithra/workspace/NewDemoService/WebServicesJaxWs/NewDemoService.war After deploying the war file I can access the WSDL file I want to access using the URL http://localhost:8080/NewDemoService/NewDemoService?WSDL To change default 8080 (HTTP) port which specifies where the web application context roots are available for a Web browser to connect to, you can use the --instanceport parameter when creating the domain. See the following command. create-domain --adminport 5000 --profile developer --user admin --instanceport 9000 domain2 Undeploying a war file To undeploy NewDemoService.war file you need to use the following command. Note that here you don't have to use the full name but literal "NewDemoService". undeploy --port 5000 --host localhost NewDemoService Stopping a domain To stop the domain "domain2" use the following command. stop-domain domain2 After this if you try to deploy to this particular domain, it will complain. Auto Deploy To perform auto deploy, copy NewDemoService.war file in to [glassfish-installation-directory]/domains/domain2/autodeploy directory. If autodeploy succeeds, you will see NewDemoService.war_deployed has created. This will deploy your war file automatically.

As you may have seen from my past performance related articles and HashMap case studies, Java thread safety problems can bring down your Java EE application and the Java EE container fairly easily. One of most common problems I have observed when troubleshooting Java EE performance problems is infinite looping triggered from the non-thread safe HashMap get() and put() operations. This problem is known since several years but recent production problems have forced me to revisit this issue one more time. This article will revisit this classic thread safety problem and demonstrate, using a simple Java program, the risk associated with a wrong usage of the plain old java.util.HashMap data structure involved in a concurrent threads context. This proof of concept exercise will attempt to achieve the following 3 goals: Revisit and compare the Java program performance level between the non-thread safe and thread safe Map data structure implementations (HashMap, Hashtable, synchronized HashMap, ConcurrentHashMap) Replicate and demonstrate the HashMap infinite looping problem using a simple Java program that everybody can compile, run and understand Review the usage of the above Map data structures in a real-life and modern Java EE container implementation such as JBoss AS7 For more detail on the ConcurrentHashMap implementation strategy, I highly recommend the great article from Brian Goetz on this subject. Tools and server specifications As a starting point, find below the different tools and software’s used for the exercise: Sun/Oracle JDK & JRE 1.7 64-bit Eclipse Java EE IDE Windows Process Explorer (CPU per Java Thread correlation) JVM Thread Dump (stuck thread analysis and CPU per Thread correlation) The following local computer was used for the problem replication process and performance measurements: Intel(R) Core(TM) i5-2520M CPU @ 2.50Ghz (2 CPU cores, 4 logical cores) 8 GB RAM Windows 7 64-bit * Results and performance of the Java program may vary depending of your workstation or server specifications. Java program In order to help us achieve the above goals, a simple Java program was created as per below: The main Java program is HashMapInfiniteLoopSimulator.java A worker Thread class WorkerThread.java was also created The program is performing the following: Initialize different static Map data structures with initial size of 2 Assign the chosen Map to the worker threads (you can chose between 4 Map implementations) Create a certain number of worker threads (as per the header configuration). 3 worker threads were created for this proof of concept NB_THREADS = 3; Each of these worker threads has the same task: lookup and insert a new element in the assigned Map data structure using a random Integer element between 1 – 1 000 000. Each worker thread perform this task for a total of 500K iterations The overall program performs 50 iterations in order to allow enough ramp up time for the HotSpot JVM The concurrent threads context is achieved using the JDK ExecutorService As you can see, the Java program task is fairly simple but complex enough to generate the following critical criteria’s: Generate concurrency against a shared / static Map data structure Use a mix of get() and put() operations in order to attempt to trigger internal locks and / or internal corruption (for the non-thread safe implementation) Use a small Map initial size of 2, forcing the internal HashMap to trigger an internal rehash/resize Finally, the following parameters can be modified at your convenience: ## Number of worker threads private static final int NB_THREADS = 3; ## Number of Java program iterations private static final int NB_TEST_ITERATIONS = 50; ## Map data structure assignment. You can choose between 4 structures // Plain old HashMap (since JDK 1.2) nonThreadSafeMap = new HashMap(2); // Plain old Hashtable (since JDK 1.0) threadSafeMap1 = new Hashtable(2); // Fully synchronized HashMap threadSafeMap2 = new HashMap(2); threadSafeMap2 = Collections.synchronizedMap(threadSafeMap2); // ConcurrentHashMap (since JDK 1.5) threadSafeMap3 = new ConcurrentHashMap(2); /*** Assign map at your convenience ****/ assignedMapForTest = threadSafeMap3; Now find below the source code of our sample program. #### HashMapInfiniteLoopSimulator.java package org.ph.javaee.training4; import java.util.Collections; import java.util.Map; import java.util.HashMap; import java.util.Hashtable; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; /** * HashMapInfiniteLoopSimulator * @author Pierre-Hugues Charbonneau * */ public class HashMapInfiniteLoopSimulator { private static final int NB_THREADS = 3; private static final int NB_TEST_ITERATIONS = 50; private static Map assignedMapForTest = null; private static Map nonThreadSafeMap = null; private static Map threadSafeMap1 = null; private static Map threadSafeMap2 = null; private static Map threadSafeMap3 = null; /** * Main program * @param args */ public static void main(String[] args) { System.out.println("Infinite Looping HashMap Simulator"); System.out.println("Author: Pierre-Hugues Charbonneau"); System.out.println("http://javaeesupportpatterns.blogspot.com"); for (int i=0; i(2); // Plain old Hashtable (since JDK 1.0) threadSafeMap1 = new Hashtable(2); // Fully synchronized HashMap threadSafeMap2 = new HashMap(2); threadSafeMap2 = Collections.synchronizedMap(threadSafeMap2); // ConcurrentHashMap (since JDK 1.5) threadSafeMap3 = new ConcurrentHashMap(2); // ConcurrentHashMap /*** Assign map at your convenience ****/ assignedMapForTest = threadSafeMap3; long timeBefore = System.currentTimeMillis(); long timeAfter = 0; Float totalProcessingTime = null; ExecutorService executor = Executors.newFixedThreadPool(NB_THREADS); for (int j = 0; j < NB_THREADS; j++) { /** Assign the Map at your convenience **/ Runnable worker = new WorkerThread(assignedMapForTest); executor.execute(worker); } // This will make the executor accept no new threads // and finish all existing threads in the queue executor.shutdown(); // Wait until all threads are finish while (!executor.isTerminated()) { } timeAfter = System.currentTimeMillis(); totalProcessingTime = new Float( (float) (timeAfter - timeBefore) / (float) 1000); System.out.println("All threads completed in "+totalProcessingTime+" seconds"); } } } #### WorkerThread.java package org.ph.javaee.training4; import java.util.Map; /** * WorkerThread * * @author Pierre-Hugues Charbonneau * */ public class WorkerThread implements Runnable { private Map map = null; public WorkerThread(Map assignedMap) { this.map = assignedMap; } @Override public void run() { for (int i=0; i<500000; i++) { // Return 2 integers between 1-1000000 inclusive Integer newInteger1 = (int) Math.ceil(Math.random() * 1000000); Integer newInteger2 = (int) Math.ceil(Math.random() * 1000000); // 1. Attempt to retrieve a random Integer element Integer retrievedInteger = map.get(String.valueOf(newInteger1)); // 2. Attempt to insert a random Integer element map.put(String.valueOf(newInteger2), newInteger2); } } } Performance comparison between thread safe Map implementations The first goal is to compare the performance level of our program when using different thread safe Map implementations: Plain old Hashtable (since JDK 1.0) Fully synchronized HashMap (via Collections.synchronizedMap()) ConcurrentHashMap (since JDK 1.5) Find below the graphical results of the execution of the Java program for each iteration along with a sample of the program console output. # Output when using ConcurrentHashMap Infinite Looping HashMap Simulator Author: Pierre-Hugues Charbonneau http://javaeesupportpatterns.blogspot.com All threads completed in 0.984 seconds All threads completed in 0.908 seconds All threads completed in 0.706 seconds All threads completed in 1.068 seconds All threads completed in 0.621 seconds All threads completed in 0.594 seconds All threads completed in 0.569 seconds All threads completed in 0.599 seconds ……………… As you can see, the ConcurrentHashMap is the clear winner here, taking in average only half a second (after an initial ramp-up) for all 3 worker threads to concurrently read and insert data within a 500K looping statement against the assigned shared Map. Please note that no problem was found with the program execution e.g. no hang situation. The performance boost is definitely due to the improved ConcurrentHashMap performance such as the non-blocking get() operation. The 2 other Map implementations performance level was fairly similar with a small advantage for the synchronized HashMap. HashMap infinite looping problem replication The next objective is to replicate the HashMap infinite looping problem observed so often from Java EE production environments. In order to do that, you simply need to assign the non-thread safe HashMap implementation as per code snippet below: /*** Assign map at your convenience ****/ assignedMapForTest = nonThreadSafeMap; Running the program as is using the non-thread safe HashMap should lead to: No output other than the program header Significant CPU increase observed from the system At some point the Java program will hang and you will be forced to kill the Java process What happened? In order to understand this situation and confirm the problem, we will perform a CPU per Thread analysis from the Windows OS using Process Explorer and JVM Thread Dump. 1 - Run the program again then quickly capture the thread per CPU data from Process Explorer as per below. Under explore.exe you will need to right click over the javaw.exe and select properties. The threads tab will be displayed. We can see overall 4 threads using almost all the CPU of our system. 2 – Now you have to quickly capture a JVM Thread Dump using the JDK 1.7 jstack utility. For our example, we can see our 3 worker threads which seems busy/stuck performing get() and put() operations. ..\jdk1.7.0\bin>jstack 272 2012-08-29 14:07:26 Full thread dump Java HotSpot(TM) 64-Bit Server VM (21.0-b17 mixed mode): "pool-1-thread-3" prio=6 tid=0x0000000006a3c000 nid=0x18a0 runnable [0x0000000007ebe000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(Unknown Source) at org.ph.javaee.training4.WorkerThread.run(WorkerThread.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) "pool-1-thread-2" prio=6 tid=0x0000000006a3b800 nid=0x6d4 runnable [0x000000000805f000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(Unknown Source) at org.ph.javaee.training4.WorkerThread.run(WorkerThread.java:29) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) "pool-1-thread-1" prio=6 tid=0x0000000006a3a800 nid=0x2bc runnable [0x0000000007d9e000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(Unknown Source) at org.ph.javaee.training4.WorkerThread.run(WorkerThread.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) .............. 3 – CPU per thread correlation It is now time to convert the Process Explorer thread ID DECIMAL format to HEXA format as per below. The HEXA value allows us to map and identify each thread as per below: ## TID: 1748 (nid=0X6D4) Thread name: pool-1-thread-2 CPU @25.71% Task: Worker thread executing a HashMap.get() operation at java.util.HashMap.get(Unknown Source) at org.ph.javaee.training4.WorkerThread.run(WorkerThread.java:29) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ## TID: 700 (nid=0X2BC) Thread name: pool-1-thread-1 CPU @23.55% Task: Worker thread executing a HashMap.put() operation at java.util.HashMap.put(Unknown Source) at org.ph.javaee.training4.WorkerThread.run(WorkerThread.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ## TID: 6304 (nid=0X18A0) Thread name: pool-1-thread-3 CPU @12.02% Task: Worker thread executing a HashMap.put() operation at java.util.HashMap.put(Unknown Source) at org.ph.javaee.training4.WorkerThread.run(WorkerThread.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ## TID: 5944 (nid=0X1738) Thread name: pool-1-thread-1 CPU @20.88% Task: Main Java program execution "main" prio=6 tid=0x0000000001e2b000 nid=0x1738 runnable [0x00000000029df000] java.lang.Thread.State: RUNNABLE at org.ph.javaee.training4.HashMapInfiniteLoopSimulator.main(HashMapInfiniteLoopSimulator.java:75) As you can see, the above correlation and analysis is quite revealing. Our main Java program is in a hang state because our 3 worker threads are using lot of CPU and not going anywhere. They may appear "stuck" performing HashMap get() & put() but in fact they are all involved in an infinite loop condition. This is exactly what we wanted to replicate. HashMap infinite looping deep dive Now let’s push the analysis one step further to better understand this looping condition. For this purpose, we added tracing code within the JDK 1.7 HashMap Java class itself in order to understand what is happening. Similar logging was added for the put() operation and also a trace indicating that the internal & automatic rehash/resize got triggered. The tracing added in get() and put() operations allows us to determine if the for() loop is dealing with circular dependency which would explain the infinite looping condition. #### HashMap.java get() operation public V get(Object key) { if (key == null) return getForNullKey(); int hash = hash(key.hashCode()); /*** P-H add-on- iteration counter ***/ int iterations = 1; for (Entry e = table[indexFor(hash, table.length)]; e != null; e = e.next) { /*** Circular dependency check ***/ Entry currentEntry = e; Entry nextEntry = e.next; Entry nextNextEntry = e.next != null?e.next.next:null; K currentKey = currentEntry.key; K nextNextKey = nextNextEntry != null?(nextNextEntry.key != null?nextNextEntry.key:null):null; System.out.println("HashMap.get() #Iterations : "+iterations++); if (currentKey != null && nextNextKey != null ) { if (currentKey == nextNextKey || currentKey.equals(nextNextKey)) System.out.println(" ** Circular Dependency detected! ["+currentEntry+"]["+nextEntry+"]"+"]["+nextNextEntry+"]"); } /***** END ***/ Object k; if (e.hash == hash && ((k = e.key) == key || key.equals(k))) return e.value; } return null; } HashMap.get() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.resize() in progress... HashMap.put() #Iterations : 1 HashMap.put() #Iterations : 2 HashMap.resize() in progress... HashMap.resize() in progress... HashMap.put() #Iterations : 1 HashMap.put() #Iterations : 2 HashMap.put() #Iterations : 1 HashMap.get() #Iterations : 1 HashMap.get() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.get() #Iterations : 1 HashMap.get() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.get() #Iterations : 1 HashMap.put() #Iterations : 1 ** Circular Dependency detected! [362565=362565][333326=333326]][362565=362565] HashMap.put() #Iterations : 2 ** Circular Dependency detected! [333326=333326][362565=362565]][333326=333326] HashMap.put() #Iterations : 1 HashMap.put() #Iterations : 1 HashMap.get() #Iterations : 1 HashMap.put() #Iterations : 1 ............................. HashMap.put() #Iterations : 56823 Again, the added logging was quite revealing. We can see that following a few internal HashMap.resize() the internal structure became affected, creating circular dependency conditions and triggering this infinite looping condition (#iterations increasing and increasing...) with no exit condition. It is also showing that the resize() / rehash operation is the most at risk of internal corruption, especially when using the default HashMap size of 16. This means that the initial size of the HashMap appears to be a big factor in the risk & problem replication. Finally, it is interesting to note that we were able to successfully run the test case with the non-thread safe HashMap by assigning an initial size setting at 1000000, preventing any resize at all. Find below the merged graph results: The HashMap was our top performer but only when preventing an internal resize. Again, this is definitely not a solution to the thread safe risk but just a way to demonstrate that the resize operation is the most at risk given the entire manipulation of the HashMap performed at that time. The ConcurrentHashMap, by far, is our overall winner by providing both fast performance and thread safety against that test case. JBoss AS7 Map data structures usage We will now conclude this article by looking at the different Map implementations within a modern Java EE container implementation such as JBoss AS 7.1.2. You can obtain the latest source code from the github master branch. Find below the report: Total JBoss AS7.1.2 Java files (August 28, 2012 snapshot): 7302 Total Java classes using java.util.Hashtable: 72 Total Java classes using java.util.HashMap: 512 Total Java classes using synchronized HashMap: 18 Total Java classes using ConcurrentHashMap: 46 Hashtable references were found mainly within the test suite components and from naming and JNDI related implementations. This low usage is not a surprise here. References to the java.util.HashMap were found from 512 Java classes. Again not a surprise given how common this implementation is since the last several years. However, it is important to mention that a good ratio was found either from local variables (not shared across threads), synchronized HashMap or manual synchronization safeguard so “technically” thread safe and not exposed to the above infinite looping condition (pending/hidden bugs is still a reality given the complexity with Java concurrency programming…this case study involving Oracle Service Bus 11g is a perfect example). A low usage of synchronized HashMap was found with only 18 Java classes from packages such as JMS, EJB3, RMI and clustering. Finally, find below a breakdown of the ConcurrentHashMap usage which was our main interest here. As you will see below, this Map implementation is used by critical JBoss components layers such as the Web container, EJB3 implementation etc. ## JBoss Single Sign On Used to manage internal SSO ID's involving concurrent Thread access Total: 1 ## JBoss Java EE & Web Container Not surprising here since lot of internal Map data structures are used to manage the http sessions objects, deployment registry, clustering & replication, statistics etc. with heavy concurrent Thread access. Total: 11 ## JBoss JNDI & Security Layer Used by highly concurrent structures such as internal JNDI security management. Total: 4 ## JBoss domain & managed server management, rollout plans... Total: 7 ## JBoss EJB3 Used by data structures such as File Timer persistence store, application Exception, Entity Bean cache, serialization, passivation... Total: 8 ## JBoss kernel, Thread Pools & protocol management Used by high concurrent Threads Map data structures involved in handling and dispatching/processing incoming requests such as HTTP. Total: 3 ## JBoss connectors such as JDBC/XA DataSources... Total: 2 ## Weld (reference implementation of JSR-299: Contexts and Dependency Injection for the JavaTM EE platform) Used in the context of ClassLoader and concurrent static Map data structures involving concurrent Threads access. Total: 3 ## JBoss Test Suite Used in some integration testing test cases such as an internal Data Store, ClassLoader testing etc. Total: 3 Final words I hope this article has helped you revisit this classic problem and understand one of the common problems and risks associated with a wrong usage of the non-thread safe HashMap implementation. My main recommendation to you is to be careful when using an HashMap in a concurrent threads context. Unless you are a Java concurrency expert, I recommend that you use ConcurrentHashMap instead which offers a very good balance between performance and thread safety. As usual, extra due diligence is always recommended such as performing cycles of load & performance testing. This will allow you to detect thread safety and / or performance problems before you promote the solution to your client production environment. Please provide any comments and share your experience with ConcurrentHashMap or HashMap implementations and troubleshooting.

many times i end up using source files from somebody else. and such files might have a lot of empty lines in it i want to get removed. the question is: how to get rid of empty or blank lines in the eclipse editor view? or even better: how to merge multiple empty lines into one? shortcut: ctrl+d deleting a line is easy: i place the cursor on a line and press ctrl+d (for line d elete). that works as well for multiple line selection. using the ‘ mother of all shortcuts ‘ reveals even more shortcut options: delete commands this approach is fine, but manual. there must be something better. search-and-replace: regular expression and there is: to automatically remove empty lines, the search-and-replace functionality helps. for this i press ctrl-f (for find) and configure it like this: regular expression to remove empty lines the magic is using a regular expression checkbox. it uses the regular expression ^\s*\n to find one or multiple empty lines and replaces it with ‘nothing’. that way i can clean up a full file and get rid of all empty lines. if i do not remember the syntax of regular expressions any more, then there is help too: the dialog has content assist available: content assist so that way pressing ctrl+space helps a lot: content assist help merging empty lines with this in mind, it is easy to do something more advanced : to merge multiple empty lines into a single one. again, a regular expression does the magic work: merging multiple empty lines into a single one with this, the regular expression ^\s*\n finds one or multiple empty lines (as before), and it replaces it with a single empty line: \r now the sources need much less screen real estate . happy removing:-)

In Java, you can easily implement some business logic in Plain Old Java Object (POJO) classes, and then able to run them in a fancy server or framework without much hassle. There many server/frameworks, such as JBossAS, Spring or Camel etc, that would allow you to deploy POJO without even hardcoding to their API. Obviously you would get advance features if you willing to couple to their API specifics, but even if you do, you can keep these to minimal by encapsulating your own POJO and their API in a wrapper. By writing and designing your own application as simple POJO as possible, you will have the most flexible ways in choose a framework or server to deploy and run your application. One effective way to write your business logic in these environments is to use Service component. In this article I will share few things I learned in writing Services. What is a Service? The word Service is overly used today, and it could mean many things to different people. When I say Service, my definition is a software component that has minimal of life-cycles such as init, start, stop, and destroy. You may not need all these stages of life-cycles in every service you write, but you can simply ignore ones that don't apply. When writing large application that intended for long running such as a server component, definining these life-cycles and ensure they are excuted in proper order is crucial! I will be walking you through a Java demo project that I have prepared. It's very basic and it should run as stand-alone. The only dependency it has is the SLF4J logger. If you don't know how to use logger, then simply replace them with System.out.println. However I would strongly encourage you to learn how to use logger effectively during application development though. Also if you want to try out the Spring related demos, then obviously you would need their jars as well. Writing basic POJO service You can quickly define a contract of a Service with life-cycles as below in an interface. package servicedemo; public interface Service { void init(); void start(); void stop(); void destroy(); boolean isInited(); boolean isStarted(); } Developers are free to do what they want in their Service implementation, but you might want to give them an adapter class so that they don't have to re-write same basic logic on each Service. I would provide an abstract service like this: package servicedemo; import java.util.concurrent.atomic.*; import org.slf4j.*; public abstract class AbstractService implements Service { protected Logger logger = LoggerFactory.getLogger(getClass()); protected AtomicBoolean started = new AtomicBoolean(false); protected AtomicBoolean inited = new AtomicBoolean(false); public void init() { if (!inited.get()) { initService(); inited.set(true); logger.debug("{} initialized.", this); } } public void start() { // Init service if it has not done so. if (!inited.get()) { init(); } // Start service now. if (!started.get()) { startService(); started.set(true); logger.debug("{} started.", this); } } public void stop() { if (started.get()) { stopService(); started.set(false); logger.debug("{} stopped.", this); } } public void destroy() { // Stop service if it is still running. if (started.get()) { stop(); } // Destroy service now. if (inited.get()) { destroyService(); inited.set(false); logger.debug("{} destroyed.", this); } } public boolean isStarted() { return started.get(); } public boolean isInited() { return inited.get(); } @Override public String toString() { return getClass().getSimpleName() + "[id=" + System.identityHashCode(this) + "]"; } protected void initService() { } protected void startService() { } protected void stopService() { } protected void destroyService() { } } This abstract class provide the basic of most services needs. It has a logger and states to keep track of the life-cycles. It then delegate new sets of life-cycle methods so subclass can choose to override. Notice that the start() method is checking auto calling init() if it hasn't already done so. Same is done in destroy() method to the stop() method. This is important if we're to use it in a container that only have two stages life-cycles invocation. In this case, we can simply invoke start() and destroy() to match to our service's life-cycles. Some frameworks might go even further and create separate interfaces for each stage of the life-cycles, such as InitableService or StartableService etc. But I think that would be too much in a typical app. In most of the cases, you want something simple, so I like it just one interface. User may choose to ignore methods they don't want, or simply use an adaptor class. Before we end this section, I would throw in a silly Hello world service that can be used in our demo later. package servicedemo; public class HelloService extends AbstractService { public void initService() { logger.info(this + " inited."); } public void startService() { logger.info(this + " started."); } public void stopService() { logger.info(this + " stopped."); } public void destroyService() { logger.info(this + " destroyed."); } } Managing multiple POJO Services with a container Now we have the basic of Service definition defined, your development team may start writing business logic code! Before long, you will have a library of your own services to re-use. To be able group and control these services into an effetive way, we want also provide a container to manage them. The idea is that we typically want to control and manage multiple services with a container as a group in a higher level. Here is a simple implementation for you to get started: package servicedemo; import java.util.*; public class ServiceContainer extends AbstractService { private List services = new ArrayList(); public void setServices(List services) { this.services = services; } public void addService(Service service) { this.services.add(service); } public void initService() { logger.debug("Initializing " + this + " with " + services.size() + " services."); for (Service service : services) { logger.debug("Initializing " + service); service.init(); } logger.info(this + " inited."); } public void startService() { logger.debug("Starting " + this + " with " + services.size() + " services."); for (Service service : services) { logger.debug("Starting " + service); service.start(); } logger.info(this + " started."); } public void stopService() { int size = services.size(); logger.debug("Stopping " + this + " with " + size + " services in reverse order."); for (int i = size - 1; i >= 0; i--) { Service service = services.get(i); logger.debug("Stopping " + service); service.stop(); } logger.info(this + " stopped."); } public void destroyService() { int size = services.size(); logger.debug("Destroying " + this + " with " + size + " services in reverse order."); for (int i = size - 1; i >= 0; i--) { Service service = services.get(i); logger.debug("Destroying " + service); service.destroy(); } logger.info(this + " destroyed."); } } From above code, you will notice few important things: We extends the AbstractService, so a container is a service itself. We would invoke all service's life-cycles before moving to next. No services will start unless all others are inited. We should stop and destroy services in reverse order for most general use cases. The above container implementation is simple and run in synchronized fashion. This mean, you start container, then all services will start in order you added them. Stop should be same but in reverse order. I also hope you would able to see that there is plenty of room for you to improve this container as well. For example, you may add thread pool to control the execution of the services in asynchronized fashion. Running POJO Services Running services with a simple runner program. In the simplest form, we can run our POJO services on our own without any fancy server or frameworks. Java programs start its life from a static main method, so we surely can invoke init and start of our services in there. But we also need to address the stop and destroy life-cycles when user shuts down the program (usually by hitting CTRL+C.) For this, the Java has the java.lang.Runtime#addShutdownHook() facility. You can create a simple stand-alone server to bootstrap Service like this: package servicedemo; import org.slf4j.*; public class ServiceRunner { private static Logger logger = LoggerFactory.getLogger(ServiceRunner.class); public static void main(String[] args) { ServiceRunner main = new ServiceRunner(); main.run(args); } public void run(String[] args) { if (args.length < 1) throw new RuntimeException("Missing service class name as argument."); String serviceClassName = args[0]; try { logger.debug("Creating " + serviceClassName); Class serviceClass = Class.forName(serviceClassName); if (!Service.class.isAssignableFrom(serviceClass)) { throw new RuntimeException("Service class " + serviceClassName + " did not implements " + Service.class.getName()); } Object serviceObject = serviceClass.newInstance(); Service service = (Service)serviceObject; registerShutdownHook(service); logger.debug("Starting service " + service); service.init(); service.start(); logger.info(service + " started."); synchronized(this) { this.wait(); } } catch (Exception e) { throw new RuntimeException("Failed to create and run " + serviceClassName, e); } } private void registerShutdownHook(final Service service) { Runtime.getRuntime().addShutdownHook(new Thread() { public void run() { logger.debug("Stopping service " + service); service.stop(); service.destroy(); logger.info(service + " stopped."); } }); } } With abover runner, you should able to run it with this command: $ java demo.ServiceRunner servicedemo.HelloService Look carefully, and you'll see that you have many options to run multiple services with above runner. Let me highlight couple: Improve above runner directly and make all args for each new service class name, instead of just first element. Or write a MultiLoaderService that will load multiple services you want. You may control argument passing using System Properties. Can you think of other ways to improve this runner? Running services with Spring The Spring framework is an IoC container, and it's well known to be easy to work POJO, and Spring lets you wire your application together. This would be a perfect fit to use in our POJO services. However, with all the features Spring brings, it missed a easy to use, out of box main program to bootstrap spring config xml context files. But with what we built so far, this is actually an easy thing to do. Let's write one of our POJO Service to bootstrap a spring context file. package servicedemo; import org.springframework.context.ConfigurableApplicationContext; import org.springframework.context.support.FileSystemXmlApplicationContext; public class SpringService extends AbstractService { private ConfigurableApplicationContext springContext; public void startService() { String springConfig = System.getProperty("springContext", "spring.xml); springContext = new FileSystemXmlApplicationContext(springConfig); logger.info(this + " started."); } public void stopService() { springContext.close(); logger.info(this + " stopped."); } } With that simple SpringService you can run and load any spring xml file. For example try this: $ java -DspringContext=config/service-demo-spring.xml demo.ServiceRunner servicedemo.SpringService Inside the config/service-demo-spring.xml file, you can easily create our container that hosts one or more service in Spring beans. Notice that I only need to setup init-method and destroy-method once on the serviceContainer bean. You can then add one or more other service such as the helloService as much as you want. They will all be started, managed, and then shutdown when you close the Spring context. Note that Spring context container did not explicitly have the same life-cycles as our services. The Spring context will automatically instanciate all your dependency beans, and then invoke all beans who's init-method is set. All that is done inside the constructor of FileSystemXmlApplicationContext. No explicit init method is called from user. However at the end, during stop of the service, Spring provide the springContext#close() to clean things up. Again, they do not differentiate stop from destroy. Because of this, we must merge our init and start into Spring's init state, and then merge stop and destroy into Spring's close state. Recall our AbstractService#destory will auto invoke stop if it hasn't already done so. So this is trick that we need to understand in order to use Spring effectively. Running services with JEE app server In a corporate env, we usually do not have the freedom to run what we want as a stand-alone program. Instead they usually have some infrustructure and stricter standard technology stack in place already, such as using a JEE application server. In these situation, the most portable way to run POJO services is in a war web application. In a Servlet web application, you can write a class that implements javax.servlet.ServletContextListener and this will provide you the life-cycles hook via contextInitialized and contextDestroyed. In there, you can instanciate your ServiceContainer object and call start and destroy methods accordingly. Here is an example that you can explore: package servicedemo; import java.util.*; import javax.servlet.*; public class ServiceContainerListener implements ServletContextListener { private static Logger logger = LoggerFactory.getLogger(ServiceContainerListener.class); private ServiceContainer serviceContainer; public void contextInitialized(ServletContextEvent sce) { serviceContainer = new ServiceContainer(); List services = createServices(); serviceContainer.setServices(services); serviceContainer.start(); logger.info(serviceContainer + " started in web application."); } public void contextDestroyed(ServletContextEvent sce) { serviceContainer.destroy(); logger.info(serviceContainer + " destroyed in web application."); } private List createServices() { List result = new ArrayList(); // populate services here. return result; } } You may configure above in the WEB-INF/web.xml like this: servicedemo.ServiceContainerListener The demo provided a placeholder that you must add your services in code. But you can easily make that configurable using the web.xml for context parameters. If you were to use Spring inside a Servlet container, you may directly use their org.springframework.web.context.ContextLoaderListener class that does pretty much same as above, except they allow you to specify their xml configuration file using the contextConfigLocation context parameter. That's how a typical Spring MVC based application is configure. Once you have this setup, you can experiment our POJO service just as the Spring xml sample given above to test things out. You should see our service in action by your logger output. PS: Actually what we described here are simply related to Servlet web application, and not JEE specific. So you can use Tomcat server just fine as well. The importance of Service's life-cycles and it's real world usage All the information I presented here are not novelty, nor a killer design pattern. In fact they have been used in many popular open source projects. However, in my past experience at work, folks always manage to make these extremely complicated, and worse case is that they completely disregard the importance of life-cycles when writing services. It's true that not everything you going to write needs to be fitted into a service, but if you find the need, please do pay attention to them, and take good care that they do invoked properly. The last thing you want is to exit JVM without clean up in services that you allocated precious resources for. These would become more disastrous if you allow your application to be dynamically reloaded during deployment without exiting JVM, in which will lead to system resources leakage. The above Service practice has been put into use in the TimeMachine project. In fact, if you look at the timemachine.scheduler.service.SchedulerEngine, it would just be a container of many services running together. And that's how user can extend the scheduler functionalities as well, by writing a Service. You can load these services dynamically by a simple properties file.

these days i’m playing with backbone and using public api as a source. the web browser has one horrible feature: it don’t allow you to fetch any external resource to our host due to the cross-origin restriction. for example if we have a server at localhost we cannot perform one ajax request to another host different than localhost. nowadays there is a header to allow it: access-control-allow-origin . the problem is that the remote server must set up this header. for example i was playing with github’s api and github doesn’t have this header. if the server is my server, is pretty straightforward to put this header but obviously i’m not the sysadmin of github, so i cannot do it. what the solution? one possible solution is, for example, create a proxy server at localhost with php. with php we can use any remote api with curl (i wrote about it here and here for example). it’s not difficult, but i asked myself: can we create a dummy proxy server with php to handle any request to localhost and redirects to the real server, instead of create one proxy for each request?. let’s start. problably there is one open source solution (tell me if you know it) but i’m on holidays and i want to code a little bit (i now, it looks insane but that’s me ). the idea is: ... $proxy->register('github', 'https://api.github.com'); ... and when i type: http://localhost/github/users/gonzalo123 and create a proxy to : https://api.github.com/users/gonzalo123 the request method is also important. if we create a post request to localhost we want a post request to github too. this time we’re not going to reinvent the wheel, so we will use symfony componets so we will use composer to start our project: we create a conposer.json file with the dependencies: { "require": { "symfony/class-loader":"dev-master", "symfony/http-foundation":"dev-master" } } now php composer.phar install and we can start coding. the script will look like this: register('github', 'https://api.github.com'); $proxy->run(); foreach($proxy->getheaders() as $header) { header($header); } echo $proxy->getcontent(); as we can see we can register as many servers as we want. in this example we only register github. the application only has two classes: restproxy , who extracts the information from the request object and calls to the real server through curlwrapper . request = $request; $this->curl = $curl; } public function register($name, $url) { $this->map[$name] = $url; } public function run() { foreach ($this->map as $name => $mapurl) { return $this->dispatch($name, $mapurl); } } private function dispatch($name, $mapurl) { $url = $this->request->getpathinfo(); if (strpos($url, $name) == 1) { $url = $mapurl . str_replace("/{$name}", null, $url); $querystring = $this->request->getquerystring(); switch ($this->request->getmethod()) { case 'get': $this->content = $this->curl->doget($url, $querystring); break; case 'post': $this->content = $this->curl->dopost($url, $querystring); break; case 'delete': $this->content = $this->curl->dodelete($url, $querystring); break; case 'put': $this->content = $this->curl->doput($url, $querystring); break; } $this->headers = $this->curl->getheaders(); } } public function getheaders() { return $this->headers; } public function getcontent() { return $this->content; } } the restproxy receive two instances in the constructor via dependency injection (curlwrapper and request). this architecture helps a lot in the tests , because we can mock both instances. very helpfully when building restproxy. the restproxy is registerd within packaist so we can install it using composer installer: first install componser curl -s https://getcomposer.org/installer | php and create a new project: php composer.phar create-project gonzalo123/rest-proxy proxy if we are using php5.4 (if not, what are you waiting for?) we can run the build-in server cd proxy php -s localhost:8888 -t www/ now we only need to open a web browser and type: http://localhost:8888/github/users/gonzalo123 the library is very minimal (it’s enough for my experiment) and it does’t allow authorization. of course full code is available in github .

First, read this. Why passwords have never been weaker—and crackers have never been stronger. There are numerous important lessons in this article. One of the small lessons is that changing your password every sixty or ninety days is farcical. The rainbow table algorithms can crack a badly-done password in minutes. Every 60 days, the cracker has to spend a few minutes breaking your new password. Why bother changing it? It only annoys the haxorz; they'll be using your account within a few minutes. However. That practice is now so ingrained that it's difficult to dislodge from the heads of security consultants. The big lesson, however, is profound. Work Experience Recently, I got a request from a developer on how to encrypt a password. We have a Python back-end and the developer was asking which crypto package to download and how to install it. "Crypto?" I asked. "Why do we need crypto?" "To encrypt passwords," they replied. I spat coffee on my monitor. I felt like hitting Caps Lock in the chat window so I could respond like this: "NEVER ENCRYPT A PASSWORD, YOU DOLT." I didn't, but I felt like it. Much Confusion The conversation took hours. Chat can be slow that way. Also, I can be slow because I need to understand what's going on before I reply. I'm a slow thinker. But the developer also needed to try stuff and provide concrete code examples, which takes time. At the time, I knew that passwords must be hashed with salt. I hadn't read the Ars Technica article cited above, so I didn't know why computationally intensive hash algorithms are best for this. We had to discuss hash algorithms. We had to discuss algorithms for generating unique salt. We had to discuss random number generators and how to use an entropy source for a seed. We had to discuss http://www.ietf.org/rfc/rfc2617.txt in some depth, since the algorithms in section 3.2.2. show some best practices in creating hash summaries of usernames, passwords, and realms. All of this was, of course, side topics before we got to the heart of the matter. What's Been Going On After several hours, my "why" questions started revealing things. The specific user story, for example, was slow to surface. Why? Partly because I didn't demand it early enough. But also, many technology folks will conceive of a "solution" and pursue that technical concept no matter how difficult or bizarre. In some cases, the concept doesn't really solve the problem. I call this the "Rat Holes of Lost Time" phenomena: we chase some concept through numerous little rat-holes before we realize there's a lot of activity but no tangible progress. There's a perceptual narrowing that occurs when we focus on the technology. Often, we're not actually solving the problem. IT people leap past the problem into the solution as naturally as they breathe. It's a hard habit to break. It turned out that they were creating some additional RESTful web services. They knew that the RESTful requests needed proper authentication. But, they were vague on the details of how to secure the new RESTful services. So they were chasing down their concept: encrypt a password and provide this encrypted password with each request. They were half right, here. A secure "token" is required. But an encrypted password is a terrible token. Use The Framework, Luke What's most disturbing about this is the developer's blind spot. For some reason, the existence of other web services didn't enter into this developer's head. Why didn't they read the code for the services created on earlier sprints? We're using Django. We already have a RESTful web services framework with a complete (and high quality) security implementation. Nothing more is required. Use the RESTful authentication already part of Django. In most cases, HTTPS is used to encrypt at the socket layer. This means that Basic Authentication is all that's required. This is a huge simplification, since all the RESTful frameworks already offer this. The Django Rest Framework has a nice authentication module. When using Piston, it's easy to work with their Authentication handler. It's possible to make RESTful requests with Digest Authentication, if SSL is not being used. For example, Akoha handles this. It's easy to extend a framework to add Digest in addition to Basic authentication. For other customers, I created an authentication handler between Piston and ForgeRock OpenAM so that OpenAM tokens were used with each RESTful request. (This requires some care to create a solution that is testable.) Bottom Lines Don't encrypt passwords. Ever. Don't write your own hash and salt algorithm. Use a framework that offers this to you. Read the Ars Technica article before doing anything password-related.

Today I learned a neat trick to organize imports in Eclipse. Of course, one can use Ctrl + Shift + O to remove the unused imports at file level. But what if you want to remove the unused imports for several files, may be at package level? Simple – in the Package Explorer window, right click on the package that you want to modify and then select source -> Organize Imports which will analyse all the files inside that package and then remove the unused imports. One more nifty trick is that you can automatically organize the imports when you save the file. To enable this, go to Windows -> Preferences -> Java -> Editor -> Save Actions and then enable Perform the selected action on save -> Organize imports. After this, whenever you save a java file, eclipse will remove the unused imports automatically.

The more I use dependency injection (DI) in my code, the more it alters the way I see both my design and implementation. Injection is so convenient and powerful that you end up wanting to make sure you use it as often as you can. And as it turns out, you can use it in many, many places. Let’s cover briefly the most obvious scenarios where DI, and more specifically, Guice, are a good fit: objects created either at class loading time or very early in your application. These two aspects are covered by either direct injection or by providers, which allow you to start building some of your object graph before you can inject more objects. I won’t go too much in details about these two use cases since they are explained in pretty much any Guice tutorial you can find on the net. Once the injector has created your graph of objects, you are pretty much back to normal and instantiating your “runtime objects” (the objects you create during the life time of your application) the normal way, most likely with “new” or factories. However, you will quickly start noticing that you need some runtime information to create these objects, other parts of them could be injected. Let’s take the following example: we have a GeoService interface that provides various geolocation functions, such as telling you if two addresses are close to each other: public interface GeoService { /** * @return true if the two addresses are within @param{miles} * miles of each other. */ boolean isNear(Address address1, Address address2, int miles); } Then you have a Person class which uses this service and also needs a name and an address to be instantiated: public class Person { // Fields omitted public Person(String name, Address address, GeoService gs) { this.name = name; this.address = address; this.geoService = gs; } public boolean livesNear(Person otherPerson) { return geoService.isNear(address, otherPerson.getAddress(), 2 /* miles */); } } Something odd should jump at you right away with this class: while name and address are part of the identity of a Person object, the presence of the GeoService instance in it feels wrong. The service is a singleton that is created on start up, so a perfect candidate to be injected, but how can I achieve the creation of a Person object when some of its information is supplied by Guice and the other part by myself? Guice gives you a very elegant and flexible way to implement this scenario with “assisted injection”. The first step is to define a factory for our objects that represents exactly how we want to create them: public interface PersonFactory { Person create(String name, Address address); } Since only name and address participate in the identity of our Person objects, these are the only parameters we need to construct our objects. The other parameters should be supplied by Guice so we modify our Person constructor to let Guice know: @Inject public Person(@Assisted String name, @Assisted Address address, GeoService geoService) { this.name = name; this.address = address; this.geoService = geoService; } In this code, I have added an @Inject annotation on the constructor and an @Assisted annotation on each parameter that I will be providing. Guice will take care of injecting the rest. Finally, we connect the factory to its objects when creating the module: Module module1 = new FactoryModuleBuilder() .build(PersonFactory.class); The important part here is to realize that we will never instantiate PersonFactory: Guice will. From now on, all we need to do whenever we want to instantiate a Person object is to ask Guice to hand us a factory: @Inject private PersonFactory personFactory; // ... Person p = personFactory.create("Bob", new Address("1 Ocean st")); If you want to find out more, take a look at the main documentation for assisted injection, which explains how to support overloaded constructors and also how to create different kinds of objects within the same factory. Wrapping up Let’s take a look at what we did. First, we started with a suspicious looking constructor: public Person(String name, Address address, GeoService s) { This constructor is suspicious because it accepts parameters that do not participate in the identity of the object (you won’t use the GeoService parameter when calculating the hash code of a Person object). Instead, we replaced this constructor with a factory that only accepts identity fields: public interface PersonFactory { Person create(String name, Address address); } and we let Guice’s assisted injection take care of creating a fully formed object for us. This observation leads us to the Identity Constructor rule: If a constructor accepts parameters that are not used to define the identity of the objects, consider injecting these parameters. Once you start looking at your objects with this rule in mind, you will be surprised to find out how many of them can benefit from assisted injection.