rate this eclipse is good, but like any other tool: it gets better after i have it customized for my special needs. eclipse stores a lot of settings in the workspace, see my article about copy my workspace settings . but is there a way to apply some settings to every workspace? at least to the new ones? because importing/exporting the settings can get really tedious as i have many workspace. and indeed, there are global settings in eclipse. and i want to have them changed… warning: changing eclipse global preferences might break an eclipse installation. so better have a backup of the changed files at hand! i’m using here the eclipse based codewarrior for mcu10.2 , but things are pretty much the same for any eclipse based product (see the documentation in defining your own global preferences ). question: where are the global preferences stored? the first thing to check is the eclipse\configuration\.settings folder: here some plugins store their global preferences. for example: the recent workspace settings are in org.eclipse.ui.ide.prefs. #fri apr 06 16:46:14 cest 2012 recent_workspaces_protocol=3 max_recent_workspaces=10 show_workspace_selection_dialog=true eclipse.preferences.version=1 recent_workspaces=c\:\\tmp\\wsp_test\nc\:\\tmp\\wsp_10.2 but what about all the other settings? looking at the codewarrior installation, inside the eclipse folder, i find the cwide.ini file. cwide.ini file this file defines the eclipse startup options for launching the ide (cwide.exe for codewarrior). the interesting part is this line: -declipse.plugincustomization=cwide.properties this tells eclipse to use the cwide.properties as a default configuration file. if i inspect that file, it has the following content: org.eclipse.debug.ui/org.eclipse.debug.ui.switch_perspective_on_suspend=always org.eclipse.debug.ui/org.eclipse.debug.ui.switch_to_perspective=always org.eclipse.ui.editors/spellingengine=org.eclipse.cdt.internal.ui.text.spelling.cspellingengine ok, that gives me an idea how settings could look like. but the question is: how to know the settings and syntax? what works (most of the time) is following approach: launch eclipse with a new workspace export the settings using file > export > general > preferences to a file change the setting in window > preferences export the settings using file > export > general > preferences to a different file compare/inspect the exported information and find the settings apply the settings to the cwide.properties file, without the /instance/ part restart the ide and check if it works with a new workspace the last check is necessary as not all settings might work that way, see this forum post . this is maybe best illustrated with an example. i have configured my workspace to use 2 for tab width and to insert spaces for tabs: changed preferences for tabs if i compare the two exported .epf files, this gives me: diffing eclipse preference files that means the two following lines are configuring what i have changed: /instance/org.eclipse.ui.editors/tabwidth=2 /instance/org.eclipse.ui.editors/spacesfortabs=true for the cwide.properties file i need to cut off the /instance/ part, so i have this added to the cwide.properties : # set tab width to 2 org.eclipse.ui.editors/tabwidth=2 # using spaces for tabs org.eclipse.ui.editors/spacesfortabs=true note: preferences are applied in following order: global preferences, then local (workspace) preferences this does not overwrite an existing setting of my workspace. as i can see from above diff, my initial workspace settings do not have any settings for tabwidth and spacesfortabs. creating a new workspace use and apply my new settings. but once i have the them, they will not be overwritten with new global ones. which makes sense: the local settings are winning. note: post a comment if you know an elegant way how to enforce/overwrite workspace settings with global ones.

The subject of build dependencies is neither a trivial nor a minor one. Various build tools approach this subject from different perspectives contributing various solutions, each with its own strengths and weaknesses. Maven and Gradle users who are familiar with release and snapshot dependencies may not know about TeamCity snapshot dependencies or assume they’re somehow related to Maven (which isn’t true). TeamCity users who are familiar with artifact and snapshot dependencies may not know that adding an Artifactory plugin allows them to use artifact and build dependencies as well, on top of those provided by TeamCity. Some of the names mentioned above seem not to be established enough while others may require a discussion about their usage patterns. Having this in mind I’ve decided to explore each solution in its own blog post, setting a goal of providing enough information so that people can choose what works best. The first post explored Maven snapshot and release dependencies. This is the second post, which covers artifact and snapshot dependencies provided by TeamCity and the third and final part will cover the artifact and build dependencies provided by TeamCity Artifactory plugin. Non-Maven Dependencies While Maven-based dependencies management and artifact repositories are very common and widespread in Java, there are cases where you may still find them insufficient or inadequate for your needs. For starters, you may not be developing in Java or perhaps your build tool is not providing built-in integration with Maven repositories, as is the case with Ant (or its Gant and NAnt spin-offs), SCons, Rake or MSBuild. Secondly, snapshot Maven dependencies provide their own set of challenges covered in the previous blog post, making it harder to ensure correct snapshot dependency is used in a chain of builds. In order to address these scenarios, TeamCity provides two ways to connect dependent build configurations and their outcomes: artifact and snapshot dependencies. TeamCity Artifact Dependencies The idea of artifact dependencies in TeamCity is very simple: download the artifacts produced by an other build before the current one begins. After the artifacts are downloaded to the folder specified (checkout directory by default), your build script can use them to achieve its goals. You can find configuration details in TeamCity documentation. Naturally, this scheme is not suitable for build tools with automatic dependencies management, but it works well with build or shell scripts accepting and expecting local paths, relative to the checkout directory. Note that the copying works not only for the produced build binaries, but for any kind of binary or text files, like the TeamCity coverage report as demonstrated on the screenshot above. There is one important detail about specifying artifact dependencies and that is “Get artifacts from” configuration where you specify what type of build should files be taken from. Possible values of this field are “last successful”, “finished”, “pinned”, or “tagged build”, as well as the build number or “Build from the same chain”. While most values should be trivial to understand with “Last successful build” being the default and generally suitable option, the definition of “same chain” build is directly related to TeamCity snapshot dependencies. TeamCity Snapshot Dependencies Imagine a monolithic multi-step build process (build, test, package, deploy) which you decide to split into multiple smaller builds, invoked sequentially, forming a chain of executions. Doing so allows one to configure or trigger every chain step separately and run certain steps in parallel in order to speedup the process (like executing tests or building independent components). Most of all, it makes the overall maintenance significantly easier. However, while doing so you need to ensure every chain step uses the same consistent set of sources pulled from VCS even if newer commits are made all the while chain steps are running. That’s what TeamCity snapshot dependencies are for: they connect several build configurations into a single chain of execution, called build chain, with every step using the same set of sources, regardless of VCS updates. Note that the TeamCity use of the term “snapshot dependencies” may confuse people familiar with Maven snapshot dependencies which are two unrelated concepts. Snapshot dependencies are configured similarly to artifact dependencies. You can find configuration details in TeamCity documentation. Using Artifact and Snapshot Dependencies Together When applicable, it is recommended to define both kinds of dependencies between build configurations, as this ensures not only a consistent set of sources used throughout a chain steps but also a consistent flow of artifacts produced. Now the definition of “Build from the same chain” in artifact dependency mentioned above becomes clear, as this is the only meaningful option in this scenario. In a way, you can think of build chain steps running in isolation from VCS updates after the first sources’ “snapshot” is taken. Chain artifacts are either re-created from the same sources or passed through chain steps with artifact dependencies. This makes chain steps consistent, reproducible and always up-to-date (when applied to using chain artifacts), something that can’t be easily achieved with Maven snapshot dependencies. Build Chains Visibility in TeamCity 7.0 TeamCity 7.0 took the notion of build chains to a whole new level by providing build chains a new UI, making chain steps visible and re-runnable. Once you have snapshot dependencies defined, a new “Build Chains” tab appears in project reports, providing a visual representation of all related build chains and a way to re-run any chain step manually, using the same set of sources pulled originally. Build Chain Triggering Having build configurations connected with snapshot dependencies and, therefore, their builds grouped into build chains not only makes them more consistent regarding the sources used, it also impacts the way builds are added to the build queue: after a certain chain step is triggered, the default behavior is to add all preceding chain steps as well, keeping their respective order, in addition to the one that was triggered initially. Let me repeat it for more clarity: triggering certain chain configuration adds preceding (those to the left of it) and not subsequent (to the right of it) configurations to the build queue, although it may seem counterintuitive at first. The idea is to mark the location where chain execution stops, which is exactly the configuration that was triggered initially; it becomes the last execution step. To trigger subsequent chain steps upon VCS changes found in a chain configuration, you can add a VCS trigger with the “Trigger on changes in snapshot dependencies” option to the configuration that would be the last execution step. This configuration is then triggered whenever any of the preceding chain steps is updated, which schedules the whole chain for execution. Having this behavior in mind, you therefore need to decide which configurations are triggered automatically and which should be run manually. Usually, earlier chain steps having no impact on external environment can be triggered automatically by VCS trigger but final chain steps, potentially modifying external systems, are invoked manually after a human verification of the previous chain results. The process of running the final chain steps manually is usually referred to as “promoting” previously finished builds. Sample Build Chain: Compile, Test, Deploy Imagine three sample build configurations, "Compile", "Test" and "Deploy" connected into a build chain: "Deploy" is snapshot dependent on "Test" which is snapshot dependent on "Compile". In this sample scenario the "Compile" and "Test" configurations are triggered automatically while "Deploy" is triggered manually, following the recommendations given above. VCS changes in "Compile" configuration only trigger an execution of this chain step, while VCS changes in "Test" configuration trigger "Compile" and "Test" execution (in that order). Once a "Compile" configuration is added to the builds queue, its sources’ timestamp is recorded on the server to be used in all subsequent chain steps. If any of the chain steps is connected to a different VCS root, its sources are also pulled according to the same timestamp. Promoting Finished Builds As soon as the automatic chain execution stops (after running "Test"), you can continue it by clicking the corresponding “Run” button on the "Deploy" configuration that was not triggered (see the build chain screenshot above). Alternatively, it is possible to promote a finished "Test" build through its “Build Actions” and invoke configurations which are snapshot dependent on it – "Deploy" configuration in this case. Summary This article has provided an overview of TeamCity artifact and snapshot dependencies, build chains, how their steps are triggered and how finished builds are promoted. I hope you now have a good understanding of how it works and of when it is appropriate (or not) to use TeamCity build dependencies in addition to those provided by build tools such as Maven. Please, refer to the TeamCity documentation for more information about this subject: Dependent Build Build Chain The final blog post in the series will uncover how you can use the TeamCity Artifactory plugin in order to achieve a behavior which is similar to build chains for projects with Maven-based dependency management. Stay tuned!

i recently answered this question in stackoverflow : what is an utility tree and what is it’s purpose in case of architecture tradeoff analysis method(atam)? i did answer the question there but here’s a better explanation with lots of examples based on the initial version for chapter 1 of soa patterns (which didn’t make it into the final version of the book). there are two types of requirements for software projects: functional and non-functional requirements. functional requirements are the requirements for what the solution must do (which are usually expressed as use cases or stories). the functional requirements are what the users (or systems) that interact with the system do with the system (fill in an order, update customer details, authorize a loan etc.). non-functional requirements are attributes the system is expected to have or manifest. these usually include requirements in areas such as performance, security, availability etc. a better name for non-functional requirements is “quality attributes” . below are some formal definitions from ieee standad 1061 “standard for a software quality metrics methodology” for quality attributes and related terms: quality attribute a characteristic of software, or a generic term applying to quality factors, quality subfactors, or metric values. quality factor a management-oriented attribute of software that contributes to its quality. quality subfactor a decomposition of a quality factor or quality subfactor to its technical components. metric value a metric output or an element that is from the range of a metric. software quality metric a function whose inputs are software data and whose output is a single numerical value that can beinterpreted as the degree to which software possesses a given attribute that affects its quality. most of the requirements that drive the design of a software architecture comes from system’s quality attributes. the reason for this is that that the effect of quality attributes is usually system-wide (e.g. you wouldn’t want your system to have good performance only in the ui – you want the system to perform well no matter what) – which is exactly what software architecture is concerned with. note however, that few requirements might still come from functional requirements) [1] . the question is how do we find out what those requirements are? the answer to that is also in the software architecture definition. the source for quality attributes are the stakeholders. so what or who are these “stakeholders”? well, a stakeholder is just about anyone who has a vested interest in the project. a typical system has a lot of stakeholders starting from the (obvious) customer, the end-users (those people in the customer organization/dept that will actually use the software) and going to the operations personnel (it – those who will have to keep the solution running), the development team, testers, maintainers, management. in some systems the stakeholders can even be the shareholders or even the general public (imagine for example, that you build a new dispatch system for a 911 center). one of the architect’s roles is to analyze the quality attributes and define an architecture that will enable delivering all the functional requirements while supporting the quality attributes. as can be expected ,sometimes quality attributes are in conflict with each other – the most obvious examples are performance vs. security or flexibility vs. simplicity and the architect’s role is to strike a balance between the different quality attributes (and the stakeholders) to make sure the overall quality of the system is maximized. contextual solutions (e.g. patterns) can be devised to solve specific quality attributes need. however saying that a system needs to have “good performance” or that it needs to be “testable” doesn’t really help us know what to do. in order for us to be able to discern which patterns apply to specific quality attribute , we need a better understanding of quality attributes besides the formal definition, something that is more concrete. the way to get that concrete understanding of the effect of quality attributes is to use scenarios. scenarios are short, “user story”-like proses that demonstrate how a quality attribute is manifested in the system using a functional situation quality attributes scenarios originated as a way to evaluate software architecture. the software engineering institute developed several evaluation methodologies, like architecture tradeoff analysis method (clements, kazman and klein, 2002) that heavily build on scenarios to contrast and compare how the different quality attributes are met by candidate architectures. atam (and similar evaluation methods like laaam which is part of msf 4.0) suggest building a “utility tree” which represent the overall usefulness of the system. the scenarios serve as the leafs of the utility tree and the architecture is evaluated by considering how the architecture makes the scenarios possible. i found that using scenarios and the utility tree approach early in the design of the architecture (see writings about saf ) can greatly enhance the quality of the architecture that is produced. when you examine the scenarios you can also prioritize them and better balance conflicting attributes. the scenarios can be used as an input to make sure the quality attributes are actually met. furthermore you can use the scenarios to help identify the strategies or patterns applicable to make the scenarios possible (and thus ensure the quality attributes are met) within the system. we usually group scenarios into a “utility tree” which is a representation of the total usefulness (“utility”) of a system . as you can see in the diagram below we have the key quality attributes (performance, security etc.). each of the quality attributes has sub categories (e.g. performance is broken into latency, data loss etc.). each sub category is demonstrated by a scenario that we expect the system to manifest. the tree representation helps get the whole picture but the important bits here are the scenarios so let’s explore them some more. scenarios are expressed as statements that have 3 parts: a stimulus , a context and a response . the stimulus is the action taken (by the system / user/ other system / any other person); response is how the system is expected to behave when the stimulus occur, and the context specifies the environment or conditions under which we expect the to get the response. for example in the following scenario: “when you perform a database operation , under normal condition, it should take less than 100 miliseconds.” “under normal condition” is the context “when you perform a database operation” is the stimulus “it should take less than 100 millisecond” is the response expected from the system. here are a couple of additional examples for quality attribute scenarios: performance –>latency -> under normal conditions a client consuming multiple services should have latency less than 5 seconds. security->authentications -> under all conditions, any call to a service should be authenticated using x.509 certificate you can also check out this document for a few more scenario examples from a system i worked on in the past [1] design has the ratios reversed i.e. most of the requirements for design come from functional requirements and a few requirements might come from the quality attributes. illustration by epsos.de

Learn how Apache Camel implements the EIPs and offers a standardized, internal domain-specific language (DSL) to integrate applications.

HMCA is a great tool for protecting a REST service. Read why.

This article will describe the complete root cause analysis of a recent Java deadlock problem observed from a Weblogic 11g production system running on the IBM JVM 1.6.This case study will also demonstrate the importance of mastering Thread Dump analysis skills; including for the IBM JVM Thread Dump format. Environment specification Java EE server: Oracle Weblogic Server 11g & Spring 2.0 OS: AIX 5.3 Java VM: IBM JRE 1.6.0 Platform type: Portal & ordering application Monitoring and troubleshooting tools JVM Thread Dump (IBM JVM format) Compuware Server Vantage (Weblogic JMX monitoring & alerting) Problem overview A major stuck Threads problem was observed & reported from Compuware Server Vantage and affecting 2 of our Weblogic 11g production managed servers causing application impact and timeout conditions from our end users. Gathering and validation of facts As usual, a Java EE problem investigation requires gathering of technical and non-technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause: · What is the client impact? MEDIUM (only 2 managed servers / JVM affected out of 16) · Recent change of the affected platform? Yes (new JMS related asynchronous component) · Any recent traffic increase to the affected platform? No · How does this problem manifest itself? A sudden increase of Threads was observed leading to rapid Thread depletion · Did a Weblogic managed server restart resolve the problem? Yes, but problem is returning after few hours (unpredictable & intermittent pattern) - Conclusion #1: The problem is related to an intermittent stuck Threads behaviour affecting only a few Weblogic managed servers at the time - Conclusion #2: Since problem is intermittent, a global root cause such as a non-responsive downstream system is not likely Thread Dump analysis – first pass The first thing to do when dealing with stuck Thread problems is to generate a JVM Thread Dump. This is a golden rule regardless of your environment specifications & problem context. A JVM Thread Dump snapshot provides you with crucial information about the active Threads and what type of processing / tasks they are performing at that time. Now back to our case study, an IBM JVM Thread Dump (javacore.xyz format) was generated which did reveal the following Java Thread deadlock condition below: 1LKDEADLOCK Deadlock detected !!! NULL --------------------- NULL 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012CC08B00) 3LKDEADLOCKWTR is waiting for: 4LKDEADLOCKMON sys_mon_t:0x0000000126171DF8 infl_mon_t: 0x0000000126171E38: 4LKDEADLOCKOBJ weblogic/jms/frontend/FESession@0x07000000198048C0/0x07000000198048D8: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012E560500) 3LKDEADLOCKWTR which is waiting for: 4LKDEADLOCKMON sys_mon_t:0x000000012884CD60 infl_mon_t: 0x000000012884CDA0: 4LKDEADLOCKOBJ weblogic/jms/frontend/FEConnection@0x0700000019822F08/0x0700000019822F20: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012CC08B00) This deadlock situation can be translated as per below: - Weblogic Thread #8 is waiting to acquire an Object monitor lock owned by Weblogic Thread #10 - Weblogic Thread #10 is waiting to acquire an Object monitor lock owned by Weblogic Thread #8 Conclusion: both Weblogic Threads #8 & #10 are waiting on each other; forever! Now before going any deeper in this root cause analysis, let me provide you a high level overview on Java Thread deadlocks. Java Thread deadlock overview Most of you are probably familiar with Java Thread deadlock principles but did you really experience a true deadlock problem? From my experience, true Java deadlocks are rare and I have only seen ~5 occurrences over the last 10 years. The reason is that most stuck Threads related problems are due to Thread hanging conditions (waiting on remote IO call etc.) but not involved in a true deadlock condition with other Thread(s). A Java Thread deadlock is a situation for example where Thread A is waiting to acquire an Object monitor lock held by Thread B which is itself waiting to acquire an Object monitor lock held by Thread A. Both these Threads will wait for each other forever. This situation can be visualized as per below diagram: Thread deadlock is confirmed…now what can you do? Once the deadlock is confirmed (most JVM Thread Dump implementations will highlight it for you), the next step is to perform a deeper dive analysis by reviewing each Thread involved in the deadlock situation along with their current task & wait condition.Find below the partial Thread Stack Trace from our problem case for each Thread involved in the deadlock condition: ** Please note that the real application Java package name was renamed for confidentiality purposes ** Weblogic Thread #8 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000012CC08B00, j9thread_t:0x00000001299E5100, java/lang/Thread:0x070000001D72EE00, state:B, prio=1 (native thread ID:0x111200F, native priority:0x1, native policy:UNKNOWN) Java callstack: at weblogic/jms/frontend/FEConnection.stop(FEConnection.java:671(Compiled Code)) at weblogic/jms/frontend/FEConnection.invoke(FEConnection.java:1685(Compiled Code)) at weblogic/messaging/dispatcher/Request.wrappedFiniteStateMachine(Request.java:961(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.syncRequest(DispatcherImpl.java:184(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.dispatchSync(DispatcherImpl.java:212(Compiled Code)) at weblogic/jms/dispatcher/DispatcherAdapter.dispatchSync(DispatcherAdapter.java:43(Compiled Code)) at weblogic/jms/client/JMSConnection.stop(JMSConnection.java:863(Compiled Code)) at weblogic/jms/client/WLConnectionImpl.stop(WLConnectionImpl.java:843) at org/springframework/jms/connection/SingleConnectionFactory.closeConnection(SingleConnectionFactory.java:342) at org/springframework/jms/connection/SingleConnectionFactory.resetConnection(SingleConnectionFactory.java:296) at org/app/JMSReceiver.receive() …………………………………………………………………… Weblogic Thread #10 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000012E560500, j9thread_t:0x000000012E35BCE0, java/lang/Thread:0x070000001ECA9200, state:B, prio=1 (native thread ID:0x4FA027, native priority:0x1, native policy:UNKNOWN) Java callstack: at weblogic/jms/frontend/FEConnection.getPeerVersion(FEConnection.java:1381(Compiled Code)) at weblogic/jms/frontend/FESession.setUpBackEndSession(FESession.java:755(Compiled Code)) at weblogic/jms/frontend/FESession.consumerCreate(FESession.java:1025(Compiled Code)) at weblogic/jms/frontend/FESession.invoke(FESession.java:2995(Compiled Code)) at weblogic/messaging/dispatcher/Request.wrappedFiniteStateMachine(Request.java:961(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.syncRequest(DispatcherImpl.java:184(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.dispatchSync(DispatcherImpl.java:212(Compiled Code)) at weblogic/jms/dispatcher/DispatcherAdapter.dispatchSync(DispatcherAdapter.java:43(Compiled Code)) at weblogic/jms/client/JMSSession.consumerCreate(JMSSession.java:2982(Compiled Code)) at weblogic/jms/client/JMSSession.setupConsumer(JMSSession.java:2749(Compiled Code)) at weblogic/jms/client/JMSSession.createConsumer(JMSSession.java:2691(Compiled Code)) at weblogic/jms/client/JMSSession.createReceiver(JMSSession.java:2596(Compiled Code)) at weblogic/jms/client/WLSessionImpl.createReceiver(WLSessionImpl.java:991(Compiled Code)) at org/springframework/jms/core/JmsTemplate102.createConsumer(JmsTemplate102.java:204(Compiled Code)) at org/springframework/jms/core/JmsTemplate.doReceive(JmsTemplate.java:676(Compiled Code)) at org/springframework/jms/core/JmsTemplate$10.doInJms(JmsTemplate.java:652(Compiled Code)) at org/springframework/jms/core/JmsTemplate.execute(JmsTemplate.java:412(Compiled Code)) at org/springframework/jms/core/JmsTemplate.receiveSelected(JmsTemplate.java:650(Compiled Code)) at org/springframework/jms/core/JmsTemplate.receiveSelected(JmsTemplate.java:641(Compiled Code)) at org/app/JMSReceiver.receive() …………………………………………………………… As you can see in the above Thread Strack Traces, such deadlock did originate from our application code which is using the Spring framework API for the JMS consumer implementation (very useful when not using MDB’s). The Stack Traces are quite interesting and revealing that both Threads are in a race condition against the same Weblogic JMS consumer session / connection and leading to a deadlock situation: - Weblogic Thread #8 is attempting to reset and close the current JMS connection - Weblogic Thread #10 is attempting to use the same JMS Connection / Session in order to create a new JMS consumer - Thread deadlock is triggered! Root cause: non Thread safe Spring JMS SingleConnectionFactory implementation A code review and a quick research from Spring JIRA bug database did reveal the following Thread safe defect below with a perfect correlation with the above analysis: # SingleConnectionFactory's resetConnection is causing deadlocks with underlying OracleAQ's JMS connection https://jira.springsource.org/browse/SPR-5987 A patch for Spring SingleConnectionFactory was released back in 2009 which did involve adding proper synchronized{} block in order to prevent Thread deadlock in the event of a JMS Connection reset operation: synchronized (connectionMonitor) { //if condition added to avoid possible deadlocks when trying to reset the target connection if (!started) { this.target.start(); started = true; } } Solution Our team is currently planning to integrate this Spring patch in to our production environment shortly. The initial tests performed in our test environment are positive. Conclusion I hope this case study has helped understand a real-life Java Thread deadlock problem and how proper Thread Dump analysis skills can allow you to quickly pinpoint the root cause of stuck Thread related problems at the code level. Please don’t hesitate to post any comment or question.

When you manage and develop infrastructure, you'll work with tests and monitoring solutions that ensure the quality of your end product (code or infrastructure). For code quality, you can have unit tests, functional tests, and integration tests etc. Similarly, you might have system monitoring, dependancy monnitoring, application monitoring, BAM, CEP, etc. In this post I'll narrate few of them: System monitoring : Watches CPU load, free memory (RAM), disk space etc. SNMP based hardware monitoring, etc. Dependency monitoring : Checks web server processes, web server states , %CPU consumption, RSS, etc. Integration : Tracks third party or other integration points whether they are available or not. BAM : Business activity monitoring. Records KPI or key performance indicators, which will in turn define the state of your business (quantitatively). This could include sucessful transactions per day or month. Process instrumentation or tracing : Includes kprobes, system tap, or other tracing like methodologies like DTrace, which lets you monitor at the individual method level. These are predominantly used for language or other interpreter optimizations. Complex event processing : Though not directly related, some of these monitoring solutions can or should use some form of complex event processing to deduce meaningful information. This is only important (or even significant) if the volume of data is large. Depending upon your problem you should employ one of more of these solutions. There are plenty of open source solutions/tooling available for all of them. BAM is kinda tricky, not BAM in itself, but defining the KPI part is bit trippy.

Continuing on the theme of temporary storage for transient messages used within Spring Integration flows, the claim-check model offers configurable storage for message payloads. The advantage in using this Enterprise Integration pattern, compared against header enrichment, is that objects don't have to be packed into the header using a Header Enrichment technique. They can be stored in a local Java Map, an IMDB, cache or anything else that be used to hold data. Several advantages using this approach are evident. Firstly, performance and efficiency. When using header enrichment, if message payloads need to be managed outside of the JVM that generates the enriched message header, the object will not be available unless it's serialised and transported around the distributed application. This could be costly in terms of performance and transport efficiency. The key factor here is the frequency of remote dispatch and the size of the header object. In specific circumstances the claim-check pattern may offer an advantage here, objects can be serialised and/or transformed into a storage specific format and stored internally in memory or externally in a data store. Secondly, accessibility. It's conceivable that message payloads undergoing claim-check processing may need to be accessed by third party applications that are unable to receive Spring Integration messages. The claim-check pattern allows this type of processing to take place. Thirdly, resiliency is offered. A data store can be chosen that guarantees persistence for messages in order that they can be recovered following failure. The following code details how the claim-check pattern can be used: The gateway used is specified as the following Java class: package com.l8mdv.sample; import org.springframework.integration.Message; import org.springframework.integration.annotation.Gateway; public interface ClaimCheckGateway { public static final String CLAIM_CHECK_ID = "CLAIM_CHECK_ID"; @Gateway (requestChannel = "claim-check-in-channel") public Message send(Message message); } Lastly, this can all be tested by using the following JUnit test case: package com.l8mdv.sample; import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.integration.Message; import org.springframework.integration.support.MessageBuilder; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import static com.l8mdv.sample.ClaimCheckGateway.CLAIM_CHECK_ID; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration( locations = {"classpath:META-INF/spring/claim-check.xml"} ) public class ClaimCheckIntegrationTest { @Autowired ClaimCheckGateway claimCheckGateway; @Test public void locatePayloadInHeader() { String payload = "Sample test message."; Message message = MessageBuilder.withPayload(payload).build(); Message response = claimCheckGateway.send(message); Assert.assertTrue(response.getPayload().equals(payload)); Assert.assertTrue(response.getHeaders().get(CLAIM_CHECK_ID) != null); } }

Looking for the best Eclipse shortcuts? Here are the top 10.

A few weeks ago, I co-authored an article (with my colleague Rama Ramani) about how the Screen Actors Guild Awards website migrated its Drupal deployment from LAMP to Windows Azure: Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure. Since then, Rama and another colleague, Jason Roth, have been working on writing up how the SAG Awards website was managed and monitored in Windows Azure. The article below is the fruit of their work…a very interesting/educational read. Overview Drupal is an open source content management system that runs on PHP. Windows Azure offers a flexible platform for hosting, managing, and scaling Drupal deployments. This paper focuses on an approach to host Drupal sites on Windows Azure, based on learning from a BPD Customer Programs Design Win engagement with the Screen Actors Guild Awards Drupal website. This paper covers guidelines and best practices for managing an existing Drupal web site in Windows Azure. For more information on how to migrate Drupal applications to Windows Azure, see Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure. The target audience for this paper is Drupal administrators who have some exposure to Windows Azure. More detailed pointers to Windows Azure content is provided throughout the paper as links. Drupal Application Architecture on Windows Azure Before reviewing the management and monitoring guidelines, it is important to understand the architecture of a typical Drupal deployment on Windows Azure. First, the following diagram displays the basic architecture of Drupal running on Windows and IIS7. In the Windows Server scenario, you could have one or more machines hosting the web site in a farm. Those machines would either persist the site content to the file system or point to other network shares. For Windows Azure, the basic architecture is the same, but there are some differences. In Windows Azure the site is hosted on a web role. A web role instance is hosted on a Windows Server 2008 virtual machine within the Windows Azure datacenter. Like the web farm, you can have multiple instances running the site. But there is no persistence guarantee for the data on the file system. Because of this, much of the shared site content should be stored in Windows Azure Blob storage. This allows them to be highly available and durable. Usually, a large portion of the site caters to static content which lends well to caching. And caching can be applied in a set of places – browser level caching, CDN to cache content in the edge closer to the browser clients, caching in Azure to reduce the load on backend, etc. Finally, the database can be located in SQL Azure. The following diagram shows these differences. For monitoring and management, we will look at Drupal on Windows Azure from three perspectives: Availability: Ensure the web site does not go down and that all tiers are setup correctly. Apply best practices to ensure that the site is deployed across data centers and perform backup operations regularly. Scalability: Correctly handle changes in user load. Understand the performance characteristics of the site. Manageability: Correctly handle updates. Make code and site changes with no downtime when possible. Although some management tasks span one or more of these categories, it is still helpful to discuss Drupal management on Windows Azure within these focus areas. Availability One main goal is that the Drupal site remains running and accessible to all end-users. This involves monitoring both the site and the SQL Azure database that the site depends on. In this section, we will briefly look at monitoring and backup tasks. Other crossover areas that affect availability will be discussed in the next section on scalability. Monitoring With any application, monitoring plays an important role with managing availability. Monitoring data can reveal whether users are successfully using the site or whether computing resources are meeting the demand. Other data reveals error counts and possibly points to issues in a specific tier of the deployment. There are several monitoring tools that can be used. The Windows Azure Management Portal. Windows Azure diagnostic data. Custom monitoring scripts. System Center Operations Manager. Third party tools such as Azure Diagnostics Manager and Azure Storage Explorer. The Windows Azure Management Portal can be used to ensure that your deployments are successful and running. You can also use the portal to manage features such as Remote Desktop so that you can directly connect to machines that are running the Drupal site. Windows Azure diagnostics allows you to collect performance counters and logs off of the web role instances that are running the Drupal site. Although there are many options for configuring diagnostics in Azure, the best solution with Drupal is to use a diagnostics configuration file. The following configuration file demonstrates some basic performance counters that can monitor resources such as memory, processor utilization, and network bandwidth. For more information about setting up diagnostic configuration files, see How to Use the Windows Azure Diagnostics Configuration File. This information is stored locally on each role instance and then transferred to Windows Azure storage per a defined schedule or on-demand. See Getting Started with Storing and Viewing Diagnostic Data in Windows Azure Storage. Various monitoring tools, such as Azure Diagnostics Manager, help you to more easily analyze diagnostic data. Monitoring the performance of the machines hosting the Drupal site is only part of the story. In order to plan properly for both availability and scalability, you should also monitor site traffic, including user load patterns and trends. Standard and custom diagnostic data could contribute to this, but there are also third-party tools that monitor web traffic. For example, if you know that spikes occur in your application during certain days of the week, you could make changes to the application to handle the additional load and increase the availability of the Drupal solution. Backup Tasks To remain highly available, it is important to backup your data as a defense-in-depth strategy for disaster recovery. This is true even though SQL Azure and Windows Azure Storage both implement redundancy to prevent data loss. One obvious reason is that these services cannot prevent administrator error if data is accidentally deleted or incorrectly changed. SQL Azure does not currently have a formal backup technology, although there are many third-party tools and solutions that provide this capability. Usually the database size for a Drupal site is relatively small. In the case of SAG Awards, it was only ~100-150 MB. So performing an entire backup using any strategy was relatively fast. If your database is much larger, you might have to test various backup strategies to find the one that works best. Apart from third-party SQL Azure backup solutions, there are several strategies for obtaining a backup of your data: · Use the Drush tool and the portabledb-export command. · Periodically copy the database using the CREATE DATABASE Transact-SQL command. · Use Data-tier applications (DAC) to assist with backup and restore of the database. SQL Azure backup and data security techniques are described in more detail in the topic, Business Continuity in SQL Azure. Note that bandwidth costs accrue with any backup operation that transfers information outside of the Windows Azure datacenter. To reduce costs, you can copy the database to a database within the same datacenter. Or you can export the data-tier applications to blob storage in the same datacenter. Another potential backup task involves the files in Blob storage. If you keep a master copy of all media files uploaded to Blob storage, then you already have an on-premises backup of those files. However, if multiple administrators are loading files into Blob storage for use on the Drupal site, it is a good idea to enumerate the storage account and to download any new files to a central location. The following PHP script demonstrates how this can be done by backing up all files in Blob storage after a specified modification date. setProxy(true, 'YOUR_PROXY_IF_NEEDED', 80); $blobs = (array)$blobObj->listBlobs(AZURE_STORAGE_CONTAINER, '', '', 35000); backupBlobs($blobs, $blobObj); function backupBlobs($blobs, $blobObj) { foreach ($blobs as $blob) { if (strtotime($blob->lastmodified) >= DEFAULT_BACKUP_FROM_DATE && strtotime($blob->lastmodified) <= DEFAULT_BACKUP_TO_DATE) { $path = pathinfo($blob->name); if ($path['basename'] != '$$$.$$$') { $dir = $path['dirname']; $oldDir = getcwd(); if (handleDirectory($dir)) { chdir($dir); $blobObj->getBlob( AZURE_STORAGE_CONTAINER, $blob->name, $path['basename'] ); chdir($oldDir); } } } } } function handleDirectory($dir) { if (!checkDirExists($dir)) { return mkdir($dir, 0755, true); } return true; } function checkDirExists($dir) { if(file_exists($dir) && is_dir($dir)) { return true; } return false; } ?> This script has a dependency on the Windows Azure SDK for PHP. Also note there are several parameters that you must modify such as the storage account, secret, and backup location. As with SQL Azure, bandwidth and transaction charges apply to a backup script like this. Scalability Drupal sites on Windows Azure can scale as load increased through typical strategies of scale-up, scale-out, and caching. The following sections describe the specifics of how these strategies are implemented in Windows Azure. Typically you make scalability decisions based on monitoring and capacity planning. Monitoring can be done in staging during testing or in production with real-time load. Capacity planning factors in projections for changes in user demand. Scale Up When you configure your web role prior to deployment, you have the option of specifying the Virtual Machine (VM) size, such as Small or ExtraLarge. Each size tier adds additional memory, processing power, and network bandwidth to each instance of your web role. For cost efficiency and smaller units of scale, you can test your application under expected load to find the smallest virtual machine size that meets your requirements. The workload usually in most popular Drupal websites can be separated out into a limited set of Drupal admins making content changes and a large user base who perform mostly read-only workload. End users can be allowed to make ‘writes’, such as uploading blogs or posting in forums, but those changes are not ‘content changes’. Drupal admins are setup to operate without caching so that the writes are made directly to SQL Azure or the corresponding backend database. This workload performs well with Large or ExtraLarge VM sizes. Also, note that the VM size is closely tied to all hardware resources, so if there are many content-rich pages that are streaming content, then the VM size requirements are higher. To make changes to the Virtual Machine size setting, you must change the vmsize attribute of the WebRole element in the service definition file, ServiceDefinition.csdef. A virtual machine size change requires existing applications to be redeployed. Scale Out In addition to the size of each web role instance, you can increase or decrease the number of instances that are running the Drupal site. This spreads the web requests across more servers, enabling the site to handle more users. To change the number of running instances of your web role, see How to Scale Applications by Increasing or Decreasing the Number of Role Instances. Note that some configuration changes can cause your existing web role instances to recycle. You can choose to handle this situation by applying the configuration change and continue running. This is done by handling the RoleEnvironment.Changing event. For more information see, How to Use the RoleEnvironment.Changing Event. A common question for any Windows Azure solution is whether there is some type of built-in automatic scaling. Windows Azure does not provide a service that provides auto-scaling. However, it is possible to create a custom solution that scales Azure services using the Service Management API. For an example of this approach, see An Auto-Scaling Module for PHP Applications in Windows Azure. Caching Caching is an important strategy for scaling Drupal applications on Windows Azure. One reason for this is that SQL Azure implements throttling mechanisms to regulate the load on any one database in the cloud. Code that uses SQL Azure should have robust error handling and retry logic to account for this. For more information, see Error Messages (SQL Azure Database). Because of the potential for load-related throttling as well as for general performance improvement, it is strongly recommended to use caching. Although Windows Azure provides a Caching service, this service does not currently have interoperability with PHP. Because of this, the best solution for caching in Drupal is to use a module that uses an open-source caching technology, such as Memcached. Outside of a specific Drupal module, you can also configure Memcached to work in PHP for Windows Azure. For more information, see Running Memcached on Windows Azure for PHP. Here is also an example of how to get Memcached working in Windows Azure using a plugin: Windows Azure Memcached plugin. In a future paper, we hope to cover this architecture in more detail. For now, here are several design and management considerations related to caching. Area Consideration Design and Implementation For a technology like Memcached, will the cache be collocated (spread across all web role instances)? Or will you attempt to setup a dedicated cache ring with worker roles that only run Memcached? Configuration What memory is required and how will items in the cache be invalidated? Performance and Monitoring What mechanisms will be used to detect the performance and overall health of the cache? For ease of use and cost savings, collocation of the cache across the web role instances of the Drupal site works best. However, this assumes that there is available reserve memory on each instance to apply toward caching. It is possible to increase the virtual machine size setting to increase the amount of available memory on each machine. It is also possible to add additional web role instances to add to the overall memory of the cache while at the same time improving the ability of the web site to respond to load. It is possible to create a dedicated cache cluster in the cloud, but the steps for this are beyond the scope of this paper[RR1] . For Windows Azure Blob storage, there is also a caching feature built into the service called the Content Delivery Network (CDN). CDN provides high-bandwidth access to files in Blob storage by caching copies of the files in edge nodes around the world. Even within a single geographic region, you could see performance improvements as there are many more edge nodes than Windows Azure datacenters. For more information, see Delivering High-Bandwidth Content with the Windows Azure CDN. Manageability It is important to note that each hosted service has a Staging environment and a Production environment. This can be used to manage deployments, because you can load and test and application in staging before performing a VIP swap with production. From a manageability standpoint, Drupal has an advantage on Windows Azure in the way that site content is stored. Because the data necessary to serve pages is stored in the database and blob storage, there is no need to redeploy the application to change the content of the site. Another best practice is to use a separate storage account for diagnostic data than the one that is used for the application itself. This can improve performance and also helps to separate the cost of diagnostic monitoring from the cost of the running application. As mentioned previously, there are several tools that can assist with managing Windows Azure applications. The following table summarizes a few of these choices. Tool Description Windows Azure Management Portal The web interface of the Windows Azure management portal shows deployments, instance counts and properties, and supports many different common management and monitoring tasks. Azure Diagnostics Managerq[RR2] [JR3] A Red Gate Software product that provides advanced monitoring and management of diagnostic data. This tool can be very useful for easily analyzing the performance of the Drupal site to determine appropriate scaling decisions. Azure Storage Explorer A tool created by Neudesic for viewing Windows Azure storage account. This can be useful for viewing both diagnostic data and the files in Blob storage.

An old customer recently asked me if I had a solution for how to integrate between their existing JMS infrastructure on Websphere MQ with RabbitMQ. Although I know that RabbitMQ has the shovel plugin which can bridge between Rabbit instances I've yet not found a good plugin for JMS <-> AMQP forwarding. The first thing that came to my mind was to utilize a Spring Integration mediation as SI has excellent support for both JMS and Rabbit. Curious as I am I started a PoC and this is the result. It takes messages of a JMS queue and forwards to an AMQP exchange that is bound to a queue the consumer application is supposed to listen to. I used an external HornetQ instance in JBoss 6.1 as the JMS Provider, but I am 100% secure that the same setup would work for Websphere MQ as they both implement JMS. Be aware that I've done no performance tweaking or QoS setup yet as this is just a proof-of-concept. For a real setup you'd probably have to think about delivery guarantees versus performance and etc... The code will be available at a GitHub repository near you soon.. SpringContext in XML: org.jnp.interfaces.NamingContextFactory jnp://localhost:1099 org.jnp.interfaces:org.jboss.naming ConnectionFactory Maven POM: 4.0.0 org.rl si.jmstorabbit 0.0.1-SNAPSHOT jar si.jmstorabbit http://maven.apache.org UTF-8 2.2.5.Final 2.1.0.RELEASE springsource-release http://repository.springsource.com/maven/bundles/release false springsource-external http://repository.springsource.com/maven/bundles/external false org.springframework.integration spring-integration-core ${spring.integration.version} org.springframework.integration spring-integration-file ${spring.integration.version} org.springframework.integration spring-integration-amqp ${spring.integration.version} org.springframework.integration spring-integration-jms ${spring.integration.version} junit junit 3.8.1 test org.springframework spring-context 3.0.7.RELEASE jboss jnp-client 4.2.2.GA org.hornetq hornetq-core-client ${hornet.version} org.hornetq hornetq-jms-client ${hornet.version} org.hornetq hornetq-jms ${hornet.version} jboss jboss-common-client 3.2.3 org.jboss.netty netty 3.2.7.Final javax.jms jms 1.1

See original post at https://muhammadkhojaye.blogspot.com/2012/04/how-to-run-amazon-elastic-mapreduce-job.html Introduction Amazon EMR is a web service which can be used to easily and efficiently process enormous amounts of data. It uses a hosted Hadoop framework running on the web-scale infrastructure of Amazon EC2 and Amazon S3. Amazon EMR removes most of the cumbersome details of Hadoop while taking care of provisioning of Hadoop, running the job flow, terminating the job flow, moving the data between Amazon EC2 and Amazon S3, and optimizing Hadoop. In this tutorial, we will use a developed WordCount Java example using Hadoop and thereafter, we execute our program on Amazon Elastic MapReduce. Prerequisites You must have valid AWS account credentials. You should also have a general familiarity with using the Eclipse IDE before you begin. The reader can also use any other IDE of their choice. Step 1 – Develop MapReduce WordCount Java Program In this section, we are first going to develop a WordCount application. A WordCount program will determine how many times different words appear in a set of files. In Eclipse (or whatever the IDE you are using), Create simple Java Project with the name "WordCount". Create a java class name Map and override the map method as follow, public class Map extends Mapper { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } Create a java class named Reduce and override the reduce method as shown below, public class Reduce extends Reducer { @Override protected void reduce(Text key, java.lang.Iterable values, org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable value : values) { sum += value.get(); } context.write(key, new IntWritable(sum)); } } Create a java class named WordCount and defined the main method as below, public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setJarByClass(WordCount.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } Export the WordCount program in a jar using eclipse and save it to some location on disk. Make sure that you have provided the Main Class (WordCount.jar) during extraction ofu8u the jar file as shown below. Our jar is ready!!! Step 2 – Upload the WordCount JAR and Input Files to Amazon S3 Now we are going to upload the WordCount jar to Amazon S3. First, go to the following URL: https://console.aws.amazon.com/s3/home Next, click “Create Bucket”, give your bucket a name, and click the “Create” button. Select your new S3 bucket in the left-hand pane. Upload the WordCount JAR and sample input file for counting the words. Step 3 – Running an Elastic MapReduce job Now that the JAR is uploaded into S3, all we need to do is to create a new Job flow. let's execute the steps below. (I encourage readers to check out the following link for details regarding each step, How to Create a Job Flow Using a Custom JAR ) Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/ Click Create New Job Flow. In the DEFINE JOB FLOW page, enter the following details, a) Job Flow Name = WordCountJob b) Select Run your own applications) Select Custom JAR in the drop-down list) Click Continue In the SPECIFY PARAMETERS page, enter values in the boxes using the following table as a guide, and then click Continue.JAR Location = bucketName/jarFileLocationJAR Arguments =s3n://bucketName/inputFileLocations3n://bucketName/outputpath Please note that the output path must be unique each time we execute the job. The Hadoop always create a folder with the same name specified here. After executing the job, just wait and monitor your job that runs through the Hadoop flow. You can also look for errors by using the Debug button. The job should be complete within 10 to 15 minutes (can also depend on the size of the input). After completing the job, You can view results in the S3 Browser panel. You can also download the files from S3 and can analyze the outcome of the job. Amazon Elastic MapReduce Resources Amazon Elastic MapReduce Documentation,http://aws.amazon.com/documentation/elasticmapreduce/ Amazon Elastic MapReduce Getting Started Guide,http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/ Amazon Elastic MapReduce Developer Guide,http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/ Apache Hadoop,http://hadoop.apache.org/ See more at https://muhammadkhojaye.blogspot.com/2012/04/how-to-run-amazon-elastic-mapreduce-job.html

In my previous post I showed you how to install and use Redis with Node.js. Today I’m going to take that a step further and walk through some of the things you can do with node_redis using Redis’s TTL and EXPIRE commands. Note: If you haven’t gone through my previous article make sure to do that now as I’ll assume you have Node.js and Redis up and running. Create a new folder and put a new text file in it called: app.js Inside the app.js file we will add some simple code to set a value that doesn’t have a time to live (or expiration on it): var redis = require("redis") , client = redis.createClient(); client.on("error", function (err) { console.log("Error " + err); }); client.on("connect", runSample); function runSample() { // Set a value client.set("string key", "Hello World", function (err, reply) { console.log(reply.toString()); }); // Get a value client.get("string key", function (err, reply) { console.log(reply.toString()); }); } When we connect to Redis and everything is ready the runSample function is called which in turn sets a value and then reads it back. Expected output: OK Hello World Lets set a timeout on a value using the EXPIRE command and see what happens. Replace the original code with this: var redis = require('redis') , client = redis.createClient(); client.on('error', function (err) { console.log('Error ' + err); }); client.on('connect', runSample); function runSample() { // Set a value with an expiration client.set('string key', 'Hello World', redis.print); // Expire in 3 seconds client.expire('string key', 3); // This timer is only to demo the TTL // Runs every second until the timeout // occurs on the value var myTimer = setInterval(function() { client.get('string key', function (err, reply) { if(reply) { console.log('I live: ' + reply.toString()); } else { clearTimeout(myTimer); console.log('I expired'); client.quit(); } }); }, 1000); } Note: Be aware that the timer I use is just to demo the EXPIRE, you should be very careful about using timers in production Nodejs projects. Run the program. Expected results: Reply: OK I live: Hello World I live: Hello World I live: Hello World I expired Now we will check to see how much time a value has left before it expires: var redis = require('redis') , client = redis.createClient(); client.on('error', function (err) { console.log('Error ' + err); }); client.on('connect', runSample); function runSample() { // Set a value client.set('string key', 'Hello World', redis.print); // Expire in 3 seconds client.expire('string key', 3); // This timer is only to demo the TTL // Runs every second until the timeout // occurs on the value var myTimer = setInterval(function() { client.get('string key', function (err, reply) { if(reply) { console.log('I live: ' + reply.toString()); client.ttl('string key', writeTTL); } else { clearTimeout(myTimer); console.log('I expired'); client.quit(); } }); }, 1000); } function writeTTL(err, data) { console.log('I live for this long yet: ' + data); } Run the program. Expected results: Reply: OK I live: Hello World I live for this long yet: 2 I live: Hello World I live for this long yet: 1 I live: Hello World I live for this long yet: 0 I expired

As you know I'm open-sourcing and completely overhauling my PhD system. One of my goals was to replace internal JMS queues with AMQP. Today I'll show you how I did it and why I was forced to change RabbitMQ to Apache Qpid. AMQP In short. AMQP is an open standard application layer protocol for message-oriented middleware. The most important feature is that AMQP is a wire-level protocol and is interoperable by design. JMS is just an API. Altough JMS brokers can be used in .NET applications (see my post: ActiveMQ and .NET combined!), the whole JMS specification does not guarantee interoperability. Also, the AMQP standard is by design more flexible and powerful (e.g., supports two-way communication by design) - they simply learnt from JMS mistakes :). Oh, forgot to mention. The AMQP was originally developed by banks :) so I don't have to say that AMQP is secure, fault-tolerant, and so on. RabbitMQ RabbitMQ is the most mature AMQP broker. RabbitMQ is written in Erlang so you have to download that first (RabbitMQ Windows installer does it for you). Download it from here: http://www.rabbitmq.com/. I also recommend installing the web management console. From Rabbit's sbin directory execute: rabbitmq-plugins enable rabbitmq_management If you're on Windows and you installed a Rabbit service you have to restart it. That's it. Spring Well, it turned out that VMware bought RabbitMQ and SpringSource developers are now developing it. Given this fact, you shouldn't be surprised that Spring - RabbitMQ integration is childishly simple. Add spring-rabbit dependency to your Maven project, and then in Spring configuration paste the following: The default configuration assumes that RabbitMQ is running on a local server using the default port and default credentials (guest/guest). Of course all these settings are configurable. To sent a message to "myqueue" queue, just inject an instance of AmqpTemplate into your service and send the message. An example would be: @Service public class HomeController { @Autowired private AmqpTemplate amqpTemplate; public void sendMessage(Bundle bundle) throws IOException { byte[] body = IOUtils.toByteArray(bundle.getInputStream()); MessageProperties messageProperties = new MessageProperties(); messageProperties.setContentType(bundle.getContentType()); messageProperties.setContentLength(bundle.getSize()); messageProperties.setTimestamp(new Date()); messageProperties.setDeliveryMode(MessageDeliveryMode.PERSISTENT); Message message = new Message(body, messageProperties); amqpTemplate.send(message); } } You can open the web console http://localhost:55672/mgmt/ and see 1 message in "myqueue" queue. Apache Camel To read a message from Apache Camel you first have to add camel-amqp dependency to your POM. Then just copy and paste the following route definition: Run the route by executing mvn:camel-run and... you'll see an error. Making a long story short, Apache Camel 2.9.0 doesn't work with RabbitMQ. This is because the camel-amqp component is using the Apache Qpid client under the hood. The current Qpid version is 0.14, but Qpid guys forgot to upload new jars to the Maven public repo. Thus camel-amqp is still using Qpid 0.12 whose client doesn't seem to negotiate protocols. Even if you exclude qpid-commons and qpid-client dependencies and explicitly add Qpid 0.14 ones (download them and install in your local repo) there will be an exception thrown from the camel-amqp component as there is no longer a default ConnectionFactory constructor. Thus I was forced to install Qpid. Qpid I downloaded the Java server and simply ran it. There is no web management console, but that's OK. You can use JConsole for JMX. Spring AMQP and Qpid In order to make Spring AMQP work with Qpid copy and paste the following configuration: As you can see in the above snippet I explicitly created AMPQComponent with connectionFactory set to Apache Qpid AMQConnectionFactory object. Source code and working example This solution is a part of the Qualitas project. I use Spring MVC to handle uploads of business processes bundles (e.g., zipped archive of a WS-BPEL process) and send it to an AMQP queue. Then Apache Camel consumes the message, does additional processing of the bundle, and installs it on a remote business process execution engine. The projects you are most interested in are: qualitas-webapp (Spring MVC sending messages to AMQP) qualitas-internall-installation (Apache Camel route consuming messages from AMQP) To check out 0.0.2-SNAPSHOT tag from here: http://code.google.com/p/qualitas/source/browse/. Qualitas Read more about Qualitas project here: http://code.google.com/p/qualitas/. Happy to welcome new developers on board! cheers, Łukasz

i’ve done a few posts recently using d3.js and now i want to show you how to use two other great javascript libraries to visualize your graphs. we’ll start with sigma.js and soon i’ll do another post with three.js . we’re going to create our graph and group our nodes into five clusters. you’ll notice later on that we’re going to give our clustered nodes colors using rgb values so we’ll be able to see them move around until they find their right place in our layout. we’ll be using two sigma.js plugins, the gefx (graph exchange xml format) parser and the forceatlas2 layout. you can see what a gefx file looks like below. notice it comes from gephi which is an interactive visualization and exploration platform, which runs on all major operating systems, is open source, and is free. ... ... in order to build this file, we will need to get the nodes and edges from the graph and create an xml file. get '/graph.xml' do @nodes = nodes @edges = edges builder :graph end we’ll use cypher to get our nodes and edges: def nodes neo = neography::rest.new cypher_query = " start node = node:nodes_index(type='user')" cypher_query << " return id(node), node" neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0]}.merge(n[1]["data"])} end we need the node and relationship ids, so notice i’m using the id() function in both cases. def edges neo = neography::rest.new cypher_query = " start source = node:nodes_index(type='user')" cypher_query << " match source -[rel]-> target" cypher_query << " return id(rel), id(source), id(target)" neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0], "source" => n[1], "target" => n[2]} } end so far we have seen graphs represented as json, and we’ve built these manually. today we’ll take advantage of the builder ruby gem to build our graph in xml. xml.instruct! :xml xml.gexf 'xmlns' => "http://www.gephi.org/gexf", 'xmlns:viz' => "http://www.gephi.org/gexf/viz" do xml.graph 'defaultedgetype' => "directed", 'idtype' => "string", 'type' => "static" do xml.nodes :count => @nodes.size do @nodes.each do |n| xml.node :id => n["id"], :label => n["name"] do xml.tag!("viz:size", :value => n["size"]) xml.tag!("viz:color", :b => n["b"], :g => n["g"], :r => n["r"]) xml.tag!("viz:position", :x => n["x"], :y => n["y"]) end end end xml.edges :count => @edges.size do @edges.each do |e| xml.edge:id => e["id"], :source => e["source"], :target => e["target"] end end end end you can get the code on github as usual and see it running live on heroku. you will want to see it live on heroku so you can see the nodes in random positions and then move to form clusters. use your mouse wheel to zoom in, and click and drag to move around. credit goes out to alexis jacomy and mathieu jacomy . you’ve seen me create numerous random graphs, but for completeness here is the code for this graph. notice how i create 5 clusters and for each node i assign half its relationships to other nodes in their cluster and half to random nodes? this is so the forceatlas2 layout plugin clusters our nodes neatly. def create_graph neo = neography::rest.new graph_exists = neo.get_node_properties(1) return if graph_exists && graph_exists['name'] names = 500.times.collect{|x| generate_text} clusters = 5.times.collect{|x| {:r => rand(256), :g => rand(256), :b => rand(256)} } commands = [] names.each_index do |n| cluster = clusters[n % clusters.size] commands << [:create_node, {:name => names[n], :size => 5.0 + rand(20.0), :r => cluster[:r], :g => cluster[:g], :b => cluster[:b], :x => rand(600) - 300, :y => rand(150) - 150 }] end names.each_index do |from| commands << [:add_node_to_index, "nodes_index", "type", "user", "{#{from}"] connected = [] # create clustered relationships members = 20.times.collect{|x| x * 10 + (from % clusters.size)} members.delete(from) rels = 3 rels.times do |x| to = members[x] connected << to commands << [:create_relationship, "follows", "{#{from}", "{#{to}"] unless to == from end # create random relationships rels = 3 rels.times do |x| to = rand(names.size) commands << [:create_relationship, "follows", "{#{from}", "{#{to}"] unless (to == from) || connected.include?(to) end end batch_result = neo.batch *commands end

Message groups in JMS provide a way to identify a set of related messages. The messages could be related by anything - a customer order number, for example. Basically a JMS broker provides a guarantee that any messages that belong to a specific group will always be consumed by a common consumer. For instance, imagine that we’ve used the splitter pattern to split out line items from an order but want to aggregate those line items together later in a route. In order to perform that aggregation you need to guarantee that all of the messages being aggregated together are consumed by the same consumer. Below is an example of using message groups with ActiveMQ within Apache Camel. package com.brinksys.camel; import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.activemq.broker.BrokerService; import org.apache.activemq.camel.component.ActiveMQComponent; import org.apache.activemq.pool.PooledConnectionFactory; import org.apache.camel.CamelContext; import org.apache.camel.Exchange; import org.apache.camel.Processor; import org.apache.camel.ProducerTemplate; import org.apache.camel.builder.RouteBuilder; import org.apache.camel.impl.DefaultCamelContext; import java.util.concurrent.TimeUnit; public class App { private static BrokerService broker; public static void main(String[] args) throws Exception { try { startBroker(); CamelContext ctx = createCamelContext(); ctx.start(); ctx.addRoutes(new RouteBuilder() { @Override public void configure() throws Exception { /* Our direct route will take a message, and set the message to group 1 if the body is an integer, * otherwise set the group to 2. * * This demonstrates the following concepts: * 1) Header Manipulation * 2) Checking the payload type of the body and using it in a choice. * 3) JMS Message groups */ from("direct:begin") .choice() .when(body().isInstanceOf(Integer.class)).setHeader("JMSXGroupID",constant("1")) .otherwise().setHeader("JMSXGroupID",constant("2")) .end() .to("amq:queue:Message.Group.Test"); /* These two are competing consumers */ from("amq:queue:Message.Group.Test").routeId("Route A").log("Received: ${body}"); from("amq:queue:Message.Group.Test").routeId("Route B").log("Received: ${body}"); } }); sendMessages(ctx.createProducerTemplate()); Thread.sleep(TimeUnit.SECONDS.toMillis(10)); stopBroker(); } catch (Exception e) { e.printStackTrace(); } } private static CamelContext createCamelContext() throws Exception { CamelContext camelContext = new DefaultCamelContext(); ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory("vm://localhost/"); PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory(activeMQConnectionFactory); pooledConnectionFactory.setMaxConnections(8); pooledConnectionFactory.setMaximumActive(500); ActiveMQComponent activeMQComponent = ActiveMQComponent.activeMQComponent(); activeMQComponent.setUsePooledConnection(true); activeMQComponent.setConnectionFactory(pooledConnectionFactory); camelContext.addComponent("amq", activeMQComponent); return camelContext; } private static void sendMessages(ProducerTemplate pt) throws Exception { for (int i = 0; i < 10; i++) { pt.sendBody("direct:begin", Integer.valueOf(i)); } for (int i = 0; i < 10; i++) { pt.sendBody("direct:begin", "next group"); } pt.sendBody("direct:begin", Integer.valueOf(1)); pt.sendBody("direct:begin", "foo"); pt.sendBody("direct:begin", Integer.valueOf(2)); } private static void startBroker() throws Exception { broker = new BrokerService(); broker.addConnector("vm://localhost"); broker.start(); } private static void stopBroker() throws Exception { broker.stop(); } } The result of running this main method is as follows: 2445 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 0 2447 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 1 2460 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 2 2466 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 3 2472 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 4 2479 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 5 2482 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 6 2485 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 7 2488 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 8 2490 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 9 2493 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2496 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2499 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2501 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2504 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2505 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2508 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2510 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2513 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2515 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: next group 2517 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 1 2535 [Camel (camel-1) thread #1 - JmsConsumer[Message.Group.Test]] INFO Route B - Received: foo 2538 [Camel (camel-1) thread #0 - JmsConsumer[Message.Group.Test]] INFO Route A - Received: 2 You’ll notice that all messages with a groupId of 1 are consumed by one route and the messages with a groupId of 2 are consumed by the other consumer. You’ll also see how relatively simple it is to inspect the body of our original message to check it’s type and set the header in the route that begins our orchestration. If you wish to run this source code, I’ve set up a little Git repository on github for hosting some camel examples. As of the time I write this, only the message group example is available, but others should appear soon.

Microsoft offered programmers several different ways of dealing with the asynchronous programming since .NET 1.0. The first model was Asynchronous programming model or APM for short. The pattern is implemented with two methods named BeginOperation and EndOperation. .NET 4 introduced new pattern – Task Asynchronous Pattern and with the introduction of .NET 4.5, Microsoft added language support for language integrated asynchronous coding style. You can check the MSDN for more samples and information. I will assume that you are familiar with it and have written code using it. You can wrap existing APM pattern into TPL pattern using the Task.Factory.FromAsync methods. For example: public static Task> ExecuteAsync(this DataServiceQuery query, object state) { return Task.Factory.FromAsync>(query.BeginExecute, query.EndExecute, state); } It is easy to wrap most of the asynchronous functions this way, but some cannot be since the wrapper functions assume that the last two parameters to the BeginOperation are AsyncCallback and object, and there are some versions of asynchronous operations that have different specifications. Examples: Extra parameters after the object state parameter: IAsyncResult DataServiceContext.BeginExecuteBatch( AsyncCallback callback, object state, params DataServiceRequest[] queries); Missing the expected object state parameter and different return type: ICancelableAsyncResult BeginQuery(AsyncCallback callBack); WorkItemCollection EndQuery(ICancelableAsyncResult car); Short solution for the first example The short and elegant way for wrapping the first example is to provide the following wrapper: public static Task ExecuteBatchAsync(this DataServiceContext context, object state, params DataServiceRequest[] queries) { if (context == null) throw new ArgumentNullException("context"); return Task.Factory.FromAsync( context.BeginExecuteBatch(null, state, queries), context.EndExecuteBatch); } We simply call the Begin method ourselves and then wrap it using an another overload for FromAsync function. The longer way However, we can fully wrap it ourselves by simulating what the FromAsync wrapper does. The complete code is listed below. public static Task ExecuteBatchAsync(this DataServiceContext context, object state, params DataServiceRequest[] queries) { // this will be our sentry that will know when our async operation is completed var tcs = new TaskCompletionSource(); try { context.BeginExecuteBatch((iar) => { try { var result = context.EndExecuteBatch(iar as ICancelableAsyncResult); tcs.TrySetResult(result); } catch (OperationCanceledException ex) { // if the inner operation was canceled, this task is cancelled too tcs.TrySetCanceled(); } catch (Exception ex) { // general exception has been set bool flag = tcs.TrySetException(ex); if (flag && ex as ThreadAbortException != null) { tcs.Task.m_contingentProperties.m_exceptionsHolder.MarkAsHandled(false); } } }, state, queries); } catch { tcs.TrySetResult(default(DataServiceResponse)); // propagate exceptions to the outside throw; } return tcs.Task; } Besides educational benefits, writing the full wrapper code allows us to add cancellation, logging and diagnostic information. Once we understand how to wrap APM pattern, We can now tackle the second problem easily. Handling the BeginQuery/EndQuery We will first create our own wrapper function in the spirit of the above code with the notable difference that we use the ICancelableAsyncResult interface instead of the IAsyncResult. public static class TaskEx { public static Task FromAsync(Func beginMethod, Func endMethod) { if (beginMethod == null) throw new ArgumentNullException("beginMethod"); if (endMethod == null) throw new ArgumentNullException("endMethod"); var tcs = new TaskCompletionSource(); try { beginMethod((iar) => { try { var result = endMethod(iar as ICancelableAsyncResult); tcs.TrySetResult(result); } catch (OperationCanceledException ex) { tcs.TrySetCanceled(); } catch (Exception ex) { bool flag = tcs.TrySetException(ex); if (flag && ex as ThreadAbortException != null) { tcs.Task.m_contingentProperties.m_exceptionsHolder.MarkAsHandled(false); } } }); } catch { tcs.TrySetResult(default(TResult)); throw; } return tcs.Task; } } The code is pretty self-explanatory and we can go ahead with the wrapping. There are four different operations that are exposed both in synchronous and asynchronous version: Query, LinkQuery, CountOnlyQuery and RegularQuery. The extension methods are short since we have already created our generic wrapper above: public static Task RunQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginQuery, query.EndQuery); } public static Task RunLinkQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginLinkQuery, query.EndLinkQuery); } public static Task RunCountOnlyQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginCountOnlyQuery, query.EndCountOnlyQuery); } public static Task RunRegularQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginRegularQuery, query.EndRegularQuery); } That is it for today, you can write your own handy extensions easily for APM functions out there.

Since the very early days of Java (and C-like languages overall), the canonical way to start your program has been something like this: public class A { public static void main(String[] args) { new A().run(args); } public void run(String[] args) { // Your application starts here } } If you are still doing this, I’m here to tell you it’s time to stop. Letting go of ‘new’ First, install Guice in your project: com.google.inject guice 3.0 and then, modify your main method as follows: public class A { public static void main(String[] args) { Injector.getInstance(A.class).run(args); } } So, what does this buy you exactly? You will find a lot of articles explaining the various benefits of Guice, such as being able to substitute different environments on the fly, but I’m going to use a different angle in this article. Let’s start by assuming the existence of a Config class that contains various configuration parameters. I’ll just hardcode them for now and use fields to make the class smaller: public class Config { String host = "com.example.com"; int port = 1234; } This class is a singleton, it is instantiated somewhere in your main class and not used anywhere else at the moment. One day, you realize you need this instance in another class which happens to be deep in your runtime hierarchy, which we will call Deep. For example, if you put a break point in the method where you need this config object, your debugger would show you stack frames similar to this: com.example.A.main() com.example.B.f(int, String) com.example.C.g(String) com.example.Deep.h(Foo, int) The easy and wrong way to solve this problem is to make the Config instance static on some class (probably A) and access it directly from Deep. I’m hoping I don’t need to explain why this is a bad idea: not only do you want to avoid using statics, but you also want to make sure that each object is exposed only to objects that need them, and making the Config object static would make your instance visible to your entire code base. Not a good thing. The second thought is to pass the object down the stack, so you modify all the signatures as follows: com.example.A.main() com.example.B.f(int, String, Config) com.example.C.g(String, Config) com.example.Deep.h(Foo, int, Config) This is a bit better since you have severely restricted the exposure of the Config object, but note that you are still making it available to more methods than really need to: B#f and C#g have really nothing to do with this object, and a little sting of discomfort hits you when you start writing the Javadoc: public class C { ... /** * @param config This method doesn't really use this parameter, * it just passes it down so Deep#h can use it. */ public void g(String s, Config config) { Unnecessary exposure is actually not the worst part of this approach, the problem is that it changes all these signatures along the way, which is certainly undesirable in a private API and absolutely devastating in a public API. And of course, it’s absolutely not scalable: if you keep adding a parameter to your method whenever you need access to a certain object, you will soon be dealing with methods that take ten parameters, most of which they just pass down the chain. Here is how we solve this problem with dependency injection (performed by Guice in this example, but this is applicable to any library that implements JSR 330, obviously): public class Deep { @Inject private Config config; and we’re done. That’s it. You don’t need to modify the Config class in any way, nor do you need to make any change in any of the classes that separate Deep from your main class. With this, you have also minimized the exposure of the Config object to just the class that needs it. Injecting right There are various ways you can inject object into your class but I’ll just mention the two that, I think, are the most important. I just showed “field injection” in the previous paragraph, but be aware that you can also prefer to use “constructor injection”: public class Deep { private final Config config; @Inject public Deep(Config config) { this.config = config; } This time, you are adding a parameter to the constructor of your Deep class (which shouldn’t worry you too much since you will never invoke it directly, Guice will) and you assign the parameter to the field in the constructor. The benefit is that you can declare your field final. The downside, obviously, is that this approach is much more verbose. Personally, I see little point in final fields since I have hardly ever encountered a bug that was due to accidentally reassigning a field, so I tend to use field injection whenever I can. Taking it to the next level Obviously, the kind of configuration object I used as an example if not very realistic. Typically, a configuration will not hardcode values like I did and will, instead, read them from some external source. Similarly, you will want to inject objects that can’t necessarily be instantiated so early in the lifecycle of your application, such as servlet contexts, database connections, or implementations of your own interfaces. This topic itself would probably cover several chapters of a book dedicated to dependency injection, so I’ll just summarize it: not all objects can be injected this way, and one benefit of using a dependency injection framework in your code is that it will force you to think about what life cycle category your objects belong to. Having said that, if you want to find out how Guice can inject objects that get created at a later time in your application life cycle, look up the Javadoc for the Provider class. Wrapping up I hope this quick introduction to dependency injection piqued your interest and that you will consider using it in your project since it has so much more to offer than what I described in this post. If you want to learn more, I suggest starting with the excellent Guice documentation.

The Map Reduce Framework works in two main phases to process the data, which are the "map" phase and the "reduce" phase.

The one major difference between IDEA and Eclipse is that IDEA "feels context", which effectively makes IDEA intelligent.