DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Popular Topics

article thumbnail
Enterprise Integration Patterns with Apache Camel Refcard Now Available!
Apache Camel is a powerful open source integration platform based on Enterprise Integration Patterns with Bean Integration. This Refcard provides you with eleven of the most essential patterns that anyone working with integration must know. This Refcard is targeted for software developers and enterprise architects, but anyone in the integration space can benefit as well. Download Now! About the Author: Claus Ibsen is a passionate open-source enthusiast who specializes in the integration space. As an engineer in the FuseSource Open Source Division he works full time on Apache Camel, FUSE Mediation Router (Apache Camel Enterprise) and related projects. Claus is very active in the Apache Camel and Fuse communities, writing blogs, twittering, assisting on the forums, irc channels and is driving the Apache Camel roadmap.
March 30, 2009
by Wei Ling Chen
· 4,841 Views
article thumbnail
Apache Camel: Integration Nirvana
Take any integration project and you have multiple applications talking over multiple transports on multiple platforms. As you can imagine, in large enterprise applications this can get complex very fast. Much of the complexity stems from two issues: 1. dealing with the specifics of applications and transports, and 2. coming up with good solutions to integration problems. Making your applications speak transports and APIs is relatively easy on its own. I'm sure everyone knows how to send JMS messages to their broker of choice; though it still requires in depth knowledge of the JMS specification, which many developers may not have. On top of that, what happens when you want to route that JMS message to another application? You then have to take care of mapping the JMS message to the application plus handle any new concepts related to the application. Add a dozen other applications into the mix and you've got quite a headache on your hands. Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations. Apache Camel was created with the intention of addressing these two issues. In this article I'll show you how it actually does this. What is Camel? Apache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers. It does this by providing: • concrete implementations of all the widely used EIPs • connectivity to a great variety of transports and APIs • easy to use Domain Specific Language (DSL) to wire EIPs and transports together Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and the Domain Specific Language (DSL). There is of course a lot more going on here under the hood but we'll leave that for another discussion. Figure 1: High level view of Camel's architecture. Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 12 essential components. There are over 60 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queue aQueue and send them to a file system directory "c:/tmp", you could use URIs like "jms:aQueue" and "file:c:\tmp". Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports 41 patterns from the EIP book, 6 other integration patterns, and many other useful Processors. To wire Processors and Endpoints together, Camel defines a Java DSL. The term DSL is used a bit loosely here as it usually implies the involvement of a compiler or interpreter that can process keywords specific to a particular domain. In Camel, DSL means a fluent Java API that contains methods named like terms from the EIP book. Its best explained with an example from("jms:aQueue") .filter().xpath("/person[@name='Jon']") .to("file:c:\tmp"); Here we define a routing rule in a single Java statement that will consume messages from the "jms:aQueue" Endpoint, send them through a Message Filter Processor, which will then send on messages passing the XPath condition to the "file:c:\tmp" endpoint. Messages failing the condition will be dropped. You can also configure your routes in a XML-based Spring configuration file. This configuration file is a lot more verbose and less auto complete friendly than the Java DSL; many prefer it though because of its direct access to Spring concepts and no requirement for compilation after changes. Here is what the earlier example would look like in Spring: /person[@name='Jon'] These are the concepts that Camel was built upon. Since then many other interesting features have been added. Details of these are left up to the reader to investigate. To get you started, some of these include: • Pluggable data formats and type converters for easy message transformation between Artix Data Services, CSV, EDI, Flatpack, HL7, JAXB, JSON, XmlBeans, XStream, Zip, Camel-bindy, etc. • Pluggable languages to create expressions or predicates for use in the DSL. Some of these languages include: EL, JXPath, Mvel, OGNL, BeanShell, JavaScript, Groovy, Python, PHP, Ruby, SQL, XPath, XQuery, etc. • Support for the integration of beans and POJOs in various places in Camel. • Excellent support for testing distributed and asynchronous systems using a messaging approach • and much more... Example A motorcycle parts business, Rider Auto Parts, supplies parts to motorcycle manufacturers. Over the years they've changed the way they receive orders several times. Initially, orders were placed by uploading CSV files to an FTP server. The message format was later changed to XML. Currently they provide a web site to submit orders as XML messages over HTTP. All of these messages are converted to an internal POJO format before processing. Rider Auto Parts states to any new customers to use the web interface to place orders. However, because of existing agreements with customers, they must keep all the old message formats and interfaces up and running. Solution using EIPs Rider Auto Parts faces a pretty common problem; over years of operation businesses acquire software baggage in the form of transports/data formats that are popular at the time. Using patterns from the EIP book we can envision the solution as something like Figure 2. Figure 2: This shows the solution to Rider Auto Parts integration problem using notation from the Enterprise Integration Patterns book. So we have several patterns in use here. 1. There are two Message Endpoints; one for FTP connectivity and another for HTTP. 2. Messages from these endpoints are fed into the incomingOrderQueue Message Channel 3. The messages are consumed from the incomingOrderQueue and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file. 4. Both Message Translators convert the message content into a POJO, which is fed into the orderQueue Message Channel. The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer. Implementation using Camel As mentioned before, Camel has a small core set of components included by default. The rest of the components exist as separate modules. In applications that require many types of connectivity it is useful to figure out what Camel modules to include. Listing 1 shows the dependencies using Apache Maven for the Camel implementation of the Rider Auto Parts example. Of course, you don't need to use Apache Maven for dependencies - it is just the easiest way to rapidly add new dependencies to your applications. The list of dependencies includes support for core Camel, ActiveMQ, JAXB marshaling, CSV marshaling, and HTTP. To make the example easier to try out, I've opted to use the File endpoint instead of the FTP. If we were using the FTP endpoint we would need to add a dependency on the camel-ftp module as well. Listing 1: Maven dependencies for the Camel implementation org.apache.camel camel-core ${camel-version} org.apache.camel camel-spring ${camel-version} org.apache.activemq activemq-camel ${activemq-version} org.apache.camel camel-jaxb ${camel-version} org.apache.camel camel-csv ${camel-version} org.apache.camel camel-jetty ${camel-version} org.apache.activemq activemq-core ${activemq-version} org.apache.xbean xbean-spring ${xbean-spring-version} While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel from Spring. The Spring beans XML file is shown in Listing 2. First we start an embedded Apache ActiveMQ broker and connect Camel to it. We also load up some helper beans that we will reference from the DSL. Finally, the camelContext element tells Camel to look for routes in the org.fusesource.camel package. Routes are Java classes that extend the RouteBuilder class in Camel. Listing 2: Spring XML file that configures an embedded ActiveMQ broker, several beans used in the Camel route, and initializes the Camel Context to search for routes in the org.fusesource.camel package. org.fusesource.camel The real meat of the Camel implementation lies in the OrderRouter class (shown in Listing 3). This class extends RouteBuilder, so it will be automatically picked up and loaded by Camel's runtime. Looking back at Figure 2, we need to receive orders from an FTP (substituted with File) and HTTP endpoint, formatted as shown in Listing 4. In the DSL we can specify these incoming endpoints with two from elements. Both from elements are connected to a to("jms:incomingOrderQueue") element, which will send the messages to a queue on the ActiveMQ broker. Listing 3: Route definitions for the example. The routing rules are specified using a fluent API, referred to as Camel's DSL. public class OrderRouter extends RouteBuilder { @Override public void configure() throws Exception { JaxbDataFormat jaxb = new JaxbDataFormat("org.fusesource.camel"); // Receive orders from two endpoints from("file:src/data?noop=true").to("jms:incomingOrderQueue"); from("jetty:http://localhost:8888/placeorder") .inOnly().to("jms:incomingOrderQueue") .transform().constant("OK"); // Do the normalization from("jms:incomingOrderQueue") .convertBodyTo(String.class) .choice() .when().method("orderHelper", "isXml") .unmarshal(jaxb) .to("jms:orderQueue") .when().method("orderHelper", "isCsv") .unmarshal().csv() .to("bean:normalizer") .to("jms:orderQueue"); } } In the case of the HTTP endpoint, there are a couple of extra things to mention. First off the HTTP client will be expecting a response from the application so we have to handle that. In Camel, we have full control over what the client gets back from the HTTP endpoint. Each response is determined by the last method in our current route definition (each Java statement is a route definition). In our case we use the transform method to set the response to the constant string "OK". Since we handle the response ourselves, we don’t want any response to come from the JMS incomingOrderQueue. To send to this queue in a fire-and-forget fashion we add the inOnly modifier. It is important to note at this point that when writing Camel DSL in a modern Java IDE, selection of the next processing step is easy because of auto complete. The auto complete feature basically gives you a list of processors (i.e. EIPs) to choose from at any point in your route. Since fluent APIs chain methods together, the only method you need to remember is the from; all other methods are shown via auto complete. Listing 4: Incoming message formats; XML on top, CSV below. "name", "amount" "brake pad", "2" The next section of DSL in Listing 3 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First we specify that we want to consume messages from the incomingOrderQueue on the ActiveMQ broker. The content based routing of the messages is done with the choice and when methods. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message we have we will be using a simple Java bean shown in Listing 5. Of course, this is demonstration code only; for production cases you would want to add more thorough checking of content types. Listing 5: Java bean that contains helper methods to be used in the DSL. public class OrderHelper { public boolean isCsv(String body) { return !body.contains("> body) { List orderHeaders = body.get(0); List orderValues = body.get(1); return new Order(orderValues.get(0), Integer.parseInt(orderValues.get(1))); } } At this point, successfully normalized messages are sent to the orderQueue for processing by some other application at the Rider Auto Parts business. Conclusion In this article I've shown two common problems that an integration developer may face: dealing with the specifics of applications and transports, and coming up with good solutions to integration problems. The Apache Camel project provides a nice answer to both of these problems. As the example has shown, solving integration problems with Camel is straight forward and results in relatively concise code. In my opinion it is the closest thing to integration nirvana that we have today. Links Apache Camel – http://camel.apache.org FUSE Mediation Router (based on Apache Camel) – http://fusesource.com/products/enterprise-camel Enterprise Integration Patterns – http://www.enterpriseintegrationpatterns.com Jon’s Blog – http://janstey.blogspot.com Camel in Action book - http://www.manning.com/ibsen Article source code - http://repo.fusesource.com/maven2/org/fusesource/examples/rider-auto-example/1.0/rider-auto-example-1.0.zip Author Jonathan Anstey is a senior engineer working for Progress Software Corporation specializing in the enterprise integration space. Jon focuses mostly on Apache Camel and its Progress endorsed likeness, FUSE Mediation Router. He also works on the Apache ActiveMQ and Apache ServiceMix projects
March 23, 2009
by Jonathan Anstey
· 223,810 Views · 8 Likes
article thumbnail
How to Fix Memory Leaks in Java
Your pager hasn’t been sleeping well. It periodically wakes you up in the middle of the night to tell you that your server is firing off “OutOfMemoryError” messages. Worse still, your significant other forcibly relocated you to the couch and told you not to return until your pager stops buzzing. Sound familiar? If so, you may have a case of memory leak induced insomnia, but fortunately we’ve got a cure for what ails you. This tutorial will teach you everything you need to know to ease your suffering, including what memory leaks are, why they happen, and how to diagnose and fix ‘em. In this article we’ll focus on techniques that will enable you to address memory leaks with any commercial or free/open source memory profiler; we’re not here to recommend one tool over another. After all, the most important thing is that you fix the problem and get some rest, not the tool you use to get it done. Before You Start Ever heard the story about how Java has “automatic memory management”—you know, the one that someone in marketing upgraded to an epic tale about how you’ll never have to worry about memory leaks ever again? As is often the case, the truth is more complex than the marketing department made it out to be. While it’s true that Java’s garbage collector (GC) helps to eliminate the most common memory leak issues from applications, it is unfortunately still possible to experience memory leaks in Java. However, they happen a lot less often than they used to in the C or C++ days. Many people believe that black magic and complex tools are required to fix memory leaks. This undeserved reputation is caused by the lack of good explanations of what they are and what to do when you encounter them. But with the proper tools and knowledge to fix memory leaks, they aren’t nearly as intimidating. Methodology In this article we’ll cover everything from memory leak basics to analyzing heap dumps, so—whether you’re an experienced Java developer or encountering Java memory leaks for the first time—you’ll be better prepared to deal with memory leaks by the time you reach the conclusion. We won’t outline a series of steps, like “do ABC with commercial tool XYZ and don’t ask why,” as that approach doesn’t work and implies that remedies are more complex then they really are. Instead, we’ll give you the background information necessary to address memory leaks, with emphasis placed on particular steps you’ll need to execute. Similarly, we’ll assume that you can learn on your own how to use the memory profiler of your choice; what’s missing is an understanding of what the tool is trying to do and why, so that will be the focus of this article. Java will be used for all examples, so all information in this article directly applies to Java applications running standalone or as a part of J2EE/JEE/Tomcat-based application server. But remember, although our primary focus is on Java, most of the process of diagnosing and fixing memory leaks described herein applies to other languages with garbage collectors. So even if you’re using Ruby, C#, or Python, there should be something for you in this article. What Are Memory Leaks? Let’s start by describing how memory leaks happen in Java. Java implements automatic garbage collection (GC), and once you stop using an object you can depend on the garbage collector to collect it. While additional details of the collection process are important when tuning GC performance, for the sole purpose of fixing a memory leak we can safely ignore them. When is memory eligible for GC? Let’s take a look at an example: We don't have to do anything special to make an object eligible for GC—we just eliminate any references to it, and it "magically" disappears and stops using memory. That's why we say that Java performs "automatic" GC. Why "eligible" for GC? Because objects are not collected immediately. GC is not instantaneous and comes with some performance impacts. Consequently, Java doesn't immediately collect every object that is eligible for collection; it typically postpones collection until a more convenient time later on. The way to think about GC in Java is that it's a "lazy bachelor" that hates taking out the trash and typically postpones the process for some period of time. However, if the trash can begins to overflow, Java immediately takes it out. In other words, if memory becomes scarce, Java immediately runs GC to free memory. Since we don't need to do anything special in order to dispose of objects in Java, how do memory leaks happen in Java? Memory leaks occur when a program never stops using an object, thus keeping a permanent reference to it. Let's take a look at an example that helps illustrate this point. The following code will cause all available memory in the JVM to be exhausted: When no more memory is remaining, an OutOfMemoryError alert will be thrown and generate an exception like this: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at MemoryLeakDemo.main(MemoryLeakDemo.java:14) In the example above, we continue adding new elements to the list memoryLeakArea without ever removing them. In addition, we keep references to the memoryLeakArea, thereby preventing GC from collecting the list itself. So although there is GC available, it cannot help because we are still using memory. The more time passes the more memory we use, which in effect requires an infinite amount memory for this program to continue running. This is an example of unbounded memory leak—the longer the program runs, the more memory it takes. So even if the memory size is increased, the application will still run out of memory at a later date. Meat & Potatoes Fixing Memory Leaks As illustrated by the flowchart below, the process of fixing memory leaks is fairly simple: Memory leaks are misunderstood creatures. Just getting an OutOfMemoryError alert doesn’t necessary mean that you're suffering from a memory leak. So, before you dive into "fixing" the problem, you must first find out whether or not a memory leak actually exists. If a memory leak does in fact exist, the next step is to determine which objects are leaking and uncover the source of the memory leak. Then, you fix it. We'll skip past the initial steps and dive right into diagnosing whether or not the problem is a memory leak. Is My Program Leaking Memory? Not every OutOfMemoryError alert indicates that a program is suffering from a memory leak. Some programs simply need more memory to run. In other words, some OutOfMemoryError alerts are caused by the load, not by the passage of time, and as a result they indicate the need for more memory in a program rather than a memory leak. To distinguish between a memory leak and an application that simply needs more memory, we need to look at the "peak load" concept. When program has just started no users have yet used it, and as a result it typically needs much less memory then when thousands of users are interacting with it. Thus, measuring memory usage immediately after a program starts is not the best way to gauge how much memory it needs! To measure how much memory an application needs, memory size measurements should be taken at the time of peak load—when it is most heavily used. The graph below shows the memory usage in a healthy Java application that does not suffer from memory leaks, with the peak load occurring around 10 AM and application usage drastically decreasing at 5 PM. Naturally, the peak load on business applications often correlates with normal business hours. The application illustrated by the chart above reaches its peak load around 10 AM and needs around 900MB of memory to run. This is normal behavior for an application suffering from no memory leaks; the difference in memory requirements throughout the day is caused solely by the user load. Now, let's suppose that we have a memory leak in the application. The primary characteristic of memory leaks is that memory requirements increase as a function of time, not as a function of the load. Let's see how the application would look after running for a few days with a memory leak and the same peak user loads reached around 10 AM every day: Because peak loads on the system are similar every morning but memory usage is growing over a period of a few days, this picture indicates a strong possibility of memory leaks. If the program eventually started suffering from OutOfMemory exceptions, it would be a very strong indication that there’s a problem with memory leaks. The picture above shows a memory leak of about 100MB per day. Note that the key to this example is that the only thing changing is the amount of time the system is up—the system peak load doesn’t change over time. This is not the case for all businesses. For example, the peak load for a tax preparation service is seasonal, as there are likely more users on the system in April than July. There is one special case that should be noted here: a program that needs to be restarted periodically in order to prevent it from crashing with an OutOfMemoryError alert. Imagine that on the previous graph the max memory size was 1100MB. If the program started with about 900MB of memory used, it would take about 48 hours to crash because it leaks about 100MB of memory per day. Similarly, if the max memory size was set to 1000MB, the program would crash every 24 hours. However, if the program was regularly restarted more often than this interval, it would appear that all is fine. Regularly scheduled restarts may appear to help, but also might make “upward sloping memory use” (as shown in the previous graph) more difficult to notice because the graph is cut short before the pattern emerges. In a case like this, you’ll need to look more carefully at the memory usage, or try to increase the available memory so that it’s easier to see the pattern. Monitoring Memory in Java As you are already aware, you need to measure memory that is free and used inside of the JVM, not memory that the JVM process is using. In other words, the top/Task Manager/Activity Monitor will measure how much memory your Java process is using, but that’s not what you need. You need a tool that can look inside the JVM process and tell you how much memory is available for your program running inside the JVM. In addition, keep in mind that Java GC is not a constant process—it runs in intervals. The memory usage that you see in the JVM is usually higher then what your program needs at the moment, as GC hasn’t yet run. Remember, lazy bachelors usually have some trash in their apartments waiting to be taken out. So, what you need to investigate is not current memory usage, but rather the average usage over a long period of time. For example, if your program is currently using 100MB of memory and five seconds later it’s using 101MB, that’s not an indication of a memory leak because GC might free up memory when it eventually runs. But if your program’s memory usage increases over a long period of time under constant usage, you might have trouble on your hands. There are a couple of options for measuring the amount of memory a program uses. The simplest one, which does not require any tools and works even with production systems, is the Verbose GC log. Verbose GC Log The Verbose GC log is defined when the JVM process is started. There are a couple of switches that can be used: -verbose:gc — prints basic information about GC to the standard output -XX:+PrintGCTimeStamps — prints the times that GC executes -XX:+PrintGCDetails — prints statistics about different regions of memory in the JVM -Xloggc: — logs the results of GC in the given file The following is an example of the output generated for Tomcat running in the default configuration with all of the previous switches enabled: 1.854: [GC 1.854: [DefNew: 570K->62K(576K), 0.0012355 secs] 2623K->2175K(3980K), 0.0012922 secs] 1.871: [GC 1.871: [DefNew: 574K->55K(576K), 0.0009810 secs] 2687K->2229K(3980K), 0.0010752 secs] 1.881: [GC 1.881: [DefNew: 567K->30K(576K), 0.0007417 secs] 2741K->2257K(3980K), 0.0007947 secs] 1.890: [GC 1.890: [DefNew: 542K->64K(576K), 0.0012155 secs] 2769K->2295K(3980K), 0.0012808 secs] The most important set of numbers is located in the second column after the second -> (e.g., in the top line shown it is 2623K->2175K(3980K). These numbers indicate that as a result of GC, we are using around 2200K of memory at the end of each GC cycle. This trace is not an indication of a memory leak—it shows a short-term trend with less then a second between samples, and that’s why we must observe long-term trends. However, if the Verbose GC log showed that the program was using around 2200K of memory after running for two days, and after running for 10 days it was using 2GB of memory (even after GC had just run), we could then conclude that there’s a memory leak. All the information that needs to be collected in order to determine if a memory leak exists can be found in the results of the Verbose GC logs. The other memory monitoring tools we’ll cover in this article simply provide more information in a form that’s easier to interpret. Monitoring the Java Process The following approach works for any Java process, including standalone clients as well as application servers like JBoss and servlet containers like Tomcat. It is based on starting the Java process with JMX monitoring enabled and attaching with the JMX monitoring tools. We’ll use Tomcat in the following example. To start Tomcat or the Java process with JMX monitoring enabled, use the following options when starting JVM: -Dcom.sun.management.jmxremote — enables JMX monitoring -Dcom.sun.management.jmxremote.port= — controls the port for JMX monitoring Note that if you’re on a production system, you’ll most likely want to secure your JVM before running it with these parameters. For that, you can specify these additional options: com.sun.management.jmxremote.ssl com.sun.management.jmxremote.authenticate Once started, you can use JConsole or VisualVM to attach to the process. Note that later JDK 6 versions include VisualVM. This is an example of the JConsole monitoring Tomcat. As shown in the example below, click on the Memory tab to get memory information: Again, we're interested in the long-term trends of heap memory usage, not trends from just a few minutes of running. Finally, note that monitoring tools like Hyperic can typically show historical trends over longer periods of time than VisualVM or JConsole. In addition, tools like Hyperic allow for more fine-grained control over operations that are permitted by users. As a result, we recommend using monitoring tools for production systems use, while all the other tools discussed in this section are more appropriate for developers or "first aid" in the absence of the real monitoring tools. Speed Kills We now know how to find out whether or not a memory leak exists, and we can even determine the speed at which we're leaking memory (e.g., 100MB per day). But are we better off with a "slow" leak (e.g., 1MB per day) or a "fast" leak (e.g., 500MB/hour)? It depends. With a "slow" leak, it takes longer for the system to run out of memory. For example, if we have 256MB of free memory remaining at the peak load time, it can take quite a long time to run out of memory with a 1MB/day memory leak. On the other hand, a faster memory leak makes it easier to reproduce and fix the problem. One of the best ways to quickly determine whether there's a fast memory leak or you simply need more memory to run at the peak time is to allot more memory to the program and see what happens. If you increase the available heap in a Java program and the time between crashes increases, you are likely suffering from memory leak. Let's assume that we're lucky enough to have a fast memory leak—on the order of 100MB per hour—on our hands, and that the initial memory picture looks like this: In this example, we can expect to run out of memory after about five hours of running time. How To Find Leaked Objects in a Fast Memory Leak Somewhat surprisingly, it is much easier to debug large memory leaks than small memory leaks. The reason is that memory leaks present a sort of "needle in the haystack" type problem—you need to find the leaked objects amongst all the other objects in the program. Suppose that the program we're debugging just ran out of memory. If the program initially had 512MB of free memory and now has none, it is obvious that the leaked objects used about 512MB of memory. The figure below illustrates this example: In this situation, half of the objects in memory have been leaked! If we randomly select an object, there's a 50% chance that it has leaked. And as memory leaks usually consist of objects of a few classes, you can get a really good start on determining which objects are leaking memory by sorting memory usage based on aggregate memory use of all objects of the same class. What if the ratio is less favorable (e.g., 512MB heap size, initially used memory 480MB, and 32MB of leaked objects at the end)? If you have an easy to reproduce memory leak, you can increase the heap size because an unbounded memory leak will eventually fill any amount of memory allotted to the program. So, you can increase the heap to 1GB, reproduce the memory leak, and get 544MB of leaked objects in a 1GB of heap. If we had a way to look at the "complete picture" of memory, it would be fairly easy to pinpoint leaked objects. Fortunately, there's a way to do exactly this: heap dump the process. Dump me Gently A heap dump is a list of objects in the memory of JVM as well as the content of the memory occupied by those objects. It preserves the value of any attributes of the objects, including references to other objects. In other words, a heap dump gives you a complete picture of the memory. There are multiple tools that allow you to dump heap in a Java process: If you're using JDK 6, you can use tool called jmap on any platform. If you're using JDK 5, the situation is slightly more complex: If you're running UNIX (Linux, Solaris, OS X) with JDK 5 you can use jmap. If you're using JDK 5 update 14 or later, you can use the -XX:+HeapDumpOnCtrlBreak option when starting JVM, then use the CTRL+BREAK key combination on Windows (or CTRL + \ on UNIX) to dump the heap. If you're running Windows and using JDK 5 pre-update 14, you'll soon wish you weren't. Trying to reproduce the problem with a more recent JDK is probably the best bet here. Some tools like VisualVM and memory profilers allow you to initiate a heap dump from the GUI, but you don’t need any fancy tools here—jmap will do just fine. As it provides the most general case, we'll use jmap in the next example. Before you dump heap, be sure to keep the following issues in mind: Programs in the JVM should be paused for the duration of the heap dump, which might take anywhere from ten seconds to several minutes. Many enterprise applications—and users of those applications—don’t take kindly to pauses that long, which may cause various timeouts to expire. So don’t try this at home or in production (unless the application is already a goner)! Heap dumps are saved on disk, and the files might be fairly large. A good rule is to make sure that you have at least twice the size of the physical memory free on the disk before you initiate a memory dump. With those final words of caution out of the way, you should now be ready to run the following command: jmap -heap:live,format=b,file=FILENAME PID Note that the -F option, which will dump non-responsive programs, might be useful on UNIX systems, but is not available on Windows. Note also that JDK 6 includes the option +XX:+HeapDumpOnOutOfMemoryError that will dump heap whenever the OutOfMemoryError alert is encountered. This can be a useful option, but keep in mind that it has the potential to consume significant amounts of disk space. You now have a heap dump in the file FILENAME and are ready to analyze it. What's In "Leaked" Memory? With the heap dump complete, we can now take a look at the memory and find out what's really causing the memory leak. Suppose that objects are holding references to each other as illustrated by the picture below. For the sake of easy calculation, let's assume that each object is 100 bytes, so that all of them together occupy 600 bytes of memory. Now, suppose that the program holds reference to object A for a prolonged period of time. As a result, objects B, C, D, E, and F are all ineligible for garbage collection, and we have the following amount of memory leaking: 100 bytes for object A 500 bytes for objects B, C, D, E and F that are retained due to the retention of object A So, holding reference to object A causes a memory leak of 600 bytes. The shallow heap of object A is 100 bytes (object A itself), and the retained heap of object A is 600 bytes. Although objects A through F are all leaked, the real cause of the memory leak is the program holding reference to object A. So how can we fix the root cause of this leak? If we first identify that object F is leaked, we can follow the reference chain back through objects D, C and A to find the cause of the memory leak. However, there are some complications to this "follow the reference chain" process: Reference chains can be really long, so manually following them can be time consuming An object is sometimes retained by more then one object, and there can even be circles involved as shown in the picture below: If we start following inbound references from object F in this example, we have to choose between following object C or object D. In addition, there's the possibility of getting caught in a circle by repeatedly following the path between objects D, E and B. On this small diagram it's easy to see that the root cause is holding object A, but when you're dealing with a situation that involves hundreds of thousands of objects (as any self-respecting memory leak does) you quickly realize that manually following the reference chain be very complex and time consuming. This is where some shortcuts can come in handy: If we had a tool that allowed us to play a "what would happen if I remove this reference" type of guessing game, we could run experiments that help locate the cause. For example, we could see that if we removed reference from "Cause of Leak" to A in the diagram above, objects A through F would all be freed. Some tools (like Quest's JProbe) have this capability. If the memory leak is large and we have a tool that allows us to sort objects by retained heap, we'll get an even greater head start because the objects with the largest retained heap are usually the cause of large memory leaks. Now that we understand what memory leaks are and how they can be corrected, let's find out how to fix them by analyzing heap dumps. Tools for Dealing with Heap Dumps Strictly speaking, you don’t need any tools that are not already part of the JDK. JDK ships with a tool called jhat, which you can use to inspect the heap dump. The process of fixing memory leaks is the same with all tools, and it's our opinion that no single tool is "light years" ahead of the others when it comes to fixing memory leaks. Although jhat will get the job done, better tools often provide a few extra helpful features: Present a better summary of heap statistics. Sort objects by retained heap. In other words, some tools can tell you the memory usage of an object and all other objects that are referenced by it, as well as list the objects referenced by other objects. This makes it much faster to diagnose the cause of a memory leak. Function on machines that have less memory then the size of the heap dump. For example, they'll allow you to analyze a 16GB heap dump from your server on a machine with only 1GB of physical memory. There are several free tools that are useful for analyzing heap dumps in Java. One that's widely used and is even included with the later versions of the JVM 6 is VisualVM. VisualVM is nice tool that gives you just enough to resolve memory leaks, and it shows heap dumps and relations between objects in graphical form. Feature-wise, one step above VisualVM is the Eclipse Memory Analyzer Tool (MAT), a free tool that includes a lot of additional options. Although it's still in incubation phase as of publication of this article, MAT is free and we've found it to be extremely useful. Commercial products like JProfiler, YourKit, and JProbe are also excellent tools for debugging memory leaks. These applications include a few options that go above and beyond VisualVM and MAT, but they're certainly not necessary to successfully debug memory leaks. Unless you already have a license for one of these commercial tools, we recommend trying MAT first. Analyzing the Heap Dump for Fast Memory Leaks It's usually easy to find a fast memory leak. A lot of memory is leaked, and you simply need to find the big hog that leaked all that memory. Analyzing the heap dump is done in order to: Find objects that are "leaking" Find the root cause of the memory leak If you have a fast memory leak and are able to reproduce it in such a way as to make the leaked objects a significant portion of the final memory picture, then determining which objects are leaked is simple because they occupy a significant portion of the memory. To determine which objects are leaked, sort the classes by total memory usage of all instances of each class. The objects near the top of the list are usually the leaked objects. Then, you can follow the reference chain to them until you find the cause of the memory leak. A useful heuristic here is that if you sort all the objects by retained heap, you can find the objects that are likely the root cause of the memory leak. Again, an example: If we assume that each instance of object C is 100 bytes, then holding object A resulted in a memory leak of almost 1GB! In other words, the retained heap of object A in this example is about 1GB. So, if we were investigating this memory leak, it would be fairly obvious that we should start there. Remember, the Eclipse Memory Analyzer Tool (MAT) allows you to sort the heap dump by the retained heap usage of the objects, so it would have been easy to use MAT to determine the retained heap of object A. Slow and Small Memory Leaks It always helps if you can easily reproduce the problem, but what if the memory leak is really small and slow? What should you do if after a few days you can reproduce only a 10MB leak in a heap that contains 2GB of objects? In situations like this, looking for the cause of the problem can feel like finding a needle in the haystack. One solution in this case is the brute force approach—simply devote enough time looking at the objects to find the problem. While that will ultimately work, there's another way to address the problem if you have an idea of which operations are leaking memory. If you know or suspect which operations leak memory, you can find even relatively slow memory leaks. The secret here is to compare heap dumps, as this highlights objects that are present in one heap dump but not in another. For example, suppose that objects A, B, and C are present in the first heap dump, while objects A, B, C, D, and E are present in the second heap dump. In this case, objects D and E are the difference. Comparing heap dumps makes it easier to find memory leaks because it effectively reduces the size of the "haystack" containing your needle. Comparing heap dumps can easily reduce the size of the objects for which you need to examine heaps from 2GBs to a few KB. Many commercial profilers as well as Eclipse MAT can be used to compare heaps. Once you've selected a tool that can compare heaps, simply follow the process explained in the following diagram: In effect, what you’re doing is creating two snapshots, one before and one after executing the use case that you know is leaking memory. That way, you significantly reduce the number of objects that you need to investigate to find leaked objects. After you find the objects that leaked, you can find the cause of the memory leak as described in the previous section. Why perform garbage collection before taking a snapshot? Because some tools won’t automatically perform GC for you, and consequently the snapshots might include objects that are no longer reachable. Many modern tools will perform GC before taking the snapshot, but if you are unsure whether or not your tool performs full GC before taking a snapshot it is recommended that you do so yourself. That way you don’t have to worry about objects that are eligible for GC but have not yet been collected. Among the objects that are created during the use case, some are supposed to be there because they are the result of the use case execution (e.g., we created a new customer, and that customer object should be retained), while other objects are the memory leak. However, since the only objects present in the difference between the snapshots are the ones that were created during the use case, the size of the haystack you need to look through is significantly reduced. Practical Problems Why do memory leaks sometimes take a long time to fix if the process is this simple? In our opinion, the main reasons are: There isn’t a sufficient amount of easily available information about fixing Java memory leaks. We hope this article will help improve the situation. It can be difficult to reliably reproduce an issue, and a lot of time is typically required to reproduce an issue before you can really start addressing it. People lack the right tools for the job—in particular, memory profilers. With many free profilers available and reasonably low prices for commercial profilers, there’s no reason you shouldn’t have good tools in your toolbox. There are a few practical issues with the use of tools and techniques that might be perceived as road blocks the first time someone tries them. We’ll address some of these issues below. Fortunately, the practical issues most commonly encountered aren’t very difficult to solve. The most common problems are: You can’t load the snapshot because you don’t have enough memory in your development box. For example, this can easily happen if you have 64-bit servers with 8GB of memory allocated to the JVM. You might not have physical memory in your development box, or you might be using the 32-bit JVM on your development box, but the tool you’re using insists on loading most of the snapshot into memory. If you like the idea of having a really powerful development box, use this situation to your advantage and ask your boss for a new machine. Alternatively, try using a tool with less of an appetite for memory, like Eclipse MAT. You can’t increase your memory on the server to get a bigger set of leaked objects, yet you need a bigger set of leaked objects to speed up the process of finding the memory leak. This often happens with 32-bit JVMs. One option in this situation is to apply the techniques described above for finding slow memory leaks, although this approach requires a significant amount of time. Alternatively, you can try to reproduce problem on a 64-bit JVM, which will allow you to increase the memory. If the problem causing the memory leak is in your application, changing the JVM is extremely unlikely to “hide” the leak. The memory leak is not in objects of just one class—you’re leaking objects from thousands of classes, so it’s difficult to find leaked objects. Or, your graph of object relations is so complex that you can find the initially-leaked objects but can’t locate the cause. In either case, you’ve got quite a pickle on your hands. The only words of encouragement we can offer are that the situation will eventually improve, as the longer you track object references the better you’ll get at it. One other piece of advice: you should probably call your significant other and let him or her know that you won’t be home for dinner. You’re getting an OutOfMemoryError alert with a new, different format. For example, you might be looking at something like this: Exception in thread “pool-2-thread-1″ java.lang.OutOfMemoryError: GC overhead limit exceeded. This message can happen with some GC settings that limit the overhead of the GC. It often indicates a memory leak, although there are some corner cases in which problems with GC performance can cause it. The best way to distinguish the cause of this message (is it a GC misconfiguration or a memory leak?) is to examine the pattern of errors. If they happen with regularity (in other words, they’re a function of time the program is running), then it’s a memory leak. If they happen only occasionally under heavy loads and clear later (in other words, they’re not a function of running time), they might indicate a need to tweak the GC or increase available memory. 3rd party caching systems can be setup so that caches are allowed to expand and fill up the available memory the program doesn’t need. However, if the program no longer needs more memory, the cache automatically releases it. The point here is that in the absence of any OutOfMemoryError alerts, continually increasing memory usage does not necessary indicate a memory leak. Both “used memory grows as function of time” and “free memory eventually runs out” conditions are necessary in order to determine the presence of a memory leak. Finishing Up This article described the methodology and techniques necessary to fix memory leaks. These techniques are universal—they apply to any profiler, and some of them even apply to languages other than Java that use garbage collection. The information presented above should give you everything you need in order to use the tools of your choice to investigate and fix memory leaks. The proper usage of specific tools will be discussed in future articles, which will focus primarily on free tools and problems in Java. However, if there’s enough interest we might tackle other languages as well. If you’re interested in follow-up articles related to memory leak resolution in a different environment or language, please leave a comment or contact us at docs-at-openlogic.com. Additional Resources The following links provide additional information on the behavior of GC in Java as well as different tools that you may find useful. http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf http://java.sun.com/developer/technicalArticles/Programming/GCPortal/ http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html http://www.eclipse.org/mat/ https://visualvm.dev.java.net/ http://www.ej-technologies.com/products/jprofiler/overview.html http://www.yourkit.com/ http://www.quest.com/jprobe/ http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jmap.html http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html The above article originally appeared on Wazi, a clearinghouse for the timeliest thinking on open source software."
March 18, 2009
by Veljko Krunic
· 295,160 Views · 2 Likes
article thumbnail
Dynamic Java Programming With Rule Engine
Rules are statements that define business procedures and policies. Normally, every business application contains many embedded business rules that determine business process flow and execution. When we program these rules with static languages like Java, there is no problem at all. However, the problem is that some business rules change over time very frequently. It is very expensive to fix these changing parts since it requires new versions and software development cycles; develop, deploy on a test server, test, package, deploy on a production server etc. Here “Dynamic Programming” comes to our rescue. Our program doesn’t have to necessarily be written with a dynamic language. We can use this advantage in Java with Rule Engine mechanism. Rule Engines don’t serve only the dynamic programming. They evaluate rules and extract some inferences by using some algorithms like RETE. This inference mechanism is hardly possible with many “if-else” statements in normal program code. Rule Engines have generally 4 types of rules; Constraint Rule: We may need to define restriction or limits for some entities. “A customer sales order amount must be lower than 100 if the plant is over capacity ” Validation Rule: Some validation rules may change over time. “Production order can’t be accepted if current day is Sunday” Action Rule: Some actions may be triggered. “Send a notification e-mail if a purchase order is over $1000 to boss”. Computation Rule: Some formula may be executed. “Discount is 20% if customer is Mr. Anderson” We can use Rule Engines on the market but even we can write our own Rule Engine to benefit the dynamic programming in Java. By using Java compilers, classloaders, it is not so much hard. My favorite rule definition format is Java source code instead of some natural languages or XML. This can also reduce the overall development cost of your custom rule engine. By writing Java rule definitions, we can also use the IDE debugger features without learning a new syntax. If we develop our own Rule Engine, we may skip the inference engine part since we only want dynamic programming feature for now. Best place for plugging rule execution is database access layer. Rules should work invisibly behind the database objects. In some cases, some rules might be invoked from static program codes. The benefits that we have when using Rule Engine are: Dynamic programming with hotswapping feature that provides fix&run easiness. Minimal cost of business rule changes that occurs many times. Accumulation of business rules in one place that may result re-usability and code repetition prevention. Dynamic validations that may change over time. Constraints can be defines for default values, assertion checks etc. In any time rule may be turned off/on. Some of the features of Rule Engine may remind us database constraints and triggers. Using database may hinder portability and require DBA involvement. I think Rule Engine is better for business rule definitions. These database features may be used for other purposes. The critical question we should ask ourselves is which part of business rules should be taken into Rule Engine from the programs. The answer is “frequently changing” rules. Some changing parts can be defined within parameter tables (processing options) but we can’t define computation or many if-else statements in these tables. But we should prefer parameter table if it is enough for the problem. Links: http://java.sun.com/developer/technicalArticles/J2SE/JavaRule.html http://www.javaworld.com/javaworld/jw-04-2005/jw-0425-ruleengine.html http://www.javaworld.com/javaworld/jw-06-2006/jw-0612-dynamic.html http://www.amazon.com/Principles-Business-Addison-Wesley-Information-Technology/dp/0201788934/ref=sr_1_3?ie=UTF8&s=books&qid=1236927606&sr=1-3 http://www.amazon.com/Business-Rules-Applied-Building-Approach/dp/0471412937/ref=sr_1_1?ie=UTF8&s=books&qid=1236927606&sr=1-1
March 13, 2009
by Adam Brown
· 33,164 Views
article thumbnail
Wicket Tutorial Series: Setting Up the Project
Each day this week will feature a new article with an in-depth look at the creation process behind setting up a Java project and implementing the frontend with Apache Wicket. Wicket is a Java web application framework which allows “Designers” (people good with Dreamweaver) and “Developers” (people good with Java and Databases) to collaborate on a project with minimal chances of stepping on each other’s toes or wearing each other’s hats. The beauty of Wicket is that it uses plain xhtml pages as it’s templating markup. This means that html pages can be loaded into Dreamweaver (or whatever tool the Designer is comfortable with) and they will look very close to the same as they would when rendered on the deployment web server. Workflow The basic workflow involved in creating and maintaining html rendered by Wicket is as follows: The Designer creates the html for the website and fleshes it out with “mock” sections. For instance in the application we intend to create during our Five Days of Wicket will be a pastebin application called “Mystic Paste”. In our application we’ll have an “Add Code to Mystic Paste” page, mock data might include some user created content in the textarea of the page. All css/images, etc… are setup such that if they were to be put on a webserver, everything would work. The Developer needs to flesh out the dynamic areas of the webpage, that is, he needs to instruct Wicket where it will need to show information from the server. The developer does this by decorating the designer’s html page with special Wicket tags and attributes. Because these tags and attributes are just considered part of another namespace separate from xhtml’s, editors like Dreamweaver and browsers will simply ignore them - It is important to note: The developer will still keep the “mocked” sections of the page intact, this is so the page renders and looks fleshed out on its own. The mocked sections will be replaced by real data when rendered by Wicket. The Developer hands the file back to the Designer. The Designer is free to make further edits, so long as he/she does not remove or manipulate the Wicket tags and attributes present in the file. If the Designer does need to remove any Wicket tags or attributes, they need to consult the Developer as such an action will “break” the webpage when Wicket renders it. Example Wicket Page Here is an example of a Wicket page. This example was taken from Manning Publishing’s book “Wicket in Action”: ... Gouda Gouda is a Dutch... $1.99 add to cart Emmental Emmental is a Swiss che... $2.99 add to cart ... This looks almost 100% like a normal webpage would look, the only difference is the addition of the “wicket:XXX” attributes and tags sprinkled through the document. The parts of the document using the special Wicket namespace modifiers will be replaced/removed in the final markup when Wicket renders the page to the user’s browser. Notice the “” element? This is where your designer can put a “mocked” version of what that area of the page should look like. You as a developer can take that mocked html and divide it out into a template that is dynamically driven from the backend. Here is how the final page looks if you were to simply load the page into a web browser (or Dreamweaver) from your hard drive: Preparing for Setup Deviating a bit from the Standard Wicket Convention One of the first things a developer notices when starting out with Wicket is the convention where Wicket likes having its html template files live at the same level and in the same packages as it’s Java source files. Sure you can jump through hoops to get Wicket to load the html template files from elsewhere but a nice compromise is to simply keep your html template files within the same package directory structure but in a source folder separate from the Java classes. Why? Well quite simply to keep your designers (Dreamweaver folks!) from having to grab Java source files along with the html files they are working on. It will just confuse them and clutter their directories. You can of course stick with the typical “Java source files along side html” convention if you wish, but I find it much cleaner to separate them during design time, and have Maven combine them only at build time into the target war (which it will gladly do automagically). Project Folder Structure . |-- pom.xml |-- src | `-- main | |-- filters | |-- java | | `-- com | | `-- mysticcoders | | `-- mysticpaste | | |-- model | | |-- persistence | | | |-- exception | | | |-- hibernate | | |-- services | | `-- web | | |-- ajax | | |-- pages | | |-- panels | | `-- servlet | |-- resources | | |-- com | | | `-- mysticcoders | | | `-- mysticpaste | | | |-- spring | | | `-- web | | | |-- ajax | | | |-- pages | | | `-- panels | | `-- log4j.properties | |-- sql | `-- webapp | |-- WEB-INF | |-- images | |-- js | `-- css src/main: maven builds source and resources from this directory to the main deployable target (i.e. our war file) filters: we keep a set of “filters” files that maven can use to interpolate variables at build time. What does this mean? It means that inside your configuration files, the files you use to setup database connections or file paths, you can insert variable place holders like ${db.host}. When maven does a build, it looks up the correct filter file to use and looks for the key=value part corresponding to “db.host” and inserts it into the configuration file for you. This ensures that you are able to configure your application per environment you deploy to (i.e. DEV, QA, PROD, etc…) by having different filter files with the same keys but different values. For more information see Maven’s documentation on filtering resource. java/*: this folder will contain all of the application’s source code. Everything from the database access code, wicket code and services code for the mysticpaste application (see below). model: all “domain” classes, that is, classes that represent the objects in the application. For mysticpaste you’ll see classes like “PasteItem” which represents an item pasted to the mysticpaste. persistence: at this level of the persistence package a list of interfaces will be kept. The interfaces comprise the basic access layer the services layer will use to save, retrieve and update items to/from the paste bin. exception: the peristence layer needs to tell the services layer when things have gone wrong. It does this via delcaring and throwing exceptions. hibernate: such is our case, our persistence interfaces will be implemented via the ORM known as Hibernate. This package will store all of the custom hibernate implementations and hibernate specific classes services: The services layer will be stored here. Both the generic interfaces and their implementation classes. The persistence layer will be injected via spring. web: this folder is where our Wicket classes will reside and it’s split into several category packages which are as follows: ajax: mysticpaste uses Ajax to render portions of its UI. The wicket classes which render the xml/html to be injected dynamically into the page are stored here. pages: standard Wicket page classes which are used throughout the application are stored here panels: reusable panel classes are stored here. Panels may be included within Wicket pages for sake of templating servlet: any run of the mill servlet code we need is stored here. A good example might be an ImageUploadServlet resources/*: the resources folder will hold our non-java based files. Noteably html files and spring confguration files spring: Holds any spring configuration files needed to wire the services and persistence layer web: this folder and all subfolders mirror the packages under src/main/java/…/web and hold the .html files that the Wicket page/panel classes use as their templates. As described above, a “standard” wicket application simply stores the .html files along side their Wicket classes under src/main/java/…/web, however we want to keep these files separate from the Java source so as to keep the directory our designers checkout from version control contianing only the files they need to work on. sql: any sql scripts we need to keep handy for building the mysticpaste database. webapp: this folder will keep the files which live at the base directory of our war file WEB-INF: where you keep your web.xml file images: any image resource, .gif/.png/.jpg files your webapp will reference js: javascript files your webapp will reference css: style sheets your webapp uses src/test/*: All files which reside under this folder are test classes and resources needed to support the tests. Maven will build everything under src/main/java and add it to the class path of the JUnit or TestNG classes you create. java: JUnit or TestNG test classes which will be run during a build resources: resource files which are needed to support the tests Getting Started Since we are using Maven as our build tool we can take advantage of the fact that the fine folks at the Wicket project have created a specialized “archetype“ which creates a skeleton web application complete with a folder structure which mimics roughly what we have outlined above and Maven pom.xml file used to build a war. The Wicket contributors have even gone one step further and have created a little web page which will, based off a few drop down options, generate the maven command you need to execute in order to create the boiler plate Wicket project. You can find this web page over on the Apache Wicket site under the “Quick Start” link. Copying the above Maven command creates a Skeleton Wicket Project To be precise, the command I used was: mvn archetype:create -DarchetypeGroupId=org.apache.wicket -DarchetypeArtifactId=wicket-archetype-quickstart -DarchetypeVersion=1.3.5 -DgroupId=com.mysticcoders -DartifactId=mysticpaste And I ended up with the following folder structure: . `-- mysticpaste |-- pom.xml `-- src |-- main | |-- java | | `-- com | | `-- mysticcoders | | |-- HomePage.html | | |-- HomePage.java | | `-- WicketApplication.java | |-- resources | | `-- log4j.properties | `-- webapp | `-- WEB-INF | `-- web.xml `-- test `-- java `-- com `-- mysticcoders |-- Start.java `-- TestHomePage.java Now obviously we’ll have to rearrange a few things, for instance I want my base package to be com.mysticcoders.mysticpaste, but that’s easy enough to do once we are in an IDE. For now, let’s test this example webapp out and see if it works. To do that switch into the mysticpaste directory (the directory that has pom.xml in it) and type the following: mvn jetty:run This will start up a Jetty webapp container running on port 8080 (if you have something running there already, use the -Djetty.port= option). Startup a webbrowser and navigate to http://localhost:8080/mysticpaste/ You should see: Your IDE Sooner or later you’re going to want to crack open your IDE and start hacking away. Maven makes this extremely easy by allowing you to create IDE specific project files based off of the Maven pom.xml file. Eclipse mvn eclipse:eclipse For eclipse you’ll also have to set the M2_REPO classpath variable for the workspace your project resides under. Do this by entering the following command: mvn -Declipse.workspace= eclipse:add-maven-repo IntelliJ IDEA mvn idea:idea -OR- in IDEA 7+ simply open the pom.xml file Netbeans Netbeans supports maven out of the box, just “Open Project” and choose the mysticpaste directory that contains the pom.xml file When generating the project files through Maven, the project is setup such that classpath entries point to your local Maven repository (i.e. ~/.m2/repository, or C:Documents and Settingsyourusername.M2repository on Windows). It also sets up src/main/java, src/main/resources as “source folders”. You may add other folders to the source folder list as per your IDE if needed, the only thing you have to remember is if you ever use mvn eclipse:clean followed by mvn eclipse:eclipse again, those other source folders will have to be readded through your IDE. Instead, you should add the source/resource folders directly to your pom.xml, this way they will be maintained. Spring The Mystic Paste application will use Spring, and really you should too. Unless you have been hiding under a rock or work in a corporate environment so lame as to which technologies newer than 2002 are forbidden you should learn to accept Spring as a defacto standard. Dependency injection for the win! We add the following to our pom.xml: org.apache.wicket wicket-spring-annot org.springframework spring org.springframework spring-test org.springframework spring-tx wicket-spring-annot: allows us to wire our Wicket application via handy dependency injection annotations (i.e. @SpringBean, see Wicket documentation for more detail) spring: is just the core spring libraries spring-test: is a set of Spring integration classes for Unit testing spring-tx: is the Spring Transaction Management api for declarative transactions web.xml additions for Spring In order for Spring to manage our Wicket application we need to setup the Wicket filter with a Spring-aware application factory. This allows us to wire our Wicket Application class in our applicationContext.xml file, which is really handy if you have a services and configuration settings you want to inject into the Wicket Application object so the rest of your application can access them. To do this, we change the original Wicket filter like so: wicket.mysticpaste org.apache.wicket.protocol.http.WicketFilter applicationFactoryClassName org.apache.wicket.spring.SpringWebApplicationFactory As well, we want our Spring context to be available to our webapp if ever there is a need for one of our pages to access the Spring managed beans directly: contextConfigLocation classpath:com/mysticcoders/mysticpaste/spring/applicationContext.xml org.springframework.web.context.ContextLoaderListener Hibernate Hibernate is our ORM of choice, it will allow us to persist and retrieve our model objects to and from the underlying database, whatever that database may be. We add the following to our pom.xml: org.hibernate hibernate-annotations c3p0 c3p0 commons-dbcp commons-dbcp javax.transaction jta 1.0.1B hibernate-annotations: used so we can annotate our model classes with mapping information, instead of having to create a separate mysticpaste.hbm.xml file. c3p0: provides a connection pooling library Hibernate can use commons-dbcp: another connection pooling library, we’ll add it as well and decide whether to use it or c3p0 later jta: this is the Java Transaction API which is needed by Hibernate (Hibernate provides an implementation of the API) web.xml additions for Hibernate To have a Hibernate Session open and ready for our webapplication during a Request Cycle we need to setup a Hibernate filter like so (otherwise, good luck getting lazy loading working!): open.hibernate.session.in.view org.springframework.orm.hibernate3.support.OpenSessionInViewFilter open.hibernate.session.in.view /* As the comment states above, make sure this filter-mapping exists *before* your wicket.mysticpaste filter or else it just plain won’t work. Database For the Mystic Paste we decided to use the freely available PostgreSQL. Adding support for PostgreSQL is very easy, unlike with some of the commercial DBMSes where you have to download and install their JDBC driver into your repository. To add support for Postgres, we simply add the following to our pom.xml: postgresql postgresql Servlets Regardless of which webapplication framework you choose there are just some times when a plain jane Servlet comes in really handy. If you have a need for Servlets and the Servlet must have access to the Wicket session add the following to your web.xml: wicket.session org.apache.wicket.protocol.http.servlet.WicketSessionFilter filterName wicket.mysticpaste And then, after your other filter-mappings add the following (assuming you mount your servlet-mappings under /servlet/): wicket.session /servlet/* Maven Filters and Profiles In order to build our Mystic Paste project for various environments (DEV/QA/PROD) we need to implement both Maven profiles and filters. Filters Filters allow you to place variables inside your configuration files and have those variables filled in durring build time. This is very handy for setting environment specific things such as database connection information. Enabling filters is quite easy, we open up the pom.xml file and find the section for and set the value for the element to true as follows: true src/main/resources . . . But for filtering to work, we need to specify a filters file. It’s not enough to specify only one filter file because we need to specify different filters per environment and we’ll do that by using Maven Profiles. Profiles To setup a profile, create a new set of elements following the section in your pom.xml file. Like so: DEV DEV DEV QA QA PROD PROD and just above your tag underneith your tag you would add the following elements: mysticpaste src/main/filters/filters-${env}.properties true src/main/resources src/main/filters will contain the following files. |-- pom.xml |-- src | `-- main | |-- filters | | `-- filters-DEV.properties | | `-- filters-QA.properties | | `-- filters-PROD.properties filters-DEV.properties jdbc.url=jdbc:postgresql://localhost:5432/mysticpaste jdbc.user=mysticpaste jdbc.password=password image.upload.path=/tmp/pasetbin/userimages image.upload.size.limit=4096K filters-PROD.properties jdbc.url=jdbc:postgresql://192.168.2.10:5432/mysticpaste jdbc.user=mysticpaste jdbc.password=CrYp71c image.upload.path=/mysticpaste/userimages image.upload.size.limit=4096K Now within any file under src/main/resources that has variables of the form ${variable.name} will have those variables replaced with the values specified in the proper filters file located under src/main/filters. For instance here is an example of a Spring applicationContext.xml file which will be interpolated with proper variables values at build time: applicationContext.xml To determine which filters file will be used depends on the profile chosen when building. For example, to build to production using the filters-PROD.properties we would execute the following: mvn clean deploy -P PROD The profile you use with the -P switch must match one of the values of the element for a profile. Conclusion Although it’s quite easy to get started with the Maven QuickStart project it is sometimes a bit frustrating putting some of the additonal pieces together. Building to several environments, setting up depenencies not included in the QuickStart project and strucuturing your project in an effort to make life easy for yourself as a developer and for your designer. I hope our Day 1 tutorial leaves you with a good sense of how a Wicket project is setup, now we can move onto coding the app! In the next four days we will be covering: Writing the Tests Designing the backend Designing the Wicket components Putting it all together Mystic Coders, LLC has been coding web magic since 2000. Mystic is a full-service Development Agency specializing in Enterprise development with Java. They are usually involved in developing enterprise-grade software for companies large and small, and have experience working in diverse industries, including b2b, b2c, and government-based projects. Mystic has done work with large companies such as LeapFrog, Nestlé, Harrah's Entertainment and the Los Angeles Conventions & Visitor's Bureau, among others. Andrew Lombardi, CTO of Mystic, is available for speaking engagements. For more about Mystic, check us out at http://www.mysticcoders.com
March 10, 2009
by Craig Tataryn
· 30,166 Views
article thumbnail
JAX-RPC client with Maven2
Recently I needed to make my Maven2 web project communicate with an old style RPC encoded web service. We run on GlassFish which comes with JAX-RPC RI built-in, so I was hoping to find a way to use it without having to bundle another runtime such as Axis into the project. Specifically I was looking for a way to make my build-script generate client stubs from a WSDL file in the project tree. I want the client stubs to use standardized JAX-RPC APIs which are serviced by the implementation provided by GlassFish. I found that the JAX-WS RI has a Maven2 plugin but not the JAX-RPC RI. I did a lot of googling, and posted messages in a few mailing lists. Surprisingly there is almost nobody using JAX-RPC on Maven2 projects because there are barely any examples online and nobody could answer my question. I'm surprised because there are a lot of legacy systems out there that developers need to integrate with, so I can't be the only one still needing JAX-RPC support. Some people suggested that I update the web service to modern RPC literal to make client-side development easier with JAX-WS, but that is not an option. Others suggested that I use Maven's antrun plugin to call out to an ant build script that does JAX-RPC work. Finally I found an all Maven solution that works for me: org.apache.axis axis 1.4 compile org.apache.axis axis-jaxrpc 1.4 compile commons-discovery commons-discovery 0.4 compile wsdl4j wsdl4j 1.6.2 compile org.codehaus.mojo axistools-maven-plugin 1.3 wsdl2java com.mycompany.service.client src/main/resources/META-INF/wsdl target/generated-sources/wsdl2java private MyServiceSEI getMyServicePort() throws ServiceException { MyServiceLocator locator = new MyServiceLocator(); MyServiceSEI port; Stub stub; port = locator.getMyServiceSEIPort(); // OPTIONALLY configure URL and HTTP BASIC authentication stub = (Stub) port; stub._setProperty(Stub.ENDPOINT_ADDRESS_PROPERTY, "http://hostname/ContextRoot/ServicePort"); stub._setProperty(Stub.USERNAME_PROPERTY, "user"); stub._setProperty(Stub.PASSWORD_PROPERTY, "secret"); return port; } It works, but I think the client stubs are dependent on the Axis 1 runtime instead of using the GlassFish provided JAX-RPC runtime through the standardized JAX-RPC APIs. I've wasted enough time on this so I moved on. If you know a better way to do this, please let me know. No RESTful services rants please! I think SOAP and REST both have their uses depending on what you are doing. From my perspective, JAX-WS SOAP development is effortless and requires significantly less code than REST because I don't have to write the building/parsing code. After examining raw SOAP messages without extra WS-* headers added in, I decided that overhead is not a valid argument against SOAP. When you need more advanced features they are available to you without having to re-invent the wheel. I think most Java developers' negative impressions on SOAP comes from the JAX-RPC and Axis days, and because they haven't tried JAX-WS yet. From http://www.ryandelaplante.com/
March 9, 2009
by Ryan Developer
· 36,941 Views
article thumbnail
Performance Monitoring Using Glassbox
The industry is recognizing the fact that performance testing & engineering should be part of the project execution road map starting from the requirements gathering phase. At many times during project executions, performance engineering related activities are executed based on customer need or slow response time of application after development phase gets completed. Glassbox can be leveraged (by developers/testers/business users) during and after the development cycle to monitor the response times of requests with-out being aware of underlying application structure and code details. Analysis generated by Glassbox gives direct pointers on where is the bottleneck which causes slow response time for that particular request/page/URL. About Glassbox Glassbox is an open source web application which aid in performance monitoring and troubleshooting of multiple web applications deployed in container. Troubleshooting It contains the built-in knowledge repository of common problems which are used to pinpoint the issues and suggestions on causes as Java code executes. Performance Monitoring It monitors the requests as Java code executes and provides details about response times. Glassbox web client (AJAX GUI) provides nice summary dashboard view which contains various attributes like (server-name, application name, operation/request-URL, average time, no. of executions, status (slow / OK) and analysis details). By default, an operation that takes more than 1 sec execution time is marked as SLOW status. Such SLA can be modified using Glassbox properties file. Analysis part describes the problem precisely and very clearly in plain English words, rather than displaying large code/exception trace. This definitely increases developer productivity by reducing developer’s time spent in log files and using IDE debuggers. Internals The two main components of Glassbox are Monitor and Agent. Monitor uses Aspect-Oriented Programming (AOP) to monitor the JVM activity. Agent diagnoses and presents the monitoring results and uses knowledge repository to cross reference the problem with suggestions/solutions. Glassbox agent supports viewing of the analysis results using JMX (eg. Java 5 JConsole) Consoles. Glassbox extensively uses the AOP approach internally to monitor the Java code. This gives the benefit of not making any changes to source code or build-process and hence can work with any legacy web application/jar file as well. Technologies Glassbox should work on any application server that supports Servlet 2.3 or later. The servers where Glassbox is tested and installation process is automated are Apache Tomcat, weblogic, websphere, Resin, Oracle OC4J, websphere, Resin, Jetty & GlassFish. Overhead Having Glassbox application running on same container would generate a performance overhead. Typically this would affect the response time and memory overhead. Hence it is recommended to start the Glassbox application only when it’s required for performance monitoring. Licensing Glassbox is an open source project, it is free to download and run. Glassbox uses the GNU Lesser General Public License to distribute software and documentation. Demo Application Development & Deployment to Tomcat To test the capabilities of Glassbox, a sample application is developed which has a TestServlet class. This servlet calls DelayGenerator class’s generateDelay() method. This method calls Thread class’s sleep() method which suspends the execution of servlet. A counter is being initialized in DelayGenerator class which determines the time interval till which servlet is needed to be suspended. TestServlet.java /** * File: TestServlet.java * @author Viral Thakkar */ package com.infosys.star.glassbox; import java.io.IOException; import java.io.PrintWriter; import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; public class TestServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { DelayGenerator delayObj = new DelayGenerator(); int delay = delayObj.generateDelay(); response.setContentType("text/html"); PrintWriter out = response.getWriter(); out.println(""); out.println(" Hello World from Test Servlet : "+delay+" milliseconds "); out.println(""); out.flush(); } } DelayGenerator.java /** * File: DelayGenerator.java * @author Viral Thakkar */ package com.infosys.star.glassbox; public class DelayGenerator { private static int counter = 1; public int generateDelay() { try { Thread.sleep(counter * 100); counter++; } catch (InterruptedException e) { e.printStackTrace(); } return counter*100; } } Glassbox Installation & Integration to Apache Tomcat 6.0 Glassbox installation is very straightforward for non-clustered environment for the server where it’s automated. Simply drop the glassbox.war file at the appropriate folder inside server folder or perform the server specific steps/configuration to deploy the war file. Browse to server url with context root as glassbox – http://<>:/glassbox. Follow the instructions available on this page. According to specific server, this page would suggest the configuration changes for a server. Please refer to Glassbox User Guide document for details on how to install Glassbox for clustered application server environment. For Apache Tomcat 6.0- Add following command line arguments to Tomcat’s Java options: -Dglassbox.install.dir=C:\Tomcat6.0\lib\glassbox -Djava.rmi.server.useCodebaseOnly=true -javaagent:C:\Tomcat6.0\lib\aspectjweaver.jar Monitoring & Technical Analysis Glassbox web client (URL- http://<>:<>/glassbox ) shows the summary and detailed view of all the requests/operations that container/JVM has executed. Summary Section View Different attributes (columns) which gets displayed in this table are as below - Attribute / Column Name Comments Status This indicates whether operation/request is performing OK, SLOW or FAILING Analysis For SLOW/FAILING status, this value provides the small summary of the cause of the problem. Operation This is name of the operation/request of an application Server Name of the server where monitoring is being done. In a clustered environment, this allows to distinguish operations on different servers. Executions This value indicates how many times this operation has run since the application server was started or Glassbox’s statistics were last reset. Click the request in above summary table to view its detailed analysis in below detailed section. Detailed Section View The details area provides information relating to operations selected in the summary table. Different sub-sections which gets displayed in this view are as below - Sub-section Name Comments Executive Summary High level summary view of the selected operation gets displayed in a table format. This is neat view to senior stake holders who are not interested in technical details. Technical Summary This section contains more technical details in paragraph and table representation formats to provide insight into root cause of the problem if any, like which operation, query is slow and statistics of same. Details like stack trace, thread lock name are provided to find and fix the problem. “Common solutions” sub section shows pointers to resolve the identified problem/s. “Glassbox has ruled out other potential problems” sub section saves time to know what problems have already been ruled out. Executive Summary View Technical Summary -> Technical Details Views Above two snapshots are parts of the Technical Details section and provide minute details at code level with line number so as to pinpoint where the problem is. Here cause is identified at Class com.infosys.star.glassbox.DelayGenerator inside Method generateDelay at line number 12 where Thread.sleep is invoked. Perform Load Testing Using JMeter and Monitor Using Glassbox Apache JMeter is used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. It can be used to make a graphical analysis of performance or to test server/script/object behavior under heavy concurrent load. Using JMeter, create a test plan that simulates 10 users requesting for 1 page 5 times. i.e. 10 x 1 x 5 = 50 HTTP requests. First step is to add a Thread Group element. The Thread Group tells JMeter the number of users to simulate, how often the users should send requests, and the how many requests they should send. Next step is to add HTTP Request element to added Thread Group. In parallel, have the Glassbox up and running to monitor response time statistics of the load generated by JMeter application. Below is the Executive summary view of above test in Glassbox web UI interface. Section “Monitoring & Technical Analysis” contains the details to understand the Glassbox generated analysis. Conclusion Glassbox is not the replacement for performance testing tool like load runner. Glassbox aids in the project to various stakeholders in finding, conveying and fixing the performance problems at all phases starting build (development) to post deployment. Glassbox application to be started/installed only during monitoring time so as to avoid the performance overhead for other applications due to CPU & memory footprint occupied by Glassbox application on the container. During load testing of the application, Glassbox turns out to be good option to figure out the root causes inside an application code. References Glassbox web site - http://www.glassbox.com/glassbox/Home.html Glassbox User Guide - http://nchc.dl.sourceforge.net/sourceforge/glassbox/Glassboxv2.0UserGuide.pdf Apache JMeter - http://jakarta.apache.org/jmeter/ Download & Support Glassbox Download Link - http://www.glassbox.com/glassbox/Downloads.html Glassbox forum Link - http://sourceforge.net/forum/forum.php?forum_id=575670 About Author Viral Thakkar is a Technical Architect with the Banking and Capital Markets vertical at Infosys. He has 9.5 years of technology consulting experience mainly on Java/JEE technologies and frameworks with large banks and financial institutions across the globe. He has been part of many small and large-scale initiatives related to application development, architecture creation and strategy definition. From http://viralpatel.net/blogs
March 5, 2009
by Viral Thakkar
· 20,622 Views
article thumbnail
Using Java, POST A Block Of XML To A Web Server
// This is a working example of POSTing a string (representing a block of XML) to a web server try { URL url = new URL( argUrl ); URLConnection con = url.openConnection(); // specify that we will send output and accept input con.setDoInput(true); con.setDoOutput(true); con.setConnectTimeout( 20000 ); // long timeout, but not infinite con.setReadTimeout( 20000 ); con.setUseCaches (false); con.setDefaultUseCaches (false); // tell the web server what we are sending con.setRequestProperty ( "Content-Type", "text/xml" ); OutputStreamWriter writer = new OutputStreamWriter( con.getOutputStream() ); writer.write( requestXml ); writer.flush(); writer.close(); // reading the response InputStreamReader reader = new InputStreamReader( con.getInputStream() ); StringBuilder buf = new StringBuilder(); char[] cbuf = new char[ 2048 ]; int num; while ( -1 != (num=reader.read( cbuf ))) { buf.append( cbuf, 0, num ); } String result = buf.toString(); System.err.println( "\nResponse from server after POST:\n" + result ); } catch( Throwable t ) { t.printStackTrace( System.out ); }
March 4, 2009
by Douglas Wyatt
· 22,630 Views
article thumbnail
Javascript Implementation Of Damerau-Levenshtein Distance
From Wikipedia: Damerau–Levenshtein distance is a "distance" (string metric) between two strings, i.e., finite sequence of symbols, given by counting the minimum number of operations needed to transform one string into the other, where an operation is defined as an insertion, deletion, or substitution of a single character, or a transposition of two characters. //based on: http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/Levenshtein_distance //and: http://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance function levenshtein( a, b ) { var i; var j; var cost; var d = new Array(); if ( a.length == 0 ) { return b.length; } if ( b.length == 0 ) { return a.length; } for ( i = 0; i <= a.length; i++ ) { d[ i ] = new Array(); d[ i ][ 0 ] = i; } for ( j = 0; j <= b.length; j++ ) { d[ 0 ][ j ] = j; } for ( i = 1; i <= a.length; i++ ) { for ( j = 1; j <= b.length; j++ ) { if ( a.charAt( i - 1 ) == b.charAt( j - 1 ) ) { cost = 0; } else { cost = 1; } d[ i ][ j ] = Math.min( d[ i - 1 ][ j ] + 1, d[ i ][ j - 1 ] + 1, d[ i - 1 ][ j - 1 ] + cost ); if( i > 1 && j > 1 && a.charAt(i - 1) == b.charAt(j-2) && a.charAt(i-2) == b.charAt(j-1) ){ d[i][j] = Math.min( d[i][j], d[i - 2][j - 2] + cost ) } } } return d[ a.length ][ b.length ]; }
February 22, 2009
by kevin mcbob
· 2,923 Views
article thumbnail
JBoss RichFaces with Spring
This article is going to show you how to build a RichFaces application with Spring.
February 16, 2009
by Max Katz
· 203,775 Views
article thumbnail
Integrate OpenOffice with Java without Installing OpenOffice
Until a few days ago, I've always needed to work with the rather cumbersome Office Bean and UNO Runtime when integrating OpenOffice into a Java application. I also had to configure a whole bunch of things to force OpenOffice to play nicely with the Java integration. Two days ago, however, I found out about ODF Toolkit. It seems to be a relatively new project, independent since last year some time, though I could be wrong. What's especially interesting is the ODFDOM: ''ODFDOM is an OpenDocument (ODF) framework. It's purpose is to provide an easy common way to create, access and manipulate ODF files, without requiring detailed knowledge of the ODF specification. It is designed to provide the ODF developer community an easy lightwork programming API, portable to any object-oriented language.'' Here's a snippet of it in action: public static void main(String[] args) { try { OdfDocument odfDoc = OdfDocument.loadDocument(new File("/home/geertjan/test.ods")); OdfFileDom odfContent = odfDoc.getContentDom(); XPath xpath = odfDoc.getXPath(); DTMNodeList nodeList = (DTMNodeList) xpath.evaluate("//table:table-row/table:table-cell[1]", odfContent, XPathConstants.NODESET); for (int i = 0; i < nodeList.getLength(); i++) { Node cell = nodeList.item(i); if (!cell.getTextContent().isEmpty()) { System.out.println(cell.getTextContent()); } } } catch (Exception ex) { //Handle... } } Let's assume that the 'test.ods' file above has this content: From the above, the code listing would print the following: Cuthbert Algernon Wilbert And, as a second example, here's me reading the first paragraph of an OpenOffice Text document: public static void main(String[] args) { try { OdfDocument odfDoc = OdfDocument.loadDocument(new File("/home/geertjan/chapter2.odt")); OdfFileDom odfContent = odfDoc.getContentDom(); XPath xpath = odfDoc.getXPath(); OdfParagraphElement para = (OdfParagraphElement) xpath.evaluate("//text:p[1]", odfContent, XPathConstants.NODE); System.out.println(para.getFirstChild().getNodeValue()); } catch (Exception ex) { //Handle... } } On my classpath I have "odfdom.jar" and "xerces-2.8.0.jar". I don't necessarily have OpenOffice installed, which means I can very easily process a whole bunch of spreadsheets (or other OpenOffice output) without (a) installing OpenOffice and (b) faster than I would otherwise do, since OpenOffice doesn't need to be started up, via the Office Bean or otherwise. In fact, Aljoscha Rittner from Sepix, who told me about this project and who is using it in his commercial applications, reports that his processing has sped up to a fraction of the original, also because he doesn't need to handle the situation where OpenOffice would crash randomly in the middle of long running processes, such as during the night when there's no human interaction for restarting it.
February 7, 2009
by Geertjan Wielenga
· 50,445 Views
article thumbnail
An Overview of Servlet 3.0
JSR 315 (Servlet 3.0) is an update to the existing Servlet 2.5 specification. Servlet 3.0 is focussed on extensibility and web framework pluggability, aligning with the goals of Java EE 6. Ease of Development (EoD) will be supported using newer language features. A reference implementation is available in the GlassFish v3 nightly build. The public review contains: Pluggability EoD Async Support Security Enhancements Miscellaneous Changes In this article we will bring you up to speed with what's happening with the Servlet 3.0 specification and give more detail on what is included. Note: this article corresponds to the public review of the specification. As it is not yet final some things may change. The Expert Group Rajiv Mordani from Sun Microsystems is the specification lead with an expert group comprised of many of the most recognisable names in the Java community: Adobe Systems Inc. Apache Software Foundation BEA Systems Ericsson AB Google Inc. Hunter, Jason IBM Icesoft Technologies Inc NCsoft Corporation Oracle Pramati Technologies Prasanna, Dhanji R. SAP AG Ship, Howard M. Lewis Suleiman, Hani Sun Microsystems, Inc. Tmax Soft, Inc. Walker, Joe Wilkins, Gregory John Pluggability Due to the popularity of so many various web frameworks, Servlet 3.0 will make it easier to use and configure the developer's framework of choice. So if you want to add in Struts, or Spring Web Flow it will be easy to do so. Methods to add Servlets and Filters If a ServletContextListener is registered and wants to add a Servlet or Filter, then at the time of context initialization the event that is fired to the Listener can add Servlets and Filters to the context. The methods are addServlet and addFilter. For more details look for the javadocs at the JCP site at http://jcp.org/aboutJava/communityprocess/pr/jsr315/index.html. Web fragments Instead of having just one monolithic web.xml that is used by the developer to declare servlets, filters and other configuration for using the framework, the framework can now include a web-fragment.xml with all it's configuration. A web-fragment is almost identical to the web.xml and can contain all the configuration needed by the framework in the META-INF directory of the framework's jar file. The container will use the information to assembe the descriptor for the application and the application needn't have to use any of the boilerplate configuration in their app for the framework. There are rules that are defined int the specification for conflict resolution, overriding and disabling fragment scanning. Along with the annotations which also can be in libraries the feature is very compelling not only for the developers that use frameworks but also for framework authors to be self sufficient in defining the configuration needed. The great benefit to this is that library providers can supply their own web.xml fragment for use in your webapp. Ease of Development Several new annotations have been defined for ease of development in Servlet 3.0, allowing you to write a Servlet without requiring a descriptor. These annotations reside in the javax.servlet.annotation package. Thanks to the addition of annotations, it is now possible to have Servlet, Filter and ServletContextListener in a war file without a web.xml. It's important to point out that this makes the web.xml optional, rather than redundant. The web.xml may be user to override metadata specified via annotations. Following discussions in the expert group and the community, it was decided that method level annotations were not to be added, so you keep the doGet, doPost methods and need to extend HttpServlet to use them. Let's take a closer look at the annotations present in the Servlet 3.0 specification: Servlet Annotation In Servlet 3.0, servlet metadata can be specified using @WebServlet @WebServlet(name="mytest", urlPatterns={"/myurl"}, initParams={ @InitParam(name="n1", value="v1"), @InitParam(name="n2", value="v2") }) public class TestServlet extends javax.servlet.http.HttpServlet { .... } TestServlet extends HttpServlet, while the meta data provided corresponds with the web.xml as follows: Parameter Annotation Parameter web.xml Servlet name name= URL Pattern urlPatterns={ } Initialization parameters InitParams={ @InitParam{name=””, value=””} .. .. Servlet Filter Annotation ServletFilter meta data is specified using the @ServletFilter annotation public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException { .... } public void destroy() { .... } } Parameter Annotation Parameter web.xml URL Pattern urlPatterns={ } Initialization parameters InitParams={ @InitParam{name=””, value=””} .. .. Servlet Context Listener Annotation A ServletContextListener is simply marked with the @WebServletContextListener rather than needing to list it the web.xml file. @WebServletContextListener public class TestServletContextListener implements javax.servlet.ServletContextListener { .... public void contextInitialized(ServletContextEvent sce) { .... } public void contextDestroyed(ServletContextEvent sce) { .... } } Async Support The biggest change made in the specification is the addition of asynchronous processing. The main cases to cover were: Waiting for a resource to become available, such as JDBC or a call to a Web Service. Generating response asynchronously Using exisiting frameworks to generate responses after waiting the for asychronous operation to complete. While the early draft had suspend and resume, the latest specification has two variations of the startAsync() method on ServletRequest. One (startAsync() ) takes no parameters and initialises an AsyncContext with the original request and response. The ( startAsync() ) other takes a request and response as a parameter to initialise the AsyncContext with. Servlets (@WebServlet) and Filters (@ServletFilter) that support asynchronous processing need to be annotated with the supportAsync attribute set. It is illegal to call startAsync if you have a Servlet or Filter that doesn't support asynchronous processing anywhere in the request processing chain. The is also an AsyncListener that can be registered to get notification on timeout and completion of asynchronous processing of the request. This listener can be used to clean up resources if the wrapped request and response were used to create it's AsyncContext. You can forward requests back to the container using the AsyncContext.forward(path) and AsyncContext.forward() methods so frameworks like JSP can create a response. Security Enhancements HTTPServletRequest will be enhanced with methods allowing programmatic login and logout, while ServletRequest will be able to force a login.The logout method will allow an application to reset the authentication state of a request without requiring the authentication to be bound to a HTTPSession. Conclusion The draft is just beginning to be implemented within Glassfish v3. The specification will be a required piece of Java EE 6 and hence all the features described above will be available in all Java EE 6 compatible containers. The presence of the various features described above will make developing Java web appications much easier. Thanks to Rajiv Mordani for his input into this article, and to Shing Wai and Jan Luehe for the code snippets.
January 21, 2009
by James Sugrue
· 77,623 Views · 2 Likes
article thumbnail
Open Source : How Do You Stay Up To Date?
I Love the concepts and beliefs behind Open Source. I use Open Source libraries, applications etc. all the time. One of the things I have always found a challenge though, is knowing when a new release comes to be. Not only that, is this a simple point release, a major release or a security release. For the most part I want to stay up to date, especially with the libraries I use, when a security releases is made. Currently I play the, I hope I have the most stable, most secure version. Howeaver, I always thought that having a single place that I can sign-up to be informed about releases would be awesome. Do you have a way to do this or, are you subscribing to a multitude of RSS feeds and mailing lists to stay informed?
January 10, 2009
by Schalk Neethling
· 5,095 Views
article thumbnail
Which Java Version Is Used To Run Our NetBeans Installation?
Ever wondered which Java version is used to run your NetBeans instance? I have a lot of JDK's installed and everytime I install a new one it ones to be the default JDK for my computer. To know which JDK is used for running NetBeans we go to Help | About and the dialog box shows which Java version is used: To explicitly tell NetBeans which JDK to use we can use the command-line argument --jdkhome when we start NetBeans. Or we can set the property netbeans_jdkhome in the file netbeans-install-dir/etc/netbeans.conf.
December 23, 2008
by Hubert Klein Ikkink
· 16,948 Views
article thumbnail
Getting Javadoc from Maven Repositories
In my previous tip on NetBeans Zone we learned how to get available sources for a Maven dependency. We can also get Javadoc for the library. The Javadoc must be available of course otherwise NetBeans is not able to download anything. We right-click on Libraries in our project and select Download All Library Javadoc. NetBeans downloads the Javadoc for the libraries if it is available. The icon will change for the library if Javadoc is downloaded. To view the Javadoc we must right-click on the library and select View Javadoc. NetBeans opens a web browser with the Javadoc for the specific library.
December 21, 2008
by Hubert Klein Ikkink
· 12,062 Views · 2 Likes
article thumbnail
A Farewell to Heavyweight/Lightweight Conflicts
Problems resulting from the mixing of Swing and AWT components have been a constant source of confusion to Java newbies. This quote explains the reason for the difference very well: Most of the issues related to mixing AWT and Swing components are related to the mixing of so-called heavyweight and lightweight components. A heavyweight component is one that is associated with its own native screen resource (commonly known as a peer). A lightweight component is one that "borrows" the screen resource of an ancestor (which means it has no native resource of its own -- so it's "lighter"). From: Mixing Heavy and Light Components The lightweight approach, taken by Swing, is the preferred, since it consumes less resources. However, there are cases where one might need to mix the two in the same application, which can result in undesired side effects. A case in point, from the same article and where all the code used below is found, goes as follows. The menu on the right behaves correctly, while the menu on the left does not: The reason for the difference is that the badly displayed menu in the second screenshot is Swing and lightweight, while the other menu is AWT, and therefore heavyweight. The Button is an AWT component too, therefore being heavyweight. Unless one forces a lightweight component, via setDefaultLightWeightPopupEnabled(false), to behave like a heavyweight component, unwanted side effects such as the above will result. But all that is old news, of course. The new news is that already with the early access release of JDK 6 Update 12 one can, without needing to do anything at all, obtain the desired effect, which is as follows: This is exactly what one would want: the rendering of a lightweight component over a heavyweight component. Though the major features of the EA release are slated as "Windows 2008 Support" and "Java Plug-In now supports 64-bit browsers", it is this heavyweight/lightweight fix that I find the most useful (since I am neither a Windows- nor a 64-bit browser-user). If one looks through the list of issues fixed in the current JDK 6 Update 12 build, one notices that a lot of fixing has been done in the Swing & AWT packages. The final screenshot above was taken on JDK 6 Update 12, using the code from the article referenced at the start. I.e., without doing anything special at all, no fiddling with z-order or the like, the behavior was exactly as an end user would expect it to be. Hurray!
December 16, 2008
by Geertjan Wielenga
· 20,815 Views
article thumbnail
The Three Pillars of Continuous Integration
Continuous Integration commonly known as CI is a process that consists of continuously compiling, testing, inspecting, and deploying source code. In any typical CI environment, this means running a new build every time code changes within a version control repository. Martin Fowler describes CI as: A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. While CI is actually a process, the term Continuous Integration often is associated with three important tools in particular. As shown in the image the three pillars of CI are: 1. A version control repository like Subversion, or CVS. 2. A CI Server such as Hudson, or Cruise Control 3. An automated build process like Ant or Nant So, let’s look at each of these in detail: Version Control Repository: Version control repositories also known as SCM (source code management) play a crucial role in any software development environment. They also play a very important role for a successful CI process. The SCM is a central place for the team to store every needed artifact for the project. It is mandatory for the teams to put everything needed for a successful build into this repository. This includes the build scripts, property files, database scripts, all the libraries required to build the software and so on. The CI Server: For CI to function properly, we also need to have an automated process that monitors a version control repository and runs a build when any changes are detected. There are several CI servers available, both open source and commercial. Most of them are similar in their basic configuration and monitor a particular version control repository and run builds when any changes are detected. Some of the most commonly used open source CI servers are; Cruise Control, Continuum, and Hudson. Hudson is particularly interesting because of its ease of configuration and compelling plug-ins, which makes integration with test and static analysis tools much easier. Automated Build: The process of CI is about building software often, which is accomplished through the use of a build. A sturdy build strategy is by far the most important aspect of a successful CI process. In the absence of a solid build that does more than compile your code, CI withers. With automated builds, teams can reliably perform (in an automated fashion) otherwise manual tasks like compilation, testing, and even more interesting things like software inspection and deployment. Now that we have seen the important tools in our CI process, let’s see how a typical CI scenario looks like for a developer: CI server is configured to poll the version control repository continuously for changes. Developer commits code to the repository. CI server detects this change, and retrieves the latest code from the repository. This causes the CI server to invoke the build script with the given targets and options. If configured, CI Server will send out an e-mail to the specified recipients when a certain important event occurs. The CI server continues to poll for changes. Why is CI Important? This is one of the most frequently asked questions, and here are a few points to note about this powerful technique: Building software often greatly increases the likelihood that you will spot defects early, when they still are relatively manageable. Extends defect visibility. CI ensures that you have production ready software at every change. CI also ensures that you have reduced the risk of integration issues by building software at every change. CI server can also be configured to run continuous inspection which can assist the development team in finding potential bugs, bad programming practice, automatically check coding standards, and also provide valuable feedback on the quality of code being written. Over the past several months, I have assisted several companies in implementing CI. There was a little bit of resistance from the developers in the early stages when we implemented continuous feedback. But, never heard a single negative comment about this approach. If you already have a version control repository and automated builds, you are very close to the CI process. Download one of the open source CI servers, configure and setup a simple project. It should take less than an hour if you have automated build scripts. Start adding additional features like code inspections, generating reports, metrics, documentation and so on. Most important, send continuous feedback to your team. Give this process a try, you sure will be surprised to see how effective it is. And, as always share your thoughts, concerns or questions.
December 15, 2008
by Meera Subbarao
· 23,858 Views
article thumbnail
Configuring Logging in JBoss
Learn how to properly configure logs in JBoss.
November 19, 2008
by Meera Subbarao
· 223,083 Views
article thumbnail
Merge Sort In Java
// A simple (i.e., not particularly optimized) implementation of merge sort in Java. // I tested the sorting capacities on several documents of increasing size: // Lincoln's Emancipation Proclamation, Mark Twain's _Innocents Abroad_, // George Eliot's _Middlemarch_, and Leo Tolstoy's _War and Peace_. // The sort operation itself is very fast -- under 2 seconds for War and Peace. import java.io.BufferedReader; import java.io.FileReader; import java.io.IOException; import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Arrays; import java.util.Date; import java.util.List; /** * A few factoids about MergeSort: * 1) while it gets better performance on the modern PC (because it takes advantage of caching * better than, say, HeapSort) * 2) It requires more space than HeapSort -- HeapSort runs in place, while MergeSort requires * extra memory space to hold the recursive calls. So if you're short on memory, * go for HeapSort * 3) MergeSort is stable -- the order of the list never changes once it's set. * 4) MergeSort is very easy to run in parallel, for obvious reasons. * 5) If you need to chew through very large files, you can use MergeSort to write out * temporary results to a file and it will work fine. HeapSort, by comparison, * relies strongly on random access and so doesn't operate as well on disk storage media. * @author Roger * */ public class MergeTest { /** * @param args */ public static void main(String[] args) throws IOException { String[] files = { "lincoln2.txt", "lincoln.txt", "twain2.txt", "twain3.txt", "twain.txt", "eliot.txt", "tolstoy.txt" }; SimpleDateFormat format = new SimpleDateFormat("EEE MMM dd HH:mm:ss-SSS zzz yyyy"); /* String[] testCase = {"aa", "baa", "quickly", "monitor", "frontage", "beast", "feast", "obsequious", "obstetrics", "feast", "alphabet", "gnomic", "triumph" }; */ System.out.println("---------TEST MERGE W ARRAY ------------"); for (int x = 0; x < files.length; x++) { String[] testCase = readFile(files[x]); long start = System.currentTimeMillis(); Date now = new Date(start); System.out.println("File: " + files[x]); System.out.println("Total words: " + testCase.length); System.out.println("begin test: " + format.format(now)); MergeTest merge = new MergeTest(); String[] sortTest = merge.mergeSort(testCase); long end = System.currentTimeMillis(); now = new Date(end); System.out.println("end test: " + format.format(now)); System.out.println("total time: " + (end - start)); System.out.println("--------------------------\n\n"); } /* System.out.print("\nbefore: {"); for (int x = 0; x < testCase.length; x++) { System.out.println(testCase[x] + " "); } System.out.println("} "); System.out.print("after: {"); for (int x = 0; x < sortTest.length; x++) { System.out.println(sortTest[x] + " "); } System.out.println("} "); */ System.exit(0); } public String[] mergeSort(String[] list) { String [] sorted = new String[list.length]; if (list.length == 1) { sorted = list; } else { int mid = list.length/2; String[] left = null; String[] right = null; if ((list.length % 2) == 0) { left = new String[list.length/2]; right = new String[list.length/2]; } else { // left = new String[list.length/2]; right = new String[(list.length/2)+1]; } int x=0; int y=0; for ( ; x < mid; x++) { left[x] = list[x]; } for ( ; x < list.length; x++) { right[y++] = list[x]; } left = mergeSort(left); right = mergeSort(right); //sorted = merge(left,right); sorted = mergeArray(left,right); } return sorted; } private String[] merge(String[] left, String[] right) { List merged = new ArrayList(); List lstLeft = new ArrayList(Arrays.asList(left)); List lstRight = new ArrayList(Arrays.asList(right)); int comp = 0; while (!lstLeft.isEmpty() || !lstRight.isEmpty()) { if (lstLeft.isEmpty()) { merged.add(lstRight.remove(0)); } else if (lstRight.isEmpty()) { merged.add(lstLeft.remove(0)); } else { // they both have elements, compare them comp = lstLeft.get(0).compareTo(lstRight.get(0)); if (comp > 0) { merged.add(lstRight.remove(0)); } else if (comp < 0) { merged.add(lstLeft.remove(0)); } else { // they're equal -- just put one in merged.add(lstLeft.remove(0)); } } } return merged.toArray(new String[merged.size()]); } private String[] mergeArray(String[] left, String[] right) { String[] merged = new String[left.length+right.length]; int lIndex = 0; int rIndex = 0; int mIndex = 0; int comp = 0; while (lIndex < left.length || rIndex < right.length) { if (lIndex == left.length) { merged[mIndex++] = right[rIndex++]; } else if (rIndex == right.length) { merged[mIndex++] = left[lIndex++]; } else { // they both have elements. Compare them comp = left[lIndex].compareTo(right[rIndex]); if (comp > 0) { merged[mIndex++] = right[rIndex++]; } else if (comp < 0) { merged[mIndex++] = left[lIndex++]; } else { // they're equal merged[mIndex++] = left[lIndex++]; } } } return merged; } public static String[] readFile(String filename) throws IOException { BufferedReader inputStream = new BufferedReader(new FileReader(filename)); List testList = new ArrayList(); String[] sample = null; String l; while ((l = inputStream.readLine()) != null) { sample = l.split(" "); for (int x=0; x < sample.length; x++) { testList.add(sample[x]); } } inputStream.close(); return testList.toArray(new String[testList.size()]); } }
November 5, 2008
by Roger Turnau
· 2,642 Views
article thumbnail
What is Wrong with JDK Logger?
I used to use log4j because it was most popular, but recently decided to try JDK logger in my current project. I like that it is built into the JDK so there are no .jar dependencies, version conflicts, etc. All I need it to do is write to a disk file using a format String I specify, rotate logs after x KB, and to keep only y archived files. I'm just as satisfied with it as I was with log4j. A while ago I did some reading about the history of logging frameworks to try and understand the hostility many developers still have towards JDK logger. From what I can tell log4j was the first widely popular logging framework in Java. Later, its ideas and general API design were standardized in JDK 1.4. The names of some things were changed, but the concepts are the same. There were many developers who did not want to require "the new" Java 1.4 so they continued with log4j. Later the commons-logging wrapper was created to provide libraries and applications the ability to use either log4j or JDK logger depending on what is present in the environment. Even today with the pending release of Java 1.7 many developers use log4j or commons-logging instead of the built in JDK logger. My question to you is what is wrong with the JDK logger? Why do some people say it was disaster? Why don't you use it? I am not trying to start a flamewar, I am genuinely interested. Is it because log4j comes with many more built-in appenders such as NTEventLogAppender, JMSAppender, and SMTPAppender? I figure the JDK logger has only basic core handlers/appenders for the same reason JSF comes only with basic HTML UI components. From http://www.ryandelaplante.com
October 29, 2008
by Ryan Developer
· 27,729 Views
  • Previous
  • ...
  • 567
  • 568
  • 569
  • 570
  • 571
  • 572
  • 573
  • 574
  • 575
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×