By Adam Comerford, Senior Solutions Engineer In Part I we discussed important considerations when picking a shard key. In this post we will go through some recommendations when running a sharded cluster at scale. Scalability is one of the core benefits of sharding in MongoDB but this can give you a false sense of security; even with that flexibility, you still have to make smart decisions about how and when you deploy resources. In this post, we will cover a couple of common mistakes that people tend to make when it comes to running a sharded cluster. 3. Waiting too long to add a new shard (overloaded) You sharded your database and scaled horizontally for a reason, perhaps it was to add more memory or disk capacity. Whatever the reason, if your application usage grows over time so (generally) does your database utilization. Eventually, your current sharded cluster will pass a certain point, let’s call it 80% utilized (as a nice round estimate), such that it becomes problematic to add another shard. Why? Well, adding a new shard to a cluster is not free, and it is not instantaneous. It consumes resources and (initially) accepts very little traffic. Essentially, at the start of its existence, a newly added shard costs you capacity instead of adding capacity. The length of time it will stay in this state will depend on the balancer and how long it takes for a significant portion of “busy/active” chunks to move onto the new shard. It can often be easier to visualize this process, so let’s make up some hypothetical numbers and set the bar relatively low. Our imaginary existing cluster will be a set of 2 shards, with 2000 chunks (500 considered “active”) and to that we need to add a 3rd shard. This 3rd shard will eventually store one third of the active chunks (and total chunks). The question is, when does this shard stop adding overhead overall and instead become an asset? In reality, this will vary from cluster to cluster and have a lot of dependencies and variables - in other words you need to have good metrics about your cluster, particularly your load bottleneck. Therefore we will once again use our imaginations and go with a relatively low bar: when 5% of active chunks—that is, those chunks seeing most traffic—have migrated to the new shard, you should expect a net gain in performance. In our imaginary system we have evaluated our load levels, the expected impact of migrations and have determine that once that 5% threshold of active chunks has been migrated to the new shard it can be considered a net gain for the overall system. Once all chunks have been balanced, then the migration overhead disappears, but initially this will be an expected trade off. This chart shows how long it would take for new shards to reach net positive contribution in your cluster (the dotted line implies net gain): In this fabricated example, it takes almost 2 hours for the new shard to attain a viable level of active chunks and be considered a net gain for the overall system. Although these numbers are fictional, these numbers are based on setups we have seen in real systems with moderate load. From there it is relatively easy to imagine this set of migrations taking even longer on an overloaded set of shards, and taking far longer for our newly added shard to cross the threshold and become a net gain. As such it is best to be proactive and add capacity before it becomes a necessity. Possible Mitigation Strategies Manual balancing of targeted “hot” chunks (chunk that is being accessed more than others) to move activity to the new shard more quickly Add the shard at low traffic time so that there is less competition for resources Disable balancing on some collections, prioritise balancing busy collections first 4. Under-provisioning Config Servers Provisioning enough resources without being wasteful is always tricky, and all the more so in a complicated distributed system like a MongoDB sharded cluster. Everyone wants to use their hardware, virtual instances, virtual machines, containers and the like in the most efficient way possible, and get the best bang for their buck. Hence it is only natural to take a look at the various pieces of a distributed cluster and look for lower utilized pieces that could be put on less expensive resources. The most common pitfall here with MongoDB are the config servers, which are often neglected when stress testing a cluster. In testing environments and smaller deployments (unless specific measures are taken to stress them) they are relatively lightly loaded and usually identified as candidates for lesser instances/hardware. The problem is that these are critical pieces of infrastructure. They may not be heavily loaded all the time, but when they do see load and struggle to service requests, that can impact all queries (reads, writes, authentication) and add latency to all requests made of the cluster in question. In particular, the first config server in the list supplied to your mongos processes is vital. This is the config server that all mongos processes will default to read from when fetching or refreshing their view of the data distribution in your cluster. Similarly, this is the server that will be hit when attempting to authenticate a user. If it is under-provisioned and cannot service queries, or if it has problems with networking (packet loss, congestion), then the effects will be significant. Possible Mitigation Strategies Ensure the config servers are load tested, slightly over-provisioned (the first config server in particular) If using virtual machines or cloud based instances, investigate increasing available resources Turning off the balancer, disabling chunk splitting will reduce the chances of high read traffic to the config servers (no migrations, no meta data refresh) but this is only a temporary fix unless you have a perfect write distribution and may not eliminate issues completely. 5. Using the count() command on sharded collections This pitfall is very common, and it seems to hit somewhat randomly in terms of how long someone has been running a sharded environment. At some point, a question will arise along the lines of: “How are we tracking/verifying/checking how many documents we have in each collection on each shard, how balanced are they and do they agree with ?” Hopefully no one is actually constructing questions this way in your organization, but you get the basic idea. The most obvious way to do a quick check on this type of thing is to count the documents and see if the numbers make sense and/or agree with counts elsewhere. That thinking naturally leads people to the count command and they proceed to use it to gather figures for their documents and collections. Unfortunately, on a busy, mature sharded cluster, the results will very rarely be what is expected. The reason for this is that the count command as implemented today has several optimizations in place to make it faster to run in general and those speed optimizations essentially bypass a key piece of the sharding functionality needed to return accurate results in this case. This is a known bug and is being tracked in SERVER-3645, but does not stop people from consistently hitting this issue. The nature of the issue means that count will report documents in the results that it should not, for example: Documents that are being deleted as part of a chunk migrations Documents that have been left behind from previous chunk migrations (also known as orphans) Documents currently being copied as part of an in-flight chunk migration A regular query (rather than a count) will have its results filtered by the respective primary and not suffer from the same problem. Hence, if you were to manually count the results from a query client-side you would get an accurate result. This quirk of sharded environments will eventually be fixed, but for now it will inevitably crop up from time to time in all active sharded clusters used by a large team. Possible Mitigation Strategies Do counts on the client side, or use targeted, range based queries (with a primary read preference) to count instead Use cleanUpOrphaned and disable the balancer (make sure it has finished current round) when performing counts across the cluster If you want tolearn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.

JSR 199 provides the compiler API to compile the Java code inside another Java program. The following are the important classes and interfaces provided for facilitating the compilation from a Java program. JavaFileObject - Represents a compilation unit, typically a class source. SimpleJavaFileObject - Implementation of the methods defined in JavaFileObject DiagnosticCollector - Collects the compilation errors, warning into a list of Diagnostic type Diagnostic - Reports the type of the problem and details like line number, character, error reason etc. JavaFileManager - To work on the Java source and class files. JavaCompiler - The compiler instance for compiling the compilation unit. CompilationTask - A sub interface of JavaCompiler which helps to compile and return the status with diagnostic when used call method on it. Where to start To compile a Java code, we need the Java source. The source can be a physical file on the disk or a string inside the program. Using the source, we need create an instance type of JavaFileObject. Using String literal Create a class which implements JavaFileObject, here i am using SimpleJavaFileObject. We need create the path URI of the class file package com.test; import java.io.IOException; import java.net.URI; import javax.tools.SimpleJavaFileObject; public class SampleSource extends SimpleJavaFileObject { private String source; protected SampleSource(String name, String code) { super(URI.create("string:///" +name.replaceAll("\\.", "/") + Kind.SOURCE.extension), Kind.SOURCE); this.source = code ; } @Override public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException { return source ; } } Now, create the instance of JavaFileObject and from those, create the Compilation Unit (A collection of JavaFileObject) String str = "package com.test;" + "\n" + "public class Test {" + "\npublic static void test() {" + "\nSystem.out.println(\"Comiler API Test\")-;" + "" + "\n}" + "\n}"; SimpleJavaFileObject fileObject = new SampleSource("com.test.Test", str); JavaFileObject javaFileObjects[] = new JavaFileObject[] { fileObject }; Iterable compilationUnits = Arrays .asList(javaFileObjects); From File System If the source is from physical location. Then create like this. File []files = new File[]{file1, file2, file3, file4} ; Iterable units = fileManager.getJavaFileObjectsFromFiles(Arrays.asList(files)); Create a JavaFileManger We will see, how to create a fileManger now. JavaFileManager fileManager = compiler.getStandardFileManager( diagnostics, Locale.getDefault(), Charset.defaultCharset()); To get the FileManger, we need diagnostic - A DiagnosticCollector of JavaFileObject locale - The locale of the compilation charset - The charset to be used. Compiler Get the compiler instance using ToolProvider. Finally, create the CompilationTask from the compiler instance using diagnostics, file manager and compilation units (Optionally writer and compilation options). JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); CompilationTask task = compiler.getTask(null, fileManager, diagnostics, compilationOptionss, null, compilationUnits); The argument required to get the CompilationTask are out - A writer which writes the output of the compiler. Defaults to System.err if null listener - A diagnostic listener, the errors or warning can be accessed using. options - Compiler options (Ex : -d, like we give in command line using javac ) classes - Name of the classes to be processed compilationUnits - List of compilation units Compile Finally, call the method to compile. This method to be called only once otherwise it throws IllegalStateException on multiple calls. Once compiled, returns true for successful compilation otherwise false. We need to look the diagnosticCollector to get the error/warning details. boolean status = task.call(); All together Putting all together. public static void main(String[] args) { String str = "package com.test;" + "\n" + "public class Test {" + "\npublic static void test() {" + "\nSystem.out.println(\"Comiler API Test\")-;" + "" + "\n}" + "\n}"; SimpleJavaFileObject fileObject = new SampleSource("com.test.Test", str); JavaFileObject javaFileObjects[] = new JavaFileObject[] { fileObject }; Iterable compilationUnits = Arrays .asList(javaFileObjects); Iterable compilationOptionss = Arrays.asList(new String[] { "-d", "classes" }); DiagnosticCollector diagnostics = new DiagnosticCollector(); JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); JavaFileManager fileManager = compiler.getStandardFileManager( diagnostics, Locale.getDefault(), Charset.defaultCharset()); CompilationTask task = compiler.getTask(null, fileManager, diagnostics, compilationOptionss, null, compilationUnits); boolean status = task.call(); if(!status) { System.out.println("Found errors in compilation"); int errors = 1; for(Diagnostic diagnostic : diagnostics.getDiagnostics()) { printError(errors, diagnostic); errors++; } } else System.out.println("Compilation sucessfull"); try { fileManager.close(); } catch (IOException e){} } public static void printError(int number,Diagnostic diagnostic) { System.out.println(); System.out.print(diagnostic.getKind()+" : "+number+" Type : "+diagnostic.getMessage(Locale.getDefault())); System.out.print(" at column : "+diagnostic.getColumnNumber()); System.out.println(" Line number : "+diagnostic.getLineNumber()); System.out.println("Source : "+diagnostic.getSource()); } Output Output with an error will be (because of an hyphen in System.out.println in main method of Test) Found errors in compilation ERROR : 1 Type : illegal start of expression at column : 40 Line number : 4 Source : com.test.SampleSource[string:///com/test/Test.java] ERROR : 2 Type : not a statement at column : 39 Line number : 4 Source : com.test.SampleSource[string:///com/test/Test.java] To read more about JSR 199, follow the official link. Happy Learning!!!! Read more articles at blog

I have been playing around with converting an application that I have to use Spring @Configuration mechanism to configure connectivity to RabbitMQ - originally I had the configuration described using an xml bean definition file. So this was my original configuration: This is a fairly simple configuration that : sets up a connection to a RabbitMQ server, creates a durable queue(if not available) creates a durable exchange and configures a binding to send messages to the exchange to be routed to the queue based on a routing key called "rube.key" This can be translated to the following @Configuration based java configuration: @Configuration public class RabbitConfig { @Autowired private ConnectionFactory rabbitConnectionFactory; @Bean DirectExchange rubeExchange() { return new DirectExchange("rmq.rube.exchange", true, false); } @Bean public Queue rubeQueue() { return new Queue("rmq.rube.queue", true); } @Bean Binding rubeExchangeBinding(DirectExchange rubeExchange, Queue rubeQueue) { return BindingBuilder.bind(rubeQueue).to(rubeExchange).with("rube.key"); } @Bean public RabbitTemplate rubeExchangeTemplate() { RabbitTemplate r = new RabbitTemplate(rabbitConnectionFactory); r.setExchange("rmq.rube.exchange"); r.setRoutingKey("rube.key"); r.setConnectionFactory(rabbitConnectionFactory); return r; } } This configuration should look much more simpler than the xml version of the configuration. I am cheating a little here though, you should be seeing a missing connectionFactory which is just being injected into this configuration, where is that coming from..this is actually part of a Spring Boot based application and there is a Spring Boot Auto configuration for RabbitMQ connectionFactory based on whether the RabbitMQ related libraries are present in the classpath. Here is the complete configuration if you are interested in exploring further - https://github.com/bijukunjummen/rg-si-rabbit/blob/master/src/main/java/rube/config/RabbitConfig.java References: Spring-AMQP project here Spring-Boot starter project using RabbitMQ here

One of the ways Spring recommends injecting inter-dependencies between beans is shown in the following sample copied from the Spring's reference guide here: @Configuration public class AppConfig { @Bean public Foo foo() { return new Foo(bar()); } @Bean public Bar bar() { return new Bar("bar1"); } } So here, bean `foo` is being injected with a `bar` dependency. However, there is one alternate way to inject dependency that is not documented well, it is to just take the dependency as a `@Bean` method parameter this way: @Configuration public class AppConfig { @Bean public Foo foo(Bar bar) { return new Foo(bar); } @Bean public Bar bar() { return new Bar("bar1"); } } There is a catch here though, the injection is now by type, the `bar` dependency would be resolved by type first and if duplicates are found, then by name: @Configuration public static class AppConfig { @Bean public Foo foo(Bar bar1) { return new Foo(bar1); } @Bean public Bar bar1() { return new Bar("bar1"); } @Bean public Bar bar2() { return new Bar("bar2"); } } In the above sample dependency `bar1` will be correctly injected. If you want to be more explicit about it, an @Qualifer annotation can be added in: @Configuration public class AppConfig { @Bean public Foo foo(@Qualifier("bar1") Bar bar1) { return new Foo(bar1); } @Bean public Bar bar1() { return new Bar("bar1"); } @Bean public Bar bar2() { return new Bar("bar2"); } } So now the question of whether this is recommended at all, I would say yes for certain cases. For eg, had the bar bean been defined in a different @Configuration class , the way to inject the dependency then is along these lines: @Configuration public class AppConfig { @Autowired @Qualifier("bar1") private Bar bar1; @Bean public Foo foo() { return new Foo(bar1); } } I find the method parameter approach simpler here: @Configuration public class AppConfig { @Bean public Foo foo(@Qualifier("bar1") Bar bar1) { return new Foo(bar1); } }

creating uber jar’s in java is nothing particular new, even creating executable jar’s was possible with maven long before java 8. with the first release of javafx 2, oracle introduced the javafxpackager tool, which has now been renamed to javapackager (java 8 u20). this enables developers to create native executables for any common platform, even mac app store packages; the drawbacks of the javapackager are that you must create the executable on the target platform and that it will than contain the whole jre to run the application. this means that your 100kb application gets more than 40mb, depending on your target platform. the first step on your way to an executable uber jar is to build your project and to collect all dependencies in a folder or a fat jar. this folder/fat jar will be the input for the javapackager tool, which creates the executable part. solution 1: the maven-dependency-plugin the maven-dependency-plugin contains the goal “unpack-dependencies”. we can use this goal to enhance the default “target/classes” folder with all the dependencies you need to run your application. we assume that the default maven build creates all classes of your project in the “target/classes” folder and the “unpack-dependencies” plugin copies all the project dependencies to this folder. the result is a valid input for the javapackager tool, which creates the executable. the “unpack-dependencies” goal is easy to use; you just need to set the output directory to the “target/classes” folder to ensure everything is together in one folder. a typical configuration looks like this: maven-dependency-plugin 2.6 unpack-dependencies package unpack-dependencies system org.springframework.jmx ${project.build.directory}/classes to create an executable jar from the target/classes directory we use the “exec-maven-plugin” to execute the javapackager commandline tool. org.codehaus.mojo exec-maven-plugin package-jar package exec ${env.java_home}/bin/javapackager -createjar -appclass ${app.main.class} -srcdir ${project.build.directory}/classes -outdir ./target -outfile ${project.artifactid}-app -v now we can create an executable jar, which contains all the project dependencies and that can simply be executed with “java –jar myapp.jar”. in the next step we want to create a native executable or an installer. specific configuration details for the javapackager can be found here: http://docs.oracle.com/javase/8/docs/technotes/tools/unix/javapackager.html , for this tutorial we assume that we want to create a native installer. to do so, we add a second “execution” to your “exec-maven-plugin” like this: package-jar2 package exec ${env.java_home}/bin/javapackager -deploy -native installer -appclass ${app.main.class} -srcfiles ${project.build.directory}/${artifactid}-app.jar -outdir ./target -outfile ${project.artifactid}-app -v once the configuration is done, you can run “mvn clean package” and you will find your executable jar as well as the native installer in your target folder. this solution works fine in most cases, but sometimes you can get in trouble with this plugin. when you have configuration files in your project that also exist in one of your dependencies, the unpack-dependency goal will overwrite your configuration file. for example your project contains a file like “meta-inf/service/com.myconf.file” and any dependency does contain the same file. in this case solution 2 may be a better approach. solution 2: the maven-shade plugin the maven-shade plugin provides the capability to package artifacts into an uber-jar, including its dependencies. it also provides various transformers to merge configuration files or to define the manifest of your jar file. this capability allows us to create an executable uber jar without involving the javapackager, so the packager is only needed to create the native executable. to build an executable uber jar following plugin configuration is needed: org.apache.maven.plugins maven-shade-plugin 2.3 package shade junit:junit jmock:* *:xml-apis … meta-inf/services/conf.file ${app.main.class} ${maven.compile.java.version} ${maven.compile.java.version} this configuration defines a main-class entry in the manifest and merges all “meta-inf/services/conf.files” together. the resulting executable jar file is now a valid input for the javapacker to create a native installer. the configuration to create a native installer with “exec-maven-plugin” and javapacker tool is exactly the same like in solution 1. i provided two example projects on github https://github.com/amoahcp/mvndemos where you can test both configurations. these are two simple projects with a main class starting a jetty webserver on port 8080, so the only dependency is jetty. both solutions can be used for any type of java projects, even with swing or javafx. reference: http://jacpfx.org/2014/10/08/uber-jars.html

in this article i explained how spring built-in transformers works for while transforming object message to map message. sometimes the messages need to be transformed before they can be consumed to achieve a business purpose. for example, a producer uses a plain xml as its payload to produce a message, while a consumer is interested in java object or types like plain text ,name-value pairs, or json model. spring integration provides endpoints such as service activators, channel adapters, message bridges, gateways, transformers, filters, and routers. in this example how transformers endpoint transform object message to map message. references: spring integration spring with jms spring with junit mockrunner sts high level view spring-mockrunner.xml in spring-mockrunner.xml file, i defined mockqueue, mockqueueconnectionfactory for inbound queue, and outbound queue for quick testing purpose. inboundqueue is where you will publish object message from objecttomaptransformertest.java class. outboundqueue where this queue expecting mapmessage type object and this queue is listing mapmessagelistener.java class. for more information mockrunner works please check my previous article mockrunner with spring jms . pom.xml 4.0.0 org.springframework.samples spring-int-jms-basic 0.0.1-snapshot 1.6 utf-8 utf-8 3.2.3.release 1.0.13 1.7.5 4.11 org.springframework spring-context ${spring-framework.version} org.springframework spring-tx ${spring-framework.version} org.springframework.integration spring-integration-core 2.2.4.release org.springframework.integration spring-integration-jmx 2.2.4.release org.springframework.integration spring-integration-jms 2.2.4.release org.slf4j slf4j-api ${slf4j.version} compile ch.qos.logback logback-classic ${logback.version} runtime org.springframework spring-test ${spring-framework.version} test junit junit ${junit.version} test com.mockrunner mockrunner-jms 1.0.3 javax.jms jms 1.1 org.codehaus.jackson jackson-mapper-asl 1.9.3 compile spring-int-jms.xml the endpoint is configured to connect to a jms server, fetch the messages,and publish them onto a local channel i.e inputchannel. where as connection-factory, and destination referred mockqueueconnectionfactory, and mockqueue(inboundqueue) beans from spring-mockrunner.xml file. inputchannel and outputchannel defined as queue channel objecttomaptransformer: object-to-map-transformer element that takes the payload from the input channel original here mockrunner-in-queue object message and emits a name-value paired map object onto the output channel i.e outputchannel and outboundjmsadapter bean fetch this message and publish to queue i.e mockrunner-out-queue. inboundjmsadapter : inbound-channel-adapter bean is responsible for receiving messages from a jms server here it is reading from mock queue name mockrunner-in-queue see objecttomaptransformertest.java class. outboundjmsadapter : outbound-channel-adapter bean is responsible to fetch messages from the channel i.e outputchannel and publish them to jms queue or topic. in this outbounjmsadapter reading message outputchannel as mapmessage and publish to outboundqueue(mockrunner-out-queue). mapmessagelistener.java package com.spijb.listener; import javax.jms.jmsexception; import javax.jms.mapmessage; import javax.jms.session; import org.slf4j.logger; import org.slf4j.loggerfactory; import org.springframework.jms.listener.sessionawaremessagelistener; public class mapmessagelistener implements sessionawaremessagelistener { private static final logger log = loggerfactory.getlogger(mapmessagelistener.class); @override public void onmessage(mapmessage message, session session) throws jmsexception { log.info("message received \r\n"+message); } } it is plain mapmessagelistener class to print received message from queue. department.java package com.spijb.domain; import java.io.serializable; public class department implements serializable{ private static final long serialversionuid = 1l; private final integer deptno; private final string name; private final string location; public department() { deptno=10; name="sales"; location="tx"; } public department(integer dno,string name,string loc) { this.deptno=dno; this.name=name; this.location=loc; } public integer getdeptno() { return deptno; } public string getname() { return name; } public string getlocation() { return location; } @override public string tostring() { return this.deptno+"-> "+this.name+"->"+this.location; } } domain object to send as a message, by default constructor assign deptno 10 , name as sales, location as tx also provide parameter constructor. spring junit class objecttomaptransformertest.java package com.spijb.invoker; import javax.jms.jmsexception; import javax.jms.message; import javax.jms.objectmessage; import javax.jms.session; import org.junit.test; import org.junit.runner.runwith; import org.springframework.beans.factory.annotation.autowired; import org.springframework.jms.core.jmstemplate; import org.springframework.jms.core.messagecreator; import org.springframework.test.context.contextconfiguration; import org.springframework.test.context.junit4.springjunit4classrunner; import com.mockrunner.mock.jms.mockqueue; import com.spijb.domain.department; @runwith(springjunit4classrunner.class) @contextconfiguration({"classpath:spring-mockrunner.xml","classpath:spring-int-jms.xml"}) public class objecttomaptransformertest { @autowired private jmstemplate jmstemplate; @autowired private mockqueue inboundqueue; @test public void shouldsendmessage() throws interruptedexception { final department defaultdepartment = new department(); jmstemplate.send(inboundqueue,new messagecreator() { @override public message createmessage(session session) throws jmsexception { objectmessage objectmessage = session.createobjectmessage(); objectmessage.setobject(defaultdepartment); return objectmessage; } }); thread.sleep(5000); } } spring with junit class where you can send message to inputchannel i.e inboundqueue using mockrunner. output : info: started inboundjmsadapter oct 06, 2014 1:24:25 pm org.springframework.integration.endpoint.abstractendpoint start info: started org.springframework.integration.config.consumerendpointfactorybean#1 13:24:26.882 [org.springframework.jms.listener.defaultmessagelistenercontainer#0-1] info c.spijb.listener.mapmessagelistener - message received com.mockrunner.mock.jms.mockmapmessage: {location=tx, name=sales, deptno=10} oct 06, 2014 1:24:30 pm org.springframework.context.support.abstractapplicationcontext doclose info: closing org.springframework.context.support.genericapplicationcontext@5840979b: startup date [mon oct 06 13:24:25 cdt 2014]; root of context hierarchy oct 06, 2014 1:24:30 pm org.springframework.context.support.defaultlifecycleprocessor$lifecyclegroup stop info: stopping beans in phase 2147483647 in the above highlighted one is output as map.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.

Sometimes one would want to handle strings which contain characters not included in UTF-8 or the default encoding (set in mule-deploy.properties). In these scenarios a different encoding which is capable of handling these characters (such as UTF-16 or UTF-32) can be used. To do so the default encoding can be easily changed by making a few modifications according to the type of transformer being used. Changing Encoding with the Datamapper When using the datamapper with data such as XML, one can easily choose the encoding by clicking on the settings button in the mapping (this should be set properly for both input and output) : Settings button datamapper A similar panel to the one below should appear: Changing Encoding when using “simple” transformers When using transformers such as object-to-string or byte-array-to-string, one would think that setting the “encoding” attribute on the transformer would do the trick: Unfortunately this doesn’t work, since the current Mule’s behaviour is to use this property just to set the MULE_ENCODING outbound property after the transformation is done. However, instead we should make sure that MULE_ENCODING outbound property is set properly before invoking the transformer. The transformer would then be able to transform the payload correctly for us.

eclipse has a great user interface (ui). but what if i want to do things from the command line, without the gui? for example to build one or more projects in the workspace without using the eclipse ui? with this, i can do automated check-outs and do automated builds. performed a command line project build with eclipse the solution to this: there is a command line version of eclipse which i can use to run eclipse in the command line version. inside the eclipse folder on windows, there is the eclipsec program which is the command-line version of eclipse: eclipsec program, a command line version of eclipse the options of this command line version (for eclipse kepler) are described here: http://help.eclipse.org/kepler/index.jsp?topic=%2forg.eclipse.platform.doc.isv%2freference%2fmisc%2fruntime-options.html for example eclipsec.exe -nosplash -application org.eclipse.cdt.managedbuilder.core.headlessbuild -data c:\my_wsp -build k64f will launch eclipse without splash screen ( -nosplash ), uses the - application command to load the managed make builder (which is used to build projects), with -data i specify the workspace to be used, and with the -build command it will the project k64f. more options and details are shown here: http://stackoverflow.com/questions/344797/build-several-cdt-c-projects-from-commandline and a very good article with additional background information how to use it with the gnu arm eclipse plubins can be found here: http://gnuarmeclipse.livius.net/blog/headless-builds/ happy headlessing :-)

Recently I ran into very interesting problem which I thought would take me just a couple of minutes to solve: protecting Apache CXF (current release 3.0.1)/ JAX-RS REST services with Spring Security (current stable version 3.2.5) in the application running inside embedded Jetty container (current release 9.2). At the end, it turns out to be very easy, once you understand how things work together and known subtle intrinsic details. This blog post will try to reveal that. Our example application is going to expose a simple JAX-RS / REST service to manage people. However, we do not want everyone to be allowed to do that so the HTTP basic authentication will be required in order to access our endpoint, deployed at http://localhost:8080/api/rest/people. Let us take a look on thePeopleRestService class: package com.example.rs; import javax.json.Json; import javax.json.JsonArray; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; @Path( "/people" ) public class PeopleRestService { @Produces( { "application/json" } ) @GET public JsonArray getPeople() { return Json.createArrayBuilder() .add( Json.createObjectBuilder() .add( "firstName", "Tom" ) .add( "lastName", "Tommyknocker" ) .add( "email", "[email protected]" ) ) .build(); } } As you can see in the snippet above, nothing is pointing out to the fact that this REST service is secured, just couple of familiar JAX-RS annotations. Now, let us declare the desired security configuration following excellent Spring Security documentation. There are many ways to configure Spring Security but we are going to show off two of them: using in-memory authentication and using user details service, both built on top of WebSecurityConfigurerAdapter. Let us start with in-memory authentication as it is the simplest one: package com.example.config; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Configuration; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; import org.springframework.security.config.http.SessionCreationPolicy; @Configuration @EnableWebSecurity @EnableGlobalMethodSecurity( securedEnabled = true ) public class InMemorySecurityConfig extends WebSecurityConfigurerAdapter { @Autowired public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception { auth.inMemoryAuthentication() .withUser( "user" ).password( "password" ).roles( "USER" ).and() .withUser( "admin" ).password( "password" ).roles( "USER", "ADMIN" ); } @Override protected void configure( HttpSecurity http ) throws Exception { http.httpBasic().and() .sessionManagement().sessionCreationPolicy( SessionCreationPolicy.STATELESS ).and() .authorizeRequests().antMatchers("/**").hasRole( "USER" ); } } In the snippet above there two users defined: user with the role USER and admin with the roles USER,ADMIN. We also protecting all URLs (/**) by setting authorization policy to allow access only users with roleUSER. Being just a part of the application configuration, let us plug it into the AppConfig class using @Importannotation. package com.example.config; import java.util.Arrays; import javax.ws.rs.ext.RuntimeDelegate; import org.apache.cxf.bus.spring.SpringBus; import org.apache.cxf.endpoint.Server; import org.apache.cxf.jaxrs.JAXRSServerFactoryBean; import org.apache.cxf.jaxrs.provider.jsrjsonp.JsrJsonpProvider; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.DependsOn; import org.springframework.context.annotation.Import; import com.example.rs.JaxRsApiApplication; import com.example.rs.PeopleRestService; @Configuration @Import( InMemorySecurityConfig.class ) public class AppConfig { @Bean( destroyMethod = "shutdown" ) public SpringBus cxf() { return new SpringBus(); } @Bean @DependsOn ( "cxf" ) public Server jaxRsServer() { JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint( jaxRsApiApplication(), JAXRSServerFactoryBean.class ); factory.setServiceBeans( Arrays.< Object >asList( peopleRestService() ) ); factory.setAddress( factory.getAddress() ); factory.setProviders( Arrays.< Object >asList( new JsrJsonpProvider() ) ); return factory.create(); } @Bean public JaxRsApiApplication jaxRsApiApplication() { return new JaxRsApiApplication(); } @Bean public PeopleRestService peopleRestService() { return new PeopleRestService(); } } At this point we have all the pieces except the most interesting one: the code which runs embedded Jettyinstance and creates proper servlet mappings, listeners, passing down the configuration we have created. package com.example; import java.util.EnumSet; import javax.servlet.DispatcherType; import org.apache.cxf.transport.servlet.CXFServlet; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.servlet.FilterHolder; import org.eclipse.jetty.servlet.ServletContextHandler; import org.eclipse.jetty.servlet.ServletHolder; import org.springframework.web.context.ContextLoaderListener; import org.springframework.web.context.support.AnnotationConfigWebApplicationContext; import org.springframework.web.filter.DelegatingFilterProxy; import com.example.config.AppConfig; public class Starter { public static void main( final String[] args ) throws Exception { Server server = new Server( 8080 ); // Register and map the dispatcher servlet final ServletHolder servletHolder = new ServletHolder( new CXFServlet() ); final ServletContextHandler context = new ServletContextHandler(); context.setContextPath( "/" ); context.addServlet( servletHolder, "/rest/*" ); context.addEventListener( new ContextLoaderListener() ); context.setInitParameter( "contextClass", AnnotationConfigWebApplicationContext.class.getName() ); context.setInitParameter( "contextConfigLocation", AppConfig.class.getName() ); // Add Spring Security Filter by the name context.addFilter( new FilterHolder( new DelegatingFilterProxy( "springSecurityFilterChain" ) ), "/*", EnumSet.allOf( DispatcherType.class ) ); server.setHandler( context ); server.start(); server.join(); } } Most of the code does not require any explanation except the the filter part. This is what I meant by subtle intrinsic detail: the DelegatingFilterProxy should be configured with the filter name which must be exactlyspringSecurityFilterChain, as Spring Security names it. With that, the security rules we have configured are going to apply to any JAX-RS service call (the security filter is executed before the Apache CXF servlet), requiring the full authentication. Let us quickly check that by building and running the project: mvn clean package java -jar target/jax-rs-2.0-spring-security-0.0.1-SNAPSHOT.jar Issuing the HTTP GET call without providing username and password does not succeed and returns HTTP status code 401. > curl -i http://localhost:8080/rest/api/people HTTP/1.1 401 Full authentication is required to access this resource WWW-Authenticate: Basic realm="Realm" Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html; charset=ISO-8859-1 Content-Length: 339 Server: Jetty(9.2.2.v20140723) The same HTTP GET call with username and password provided returns successful response (with some JSON generated by the server). > curl -i -u user:password http://localhost:8080/rest/api/people HTTP/1.1 200 OK Date: Sun, 28 Sep 2014 20:07:35 GMT Content-Type: application/json Content-Length: 65 Server: Jetty(9.2.2.v20140723) [{"firstName":"Tom","lastName":"Tommyknocker","email":"[email protected]"}] Excellent, it works like a charm! Turns out, it is really very easy. Also, as it was mentioned before, the in-memory authentication could be replaced with user details service, here is an example how it could be done: package com.example.config; import java.util.Arrays; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; import org.springframework.security.config.http.SessionCreationPolicy; import org.springframework.security.core.authority.SimpleGrantedAuthority; import org.springframework.security.core.userdetails.User; import org.springframework.security.core.userdetails.UserDetails; import org.springframework.security.core.userdetails.UserDetailsService; import org.springframework.security.core.userdetails.UsernameNotFoundException; @Configuration @EnableWebSecurity @EnableGlobalMethodSecurity(securedEnabled = true) public class UserDetailsSecurityConfig extends WebSecurityConfigurerAdapter { @Autowired public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception { auth.userDetailsService( userDetailsService() ); } @Bean public UserDetailsService userDetailsService() { return new UserDetailsService() { @Override public UserDetails loadUserByUsername( final String username ) throws UsernameNotFoundException { if( username.equals( "admin" ) ) { return new User( username, "password", true, true, true, true, Arrays.asList( new SimpleGrantedAuthority( "ROLE_USER" ), new SimpleGrantedAuthority( "ROLE_ADMIN" ) ) ); } else if ( username.equals( "user" ) ) { return new User( username, "password", true, true, true, true, Arrays.asList( new SimpleGrantedAuthority( "ROLE_USER" ) ) ); } return null; } }; } @Override protected void configure( HttpSecurity http ) throws Exception { http .httpBasic().and() .sessionManagement().sessionCreationPolicy( SessionCreationPolicy.STATELESS ).and() .authorizeRequests().antMatchers("/**").hasRole( "USER" ); } } Replacing the @Import( InMemorySecurityConfig.class ) with @Import( UserDetailsSecurityConfig.class ) in the AppConfig class leads to the same results, as both security configurations define the identical sets of users and their roles. I hope, this blog post will save you some time and gives a good starting point, as Apache CXF and Spring Security are getting along very well under Jetty umbrella! The complete source code is available on GitHub.

Memoization is a technique used to speed up functions. Memoization may be done manually. It may also be done automatically. We can find many examples of automatic memoization on Internet. In this article, I will show how Java 8 makes it very easy to memoize functions. What is memoization Memoization consist in caching the results of functions in order to speed them up when they are called several times with the same argument. The first call implies computing and storing the result in memory before returning it. Subsequent calls with the same parameter imply only fetching the previously stored value and returning it. How to apply memoization Memoization may be applied manually by hard coding it in every function that may benefit from it. If it takes a long time to compute the return value, memoization will speed up the program. For functions that take less time to evaluate than fetching the previously stored value from memory, memoization is clearly not a good option. Hard coding memoization by hand in each function is not a good option neither because it is repeating the same principle again and again. That is why automatic memoization is desirable. What to memoize Memoization applies to functions. Prior to Java 8, Java had no functions. However, it was perfectly possible to define some. Furthermore, we used to create “functional” methods, that is methods taking an argument and returning a value based only upon this argument. These kind of method may benefit from memoization. By the way, there is a match between functional methods and functions. For example, the following method: Integer doubleValue(Integer x) { return x * 2; } corresponds to: Integer doubleValue(Integer x) { if (cache.containsKey(x)) { return cache.get(x); } else { Integer result = x * 2; cache.put(x, result) ; return result; } } In Java 8, we can make this much cleaner: Map cache = new ConcurrentHashMap<>(); Integer doubleValue(Integer x) { return cache.computeIfAbsent(x, y -> y * 2); } Our function may be modified to use the same technique: Function doubleValue = x -> cache.computeIfAbsent(x, y -> y * 2); This is pretty simple, but it has two main drawbacks: We have to repeat this modification for all functions. The map we use is exposed and could potentially be modified by another thread having nothing to do with the function. The second problem is quite easy to address. We may put the method or the function in a separate class, including the map, with private access. For example, for the method case: public class Doubler { private static Map cache = new ConcurrentHashMap<>(); public static Integer doubleValue(Integer x) { return cache.computeIfAbsent(x, y -> y * 2); } } We may then instantiate that class and use it each time we want to compute a value: Integer y = Doubler.doubleValue(x); With this solution, the map is no longer accessible from outside. We can't do the same for functions because functions are anonymous classes and such classes may not have static members. One possibility would be to pass the map to the function as an additional argument. This may be done through a closure: class Doubler { private static Map cache = new ConcurrentHashMap<>(); public static Function doubleValue = x->cache.computeIfAbsent(x, y -> y * 2); } We can use this function as follows: Integer y = Doubler.doubleValue.apply(x); This gives no advantage compared to the “method” solution. However, we may also use this function in more idiomatic examples, such as: IntStream.range(1, 10).boxed().map(Doubler.doubleValue); This is equivalent to using the method version with the following syntax: IntStream.range(1, 10).boxed().map(ThisClass::doubleValue); The main problem is that while solving the second issue, we have made the first one more acute, which make automatic memoization more desirable. Automatic memoization: requirements What we need is a way to do the following: Function f = x -> x * 2; Function g = Memoizer.memoize(f); so that we may use the memoized function as a drop in replacement for the original one. All values returned by function g will be calculated through the original function f the first time, and returned from the cache for all subsequent accesses. By contrast, if we create a third function: Function f = x -> x * 2; Function g = Memoizer.memoize(f); Function h = Memoizer.memoize(f); the values cached by g will not be returned by h. In other words, g and h will use separate caches. Implementation The Memoizer class is quite simple: public class Memoizer { private final Map cache = new ConcurrentHashMap<>(); private Memoizer() {} private Function doMemoize(final Function function) { return input -> cache.computeIfAbsent(input, function::apply); } public static Function memoize(final Function function) { return new Memoizer().doMemoize(function); } } Using this class is also extremely simple: Integer longCalculation(Integer x) { try { Thread.sleep(1_000); } catch (InterruptedException ignored) { } return x * 2; } Function f = this::longCalculation; Function g = Memoizer.memoize(f); public void automaticMemoizationExample() { long startTime = System.currentTimeMillis(); Integer result1 = g.apply(1); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = g.apply(1); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } Running the automaticMemoizationExample method will produce the following result: 2 2 1000 0 We can now make memoized function out of ordinary ones by just calling a single method! What about functions with several arguments? Short answer: nothing. There are no such things in this world as functions with several arguments. Functions are applications of one set (the source set) to another set (the target set). So, they simply can't have several arguments. But this does not solve our problem. What is the functional equivalent to a method with several arguments? Long answer: what people generally consider as functions with several arguments are in fact either: Functions of tuples Function returning functions returning functions … returning a result In either cases, we are only concerned with functions of one argument, so we can easily use our Memoizer class. Using functions of tuples would probably be the simplest choice... if Java had tuples! We could of course write tuples. But to store tuples in maps, we would have to implement equals and hashcode for them, plus we would have to define tuples for two elements (pairs), tuple for three elements, and so on. Who knows where to stop? The second option is much easier. It is based upon currying, which means applying each argument one after the other instead of applying them as a whole (the tuple). Currying a function is very easy. The only problem, in Java 8, is that writing the types is really cumbersome. Currying a “function of two arguments” (in fact a function of a pair) is easy once you master the type. Java has in fact a shortcut for functions of tuple2 which is called BiFunction. We will take this as an example. The two following functions are equivalent (from the result point of view): BiFunction h = (x, y) -> x + y; Function> hc = x -> y -> x + y; Not considering the types, there are very little differences. In the first case, the two arguments are put between parentheses, separated by a comma, which is, by the way, how tuples are written in most languages which have them! Remove the parentheses and separate the arguments with an arrow and you get the curried version. We can only regret that we have to write the type as: Function> when other languages use a simplified syntax such as: Integer -> Integer -> Integer From this, it is easy to memoized this curried version, although we cant use the same simple form as previously. We have to memoize each function: Function> mhc = Memoizer.memoize(x -> Memoizer.memoize(y -> x + y)); Same thing for a function of (a tuple of) 3 arguments (which by the way has no equivalent in Java): Function>> f3 = x -> y -> z -> x + y - z; Function>> f3m = Memoizer.memoize(x -> Memoizer.memoize(y -> Memoizer.memoize(z -> x + y – z)); Here is an example of using this memoized function “of three arguments”: Function>> f3 = x -> y -> z -> longCalculation(x) + longCalculation(y) - longCalculation(z); Function>> f3m = Memoizer.memoize(x -> Memoizer.memoize(y -> Memoizer.memoize(z -> longCalculation(x) + longCalculation(y) - longCalculation(z)))); public void automaticMemoizationExample2() { long startTime = System.currentTimeMillis(); Integer result1 = f3m.apply(2).apply(3).apply(4); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = f3m.apply(2).apply(3).apply(4); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } This example produces the following output: 2 2 3002 0 showing that the first access to method longCalculation has taken 3000 milliseconds and the second has return immediately. On the other hand, using a function of tuple may seem easier once you have the Tuple class defined. Here is an example of Tuple3: public class Tuple3 { public final T _1; public final U _2; public final V _3; public Tuple3(T t, U u, V v) { _1 = Objects.requireNonNull(t); _2 = Objects.requireNonNull(u); _3 = Objects.requireNonNull(v); } @Override public boolean equals(Object o) { if (!(o instanceof Tuple3)) return false; else { Tuple3 that = (Tuple3) o; return _1.equals(that._1) && _2.equals(that._2) && _3.equals(that._3); } } @Override public int hashCode() { return _1.hashCode() + _2.hashCode() + _3.hashCode(); } } Using this class, we may rewrite the previous example as: Function, Integer> ft = x -> longCalculation(x._1) + longCalculation(x._2) - longCalculation(x._3); Function, Integer> ftm = Memoizer.memoize(ft); public void automaticMemoizationExample3() { long startTime = System.currentTimeMillis(); Integer result1 = ftm.apply(new Tuple3<>(2, 3, 4)); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = ftm.apply(new Tuple3<>(2, 3, 4)); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } Conclusion Memoizing is about maintaining state between function calls. A memoized function is a function which behavior is dependent upon the current state. However, it will always return the same value for the same argument. Only the time needed to return the value will be different. So the memoized function is still a pure function if the original function is pure. However, there is a kind of function that may pose a problem: recursive functions that call themselves several times with the same argument may not be memoized this way. This will be addressed in a next article.

this article describes steps required to configure your eclipse for sonarqube such that developers are not required to leave the eclipse ide to manage their source code quality. please feel free to comment/suggest if i missed to mention one or more important points. also, sorry for the typos. following are the key points described later in this article: installing & configuraing sonarqube in eclipse analyzing code using sonarqube installing & configuraing sonarqube in eclipse follow the steps given on following pages to install and configure sonarqube in your eclipse ide. the page on installing sonarqube in eclipse consists of steps required to install sonarqube in the ide. once downloaded, it would prompt to restart the eclipse ide. for proceeding ahead, you must restart the eclipse ide. the page on configuring sonarqube in eclipse consists of steps required to configure eclipse for sonarqube. the key thing to note is the fact that as you click on preferences > sonarqube > servers , you see the default server entry for http://localhost:9000. initially, as per instruction page, seeing the default entry, i went ahead to next step for linking the project to sonarqube server and run the analysis. however, it throw the error such as unknowb version of sonarqube server . after a little research, i figured out that the error was thrown as the sonarqube server was not reachable although i saw it configured as http://localhost:9000. the solution is to go to preferences > sonarqube > servers , click “edit” button and you would see sonarqube server url, username and password. the key is to give username and password and run the analysis once again. analyzing code using sonarqube once you are done with installation and configuration of sonarqube server in eclipse ide, for code analysis, all you need is right click on your project and click sonarqube > analyze and that is it. the violations would appear in sonarqube analysis as shown in the screenshot below. sonarqube analysis in eclipse however, do note that these issues may not appear if you try to access on the browser. for that to happen, you need to once again run “sonar-runner” from the project root.

In my last post, I explained how to do code reviews properly. I recommended taking advantage of static analysis tools like Findbugs, PMD, Klocwork or Fortify to check for common mistakes and bad code before passing the code on to a reviewer, to make the reviewer’s job easier and reviews more effective. Some readers asked whether static analysis tools can be used instead of manual code reviews. Manual code reviews add delays and costs to development, while static analysis tools keep getting better, faster, and more accurate. So can you automate code reviews, in the same way that many teams automate functional testing? Do you need to do manual reviews too, or can you rely on technology to do the job for you? Let’s start by understanding what static analysis bug checking tools are good at, and what they aren’t. What static analysis tools can do – and what they can’t do In this article, Paul Anderson at GrammaTech does a good job of explaining how static analysis bug finding works, the trade-offs between recall (finding all of the real problems), precision (minimizing false positives) and speed, and the practical limitations of using static analysis tools for finding bugs. Static analysis tools are very good at catching certain kinds of mistakes, including memory corruption and buffer overflows (for C/C++), memory leaks, illegal and unsafe operations, null pointers, infinite loops, incomplete code, redundant code and dead code. A static analysis tool knows if you are calling a library incorrectly (as long as it recognizes the function), if you are using the language incorrectly (things that a compiler could find but doesn’t) or inconsistently (indicating that the programmer may have misunderstood something). And static analysis tools can identify code with maintainability problems, code that doesn't follow good practice or standards, is complex or badly structured and a good candidate for refactoring. But these tools can’t tell you when you have got the requirements wrong, or when you have forgotten something or missed something important – because the tool doesn't know what the code is supposed to do. A tool can find common off-by-one mistakes and some endless loops, but it won’t catch application logic mistakes like sorting in descending order instead of ascending order, or dividing when you meant to multiply, referring to buyer when it should have been seller, or lessee instead of lessor. These are mistakes that aren't going to be caught in unit testing either, since the same person who wrote the code wrote the tests, and will make the same mistakes. Tools can’t find missing functions or unimplemented features or checks that should have been made but weren't. They can’t find mistakes or holes in workflows. Or oversights in auditing or logging. Or debugging code left in by accident. Static analysis tools may be able to find some backdoors or trapdoors – simple ones at least. And they might find some concurrency problems – deadlocks, races and mistakes or inconsistencies in locking. But they will miss a lot of them too. Static analysis tools like Findbugs can do security checks for you: unsafe calls and operations, use of weak encryption algorithms and weak random numbers, using hard-coded passwords, and at least some cases of XSS, CSRF, and simple SQL injection. More advanced commercial tools that do inter-procedural and data flow analysis (looking at the sources, sinks and paths between) can find other bugs including injection problems that are difficult and time-consuming to trace by hand. But a tool can’t tell you that you forgot to encrypt an important piece of data, or that you shouldn't be storing some data in the first place. It can’t find logic bugs in critical security features, if sensitive information could be leaked, when you got an access control check wrong, or if the code could fail open instead of closed. And using one static analysis tool on its own to check code may not be enough. Evaluations of static analysis tools, such as NIST's SAMATE project (a series of comparative studies, where many tools are run against the same code), show almost no overlap between the problems found by different tools (outside of a few common areas like buffer errors) even when the tools are supposed to be doing the same kinds of checks. Which means that to get the most out of static analysis, you will need to run two or more tools against the same code (which is what SonarQube, for example, which integrates its own static analysis results with other tools, including popular free tools, does for you). If you’re paying for commercial tools, this could get very expensive fast. Tools vs. Manual Reviews Tools can find cases of bad coding or bad typing – but not bad thinking. These are problems that you will have to find through manual reviews. A 2005 study Comparing Bug Finding Tools with Reviews and Tests used Open Source bug finding tools (including Findbugs and PMD) on 5 different code bases, comparing what the tools found to what was found through code reviews and functional testing. Static analysis tools found only a small subset of the bugs found in manual reviews, although the tools were more consistent – manual reviewers missed a few cases that the tools picked up. Just like manual reviews, the tools found more problems with maintainability than real defects (this is partly because one of the tools evaluated – PMD – focuses on code structure and best practices). Testing (black box – including equivalence and boundary testing – and white box functional testing and unit testing) found fewer bugs than reviews. But different bugs. There was no overlap at all between bugs found in testing and the bugs found by the static analysis tools. Finding problems that could happen - or do happen Static analysis tools are good at finding problems that “could happen”, but not necessarily problems that “do happen”. Researchers at Colorado State University ran static analysis tools against several releases of different Open Source projects, and compared what the tools found against the changes and fixes that developers actually made over a period of a few years – to see whether the tools could correctly predict the fixes that needed to be made and what code needed to be refactored. The tools reported hundreds of problems in the code, but found very few of the serious problems that developers ended up fixing. One simple tool (Jlint) did not find anything that was actually fixed or cleaned up by developers. Of 112 serious bugs that were fixed in one project, only 3 were also found by static analysis tools. In another project, only 4 of 136 bugs that were actually reported and fixed were found by the tools. Many of the bugs that developers did fix were problems like null pointers and incorrect string operations – problems that static analysis tools should be good at catching, but didn’t. The tools did a much better job of predicting what code should be refactored: developers ended up refactoring and cleaning up more than 70% of the code structure and code clarity issues that the tools reported (PMD, a free code checking tool, was especially good for this). Ericsson evaluated different commercial static analysis tools against large, well-tested, mature applications. On one C application, a commercial tool found 40 defects – nothing that could cause a crash, but still problems that needed to be fixed. On another large C code base, 1% of the tool’s findings turned out to be bugs serious enough to fix. On the third project, they ran 2 commercial tools against an old version of a C system with known memory leaks. One tool found 32 bugs, another 16: only 3 of the bugs were found by both tools. Surprisingly, neither tool found the already known memory leaks – all of the bugs found were new ones. And on a Java system with known bugs they tried 3 different tools. None of the tools found any of the known bugs, but one of the tools found 19 new bugs that the team agreed to fix. Ericsson’s experience is that static analysis tools find bugs that are extremely difficult to find otherwise. But it’s rare to find stop-the-world bugs – especially in production code – using static analysis. This is backed up by another study on the use of static analysis (Findbugs) at Google and on the Sun JDK 1.6.0. Using the tool, engineers found a lot of bugs that were real, but not worth the cost of fixing: deliberate errors, masked errors, infeasible situations, code that was already doomed, errors in test code or logging code, errors in old code that was “going away soon” or other relatively unimportant cases. Only around 10% of medium and high priority correctness errors found by the tool were real bugs that absolutely needed to be fixed. The Case for Security So far we've mostly looked at static analysis checking for run-time correctness and general code quality, not security. Although security builds on code quality – vulnerabilities are just bugs that hackers look for and exploit – checking code for correctness and clarity isn’t enough for a secure app. A lot of investment in static analysis technology over the past 5-10 years has been in finding security problems in code, such as common problems listed in OWASP’s Top 10 or the SANS/CWE Top 25 Most Dangerous Software Errors. A couple of studies have looked at the effectiveness of static analysis tools compared to manual reviews in finding security vulnerabilities. The first study was on a large application that had 15 known security vulnerabilities found through a structured manual assessment done by security experts. Two different commercial static analysis tools were run across the code. The tools together found less than half of the known security bugs – only the simplest ones, the bugs that didn't require a deep understanding of the code or the design. And of course the tools reported thousands of other issues that needed to be reviewed and qualified or thrown away as false positives. These other issues including some run-time correctness problems, null pointers and resource leaks, and code quality findings (dead code, unused variables), but no other real security vulnerabilities beyond those already found by the manual security review. But this assumes that you have a security expert around to review the code. To find security vulnerabilities, a reviewer needs to understand the code (the language and the frameworks), and they also need to understand what kind of security problems to look for. Another study shows how difficult this is. Thirty developers were hired to do independent security code reviews of a small web app (some security experts, others web developers). They were not allowed to use static analysis tools. The app had 6 known vulnerabilities. 20% of the reviewers did not find any of the known bugs. None of the reviewers found all of the known bugs, although several found a new XSS vulnerability that the researchers hadn’t known about. On average, 10 reviewers would have had only an 80% chance of finding all of the security bugs. And, not Or Static analysis tools are especially useful for developers working in unsafe languages like C/C++ (where there is a wide choice of tools to find common mistakes) or dynamically typed scripting languages like Javascript or PHP (where unfortunately the tools aren't that good), and for teams starting off learning a new language and framework. Using static analysis is (or should be) a requirement in highly regulated, safety critical environments like medical devices and avionics. And until more developers get more training and understand more about how to write secure software, we will all need to lean on static analysis (and dynamic analysis) security testing tools to catch vulnerabilities. But static analysis isn't a substitute for code reviews. Yes, code reviews take extra time and add costs to development, even if you are smart about how you do them – and being smart includes running static analysis checks before you do reviews. If you want to move fast and write good, high-quality and secure code, you still have to do reviews.You can’t rely on static analysis alone.

the gnu arm eclipse plugins from liviu already offer several built-in actions which can be performed at the end of a build: creating flash image , create listing file and printing the code and data size: gnu arm eclipse extra post build steps but what if i need different things, or even more things? post-build steps for this there is the ‘post-build steps’ settings i can use: that command is executed at the end of the build: print size post-build step :!: the post build step is only executed if sources files have been compiled and linked. if you want to enforce that there is always a ‘true’ post build, then you need to delete some files in the pre-build step to enforce a compilation and a link phase. multiple post-build steps but what i need more than one action in the post-build step? i could call a batch or script file, but this is probably an overkill in too many cases, and adds a dependency to that script file. a better approach is to directly execute multiple commands as post-build step. unfortunately, the documentation found about the post-build step with a web-search is misleading (e.g. in the eclipse luna documentation ): “command: specifies one or more commands to execute immediately after the execution of the build. use semicolons to separate multiple commands.” unfortunately, semicolons is plain wrong (at least did not work for me) :-(. the solution is to use ‘ & ‘ (ampersand) to separate multiple commands on windows : :idea: on linux, use the ‘;’ to separate commands as noted in the documentation/help, and use ‘&’ on windows. unfortunately, this makes project not cross-platform. multiple post-build commands and this works for me, at least under windows 7 :-).

CodePro Analytix is the premier Java software testing tool for Eclipse developers.

this tip sounds very basic, but still: i get asked about this about once a week. so it must be something non-obvious in eclipse then ;-): how to add existing files to an eclipse project. as with many things in eclipse, there is not a single way to do something. there are two basic ways to do this: import drag & drop copy & paste the first is the ‘official’ way in eclipse, the other two are much faster and easier :-). 1. import to import one or multiple files, select the folder/project where i want to add the files, then use the menu file > import : menu file import alternatively, i can use the context menu: import context menu then use general > file system : import from file system in the next dialog, i can browse for the files and select them to be added: importing files from filesystem 2. drag & drop while the ‘import’ method is fine, i rarely use it: too many clicks! a easier way (at least under windows) is to drag & drop the files from the windows explorer: drag and drop to add files 3. copy & paste the third way is even easier: select the files i want to add in the explorer/file manager, and copy them ( ctrl-c on windows) paste them in the project/folder in eclipse ( ctrl-v on windows) that last method does not advanced options like if i want to link to the files, see “ link to files and folders in eclipse “, but is definitely the fastest and easiest way. happy adding :-)

Learn how to resolve the ''failure to transfer'' error encountered in Maven in this quick tutorial.

Introduction In this post I would like to put my two cents in and talk about RESTful web services. First of all I don't intend to discuss the history of RESTful services. Neither this is a tutorial about implementing RESTful web services. What I'm trying to do is discuss how we, as developers, should think about a RESTful web service and do better RESTful APIs. First of all I want to thank my friend, Filipe (@filaruina) whom worked with me at Cortex Intelligence. We talked a lot about REST and he helped me to open my eyes to a lot of aspects that I hadn't yet. There and back again This month I started working in a new company. One of my first tasks was to finish the integration of two systems which the communication was done via SOAP. When I started reading the source code I saw the common mistake some developers make (including me). The first thing we do is replace all the SOAP calls by HTTP requests. After that we use the status code to inform the result of our method. We return 200 if the method executes without any errors, and 500 when there are any error in the processing. Last, but not least, we spit the log of the operation in the response body, so the client knows what happened in the method execution and there you are! We have REST... Please, stop! RESTful for the win I'm sure that you have read a lot of tutorials about REST, status codes, http verbs and everything else. But it's time to stop memorizing REST and start thinking REST. REST is not a protocol (HTTP is). REST is not a specific technology communication (RMI is). REST is an abstraction over a protocol, it's a modern web architecture, thought to replace the specific technology communication protocols and define a way of integrating systems with very loose coupling and much more versatility. Much more than that, REST is a communication way where the communicators (the systems) are not being orchestrated by a conductor. It is more like a dance, where the dancers know what they should do to make the dance works and be beautiful. If you are using REST just as a communication protocol, stop calling it REST and start calling it "HTTP Interface", it's better. Hyping Hypermedia When we talk about REST, we are talking about Hypermedia. Period. I totally agree with Steve Klabnik in his post Rest is Over. Steve says that RESTful APIs have evolved to Hypermedia APIs. As it should be from the begining. REST isn't only about communication by HTTP requests, correct HTTP verbs or well-chosen status codes, the real thing is Hypermedia. Formally we have four constraints in REST: Identification of resources; Manipulation of resources through representations; Self-descriptive messages; Hypermedia as the engine of application state; In my experience I've seen that constraints one and two are easier to understand and use. Constraint three is often misunderstood and not well implemented. Constraint four, almost always, is left behind. But what does "Hypermedia as the engine of application state" mean? Well, it means that you can look to your application as a finite-state machine, where the actual node represents the current resource being "viewed" by the user, and the transitions are like instructions to the user on his next steps. Let's take a look at an example. Imagine that we have an online payment system, responsible for managing transactions between sellers and buyers from a lot of other services (have you heard about any services like this?). Imagine that Bob wants to buy a bycicle from Bikers World website. The first thing that the website should do is create a payment resource using the RESTful API of our payment service. It can be something like: POST /payments Host: bikersworld.com Accept: application/json Request-Body: { "transactionId": "hyx48yu9pe", "value": "150.0" } And then our server responds something like this: HTTP/1.1 204 CREATED Content-Type: application/json Response-Body: { "transactionId": "hyx48yu9pe", "value": "150.0", "status": "pending", "links": [ { "rel": "self", "method": "get", "href": "http://onlinepayments.com/payments/5" }, { "rel": "confirm", "method": "put", "href": "http://onlinepayments.com/payments/5/confirm" }, { "rel": "cancel", "method": "delete", "href": "http://onlinepayments.com/payments/5" } ] } Did you notice the "links" key in the json that the API answered? It informs us about the other possible states we can reach from where we are. Would you need to check the API docs to know what you should do next? What would you do if you wanted to cancel the order? Or to retrieve the order info? As you can see, the server request is enough to inform the user about the next states available to the resource. After creating a payment, it's easy to see that there is a requirement to confirm it, and you know all of this without reading any documentation. Another thing that can make our API even better is to use the OPTION verb to serve to our clients the options (duh!) that they have for a resource. Imagine an JSON restul API that for a OPTION request like this: OPTION /payments Host: bikersworld.com Accept: application/json It returns a JSON documentation like that: HTTP/1.1 200 OK Content-Type: application/json Response-Body: { "POST": { "description": "Create an payment", "parameters": { "transactionId": { "type": "string", "description": "The transaction id associated to the payment", "required": true }, "value": { "type": "double", "description": "The value of the payment being created" "required": true } }, "example": { "transactionId": "hyx48yu9pe", "value": "150.0" } }, ... //more useful information } I'm not saying that it is easy to apply this concepts and sometimes your domain can't be mapped very well to a state-driven model and your entities don't have a well defined lifecycle. In this case you should try to do your best to fit these concepts in your API. But don'let hypermedia overwhelm you. You can develop your API incrementally. In this post, Matt Cottingham showed a picture that I consider to be an excelent representation of the steps taken by an API during it's development. As you can see, hypermedia controls are labeled as the top level feature for a restful API, but this doesn't mean that you can't design a restful (not THAT restful) API without it. You can, but don't forget this image. As any software, our API definitions can evolve and be more restful day by day. Useful resources: Principled Design of the Modern Web Architecture, article written by Roy T. Fielding and Richard N. Taylor; Architectural Styles and the Design of Network-based Software Architectures, PhD thesis written by Roy T. Fielding; PUT or POST: The REST of the Story, blog post by John Calcote; A Short Explanation of Hypermedia Controls in RESTful Services, blog post by Matt Cottingham; REST APIs must be hypertext-driven blog post by Roy T. Fielding; Why HATEOAS, presentation by Wayne Lee; The RESTful CookBook, created by Joshua Thijssen; The HTTP OPTIONS method and potential for self-describing RESTful APIs, blog post by Zac Stewart; Netflix Rest API documentation, an awesome example of restful API. (this post was originally published in my blog)

In this article we wanted to share another configuration property that can provide surprising help when setting up your JBoss BPM Suite. Previously we outlined a basic set of configuration properties to provide you with a few tricks when installing your own JBoss BRMS or JBoss BPM Suite products. As the JBoss BPM Suite is a super set, including full JBoss BRMS functionality, the rest of this article will refer only to JBoss BPM Suite but apply to both products. In this article we will show you how to modify your JBoss EAP container configuration to point the products at a custom deployment repository by adjusting a single configuration property. Maven repository The default setup is that the products will look for your maven setting in the default settings.xml as found set in theM2_HOME variable or in the users home directory at .m2/settings.xml. The following system property can be added to JBoss EAP standalone.xml configuration file to point to any file containing your custom settings. kie.maven.settings.custom Location of the maven configuration file where it can find it's settings. Default: the M2_HOME/conf/settings.xml or users home directory .m2/settings.xml Example usage in JBoss EAP When initially setting up the product for use on JBoss EAP containers, one can adjust configuration with the help of system properties. Below we show how to configure an installation to point to our custom maven deployment repository by using a custom settings file we will call bpmsuite-settings.xml We hope this helps you with configuring your own custom deployment repositories and enables you to tie into existing continuous integration infrastructures that might exist in your organization.

Originally Written by Muhammad Irfan Here on the Percona Support team we often ask customers to retrieve disk stats to monitor disk IO and to measure block devices iops and latency. There are a number of tools available to monitor IO on Linux. iostat is one of the popular tools and Percona Toolkit, which is free, contains the pt-diskstats tool for this purpose. The pt-diskstats tool is similar to iostat but it’s more interactive and contains extended information. pt-diskstats reports current disk activity and shows the statistics for the last second (which by default is 1 second) and will continue until interrupted. The pt-diskstats tool collects samples of /proc/diskstats. In this post, I will share some examples about how to monitor and check to see if the IO subsystem is performing properly or if any disks are a limiting factor – all this by using the pt-diskstats tool. pt-diskstats output consists on number of columns and in order to interpret pt-diskstats output we need to know what each column represents. rd_s tells about number of reads per second while wr_s represents number of writes per second. rd_rt and wr_rt shows average response time in milliseconds for reads & writes respectively, which is similar to iostat tool output await column but pt-diskstats shows individual response time for reads and writes at disk level. Just a note, modern iostat splits read and write latency out, but most distros don’t have the latest iostat in their systat (or equivalent) package. rd_mrg and wr_mrg are other two important columns in pt-diskstats output. *_mrg is telling us how many of the original operations the IO elevator (disk scheduler) was able to merge to reduce IOPS, so *_mrg is telling us a quite important thing by letting us know that the IO scheduler was able to consolidate many or few operations. If rd_mrg/wr_mrg is high% then the IO workload is sequential on the other hand, If rd_mrg/wr_mrg is a low% then IO workload is all random. Binary logs, redo logs (aka ib_logfile*), undo log and doublewrite buffer all need sequential writes. qtime and stime are last two columns in pt-diskstats output where qtime reflects to time spent in disk scheduler queue i.e. average queue time before sending it to physical device and on the other hand stime is average service time which is time accumulated to process the physical device request. Note, that qtime is not discriminated between reads and writes and you can check if response time is higher for qtime than it signal towards disk scheduler. Also note that service time (stime field and svctm field in in pt-diskstats & iostat output respectively) is not reliable on Linux. If you read the iostat manual you will see it is deprecated. Along with that, there are many other parameters for pt-diskstats – you can found full documentation here. Below is an example of pt-disktats in action. I used the –devices-regex option which prints only device information that matches this Perl regex. $ pt-diskstats --devices-regex=sd --interval 5 #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.1 sda 21.6 22.8 0.5 45% 1.2 29.4 275.5 4.0 1.1 0% 40.0 145.1 65% 158 297.1 155.0 2.1 1.1 sdb 15.0 21.0 0.3 33% 0.1 5.2 0.0 0.0 0.0 0% 0.0 0.0 11% 1 15.0 0.5 4.7 1.1 sdc 5.6 10.0 0.1 0% 0.0 5.2 1.9 6.0 0.0 33% 0.0 2.0 3% 0 7.5 0.4 3.6 1.1 sdd 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 5.0 sda 17.0 14.8 0.2 64% 3.1 66.7 404.9 4.6 1.8 14% 140.9 298.5 100% 111 421.9 277.6 1.9 5.0 sdb 14.0 19.9 0.3 48% 0.1 5.5 0.4 174.0 0.1 98% 0.0 0.0 11% 0 14.4 0.9 2.4 5.0 sdc 3.6 27.1 0.1 61% 0.0 3.5 2.8 5.7 0.0 30% 0.0 2.0 3% 0 6.4 0.7 2.4 5.0 sdd 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 These are the stats from 7200 RPM SATA disks. As you can see, the write-response time is very high and most of that is made up of IO queue time. This shows the problem exactly. The problem is that the IO subsystem is not able to handle the write workload because the amount of writes that are being performed are way beyond what it can handle. It means the disks cannot service every request concurrently. The workload would actually depend a lot on where the hot data is stored and as we can see in this particular case the workload only hits a single disk out of the 4 disks. A single 7.2K RPM disk can only do about 100 random writes per second which is not a lot considering heavy workload. It’s not particularly a hardware issue but a hardware capacity issue. The kind of workload that is present and the amount of writes that are performed per second are not something that the IO subsystem is able to handle in an efficient manner. Mostly writes are generated on this server as can be seen by the disk stats. Let me show you a second example. Here you can see read latency. rd_rt is consistently between 10ms-30ms. It depends on how fast the disks are spinning and the number of disks. To deal with it possible solutions would be to optimize queries to avoid table scans, use memcached where possible, use SSD’s as it can provide good I/O performance with high concurrency. You will find this post useful on SSD’s from our CEO, Peter Zaitsev. #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.0 sdb 33.0 29.1 0.9 0% 1.1 34.7 7.0 10.3 0.1 61% 0.0 0.4 99% 1 40.0 2.2 19.5 1.0 sdb1 0.0 0.0 0.0 0% 0.0 0.0 7.0 10.3 0.1 61% 0.0 0.4 1% 0 7.0 0.0 0.4 1.0 sdb2 33.0 29.1 0.9 0% 1.1 34.7 0.0 0.0 0.0 0% 0.0 0.0 99% 1 33.0 3.5 30.2 1.0 sdb 81.9 28.5 2.3 0% 1.1 14.0 0.0 0.0 0.0 0% 0.0 0.0 99% 1 81.9 2.0 12.0 1.0 sdb1 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 1.0 sdb2 81.9 28.5 2.3 0% 1.1 14.0 0.0 0.0 0.0 0% 0.0 0.0 99% 1 81.9 2.0 12.0 1.0 sdb 50.0 25.7 1.3 0% 1.3 25.1 13.0 11.7 0.1 66% 0.0 0.7 99% 1 63.0 3.4 11.3 1.0 sdb1 25.0 21.3 0.5 0% 0.6 25.2 13.0 11.7 0.1 66% 0.0 0.7 46% 1 38.0 3.2 7.3 1.0 sdb2 25.0 30.1 0.7 0% 0.6 25.0 0.0 0.0 0.0 0% 0.0 0.0 56% 0 25.0 3.6 22.2 From the below diskstats output it seems that IO is saturated between both reads and writes. This can be noticed with high value for columns rd_s and wr_s. In this particular case, consider having disks in either RAID 5 (better for read only workload) or RAID 10 array is good option along with battery-backed write cache (BBWC) as single disk can really be bad for performance when you are IO bound. device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime sdb1 362.0 27.4 9.7 0% 2.7 7.5 525.2 20.2 10.3 35% 6.4 8.0 100% 0 887.2 7.0 0.9 sdb1 439.9 26.5 11.4 0% 3.4 7.7 545.7 20.8 11.1 34% 9.8 11.9 100% 0 985.6 9.6 0.8 sdb1 576.6 26.5 14.9 0% 4.5 7.8 400.2 19.9 7.8 34% 6.7 10.9 100% 0 976.8 8.6 0.8 sdb1 410.8 24.2 9.7 0% 2.9 7.1 403.1 18.3 7.2 34% 10.8 17.7 100% 0 813.9 12.5 1.0 sdb1 378.4 24.6 9.1 0% 2.7 7.3 506.1 16.5 8.2 33% 5.7 7.6 100% 0 884.4 6.6 0.9 sdb1 572.8 26.1 14.6 0% 4.8 8.4 422.6 17.2 7.1 30% 1.7 2.8 100% 0 995.4 4.7 0.8 sdb1 429.2 23.0 9.6 0% 3.2 7.4 511.9 14.5 7.2 31% 1.2 1.7 100% 0 941.2 3.6 0.9 The following example reflects write heavy activity but write-response time is very good, under 1ms, which shows disks are healthy and capable of handling high number of IOPS. #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.0 dm-0 530.8 16.0 8.3 0% 0.3 0.5 6124.0 5.1 30.7 0% 1.7 0.3 86% 2 6654.8 0.2 0.1 2.0 dm-0 633.1 16.1 10.0 0% 0.3 0.5 6173.0 6.1 36.6 0% 1.7 0.3 88% 1 6806.1 0.2 0.1 3.0 dm-0 731.8 16.0 11.5 0% 0.4 0.5 6064.2 5.8 34.1 0% 1.9 0.3 90% 2 6795.9 0.2 0.1 4.0 dm-0 711.1 16.0 11.1 0% 0.3 0.5 6448.5 5.4 34.3 0% 1.8 0.3 92% 2 7159.6 0.2 0.1 5.0 dm-0 700.1 16.0 10.9 0% 0.4 0.5 5689.4 5.8 32.2 0% 1.9 0.3 88% 0 6389.5 0.2 0.1 6.0 dm-0 774.1 16.0 12.1 0% 0.3 0.4 6409.5 5.5 34.2 0% 1.7 0.3 86% 0 7183.5 0.2 0.1 7.0 dm-0 849.6 16.0 13.3 0% 0.4 0.5 6151.2 5.4 32.3 0% 1.9 0.3 88% 3 7000.8 0.2 0.1 8.0 dm-0 664.2 16.0 10.4 0% 0.3 0.5 6349.2 5.7 35.1 0% 2.0 0.3 90% 2 7013.4 0.2 0.1 9.0 dm-0 951.0 16.0 14.9 0% 0.4 0.4 5807.0 5.3 29.9 0% 1.8 0.3 90% 3 6758.0 0.2 0.1 10.0 dm-0 742.0 16.0 11.6 0% 0.3 0.5 6461.1 5.1 32.2 0% 1.7 0.3 87% 1 7203.2 0.2 0.1 Let me show you a final example. I used –interval and –iterations parameters for pt-diskstats which tells us to wait for a number of seconds before printing the next disk stats and to limit the number of samples respectively. If you notice, you will see in 3rd iteration high latency (rd_rt, wr_rt) mostly for reads. Also, you can notice a high value for queue time (qtime) and service time (stime) where qtime is related to disk IO scheduler settings. For MySQL database servers we usually recommends noop/deadline instead of default cfq. $ pt-diskstats --interval=20 --iterations=3 #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 10.4 hda 11.7 4.0 0.0 0% 0.0 1.1 40.7 11.7 0.5 26% 0.1 2.1 10% 0 52.5 0.4 1.5 10.4 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.4 7.0 0.0 43% 0.0 0.1 0% 0 0.4 0.0 0.1 10.4 hda3 0.0 0.0 0.0 0% 0.0 0.0 0.4 107.0 0.0 96% 0.0 0.2 0% 0 0.4 0.0 0.2 10.4 hda5 0.0 0.0 0.0 0% 0.0 0.0 0.7 20.0 0.0 80% 0.0 0.3 0% 0 0.7 0.1 0.2 10.4 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.1 4.0 0.0 0% 0.0 4.0 0% 0 0.1 0.0 4.0 10.4 hda9 11.7 4.0 0.0 0% 0.0 1.1 39.2 10.7 0.4 3% 0.1 2.7 9% 0 50.9 0.5 1.8 10.4 drbd1 11.7 4.0 0.0 0% 0.0 1.1 39.1 10.7 0.4 0% 0.1 2.8 9% 0 50.8 0.5 1.7 20.0 hda 14.6 4.0 0.1 0% 0.0 1.4 39.5 12.3 0.5 26% 0.3 6.4 18% 0 54.1 2.6 2.7 20.0 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.4 9.1 0.0 56% 0.0 42.0 3% 0 0.4 0.0 42.0 20.0 hda3 0.0 0.0 0.0 0% 0.0 0.0 1.5 22.3 0.0 82% 0.0 1.5 0% 0 1.5 1.2 0.3 20.0 hda5 0.0 0.0 0.0 0% 0.0 0.0 1.1 18.9 0.0 79% 0.1 21.4 11% 0 1.1 0.1 21.3 20.0 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.8 10.4 0.0 62% 0.0 1.5 0% 0 0.8 1.3 0.2 20.0 hda9 14.6 4.0 0.1 0% 0.0 1.4 35.8 11.7 0.4 3% 0.2 4.9 18% 0 50.4 0.5 3.5 20.0 drbd1 14.6 4.0 0.1 0% 0.0 1.4 36.4 11.6 0.4 0% 0.2 5.1 17% 0 51.0 0.5 3.4 20.0 hda 0.9 4.0 0.0 0% 0.2 251.9 28.8 61.8 1.7 92% 4.5 13.1 31% 2 29.6 12.8 0.9 20.0 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.6 8.3 0.0 52% 0.1 98.2 6% 0 0.6 48.9 49.3 20.0 hda3 0.0 0.0 0.0 0% 0.0 0.0 2.0 23.2 0.0 83% 0.0 1.4 0% 0 2.0 1.2 0.3 20.0 hda5 0.0 0.0 0.0 0% 0.0 0.0 4.9 249.4 1.2 98% 4.0 13.2 9% 0 4.9 12.9 0.3 20.0 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 20.0 hda9 0.9 4.0 0.0 0% 0.2 251.9 21.3 24.2 0.5 32% 0.4 12.9 31% 2 22.2 10.2 9.7 20.0 drbd1 0.9 4.0 0.0 0% 0.2 251.9 30.6 17.0 0.5 0% 0.7 24.1 30% 5 31.4 21.0 9.5 You can see the busy column in pt-diskstats output which is the same as the util column in iostat – which points to utilization. Actually, pt-diskstats is quite similar to the iostat tool but pt-diskstats is more interactive and has more information. The busy percentage is only telling us for how long the IO subsystem was busy, but is not indicating capacity. So the only time you care about %busy is when it’s 100% and at the same time latency (await in iostat and rd_rt/wr_rt in diskstats output) increases over -say- 5ms. You can estimate capacity of your IO subsystem and then look at the IOPS being consumed (r/s + w/s columns). Also, the system can process more than one request in parallel (in case of RAID) so %busy can go beyond 100% in pt-diskstats output. If you need to check disk throughput, block device IOPS run the following to capture metrics from your IO subsystem and see if utilization matches other worrisome symptoms. I would suggest capturing disk stats during peak load. Output can be grouped by sample or by disk using the –group-by option. You can use the sysbench benchmark tool for this purpose to measure database server performance. You will find this link useful for sysbench tool details. $ pt-diskstats --group-by=all --iterations=7200 > /tmp/pt-diskstats.out; Conclusion: pt-diskstats is one of the finest tools from Percona Toolkit. By using this tool you can easily spot disk bottlenecks, measure the IO subsystem and identify how much IOPS your drive can handle (i.e. disk capacity).