in this article i explained how spring built-in transformers works for while transforming object message to map message. sometimes the messages need to be transformed before they can be consumed to achieve a business purpose. for example, a producer uses a plain xml as its payload to produce a message, while a consumer is interested in java object or types like plain text ,name-value pairs, or json model. spring integration provides endpoints such as service activators, channel adapters, message bridges, gateways, transformers, filters, and routers. in this example how transformers endpoint transform object message to map message. references: spring integration spring with jms spring with junit mockrunner sts high level view spring-mockrunner.xml in spring-mockrunner.xml file, i defined mockqueue, mockqueueconnectionfactory for inbound queue, and outbound queue for quick testing purpose. inboundqueue is where you will publish object message from objecttomaptransformertest.java class. outboundqueue where this queue expecting mapmessage type object and this queue is listing mapmessagelistener.java class. for more information mockrunner works please check my previous article mockrunner with spring jms . pom.xml 4.0.0 org.springframework.samples spring-int-jms-basic 0.0.1-snapshot 1.6 utf-8 utf-8 3.2.3.release 1.0.13 1.7.5 4.11 org.springframework spring-context ${spring-framework.version} org.springframework spring-tx ${spring-framework.version} org.springframework.integration spring-integration-core 2.2.4.release org.springframework.integration spring-integration-jmx 2.2.4.release org.springframework.integration spring-integration-jms 2.2.4.release org.slf4j slf4j-api ${slf4j.version} compile ch.qos.logback logback-classic ${logback.version} runtime org.springframework spring-test ${spring-framework.version} test junit junit ${junit.version} test com.mockrunner mockrunner-jms 1.0.3 javax.jms jms 1.1 org.codehaus.jackson jackson-mapper-asl 1.9.3 compile spring-int-jms.xml the endpoint is configured to connect to a jms server, fetch the messages,and publish them onto a local channel i.e inputchannel. where as connection-factory, and destination referred mockqueueconnectionfactory, and mockqueue(inboundqueue) beans from spring-mockrunner.xml file. inputchannel and outputchannel defined as queue channel objecttomaptransformer: object-to-map-transformer element that takes the payload from the input channel original here mockrunner-in-queue object message and emits a name-value paired map object onto the output channel i.e outputchannel and outboundjmsadapter bean fetch this message and publish to queue i.e mockrunner-out-queue. inboundjmsadapter : inbound-channel-adapter bean is responsible for receiving messages from a jms server here it is reading from mock queue name mockrunner-in-queue see objecttomaptransformertest.java class. outboundjmsadapter : outbound-channel-adapter bean is responsible to fetch messages from the channel i.e outputchannel and publish them to jms queue or topic. in this outbounjmsadapter reading message outputchannel as mapmessage and publish to outboundqueue(mockrunner-out-queue). mapmessagelistener.java package com.spijb.listener; import javax.jms.jmsexception; import javax.jms.mapmessage; import javax.jms.session; import org.slf4j.logger; import org.slf4j.loggerfactory; import org.springframework.jms.listener.sessionawaremessagelistener; public class mapmessagelistener implements sessionawaremessagelistener { private static final logger log = loggerfactory.getlogger(mapmessagelistener.class); @override public void onmessage(mapmessage message, session session) throws jmsexception { log.info("message received \r\n"+message); } } it is plain mapmessagelistener class to print received message from queue. department.java package com.spijb.domain; import java.io.serializable; public class department implements serializable{ private static final long serialversionuid = 1l; private final integer deptno; private final string name; private final string location; public department() { deptno=10; name="sales"; location="tx"; } public department(integer dno,string name,string loc) { this.deptno=dno; this.name=name; this.location=loc; } public integer getdeptno() { return deptno; } public string getname() { return name; } public string getlocation() { return location; } @override public string tostring() { return this.deptno+"-> "+this.name+"->"+this.location; } } domain object to send as a message, by default constructor assign deptno 10 , name as sales, location as tx also provide parameter constructor. spring junit class objecttomaptransformertest.java package com.spijb.invoker; import javax.jms.jmsexception; import javax.jms.message; import javax.jms.objectmessage; import javax.jms.session; import org.junit.test; import org.junit.runner.runwith; import org.springframework.beans.factory.annotation.autowired; import org.springframework.jms.core.jmstemplate; import org.springframework.jms.core.messagecreator; import org.springframework.test.context.contextconfiguration; import org.springframework.test.context.junit4.springjunit4classrunner; import com.mockrunner.mock.jms.mockqueue; import com.spijb.domain.department; @runwith(springjunit4classrunner.class) @contextconfiguration({"classpath:spring-mockrunner.xml","classpath:spring-int-jms.xml"}) public class objecttomaptransformertest { @autowired private jmstemplate jmstemplate; @autowired private mockqueue inboundqueue; @test public void shouldsendmessage() throws interruptedexception { final department defaultdepartment = new department(); jmstemplate.send(inboundqueue,new messagecreator() { @override public message createmessage(session session) throws jmsexception { objectmessage objectmessage = session.createobjectmessage(); objectmessage.setobject(defaultdepartment); return objectmessage; } }); thread.sleep(5000); } } spring with junit class where you can send message to inputchannel i.e inboundqueue using mockrunner. output : info: started inboundjmsadapter oct 06, 2014 1:24:25 pm org.springframework.integration.endpoint.abstractendpoint start info: started org.springframework.integration.config.consumerendpointfactorybean#1 13:24:26.882 [org.springframework.jms.listener.defaultmessagelistenercontainer#0-1] info c.spijb.listener.mapmessagelistener - message received com.mockrunner.mock.jms.mockmapmessage: {location=tx, name=sales, deptno=10} oct 06, 2014 1:24:30 pm org.springframework.context.support.abstractapplicationcontext doclose info: closing org.springframework.context.support.genericapplicationcontext@5840979b: startup date [mon oct 06 13:24:25 cdt 2014]; root of context hierarchy oct 06, 2014 1:24:30 pm org.springframework.context.support.defaultlifecycleprocessor$lifecyclegroup stop info: stopping beans in phase 2147483647 in the above highlighted one is output as map.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.

Sometimes one would want to handle strings which contain characters not included in UTF-8 or the default encoding (set in mule-deploy.properties). In these scenarios a different encoding which is capable of handling these characters (such as UTF-16 or UTF-32) can be used. To do so the default encoding can be easily changed by making a few modifications according to the type of transformer being used. Changing Encoding with the Datamapper When using the datamapper with data such as XML, one can easily choose the encoding by clicking on the settings button in the mapping (this should be set properly for both input and output) : Settings button datamapper A similar panel to the one below should appear: Changing Encoding when using “simple” transformers When using transformers such as object-to-string or byte-array-to-string, one would think that setting the “encoding” attribute on the transformer would do the trick: Unfortunately this doesn’t work, since the current Mule’s behaviour is to use this property just to set the MULE_ENCODING outbound property after the transformation is done. However, instead we should make sure that MULE_ENCODING outbound property is set properly before invoking the transformer. The transformer would then be able to transform the payload correctly for us.

eclipse has a great user interface (ui). but what if i want to do things from the command line, without the gui? for example to build one or more projects in the workspace without using the eclipse ui? with this, i can do automated check-outs and do automated builds. performed a command line project build with eclipse the solution to this: there is a command line version of eclipse which i can use to run eclipse in the command line version. inside the eclipse folder on windows, there is the eclipsec program which is the command-line version of eclipse: eclipsec program, a command line version of eclipse the options of this command line version (for eclipse kepler) are described here: http://help.eclipse.org/kepler/index.jsp?topic=%2forg.eclipse.platform.doc.isv%2freference%2fmisc%2fruntime-options.html for example eclipsec.exe -nosplash -application org.eclipse.cdt.managedbuilder.core.headlessbuild -data c:\my_wsp -build k64f will launch eclipse without splash screen ( -nosplash ), uses the - application command to load the managed make builder (which is used to build projects), with -data i specify the workspace to be used, and with the -build command it will the project k64f. more options and details are shown here: http://stackoverflow.com/questions/344797/build-several-cdt-c-projects-from-commandline and a very good article with additional background information how to use it with the gnu arm eclipse plubins can be found here: http://gnuarmeclipse.livius.net/blog/headless-builds/ happy headlessing :-)

Recently I ran into very interesting problem which I thought would take me just a couple of minutes to solve: protecting Apache CXF (current release 3.0.1)/ JAX-RS REST services with Spring Security (current stable version 3.2.5) in the application running inside embedded Jetty container (current release 9.2). At the end, it turns out to be very easy, once you understand how things work together and known subtle intrinsic details. This blog post will try to reveal that. Our example application is going to expose a simple JAX-RS / REST service to manage people. However, we do not want everyone to be allowed to do that so the HTTP basic authentication will be required in order to access our endpoint, deployed at http://localhost:8080/api/rest/people. Let us take a look on thePeopleRestService class: package com.example.rs; import javax.json.Json; import javax.json.JsonArray; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; @Path( "/people" ) public class PeopleRestService { @Produces( { "application/json" } ) @GET public JsonArray getPeople() { return Json.createArrayBuilder() .add( Json.createObjectBuilder() .add( "firstName", "Tom" ) .add( "lastName", "Tommyknocker" ) .add( "email", "[email protected]" ) ) .build(); } } As you can see in the snippet above, nothing is pointing out to the fact that this REST service is secured, just couple of familiar JAX-RS annotations. Now, let us declare the desired security configuration following excellent Spring Security documentation. There are many ways to configure Spring Security but we are going to show off two of them: using in-memory authentication and using user details service, both built on top of WebSecurityConfigurerAdapter. Let us start with in-memory authentication as it is the simplest one: package com.example.config; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Configuration; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; import org.springframework.security.config.http.SessionCreationPolicy; @Configuration @EnableWebSecurity @EnableGlobalMethodSecurity( securedEnabled = true ) public class InMemorySecurityConfig extends WebSecurityConfigurerAdapter { @Autowired public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception { auth.inMemoryAuthentication() .withUser( "user" ).password( "password" ).roles( "USER" ).and() .withUser( "admin" ).password( "password" ).roles( "USER", "ADMIN" ); } @Override protected void configure( HttpSecurity http ) throws Exception { http.httpBasic().and() .sessionManagement().sessionCreationPolicy( SessionCreationPolicy.STATELESS ).and() .authorizeRequests().antMatchers("/**").hasRole( "USER" ); } } In the snippet above there two users defined: user with the role USER and admin with the roles USER,ADMIN. We also protecting all URLs (/**) by setting authorization policy to allow access only users with roleUSER. Being just a part of the application configuration, let us plug it into the AppConfig class using @Importannotation. package com.example.config; import java.util.Arrays; import javax.ws.rs.ext.RuntimeDelegate; import org.apache.cxf.bus.spring.SpringBus; import org.apache.cxf.endpoint.Server; import org.apache.cxf.jaxrs.JAXRSServerFactoryBean; import org.apache.cxf.jaxrs.provider.jsrjsonp.JsrJsonpProvider; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.DependsOn; import org.springframework.context.annotation.Import; import com.example.rs.JaxRsApiApplication; import com.example.rs.PeopleRestService; @Configuration @Import( InMemorySecurityConfig.class ) public class AppConfig { @Bean( destroyMethod = "shutdown" ) public SpringBus cxf() { return new SpringBus(); } @Bean @DependsOn ( "cxf" ) public Server jaxRsServer() { JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint( jaxRsApiApplication(), JAXRSServerFactoryBean.class ); factory.setServiceBeans( Arrays.< Object >asList( peopleRestService() ) ); factory.setAddress( factory.getAddress() ); factory.setProviders( Arrays.< Object >asList( new JsrJsonpProvider() ) ); return factory.create(); } @Bean public JaxRsApiApplication jaxRsApiApplication() { return new JaxRsApiApplication(); } @Bean public PeopleRestService peopleRestService() { return new PeopleRestService(); } } At this point we have all the pieces except the most interesting one: the code which runs embedded Jettyinstance and creates proper servlet mappings, listeners, passing down the configuration we have created. package com.example; import java.util.EnumSet; import javax.servlet.DispatcherType; import org.apache.cxf.transport.servlet.CXFServlet; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.servlet.FilterHolder; import org.eclipse.jetty.servlet.ServletContextHandler; import org.eclipse.jetty.servlet.ServletHolder; import org.springframework.web.context.ContextLoaderListener; import org.springframework.web.context.support.AnnotationConfigWebApplicationContext; import org.springframework.web.filter.DelegatingFilterProxy; import com.example.config.AppConfig; public class Starter { public static void main( final String[] args ) throws Exception { Server server = new Server( 8080 ); // Register and map the dispatcher servlet final ServletHolder servletHolder = new ServletHolder( new CXFServlet() ); final ServletContextHandler context = new ServletContextHandler(); context.setContextPath( "/" ); context.addServlet( servletHolder, "/rest/*" ); context.addEventListener( new ContextLoaderListener() ); context.setInitParameter( "contextClass", AnnotationConfigWebApplicationContext.class.getName() ); context.setInitParameter( "contextConfigLocation", AppConfig.class.getName() ); // Add Spring Security Filter by the name context.addFilter( new FilterHolder( new DelegatingFilterProxy( "springSecurityFilterChain" ) ), "/*", EnumSet.allOf( DispatcherType.class ) ); server.setHandler( context ); server.start(); server.join(); } } Most of the code does not require any explanation except the the filter part. This is what I meant by subtle intrinsic detail: the DelegatingFilterProxy should be configured with the filter name which must be exactlyspringSecurityFilterChain, as Spring Security names it. With that, the security rules we have configured are going to apply to any JAX-RS service call (the security filter is executed before the Apache CXF servlet), requiring the full authentication. Let us quickly check that by building and running the project: mvn clean package java -jar target/jax-rs-2.0-spring-security-0.0.1-SNAPSHOT.jar Issuing the HTTP GET call without providing username and password does not succeed and returns HTTP status code 401. > curl -i http://localhost:8080/rest/api/people HTTP/1.1 401 Full authentication is required to access this resource WWW-Authenticate: Basic realm="Realm" Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html; charset=ISO-8859-1 Content-Length: 339 Server: Jetty(9.2.2.v20140723) The same HTTP GET call with username and password provided returns successful response (with some JSON generated by the server). > curl -i -u user:password http://localhost:8080/rest/api/people HTTP/1.1 200 OK Date: Sun, 28 Sep 2014 20:07:35 GMT Content-Type: application/json Content-Length: 65 Server: Jetty(9.2.2.v20140723) [{"firstName":"Tom","lastName":"Tommyknocker","email":"[email protected]"}] Excellent, it works like a charm! Turns out, it is really very easy. Also, as it was mentioned before, the in-memory authentication could be replaced with user details service, here is an example how it could be done: package com.example.config; import java.util.Arrays; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; import org.springframework.security.config.http.SessionCreationPolicy; import org.springframework.security.core.authority.SimpleGrantedAuthority; import org.springframework.security.core.userdetails.User; import org.springframework.security.core.userdetails.UserDetails; import org.springframework.security.core.userdetails.UserDetailsService; import org.springframework.security.core.userdetails.UsernameNotFoundException; @Configuration @EnableWebSecurity @EnableGlobalMethodSecurity(securedEnabled = true) public class UserDetailsSecurityConfig extends WebSecurityConfigurerAdapter { @Autowired public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception { auth.userDetailsService( userDetailsService() ); } @Bean public UserDetailsService userDetailsService() { return new UserDetailsService() { @Override public UserDetails loadUserByUsername( final String username ) throws UsernameNotFoundException { if( username.equals( "admin" ) ) { return new User( username, "password", true, true, true, true, Arrays.asList( new SimpleGrantedAuthority( "ROLE_USER" ), new SimpleGrantedAuthority( "ROLE_ADMIN" ) ) ); } else if ( username.equals( "user" ) ) { return new User( username, "password", true, true, true, true, Arrays.asList( new SimpleGrantedAuthority( "ROLE_USER" ) ) ); } return null; } }; } @Override protected void configure( HttpSecurity http ) throws Exception { http .httpBasic().and() .sessionManagement().sessionCreationPolicy( SessionCreationPolicy.STATELESS ).and() .authorizeRequests().antMatchers("/**").hasRole( "USER" ); } } Replacing the @Import( InMemorySecurityConfig.class ) with @Import( UserDetailsSecurityConfig.class ) in the AppConfig class leads to the same results, as both security configurations define the identical sets of users and their roles. I hope, this blog post will save you some time and gives a good starting point, as Apache CXF and Spring Security are getting along very well under Jetty umbrella! The complete source code is available on GitHub.

Memoization is a technique used to speed up functions. Memoization may be done manually. It may also be done automatically. We can find many examples of automatic memoization on Internet. In this article, I will show how Java 8 makes it very easy to memoize functions. What is memoization Memoization consist in caching the results of functions in order to speed them up when they are called several times with the same argument. The first call implies computing and storing the result in memory before returning it. Subsequent calls with the same parameter imply only fetching the previously stored value and returning it. How to apply memoization Memoization may be applied manually by hard coding it in every function that may benefit from it. If it takes a long time to compute the return value, memoization will speed up the program. For functions that take less time to evaluate than fetching the previously stored value from memory, memoization is clearly not a good option. Hard coding memoization by hand in each function is not a good option neither because it is repeating the same principle again and again. That is why automatic memoization is desirable. What to memoize Memoization applies to functions. Prior to Java 8, Java had no functions. However, it was perfectly possible to define some. Furthermore, we used to create “functional” methods, that is methods taking an argument and returning a value based only upon this argument. These kind of method may benefit from memoization. By the way, there is a match between functional methods and functions. For example, the following method: Integer doubleValue(Integer x) { return x * 2; } corresponds to: Integer doubleValue(Integer x) { if (cache.containsKey(x)) { return cache.get(x); } else { Integer result = x * 2; cache.put(x, result) ; return result; } } In Java 8, we can make this much cleaner: Map cache = new ConcurrentHashMap<>(); Integer doubleValue(Integer x) { return cache.computeIfAbsent(x, y -> y * 2); } Our function may be modified to use the same technique: Function doubleValue = x -> cache.computeIfAbsent(x, y -> y * 2); This is pretty simple, but it has two main drawbacks: We have to repeat this modification for all functions. The map we use is exposed and could potentially be modified by another thread having nothing to do with the function. The second problem is quite easy to address. We may put the method or the function in a separate class, including the map, with private access. For example, for the method case: public class Doubler { private static Map cache = new ConcurrentHashMap<>(); public static Integer doubleValue(Integer x) { return cache.computeIfAbsent(x, y -> y * 2); } } We may then instantiate that class and use it each time we want to compute a value: Integer y = Doubler.doubleValue(x); With this solution, the map is no longer accessible from outside. We can't do the same for functions because functions are anonymous classes and such classes may not have static members. One possibility would be to pass the map to the function as an additional argument. This may be done through a closure: class Doubler { private static Map cache = new ConcurrentHashMap<>(); public static Function doubleValue = x->cache.computeIfAbsent(x, y -> y * 2); } We can use this function as follows: Integer y = Doubler.doubleValue.apply(x); This gives no advantage compared to the “method” solution. However, we may also use this function in more idiomatic examples, such as: IntStream.range(1, 10).boxed().map(Doubler.doubleValue); This is equivalent to using the method version with the following syntax: IntStream.range(1, 10).boxed().map(ThisClass::doubleValue); The main problem is that while solving the second issue, we have made the first one more acute, which make automatic memoization more desirable. Automatic memoization: requirements What we need is a way to do the following: Function f = x -> x * 2; Function g = Memoizer.memoize(f); so that we may use the memoized function as a drop in replacement for the original one. All values returned by function g will be calculated through the original function f the first time, and returned from the cache for all subsequent accesses. By contrast, if we create a third function: Function f = x -> x * 2; Function g = Memoizer.memoize(f); Function h = Memoizer.memoize(f); the values cached by g will not be returned by h. In other words, g and h will use separate caches. Implementation The Memoizer class is quite simple: public class Memoizer { private final Map cache = new ConcurrentHashMap<>(); private Memoizer() {} private Function doMemoize(final Function function) { return input -> cache.computeIfAbsent(input, function::apply); } public static Function memoize(final Function function) { return new Memoizer().doMemoize(function); } } Using this class is also extremely simple: Integer longCalculation(Integer x) { try { Thread.sleep(1_000); } catch (InterruptedException ignored) { } return x * 2; } Function f = this::longCalculation; Function g = Memoizer.memoize(f); public void automaticMemoizationExample() { long startTime = System.currentTimeMillis(); Integer result1 = g.apply(1); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = g.apply(1); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } Running the automaticMemoizationExample method will produce the following result: 2 2 1000 0 We can now make memoized function out of ordinary ones by just calling a single method! What about functions with several arguments? Short answer: nothing. There are no such things in this world as functions with several arguments. Functions are applications of one set (the source set) to another set (the target set). So, they simply can't have several arguments. But this does not solve our problem. What is the functional equivalent to a method with several arguments? Long answer: what people generally consider as functions with several arguments are in fact either: Functions of tuples Function returning functions returning functions … returning a result In either cases, we are only concerned with functions of one argument, so we can easily use our Memoizer class. Using functions of tuples would probably be the simplest choice... if Java had tuples! We could of course write tuples. But to store tuples in maps, we would have to implement equals and hashcode for them, plus we would have to define tuples for two elements (pairs), tuple for three elements, and so on. Who knows where to stop? The second option is much easier. It is based upon currying, which means applying each argument one after the other instead of applying them as a whole (the tuple). Currying a function is very easy. The only problem, in Java 8, is that writing the types is really cumbersome. Currying a “function of two arguments” (in fact a function of a pair) is easy once you master the type. Java has in fact a shortcut for functions of tuple2 which is called BiFunction. We will take this as an example. The two following functions are equivalent (from the result point of view): BiFunction h = (x, y) -> x + y; Function> hc = x -> y -> x + y; Not considering the types, there are very little differences. In the first case, the two arguments are put between parentheses, separated by a comma, which is, by the way, how tuples are written in most languages which have them! Remove the parentheses and separate the arguments with an arrow and you get the curried version. We can only regret that we have to write the type as: Function> when other languages use a simplified syntax such as: Integer -> Integer -> Integer From this, it is easy to memoized this curried version, although we cant use the same simple form as previously. We have to memoize each function: Function> mhc = Memoizer.memoize(x -> Memoizer.memoize(y -> x + y)); Same thing for a function of (a tuple of) 3 arguments (which by the way has no equivalent in Java): Function>> f3 = x -> y -> z -> x + y - z; Function>> f3m = Memoizer.memoize(x -> Memoizer.memoize(y -> Memoizer.memoize(z -> x + y – z)); Here is an example of using this memoized function “of three arguments”: Function>> f3 = x -> y -> z -> longCalculation(x) + longCalculation(y) - longCalculation(z); Function>> f3m = Memoizer.memoize(x -> Memoizer.memoize(y -> Memoizer.memoize(z -> longCalculation(x) + longCalculation(y) - longCalculation(z)))); public void automaticMemoizationExample2() { long startTime = System.currentTimeMillis(); Integer result1 = f3m.apply(2).apply(3).apply(4); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = f3m.apply(2).apply(3).apply(4); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } This example produces the following output: 2 2 3002 0 showing that the first access to method longCalculation has taken 3000 milliseconds and the second has return immediately. On the other hand, using a function of tuple may seem easier once you have the Tuple class defined. Here is an example of Tuple3: public class Tuple3 { public final T _1; public final U _2; public final V _3; public Tuple3(T t, U u, V v) { _1 = Objects.requireNonNull(t); _2 = Objects.requireNonNull(u); _3 = Objects.requireNonNull(v); } @Override public boolean equals(Object o) { if (!(o instanceof Tuple3)) return false; else { Tuple3 that = (Tuple3) o; return _1.equals(that._1) && _2.equals(that._2) && _3.equals(that._3); } } @Override public int hashCode() { return _1.hashCode() + _2.hashCode() + _3.hashCode(); } } Using this class, we may rewrite the previous example as: Function, Integer> ft = x -> longCalculation(x._1) + longCalculation(x._2) - longCalculation(x._3); Function, Integer> ftm = Memoizer.memoize(ft); public void automaticMemoizationExample3() { long startTime = System.currentTimeMillis(); Integer result1 = ftm.apply(new Tuple3<>(2, 3, 4)); long time1 = System.currentTimeMillis() - startTime; startTime = System.currentTimeMillis(); Integer result2 = ftm.apply(new Tuple3<>(2, 3, 4)); long time2 = System.currentTimeMillis() - startTime; System.out.println(result1); System.out.println(result2); System.out.println(time1); System.out.println(time2); } Conclusion Memoizing is about maintaining state between function calls. A memoized function is a function which behavior is dependent upon the current state. However, it will always return the same value for the same argument. Only the time needed to return the value will be different. So the memoized function is still a pure function if the original function is pure. However, there is a kind of function that may pose a problem: recursive functions that call themselves several times with the same argument may not be memoized this way. This will be addressed in a next article.

this article describes steps required to configure your eclipse for sonarqube such that developers are not required to leave the eclipse ide to manage their source code quality. please feel free to comment/suggest if i missed to mention one or more important points. also, sorry for the typos. following are the key points described later in this article: installing & configuraing sonarqube in eclipse analyzing code using sonarqube installing & configuraing sonarqube in eclipse follow the steps given on following pages to install and configure sonarqube in your eclipse ide. the page on installing sonarqube in eclipse consists of steps required to install sonarqube in the ide. once downloaded, it would prompt to restart the eclipse ide. for proceeding ahead, you must restart the eclipse ide. the page on configuring sonarqube in eclipse consists of steps required to configure eclipse for sonarqube. the key thing to note is the fact that as you click on preferences > sonarqube > servers , you see the default server entry for http://localhost:9000. initially, as per instruction page, seeing the default entry, i went ahead to next step for linking the project to sonarqube server and run the analysis. however, it throw the error such as unknowb version of sonarqube server . after a little research, i figured out that the error was thrown as the sonarqube server was not reachable although i saw it configured as http://localhost:9000. the solution is to go to preferences > sonarqube > servers , click “edit” button and you would see sonarqube server url, username and password. the key is to give username and password and run the analysis once again. analyzing code using sonarqube once you are done with installation and configuration of sonarqube server in eclipse ide, for code analysis, all you need is right click on your project and click sonarqube > analyze and that is it. the violations would appear in sonarqube analysis as shown in the screenshot below. sonarqube analysis in eclipse however, do note that these issues may not appear if you try to access on the browser. for that to happen, you need to once again run “sonar-runner” from the project root.

In my last post, I explained how to do code reviews properly. I recommended taking advantage of static analysis tools like Findbugs, PMD, Klocwork or Fortify to check for common mistakes and bad code before passing the code on to a reviewer, to make the reviewer’s job easier and reviews more effective. Some readers asked whether static analysis tools can be used instead of manual code reviews. Manual code reviews add delays and costs to development, while static analysis tools keep getting better, faster, and more accurate. So can you automate code reviews, in the same way that many teams automate functional testing? Do you need to do manual reviews too, or can you rely on technology to do the job for you? Let’s start by understanding what static analysis bug checking tools are good at, and what they aren’t. What static analysis tools can do – and what they can’t do In this article, Paul Anderson at GrammaTech does a good job of explaining how static analysis bug finding works, the trade-offs between recall (finding all of the real problems), precision (minimizing false positives) and speed, and the practical limitations of using static analysis tools for finding bugs. Static analysis tools are very good at catching certain kinds of mistakes, including memory corruption and buffer overflows (for C/C++), memory leaks, illegal and unsafe operations, null pointers, infinite loops, incomplete code, redundant code and dead code. A static analysis tool knows if you are calling a library incorrectly (as long as it recognizes the function), if you are using the language incorrectly (things that a compiler could find but doesn’t) or inconsistently (indicating that the programmer may have misunderstood something). And static analysis tools can identify code with maintainability problems, code that doesn't follow good practice or standards, is complex or badly structured and a good candidate for refactoring. But these tools can’t tell you when you have got the requirements wrong, or when you have forgotten something or missed something important – because the tool doesn't know what the code is supposed to do. A tool can find common off-by-one mistakes and some endless loops, but it won’t catch application logic mistakes like sorting in descending order instead of ascending order, or dividing when you meant to multiply, referring to buyer when it should have been seller, or lessee instead of lessor. These are mistakes that aren't going to be caught in unit testing either, since the same person who wrote the code wrote the tests, and will make the same mistakes. Tools can’t find missing functions or unimplemented features or checks that should have been made but weren't. They can’t find mistakes or holes in workflows. Or oversights in auditing or logging. Or debugging code left in by accident. Static analysis tools may be able to find some backdoors or trapdoors – simple ones at least. And they might find some concurrency problems – deadlocks, races and mistakes or inconsistencies in locking. But they will miss a lot of them too. Static analysis tools like Findbugs can do security checks for you: unsafe calls and operations, use of weak encryption algorithms and weak random numbers, using hard-coded passwords, and at least some cases of XSS, CSRF, and simple SQL injection. More advanced commercial tools that do inter-procedural and data flow analysis (looking at the sources, sinks and paths between) can find other bugs including injection problems that are difficult and time-consuming to trace by hand. But a tool can’t tell you that you forgot to encrypt an important piece of data, or that you shouldn't be storing some data in the first place. It can’t find logic bugs in critical security features, if sensitive information could be leaked, when you got an access control check wrong, or if the code could fail open instead of closed. And using one static analysis tool on its own to check code may not be enough. Evaluations of static analysis tools, such as NIST's SAMATE project (a series of comparative studies, where many tools are run against the same code), show almost no overlap between the problems found by different tools (outside of a few common areas like buffer errors) even when the tools are supposed to be doing the same kinds of checks. Which means that to get the most out of static analysis, you will need to run two or more tools against the same code (which is what SonarQube, for example, which integrates its own static analysis results with other tools, including popular free tools, does for you). If you’re paying for commercial tools, this could get very expensive fast. Tools vs. Manual Reviews Tools can find cases of bad coding or bad typing – but not bad thinking. These are problems that you will have to find through manual reviews. A 2005 study Comparing Bug Finding Tools with Reviews and Tests used Open Source bug finding tools (including Findbugs and PMD) on 5 different code bases, comparing what the tools found to what was found through code reviews and functional testing. Static analysis tools found only a small subset of the bugs found in manual reviews, although the tools were more consistent – manual reviewers missed a few cases that the tools picked up. Just like manual reviews, the tools found more problems with maintainability than real defects (this is partly because one of the tools evaluated – PMD – focuses on code structure and best practices). Testing (black box – including equivalence and boundary testing – and white box functional testing and unit testing) found fewer bugs than reviews. But different bugs. There was no overlap at all between bugs found in testing and the bugs found by the static analysis tools. Finding problems that could happen - or do happen Static analysis tools are good at finding problems that “could happen”, but not necessarily problems that “do happen”. Researchers at Colorado State University ran static analysis tools against several releases of different Open Source projects, and compared what the tools found against the changes and fixes that developers actually made over a period of a few years – to see whether the tools could correctly predict the fixes that needed to be made and what code needed to be refactored. The tools reported hundreds of problems in the code, but found very few of the serious problems that developers ended up fixing. One simple tool (Jlint) did not find anything that was actually fixed or cleaned up by developers. Of 112 serious bugs that were fixed in one project, only 3 were also found by static analysis tools. In another project, only 4 of 136 bugs that were actually reported and fixed were found by the tools. Many of the bugs that developers did fix were problems like null pointers and incorrect string operations – problems that static analysis tools should be good at catching, but didn’t. The tools did a much better job of predicting what code should be refactored: developers ended up refactoring and cleaning up more than 70% of the code structure and code clarity issues that the tools reported (PMD, a free code checking tool, was especially good for this). Ericsson evaluated different commercial static analysis tools against large, well-tested, mature applications. On one C application, a commercial tool found 40 defects – nothing that could cause a crash, but still problems that needed to be fixed. On another large C code base, 1% of the tool’s findings turned out to be bugs serious enough to fix. On the third project, they ran 2 commercial tools against an old version of a C system with known memory leaks. One tool found 32 bugs, another 16: only 3 of the bugs were found by both tools. Surprisingly, neither tool found the already known memory leaks – all of the bugs found were new ones. And on a Java system with known bugs they tried 3 different tools. None of the tools found any of the known bugs, but one of the tools found 19 new bugs that the team agreed to fix. Ericsson’s experience is that static analysis tools find bugs that are extremely difficult to find otherwise. But it’s rare to find stop-the-world bugs – especially in production code – using static analysis. This is backed up by another study on the use of static analysis (Findbugs) at Google and on the Sun JDK 1.6.0. Using the tool, engineers found a lot of bugs that were real, but not worth the cost of fixing: deliberate errors, masked errors, infeasible situations, code that was already doomed, errors in test code or logging code, errors in old code that was “going away soon” or other relatively unimportant cases. Only around 10% of medium and high priority correctness errors found by the tool were real bugs that absolutely needed to be fixed. The Case for Security So far we've mostly looked at static analysis checking for run-time correctness and general code quality, not security. Although security builds on code quality – vulnerabilities are just bugs that hackers look for and exploit – checking code for correctness and clarity isn’t enough for a secure app. A lot of investment in static analysis technology over the past 5-10 years has been in finding security problems in code, such as common problems listed in OWASP’s Top 10 or the SANS/CWE Top 25 Most Dangerous Software Errors. A couple of studies have looked at the effectiveness of static analysis tools compared to manual reviews in finding security vulnerabilities. The first study was on a large application that had 15 known security vulnerabilities found through a structured manual assessment done by security experts. Two different commercial static analysis tools were run across the code. The tools together found less than half of the known security bugs – only the simplest ones, the bugs that didn't require a deep understanding of the code or the design. And of course the tools reported thousands of other issues that needed to be reviewed and qualified or thrown away as false positives. These other issues including some run-time correctness problems, null pointers and resource leaks, and code quality findings (dead code, unused variables), but no other real security vulnerabilities beyond those already found by the manual security review. But this assumes that you have a security expert around to review the code. To find security vulnerabilities, a reviewer needs to understand the code (the language and the frameworks), and they also need to understand what kind of security problems to look for. Another study shows how difficult this is. Thirty developers were hired to do independent security code reviews of a small web app (some security experts, others web developers). They were not allowed to use static analysis tools. The app had 6 known vulnerabilities. 20% of the reviewers did not find any of the known bugs. None of the reviewers found all of the known bugs, although several found a new XSS vulnerability that the researchers hadn’t known about. On average, 10 reviewers would have had only an 80% chance of finding all of the security bugs. And, not Or Static analysis tools are especially useful for developers working in unsafe languages like C/C++ (where there is a wide choice of tools to find common mistakes) or dynamically typed scripting languages like Javascript or PHP (where unfortunately the tools aren't that good), and for teams starting off learning a new language and framework. Using static analysis is (or should be) a requirement in highly regulated, safety critical environments like medical devices and avionics. And until more developers get more training and understand more about how to write secure software, we will all need to lean on static analysis (and dynamic analysis) security testing tools to catch vulnerabilities. But static analysis isn't a substitute for code reviews. Yes, code reviews take extra time and add costs to development, even if you are smart about how you do them – and being smart includes running static analysis checks before you do reviews. If you want to move fast and write good, high-quality and secure code, you still have to do reviews.You can’t rely on static analysis alone.

the gnu arm eclipse plugins from liviu already offer several built-in actions which can be performed at the end of a build: creating flash image , create listing file and printing the code and data size: gnu arm eclipse extra post build steps but what if i need different things, or even more things? post-build steps for this there is the ‘post-build steps’ settings i can use: that command is executed at the end of the build: print size post-build step :!: the post build step is only executed if sources files have been compiled and linked. if you want to enforce that there is always a ‘true’ post build, then you need to delete some files in the pre-build step to enforce a compilation and a link phase. multiple post-build steps but what i need more than one action in the post-build step? i could call a batch or script file, but this is probably an overkill in too many cases, and adds a dependency to that script file. a better approach is to directly execute multiple commands as post-build step. unfortunately, the documentation found about the post-build step with a web-search is misleading (e.g. in the eclipse luna documentation ): “command: specifies one or more commands to execute immediately after the execution of the build. use semicolons to separate multiple commands.” unfortunately, semicolons is plain wrong (at least did not work for me) :-(. the solution is to use ‘ & ‘ (ampersand) to separate multiple commands on windows : :idea: on linux, use the ‘;’ to separate commands as noted in the documentation/help, and use ‘&’ on windows. unfortunately, this makes project not cross-platform. multiple post-build commands and this works for me, at least under windows 7 :-).

CodePro Analytix is the premier Java software testing tool for Eclipse developers.

this tip sounds very basic, but still: i get asked about this about once a week. so it must be something non-obvious in eclipse then ;-): how to add existing files to an eclipse project. as with many things in eclipse, there is not a single way to do something. there are two basic ways to do this: import drag & drop copy & paste the first is the ‘official’ way in eclipse, the other two are much faster and easier :-). 1. import to import one or multiple files, select the folder/project where i want to add the files, then use the menu file > import : menu file import alternatively, i can use the context menu: import context menu then use general > file system : import from file system in the next dialog, i can browse for the files and select them to be added: importing files from filesystem 2. drag & drop while the ‘import’ method is fine, i rarely use it: too many clicks! a easier way (at least under windows) is to drag & drop the files from the windows explorer: drag and drop to add files 3. copy & paste the third way is even easier: select the files i want to add in the explorer/file manager, and copy them ( ctrl-c on windows) paste them in the project/folder in eclipse ( ctrl-v on windows) that last method does not advanced options like if i want to link to the files, see “ link to files and folders in eclipse “, but is definitely the fastest and easiest way. happy adding :-)

Learn how to resolve the ''failure to transfer'' error encountered in Maven in this quick tutorial.

Introduction In this post I would like to put my two cents in and talk about RESTful web services. First of all I don't intend to discuss the history of RESTful services. Neither this is a tutorial about implementing RESTful web services. What I'm trying to do is discuss how we, as developers, should think about a RESTful web service and do better RESTful APIs. First of all I want to thank my friend, Filipe (@filaruina) whom worked with me at Cortex Intelligence. We talked a lot about REST and he helped me to open my eyes to a lot of aspects that I hadn't yet. There and back again This month I started working in a new company. One of my first tasks was to finish the integration of two systems which the communication was done via SOAP. When I started reading the source code I saw the common mistake some developers make (including me). The first thing we do is replace all the SOAP calls by HTTP requests. After that we use the status code to inform the result of our method. We return 200 if the method executes without any errors, and 500 when there are any error in the processing. Last, but not least, we spit the log of the operation in the response body, so the client knows what happened in the method execution and there you are! We have REST... Please, stop! RESTful for the win I'm sure that you have read a lot of tutorials about REST, status codes, http verbs and everything else. But it's time to stop memorizing REST and start thinking REST. REST is not a protocol (HTTP is). REST is not a specific technology communication (RMI is). REST is an abstraction over a protocol, it's a modern web architecture, thought to replace the specific technology communication protocols and define a way of integrating systems with very loose coupling and much more versatility. Much more than that, REST is a communication way where the communicators (the systems) are not being orchestrated by a conductor. It is more like a dance, where the dancers know what they should do to make the dance works and be beautiful. If you are using REST just as a communication protocol, stop calling it REST and start calling it "HTTP Interface", it's better. Hyping Hypermedia When we talk about REST, we are talking about Hypermedia. Period. I totally agree with Steve Klabnik in his post Rest is Over. Steve says that RESTful APIs have evolved to Hypermedia APIs. As it should be from the begining. REST isn't only about communication by HTTP requests, correct HTTP verbs or well-chosen status codes, the real thing is Hypermedia. Formally we have four constraints in REST: Identification of resources; Manipulation of resources through representations; Self-descriptive messages; Hypermedia as the engine of application state; In my experience I've seen that constraints one and two are easier to understand and use. Constraint three is often misunderstood and not well implemented. Constraint four, almost always, is left behind. But what does "Hypermedia as the engine of application state" mean? Well, it means that you can look to your application as a finite-state machine, where the actual node represents the current resource being "viewed" by the user, and the transitions are like instructions to the user on his next steps. Let's take a look at an example. Imagine that we have an online payment system, responsible for managing transactions between sellers and buyers from a lot of other services (have you heard about any services like this?). Imagine that Bob wants to buy a bycicle from Bikers World website. The first thing that the website should do is create a payment resource using the RESTful API of our payment service. It can be something like: POST /payments Host: bikersworld.com Accept: application/json Request-Body: { "transactionId": "hyx48yu9pe", "value": "150.0" } And then our server responds something like this: HTTP/1.1 204 CREATED Content-Type: application/json Response-Body: { "transactionId": "hyx48yu9pe", "value": "150.0", "status": "pending", "links": [ { "rel": "self", "method": "get", "href": "http://onlinepayments.com/payments/5" }, { "rel": "confirm", "method": "put", "href": "http://onlinepayments.com/payments/5/confirm" }, { "rel": "cancel", "method": "delete", "href": "http://onlinepayments.com/payments/5" } ] } Did you notice the "links" key in the json that the API answered? It informs us about the other possible states we can reach from where we are. Would you need to check the API docs to know what you should do next? What would you do if you wanted to cancel the order? Or to retrieve the order info? As you can see, the server request is enough to inform the user about the next states available to the resource. After creating a payment, it's easy to see that there is a requirement to confirm it, and you know all of this without reading any documentation. Another thing that can make our API even better is to use the OPTION verb to serve to our clients the options (duh!) that they have for a resource. Imagine an JSON restul API that for a OPTION request like this: OPTION /payments Host: bikersworld.com Accept: application/json It returns a JSON documentation like that: HTTP/1.1 200 OK Content-Type: application/json Response-Body: { "POST": { "description": "Create an payment", "parameters": { "transactionId": { "type": "string", "description": "The transaction id associated to the payment", "required": true }, "value": { "type": "double", "description": "The value of the payment being created" "required": true } }, "example": { "transactionId": "hyx48yu9pe", "value": "150.0" } }, ... //more useful information } I'm not saying that it is easy to apply this concepts and sometimes your domain can't be mapped very well to a state-driven model and your entities don't have a well defined lifecycle. In this case you should try to do your best to fit these concepts in your API. But don'let hypermedia overwhelm you. You can develop your API incrementally. In this post, Matt Cottingham showed a picture that I consider to be an excelent representation of the steps taken by an API during it's development. As you can see, hypermedia controls are labeled as the top level feature for a restful API, but this doesn't mean that you can't design a restful (not THAT restful) API without it. You can, but don't forget this image. As any software, our API definitions can evolve and be more restful day by day. Useful resources: Principled Design of the Modern Web Architecture, article written by Roy T. Fielding and Richard N. Taylor; Architectural Styles and the Design of Network-based Software Architectures, PhD thesis written by Roy T. Fielding; PUT or POST: The REST of the Story, blog post by John Calcote; A Short Explanation of Hypermedia Controls in RESTful Services, blog post by Matt Cottingham; REST APIs must be hypertext-driven blog post by Roy T. Fielding; Why HATEOAS, presentation by Wayne Lee; The RESTful CookBook, created by Joshua Thijssen; The HTTP OPTIONS method and potential for self-describing RESTful APIs, blog post by Zac Stewart; Netflix Rest API documentation, an awesome example of restful API. (this post was originally published in my blog)

In this article we wanted to share another configuration property that can provide surprising help when setting up your JBoss BPM Suite. Previously we outlined a basic set of configuration properties to provide you with a few tricks when installing your own JBoss BRMS or JBoss BPM Suite products. As the JBoss BPM Suite is a super set, including full JBoss BRMS functionality, the rest of this article will refer only to JBoss BPM Suite but apply to both products. In this article we will show you how to modify your JBoss EAP container configuration to point the products at a custom deployment repository by adjusting a single configuration property. Maven repository The default setup is that the products will look for your maven setting in the default settings.xml as found set in theM2_HOME variable or in the users home directory at .m2/settings.xml. The following system property can be added to JBoss EAP standalone.xml configuration file to point to any file containing your custom settings. kie.maven.settings.custom Location of the maven configuration file where it can find it's settings. Default: the M2_HOME/conf/settings.xml or users home directory .m2/settings.xml Example usage in JBoss EAP When initially setting up the product for use on JBoss EAP containers, one can adjust configuration with the help of system properties. Below we show how to configure an installation to point to our custom maven deployment repository by using a custom settings file we will call bpmsuite-settings.xml We hope this helps you with configuring your own custom deployment repositories and enables you to tie into existing continuous integration infrastructures that might exist in your organization.

Originally Written by Muhammad Irfan Here on the Percona Support team we often ask customers to retrieve disk stats to monitor disk IO and to measure block devices iops and latency. There are a number of tools available to monitor IO on Linux. iostat is one of the popular tools and Percona Toolkit, which is free, contains the pt-diskstats tool for this purpose. The pt-diskstats tool is similar to iostat but it’s more interactive and contains extended information. pt-diskstats reports current disk activity and shows the statistics for the last second (which by default is 1 second) and will continue until interrupted. The pt-diskstats tool collects samples of /proc/diskstats. In this post, I will share some examples about how to monitor and check to see if the IO subsystem is performing properly or if any disks are a limiting factor – all this by using the pt-diskstats tool. pt-diskstats output consists on number of columns and in order to interpret pt-diskstats output we need to know what each column represents. rd_s tells about number of reads per second while wr_s represents number of writes per second. rd_rt and wr_rt shows average response time in milliseconds for reads & writes respectively, which is similar to iostat tool output await column but pt-diskstats shows individual response time for reads and writes at disk level. Just a note, modern iostat splits read and write latency out, but most distros don’t have the latest iostat in their systat (or equivalent) package. rd_mrg and wr_mrg are other two important columns in pt-diskstats output. *_mrg is telling us how many of the original operations the IO elevator (disk scheduler) was able to merge to reduce IOPS, so *_mrg is telling us a quite important thing by letting us know that the IO scheduler was able to consolidate many or few operations. If rd_mrg/wr_mrg is high% then the IO workload is sequential on the other hand, If rd_mrg/wr_mrg is a low% then IO workload is all random. Binary logs, redo logs (aka ib_logfile*), undo log and doublewrite buffer all need sequential writes. qtime and stime are last two columns in pt-diskstats output where qtime reflects to time spent in disk scheduler queue i.e. average queue time before sending it to physical device and on the other hand stime is average service time which is time accumulated to process the physical device request. Note, that qtime is not discriminated between reads and writes and you can check if response time is higher for qtime than it signal towards disk scheduler. Also note that service time (stime field and svctm field in in pt-diskstats & iostat output respectively) is not reliable on Linux. If you read the iostat manual you will see it is deprecated. Along with that, there are many other parameters for pt-diskstats – you can found full documentation here. Below is an example of pt-disktats in action. I used the –devices-regex option which prints only device information that matches this Perl regex. $ pt-diskstats --devices-regex=sd --interval 5 #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.1 sda 21.6 22.8 0.5 45% 1.2 29.4 275.5 4.0 1.1 0% 40.0 145.1 65% 158 297.1 155.0 2.1 1.1 sdb 15.0 21.0 0.3 33% 0.1 5.2 0.0 0.0 0.0 0% 0.0 0.0 11% 1 15.0 0.5 4.7 1.1 sdc 5.6 10.0 0.1 0% 0.0 5.2 1.9 6.0 0.0 33% 0.0 2.0 3% 0 7.5 0.4 3.6 1.1 sdd 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 5.0 sda 17.0 14.8 0.2 64% 3.1 66.7 404.9 4.6 1.8 14% 140.9 298.5 100% 111 421.9 277.6 1.9 5.0 sdb 14.0 19.9 0.3 48% 0.1 5.5 0.4 174.0 0.1 98% 0.0 0.0 11% 0 14.4 0.9 2.4 5.0 sdc 3.6 27.1 0.1 61% 0.0 3.5 2.8 5.7 0.0 30% 0.0 2.0 3% 0 6.4 0.7 2.4 5.0 sdd 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 These are the stats from 7200 RPM SATA disks. As you can see, the write-response time is very high and most of that is made up of IO queue time. This shows the problem exactly. The problem is that the IO subsystem is not able to handle the write workload because the amount of writes that are being performed are way beyond what it can handle. It means the disks cannot service every request concurrently. The workload would actually depend a lot on where the hot data is stored and as we can see in this particular case the workload only hits a single disk out of the 4 disks. A single 7.2K RPM disk can only do about 100 random writes per second which is not a lot considering heavy workload. It’s not particularly a hardware issue but a hardware capacity issue. The kind of workload that is present and the amount of writes that are performed per second are not something that the IO subsystem is able to handle in an efficient manner. Mostly writes are generated on this server as can be seen by the disk stats. Let me show you a second example. Here you can see read latency. rd_rt is consistently between 10ms-30ms. It depends on how fast the disks are spinning and the number of disks. To deal with it possible solutions would be to optimize queries to avoid table scans, use memcached where possible, use SSD’s as it can provide good I/O performance with high concurrency. You will find this post useful on SSD’s from our CEO, Peter Zaitsev. #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.0 sdb 33.0 29.1 0.9 0% 1.1 34.7 7.0 10.3 0.1 61% 0.0 0.4 99% 1 40.0 2.2 19.5 1.0 sdb1 0.0 0.0 0.0 0% 0.0 0.0 7.0 10.3 0.1 61% 0.0 0.4 1% 0 7.0 0.0 0.4 1.0 sdb2 33.0 29.1 0.9 0% 1.1 34.7 0.0 0.0 0.0 0% 0.0 0.0 99% 1 33.0 3.5 30.2 1.0 sdb 81.9 28.5 2.3 0% 1.1 14.0 0.0 0.0 0.0 0% 0.0 0.0 99% 1 81.9 2.0 12.0 1.0 sdb1 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 1.0 sdb2 81.9 28.5 2.3 0% 1.1 14.0 0.0 0.0 0.0 0% 0.0 0.0 99% 1 81.9 2.0 12.0 1.0 sdb 50.0 25.7 1.3 0% 1.3 25.1 13.0 11.7 0.1 66% 0.0 0.7 99% 1 63.0 3.4 11.3 1.0 sdb1 25.0 21.3 0.5 0% 0.6 25.2 13.0 11.7 0.1 66% 0.0 0.7 46% 1 38.0 3.2 7.3 1.0 sdb2 25.0 30.1 0.7 0% 0.6 25.0 0.0 0.0 0.0 0% 0.0 0.0 56% 0 25.0 3.6 22.2 From the below diskstats output it seems that IO is saturated between both reads and writes. This can be noticed with high value for columns rd_s and wr_s. In this particular case, consider having disks in either RAID 5 (better for read only workload) or RAID 10 array is good option along with battery-backed write cache (BBWC) as single disk can really be bad for performance when you are IO bound. device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime sdb1 362.0 27.4 9.7 0% 2.7 7.5 525.2 20.2 10.3 35% 6.4 8.0 100% 0 887.2 7.0 0.9 sdb1 439.9 26.5 11.4 0% 3.4 7.7 545.7 20.8 11.1 34% 9.8 11.9 100% 0 985.6 9.6 0.8 sdb1 576.6 26.5 14.9 0% 4.5 7.8 400.2 19.9 7.8 34% 6.7 10.9 100% 0 976.8 8.6 0.8 sdb1 410.8 24.2 9.7 0% 2.9 7.1 403.1 18.3 7.2 34% 10.8 17.7 100% 0 813.9 12.5 1.0 sdb1 378.4 24.6 9.1 0% 2.7 7.3 506.1 16.5 8.2 33% 5.7 7.6 100% 0 884.4 6.6 0.9 sdb1 572.8 26.1 14.6 0% 4.8 8.4 422.6 17.2 7.1 30% 1.7 2.8 100% 0 995.4 4.7 0.8 sdb1 429.2 23.0 9.6 0% 3.2 7.4 511.9 14.5 7.2 31% 1.2 1.7 100% 0 941.2 3.6 0.9 The following example reflects write heavy activity but write-response time is very good, under 1ms, which shows disks are healthy and capable of handling high number of IOPS. #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.0 dm-0 530.8 16.0 8.3 0% 0.3 0.5 6124.0 5.1 30.7 0% 1.7 0.3 86% 2 6654.8 0.2 0.1 2.0 dm-0 633.1 16.1 10.0 0% 0.3 0.5 6173.0 6.1 36.6 0% 1.7 0.3 88% 1 6806.1 0.2 0.1 3.0 dm-0 731.8 16.0 11.5 0% 0.4 0.5 6064.2 5.8 34.1 0% 1.9 0.3 90% 2 6795.9 0.2 0.1 4.0 dm-0 711.1 16.0 11.1 0% 0.3 0.5 6448.5 5.4 34.3 0% 1.8 0.3 92% 2 7159.6 0.2 0.1 5.0 dm-0 700.1 16.0 10.9 0% 0.4 0.5 5689.4 5.8 32.2 0% 1.9 0.3 88% 0 6389.5 0.2 0.1 6.0 dm-0 774.1 16.0 12.1 0% 0.3 0.4 6409.5 5.5 34.2 0% 1.7 0.3 86% 0 7183.5 0.2 0.1 7.0 dm-0 849.6 16.0 13.3 0% 0.4 0.5 6151.2 5.4 32.3 0% 1.9 0.3 88% 3 7000.8 0.2 0.1 8.0 dm-0 664.2 16.0 10.4 0% 0.3 0.5 6349.2 5.7 35.1 0% 2.0 0.3 90% 2 7013.4 0.2 0.1 9.0 dm-0 951.0 16.0 14.9 0% 0.4 0.4 5807.0 5.3 29.9 0% 1.8 0.3 90% 3 6758.0 0.2 0.1 10.0 dm-0 742.0 16.0 11.6 0% 0.3 0.5 6461.1 5.1 32.2 0% 1.7 0.3 87% 1 7203.2 0.2 0.1 Let me show you a final example. I used –interval and –iterations parameters for pt-diskstats which tells us to wait for a number of seconds before printing the next disk stats and to limit the number of samples respectively. If you notice, you will see in 3rd iteration high latency (rd_rt, wr_rt) mostly for reads. Also, you can notice a high value for queue time (qtime) and service time (stime) where qtime is related to disk IO scheduler settings. For MySQL database servers we usually recommends noop/deadline instead of default cfq. $ pt-diskstats --interval=20 --iterations=3 #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 10.4 hda 11.7 4.0 0.0 0% 0.0 1.1 40.7 11.7 0.5 26% 0.1 2.1 10% 0 52.5 0.4 1.5 10.4 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.4 7.0 0.0 43% 0.0 0.1 0% 0 0.4 0.0 0.1 10.4 hda3 0.0 0.0 0.0 0% 0.0 0.0 0.4 107.0 0.0 96% 0.0 0.2 0% 0 0.4 0.0 0.2 10.4 hda5 0.0 0.0 0.0 0% 0.0 0.0 0.7 20.0 0.0 80% 0.0 0.3 0% 0 0.7 0.1 0.2 10.4 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.1 4.0 0.0 0% 0.0 4.0 0% 0 0.1 0.0 4.0 10.4 hda9 11.7 4.0 0.0 0% 0.0 1.1 39.2 10.7 0.4 3% 0.1 2.7 9% 0 50.9 0.5 1.8 10.4 drbd1 11.7 4.0 0.0 0% 0.0 1.1 39.1 10.7 0.4 0% 0.1 2.8 9% 0 50.8 0.5 1.7 20.0 hda 14.6 4.0 0.1 0% 0.0 1.4 39.5 12.3 0.5 26% 0.3 6.4 18% 0 54.1 2.6 2.7 20.0 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.4 9.1 0.0 56% 0.0 42.0 3% 0 0.4 0.0 42.0 20.0 hda3 0.0 0.0 0.0 0% 0.0 0.0 1.5 22.3 0.0 82% 0.0 1.5 0% 0 1.5 1.2 0.3 20.0 hda5 0.0 0.0 0.0 0% 0.0 0.0 1.1 18.9 0.0 79% 0.1 21.4 11% 0 1.1 0.1 21.3 20.0 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.8 10.4 0.0 62% 0.0 1.5 0% 0 0.8 1.3 0.2 20.0 hda9 14.6 4.0 0.1 0% 0.0 1.4 35.8 11.7 0.4 3% 0.2 4.9 18% 0 50.4 0.5 3.5 20.0 drbd1 14.6 4.0 0.1 0% 0.0 1.4 36.4 11.6 0.4 0% 0.2 5.1 17% 0 51.0 0.5 3.4 20.0 hda 0.9 4.0 0.0 0% 0.2 251.9 28.8 61.8 1.7 92% 4.5 13.1 31% 2 29.6 12.8 0.9 20.0 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.6 8.3 0.0 52% 0.1 98.2 6% 0 0.6 48.9 49.3 20.0 hda3 0.0 0.0 0.0 0% 0.0 0.0 2.0 23.2 0.0 83% 0.0 1.4 0% 0 2.0 1.2 0.3 20.0 hda5 0.0 0.0 0.0 0% 0.0 0.0 4.9 249.4 1.2 98% 4.0 13.2 9% 0 4.9 12.9 0.3 20.0 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 20.0 hda9 0.9 4.0 0.0 0% 0.2 251.9 21.3 24.2 0.5 32% 0.4 12.9 31% 2 22.2 10.2 9.7 20.0 drbd1 0.9 4.0 0.0 0% 0.2 251.9 30.6 17.0 0.5 0% 0.7 24.1 30% 5 31.4 21.0 9.5 You can see the busy column in pt-diskstats output which is the same as the util column in iostat – which points to utilization. Actually, pt-diskstats is quite similar to the iostat tool but pt-diskstats is more interactive and has more information. The busy percentage is only telling us for how long the IO subsystem was busy, but is not indicating capacity. So the only time you care about %busy is when it’s 100% and at the same time latency (await in iostat and rd_rt/wr_rt in diskstats output) increases over -say- 5ms. You can estimate capacity of your IO subsystem and then look at the IOPS being consumed (r/s + w/s columns). Also, the system can process more than one request in parallel (in case of RAID) so %busy can go beyond 100% in pt-diskstats output. If you need to check disk throughput, block device IOPS run the following to capture metrics from your IO subsystem and see if utilization matches other worrisome symptoms. I would suggest capturing disk stats during peak load. Output can be grouped by sample or by disk using the –group-by option. You can use the sysbench benchmark tool for this purpose to measure database server performance. You will find this link useful for sysbench tool details. $ pt-diskstats --group-by=all --iterations=7200 > /tmp/pt-diskstats.out; Conclusion: pt-diskstats is one of the finest tools from Percona Toolkit. By using this tool you can easily spot disk bottlenecks, measure the IO subsystem and identify how much IOPS your drive can handle (i.e. disk capacity).

This article represents top 4 security vulnerabilities related coding practice to avoid while you are programming with Java language. Recently, I came across few Java projects where these instances were found. Please feel free to comment/suggest if I missed to mention one or more important points. Also, sorry for the typos. Following are the key points described later in this article: Executing a dynamically generated SQL statement Directly writing an Http Parameter to Servlet output Creating an SQL PreparedStatement from dynamic string Array is stored directly Executing a Dynamically Generated SQL Statement This is most common of all. One can find mention of this vulenrability at several places. As a matter of fact, many developers are also aware of this vulnerability, although this is a different thing they end up making mistakes once in a while. In several DAO classes, the instances such as following code were found which could lead to SQL injection attacks. StringBuilder query = new StringBuilder(); query.append( "select * from user u where u.name in (" + namesString + ")" ); try { Connection connection = getConnection(); Statement statement = connection.createStatement(); resultSet = statement.executeQuery(query.toString()); } Instead of above query, one could as well make use of prepared statement such as that demonstrated in the code below. It not only makes code less vulnerable to SQL injection attacks but also makes it more efficient. StringBuilder query = new StringBuilder(); query.append( "select * from user u where u.name in (?)" ); try { Connection connection = getConnection(); PreparedStatement statement = connection.prepareCall(query.toString()); statement.setString( 1, namesString ); resultSet = statement.execute(); } Directly writing an Http Parameter to Servlet Output In Servlet classes, I found instances where the Http request parameter was written as it is, to the output stream, without any validation checks. Following code demonstrate the same: public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String content = request.getParameter("some_param"); // // .... some code goes here // response.getWriter().print(content); } Note that above code does not persist anything. Code like above may lead to what is called reflected (or non-persistent) cross site scripting (XSS) vulnerability. Reflected XSS occur when an attacker injects browser executable code within a single HTTP response. As it goes by definition (being non-persistent), the injected attack does not get stored within the application; it manifests only users who open a maliciously crafted link or third-party web page. The attack string is included as part of the crafted URI or HTTP parameters, improperly processed by the application, and returned to the victim. You could read greater details on following OWASP page on reflect XSS Creating an SQL PreparedStatement from Dynamic Query String What it essentially means is the fact that although PreparedStatement was used, but the query was generated as a string buffer and not in the way recommended for prepared statement (parametrized). If unchecked, tainted data from a user would create a String where SQL injection could make it behave in unexpected and undesirable manner. One should rather make the query statement parametrized and, use the PreparedStatement appropriately. Take a look at following code to identify the vulnerable code. StringBuilder query = new StringBuilder(); query.append( "select * from user u where u.name in (" + namesString + ")" ); try { Connection connection = getConnection(); PreparedStatement statement = connection.prepareStatement(query.toString()); resultSet = statement.executeQuery(); } Array is Stored Directly Instances of this vulnerability, Array is stored directly, could help the attacker change the objects stored in array outside of program, and the program behave in inconsistent manner as the reference to the array passed to method is held by the caller/invoker. The solution is to make a copy within the object when it gets passed. In this manner, a subsequent modification of the collection won’t affect the array stored within the object. You could read the details on following stackoverflow page. Following code represents the vulnerability: // Note that values is a String array in the code below. // public void setValues(String[] somevalues) { this.values = somevalues; }

If you use Java for programming, read on to learn about tools like Eclipse IDE, the Java Development Kit, and other must-know tools.

eclipse projects have the nice features that they can link to files and folders: so instead of having the physical file, it is just a pointer to a file. this is very cool as that way i can point to shared files, or keep files in a common place referenced from projects, and so on. linked folder and file in eclipse as with most things in eclipse, there is not a single way how to do things. so i’m showing in this post several ways how to link to files and folders. creating a new link select the folder/project where to create a link to a file. use the context menu with n ew > file (or the menu file > new > other > general > file ): new file that opens a dialog to create a new file. i can select which project/folder i want to use for this: new file dialog next, click on ‘advanced’, enable ‘link to file in the file system’ and browse to the file you want to link to: link to file this will set the link to the file: specified link to file as an absolute path (c:\….) is probably not really what you want, there are eclipse path variables you can use: path variables :idea: note that these are eclipse path variables (not eclipse build variables), see “ eclipse build variables “. to use the path variable for the link, use the ‘extend…’ button and then select the file, then press ok: extending path variable this now uses a path variable for that link. note that it shows as well to which file it resolves: resolved path variable for linked file press finish, and the link will be created. note the small ‘arrow’ in the icon (see “ icon and label decorators in eclipse “) to show a linked file: linked file modifying a link if you right-click on that linked file and select ‘properties’ of it, you can see that it is really a linked file, with the link information, and you can change/edit that link any time: linked file properties linked folder as for link to files, its possible to create ‘link to folders’. it works the same way: select the folder, then use file > new > folder: creating new folder use default location : this creates a normal folder. virtual folder : this does not create a physical folder, but a virtual ‘container’ where i can place links or other virtual folders. this is useful to organize links and virtual folders, without the need for a physical folder. linked folder : like linked files, this links to a folder. :idea: the cool thing about linked (source) folders is: when i add new files to that folder where it links to, the projects with that linked folder to it will ‘see’ the extra files too, and that way new files are automatically added to the project. i do this many times, and it is like a ‘library’ folder for me: i can add a new source file to that ‘library’ folder, and every project linking to that folder will automatically have it added. :-) deleting links linked files and linked folders can be deleted from the project too. in that case, the destination of the link is *not* deleted, only the link: deleting a link to a folder using drag & drop as mentioned at the beginning: there are multiple ways to do the same thing in eclipse. instead using the top menu, or using the context menu, i can use drag & drop. to create links, i need to hold the ctrl key: drag and drop with ctrl pressed to create a link :idea: notice that during the drag&drop with ctrl key pressed the icon gets a ‘+’ to show copy/linking mode. when i drop the file: i get the usual choices how i want to create the link: link to files and folders dialog drag&drop of files and folders do not work inside eclipse. what works as well under windows is to drag&drop a file or folder from the windows explorer :-). summary links to files and folders are a cool thing in eclipse, and they can be created with menus or even simpler with drag&drop. this is not limited to inside eclipse: i can drag&drop from outside with the windows explorer and that way can link to files and folders everywhere :-) happy linking :-)

in “ semihosting with kinetis design studio ” i used printf() to exchange text and data between the target board and the host using the debug connection. kinetis design studio (kds) has that semihosting baked into its libraries. what about if using the gnu arm embedded (launchpad) tools and libraries (see “ switching arm gnu tool chain and libraries in kinetis design studio “)? actually it requires two more steps, but is very easy too. semihosting output there are three things to be in place to use semihosting with the gnu arm embedded (launchpad) libraries: option in the gnu linker settings enabling semihosting in the debugger settings initializing the gnu libraries linker option to enable semihosting for the gnu arm embedded ( launchpad ) libraries, i need to add --specs=rdimon.specs to the linker options: linker option to enable semihosting in case i’m using newlib-nano and want to use printf() and/or scanf() with floating point support, i need to pull in some symbols explicitly with the linker options ‘u': -u _scanf_float -u _printf_float debugger settings in the gnu arm eclipse plugins, i need to enable semihosting. segger j-link for segger j-link, i enable the console in the launch configuration: allocated semihosting console for segger additionally i enable semihosting options in the startup options of the debugger: enabled semihosting in the startup options for segger p&e multilink for p&e the following settings are used: semihosting settings for pne settings for openocd the following settings are used for openocd: openocd semihosting settings initializing the gnu libraries if you would now try to use semihosting with running the debugger, you probably will get error messages like this (e.g. from segger j-link): warning: semihosting command sys_flen failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. the reason is that the semihosting needs to be enabled by the application. i need to call initialise_monitor_handles() before i’m using printf() : 1 2 3 4 5 6 7 8 extern void initialise_monitor_handles( void ); /* prototype */ int main( void ) { initialise_monitor_handles(); /* initialize handles */ for (;;) { printf ( "hello world!\r\n" ); } } with this, i can use printf() and scanf() through a debugger connection. semihosting printf output summary while i don’t like printf() for many reasons, sometimes it is useful to exchange data with the host. using semihosting no physical connection is required, as the communication goes through the debugger. it is somewhat intrusive, and adds code and data overhead, but the gnu arm embedded (launchpad) libraries (both newlib and newlib-nano) have semihosting built-in. it is a matter to enable it in the linker and debugger settings, and to initialize the handles in the application. happy semihosting :-)

Exposing a REST based endpoint for a Spring Boot application or for that matter a straight Spring MVC application is straightforward, the following is a controller exposing an endpoint to create an entity based on the content POST'ed to it: @RestController @RequestMapping("/rest/hotels") public class RestHotelController { .... @RequestMapping(method=RequestMethod.POST) public Hotel create(@RequestBody @Valid Hotel hotel) { return this.hotelRepository.save(hotel); } } Internally Spring MVC uses a component called a HttpMessageConverter to convert the Http request to an object representation and back. A set of default converters are automatically registered which supports a whole range of different resource representation formats - json, xml for instance. Now, if there is a need to customize the message converters in some way, Spring Boot makes it simple. As an example consider if the POST method in the sample above needs to be little more flexible and should ignore properties which are not present in the Hotel entity - typically this can be done by configuring the Jackson ObjectMapper, all that needs to be done with Spring Boot is to create a new HttpMessageConverter bean and that would end up overriding all the default message converters, this way: @Bean public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter() { MappingJackson2HttpMessageConverter jsonConverter = new MappingJackson2HttpMessageConverter(); ObjectMapper objectMapper = new ObjectMapper(); objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); jsonConverter.setObjectMapper(objectMapper); return jsonConverter; } This works well for a Spring-Boot application, however for straight Spring MVC applications which do not make use of Spring-Boot, configuring a custom converter is a little more complicated - the default converters are not registered by default and an end user has to be explicit about registering the defaults: @Configuration public class WebConfig extends WebMvcConfigurationSupport { @Bean public MappingJackson2HttpMessageConverter customJackson2HttpMessageConverter() { MappingJackson2HttpMessageConverter jsonConverter = new MappingJackson2HttpMessageConverter(); ObjectMapper objectMapper = new ObjectMapper(); objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); jsonConverter.setObjectMapper(objectMapper); return jsonConverter; } @Override public void configureMessageConverters(List> converters) { converters.add(customJackson2HttpMessageConverter()); super.addDefaultHttpMessageConverters(); } } Here WebMvcConfigurationSupport provides a way to more finely tune the MVC tier configuration of a Spring based application. In the configureMessageConverters method, the custom converter is being registered and then an explicit call is being made to ensure that the defaults are registered also. A little more work than for a Spring-Boot based application.