When working with Entity Framework - Code First model approach, a developer creates POCO entities for database tables. The benefit of using Code First model is to have POCO entity for each table that can be used as either WCF Data Contracts or you can apply your own custom attributes to handle Security, Logging, etc. and there is no mapping needed as we used to do in Entity Framework (Model First) approach if the application architecture is n-tier based. Considering the Data Access layer, we will implement a repository pattern that encapsulates the persistence logic in a separate class. This class will be responsible to perform database operations. Let’s suppose the application is based on n-tier architecture and having 3 tiers namely Presentation, Business and Data Access. Common library contains all our POCO entities that will be used by all the layers. Presentation Layer: Contains Views, Forms Business Layer: Managers that handle logic functionality Data Access Layer: Contains Repository class that handles CRUD operations Common Library: Contain POCO entities. We will implement an interface named “IRepository” that defines the signature of all the appropriate generic methods needed to perform CRUD operation and then implement the Repository class that defines the actual implementation of each method. We can also instantiate Repository object using Dependency Injection or apply Factory pattern. Code Snippet: IRepository public interface IRepository : IDisposable { /// /// Gets all objects from database /// /// IQueryable All() where T : class; /// /// Gets objects from database by filter. /// /// Specified a filter /// IQueryable Filter(Expression> predicate) where T : class; /// /// Gets objects from database with filting and paging. /// /// /// Specified a filter /// Returns the total records count of the filter. /// Specified the page index. /// Specified the page size /// IQueryable Filter(Expression> filter, out int total, int index = 0, int size = 50) where T : class; /// /// Gets the object(s) is exists in database by specified filter. /// /// Specified the filter expression /// bool Contains(Expression> predicate) where T : class; /// /// Find object by keys. /// /// Specified the search keys. /// T Find(params object[] keys) where T : class; /// /// Find object by specified expression. /// /// /// T Find(Expression> predicate) where T : class; /// /// Create a new object to database. /// /// Specified a new object to create. /// T Create(T t) where T : class; /// /// Delete the object from database. /// /// Specified a existing object to delete. int Delete(T t) where T : class; /// /// Delete objects from database by specified filter expression. /// /// /// int Delete(Expression> predicate) where T : class; /// /// Update object changes and save to database. /// /// Specified the object to save. /// int Update(T t) where T : class; /// /// Select Single Item by specified expression. /// /// /// /// T Single(Expression> expression) where T : class; void SaveChanges(); void ExecuteProcedure(String procedureCommand, params SqlParameter[] sqlParams); } Code Snippet: Repository public class Repository : IRepository { DbContext Context; public Repository() { Context = new DBContext(); } public Repository(DBContext context) { Context = context; } public void CommitChanges() { Context.SaveChanges(); } public T Single(Expression> expression) where T : class { return All().FirstOrDefault(expression); } public IQueryable All() where T : class { return Context.Set().AsQueryable(); } public virtual IQueryable Filter(Expression> predicate) where T : class { return Context.Set().Where(predicate).AsQueryable(); } public virtual IQueryable Filter(Expression> filter, out int total, int index = 0, int size = 50) where T : class { int skipCount = index * size; var _resetSet = filter != null ? Context.Set().Where(filter).AsQueryable() : Context.Set().AsQueryable(); _resetSet = skipCount == 0 ? _resetSet.Take(size) : _resetSet.Skip(skipCount).Take(size); total = _resetSet.Count(); return _resetSet.AsQueryable(); } public virtual T Create(T TObject) where T : class { var newEntry = Context.Set().Add(TObject); Context.SaveChanges(); return newEntry; } public virtual int Delete(T TObject) where T : class { Context.Set().Remove(TObject); return Context.SaveChanges(); } public virtual int Update(T TObject) where T : class { try { var entry = Context.Entry(TObject); Context.Set().Attach(TObject); entry.State = EntityState.Modified; return Context.SaveChanges(); } catch (OptimisticConcurrencyException ex) { throw ex; } } public virtual int Delete(Expression> predicate) where T : class { var objects = Filter(predicate); foreach (var obj in objects) Context.Set().Remove(obj); return Context.SaveChanges(); } public bool Contains(Expression> predicate) where T : class { return Context.Set().Count(predicate) > 0; } public virtual T Find(params object[] keys) where T : class { return (T)Context.Set().Find(keys); } public virtual T Find(Expression> predicate) where T : class { return Context.Set().FirstOrDefault(predicate); } public virtual void ExecuteProcedure(String procedureCommand, params SqlParameter[] sqlParams){ Context.Database.ExecuteSqlCommand(procedureCommand, sqlParams); } public virtual void SaveChanges() { Context.SaveChanges(); } public void Dispose() { if (Context != null) Context.Dispose(); } } The benefit of using Repository pattern is that all the database operations will be managed centrally and in future if you want to change the underlying database connector you can add another Repository class and defines its own implementation or change the existing one.

This case study describes the complete root cause analysis and resolution of an Apache Log4j thread race problem affecting a Weblogic Portal 10.0 production environment. It will also demonstrate the importance of proper Java classloader knowledge when developing and supporting Java EE applications. This article is also another opportunity for you to improve your thread dump analysis skills and understand thread race conditions. Environment specifications Java EE server: Oracle Weblogic Portal 10.0 OS: Solaris 10 JDK: Oracle/Sun HotSpot JVM 1.5 Logging API: Apache Log4j 1.2.15 RDBMS: Oracle 10g Platform type: Web Portal Troubleshooting tools Quest Foglight for Java (monitoring and alerting) Java VM Thread Dump (thread race analysis) Problem overview Major performance degradation was observed from one of our Weblogic Portal production environments. Alerts were also sent from the Foglight agents indicating a significant surge in Weblogic threads utilization up to the upper default limit of 400. Gathering and validation of facts As usual, a Java EE problem investigation requires gathering of technical and non technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause: What is the client impact? HIGH Recent change of the affected platform? Yes, a recent deployment was performed involving minor content changes and some Java libraries changes & refactoring Any recent traffic increase to the affected platform? No Since how long this problem has been observed? New problem observed following the deployment Did a restart of the Weblogic server resolve the problem? No, any restart attempt did result in an immediate surge of threads Did a rollback of the deployment changes resolve the problem? Yes Conclusion #1: The problem appears to be related to the recent changes. However, the team was initially unable to pinpoint the root cause. This is now what we will discuss for the rest of the article. Weblogic hogging thread report The initial thread surge problem was reported by Foglight. As you can see below, the threads utilization was significant (up to 400) leading to a high volume of pending client requests and ultimately major performance degradation. As usual, thread problems require proper thread dump analysis in order to pinpoint the source of threads contention. Lack of this critical analysis skill will prevent you to go any further in the root cause analysis. For our case study, a few thread dump snapshots were generated from our Weblogic servers using the simple Solaris OS command kill -3 . Thread Dump data was then extracted from the Weblogic standard output log files. Thread Dump analysis The first step of the analysis was to perform a fast scan of all stuck threads and pinpoint a problem “pattern”. We found 250 threads stuck in the following execution path: "[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x03c4fc38 nid=0xe6 waiting for monitor entry [0x3f99e000..0x3f99f970] at org.apache.log4j.Category.callAppenders(Category.java:186) - waiting to lock <0x8b3c4c68> (a org.apache.log4j.spi.RootCategory) at org.apache.log4j.Category.forcedLog(Category.java:372) at org.apache.log4j.Category.log(Category.java:864) at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:110) at org.apache.beehive.netui.util.logging.Logger.debug(Logger.java:119) at org.apache.beehive.netui.pageflow.DefaultPageFlowEventReporter.beginPageRequest(DefaultPageFlowEventReporter.java:164) at com.bea.wlw.netui.pageflow.internal.WeblogicPageFlowEventReporter.beginPageRequest(WeblogicPageFlowEventReporter.java:248) at org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:154) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:336) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:526) at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:261) at .AppRedirectFilter.doFilter(RedirectFilter.java:83) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at .AppServletFilter.doFilter(PortalServletFilter.java:336) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(Unknown Source) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046) at weblogic.servlet.internal.ServletRequestImpl.run(Unknown Source) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200) at weblogic.work.ExecuteThread.run(ExecuteThread.java:172) As you can see, it appears that all the threads are waiting to acquire a lock on an Apache Log4j object monitor (org.apache.log4j.spi.RootCategory) when attempting to log debug information to the configured appender and log file. How did we figure that out from this thread stack trace? Let’s dissect this thread stack trace in order for you to better understand this thread race condition e.g. 250 threads attempting to acquire the same object monitor concurrently. At this point the main question is why are we seeing this problem suddenly? An increase of the logging level or load was also ruled out at this point after proper verification. The fact that the rollback of the previous changes did fix the problem did naturally lead us to perform a deeper review of the promoted changes. Before we go to the final root cause section, we will perform a code review of the affected Log4j code e.g. exposed to thread race conditions. Apache Log4j 1.2.15 code review ## org.apache.log4j.Category /** * Call the appenders in the hierrachy starting at this. If no * appenders could be found, emit a warning. * * * This method calls all the appenders inherited from the hierarchy * circumventing any evaluation of whether to log or not to log the * particular log request. * * @param event * the event to log. */ public void callAppenders(LoggingEvent event) { int writes = 0; for (Category c = this; c != null; c = c.parent) { // Protected against simultaneous call to addAppender, // removeAppender,... synchronized (c) { if (c.aai != null) { writes += c.aai.appendLoopOnAppenders(event); } if (!c.additive) { break; } } } if (writes == 0) { repository.emitNoAppenderWarning(this); } As you can see, the Catelogry.callAppenders() is using a synchronized block at the Category level which can lead to a severe thread race condition under heavy concurrent load. In this scenario, the usage of a re-entrant read write lock would have been more appropriate (e.g. such lock strategy allows concurrent “read” but single “write”). You can find reference to this known Apache Log4j limitation below along with some possible solutions. https://issues.apache.org/bugzilla/show_bug.cgi?id=41214 Does the above Log4j behaviour is the actual root cause of our problem? Not so fast… Let’s remember that this problem got exposed only following a recent deployment. The real question is what application change triggered this problem & side effect from the Apache Log4j logging API? Root cause: a perfect storm! Deep dive analysis of the recent changes deployed did reveal that some Log4j libraries at the child classloader level were removed along with the associated “child first” policy. This refactoring exercise ended-up moving the delegation of both Commons logging and Log4j at the parent classloader level. What is the problem? Before this change, the logging events were split between Weblogic Beehive Log4j calls at the parent classloader and web application logging events at the child class loader. Since each classloader had its own copy of the Log4j objects, the thread race condition problem was split in half and not exposed (masked) under the current load conditions. Following the refactoring, all Log4j calls were moved to the parent classloader (Java EE app); adding significant concurrency level to the Log4j components such as Category. This increase concurrency level along with this known Category.java thread race / deadlock behaviour was a perfect storm for our production environment. In other to mitigate this problem, 2 immediate solutions were applied to the environment: Rollback the refactoring and split Log4j calls back between parent and child classloader Reduce logging level for some appenders from DEBUG to WARNING This problem case again re-enforce the importance of performing proper testing and impact assessment when applying changes such as library and class loader related changes. Such changes can appear simple at the "surface" but can trigger some deep execution pattern changes, exposing your application(s) to known thread race conditions. A future upgrade to Apache Log4j 2 (or other logging API’s) will also be explored as it is expected to bring some performance enhancements which may address some of these thread race & scalability concerns. Please provide any comment or share your experience on thread race related problems with logging API's.

RateLimiter class was recently added to Guava libraries (since 13.0) and it is already among my favourite tools. Have a look what the JavaDoc says: [...] rate limiter distributes permits at a configurable rate. Each acquire() blocks if necessary until a permit is available [...] Rate limiters are often used to restrict the rate at which some physical or logical resource is accessed Basically this small utility class can be used e.g. to limit the number of requests per second your API wishes to handle or to throttle your own client code, avoiding denial of service of someone else's API if we are hitting it too often. Let's start from a simple example. Say we have a long running process that needs to broadcast its progress to supplied listener: def longRunning(listener: Listener) { var processed = 0 for(item <- items) { //..do work... processed += 1 listener.progressChanged(100.0 * processed / items.size) } } trait Listener { def progressChanged(percentProgress: Double) } Please forgive me the imperative style of this Scala code, but that's not the point. The problem I want to highlight becomes obvious once we start our application with some concrete listener: class ConsoleListener extends Listener { def progressChanged(percentProgress: Double) { println("Progress: " + percentProgress) } } longRunning(new ConsoleListener) Imagine that longRunning() method processes millions of items but each iteration takes just a split of a second. The amount of logging messages is just insane, not to mention console output is probably taking much more time than processing itself. You've probably faced such a problem several times and have a simple workaround: if(processed % 100 == 0) { listener.progressChanged(100.0 * processed / items.size) } There, I Fixed It! We only print progress every 100th iteration. However this approach has several drawbacks: code is polluted with unrelated logic there is no guarantee that every 100th iteration is slow enough... ... or maybe it's still too slow? What we really want to achieve is to limit the frequency of progress updates (say: two times per second). OK, going deeper into the rabbit hole: def longRunning(listener: Listener) { var processed = 0 var lastUpdateTimestamp = 0L for(item <- items) { //..do work... processed += 1 if(System.currentTimeMillis() - lastUpdateTimestamp > 500) { listener.progressChanged(100.0 * processed / items.size) lastUpdateTimestamp = System.currentTimeMillis() } } } Do you also have a feeling that we are going in the wrong direction? Ladies and gentlemen, I give you RateLimiter: var processed = 0 val limiter = RateLimiter.create(2) for (item <- items) { //..do work... processed += 1 if (limiter.tryAcquire()) { listener.progressChanged(100.0 * processed / items.size) } } Getting better? If the API is not clear: we are first creating a RateLimiter with 2 permits per second. This means we can acquire up to two permits during one second and if we try to do it more often tryAcquire() will return false (or thread will block if acquire() is used instead1). So the code above guarantees that the listener won't be called more that two times per second. As a bonus, if you want to completely get rid of unrelated throttling code from the business logic, decorator pattern to the rescue. First let's create a listener that wraps another (concrete) listener and delegates to it only at a given rate: class RateLimitedListener(target: Listener) extends Listener { val limiter = RateLimiter.create(2) def progressChanged(percentProgress: Double) { if (limiter.tryAcquire()) { target.progressChanged(percentProgress) } } } What's best about the decorator pattern is that both the code using the listener and the concrete implementation are not aware of the decorator. Also the client code became much simpler (essentially we came back to original): def longRunning(listener: Listener) { var processed = 0 for (item <- items) { //..do work... processed += 1 listener.progressChanged(100.0 * processed / items.size) } } longRunning(new RateLimitedListener(new ConsoleListener)) But we've only scratched the surface of where RateLimiter can be used! Say we want to avoid aforementioned denial of service attack or slow down automated clients of our API. It's very simple with RateLimiter and servlet filter: @WebFilter(urlPatterns=Array("/*")) class RateLimiterFilter extends Filter { val limiter = RateLimiter.create(100) def init(filterConfig: FilterConfig) {} def doFilter(request: ServletRequest, response: ServletResponse, chain: FilterChain) { if(limiter.tryAcquire()) { chain.doFilter(request, response) } else { response.asInstanceOf[HttpServletResponse].sendError(SC_TOO_MANY_REQUESTS) } } def destroy() {} } Another self-descriptive sample. This time we limit our API to handle not more than 100 requests per second (of course RateLimiter is thread safe). All HTTP requests that come through our filter are subject to rate limiting. If we cannot handle incoming request, we send HTTP 429 - Too Many Requests error code (not yet available in servlet spec). Alternatively you may wish to block the client for a while instead of eagerly rejecting it. That's fairly straightforward as well: def doFilter(request: ServletRequest, response: ServletResponse, chain: FilterChain) { limiter.acquire() chain.doFilter(request, response) } limiter.acquire() will block as long as it's needed to keep desired 100 requests per second limit. Yet another alternative is to use tryAcquire() with timeout (blocking up to given amount of time). Blocking approach is better if you want to avoid sending errors to the client. However under high load it's easy to imagine almost all HTTP threads blocked waiting for RateLimiter, eventually causing servlet container to reject connections. So dropping of clients can be only partially avoided. This filter is a good starting point to build more sophisticated solutions. Map of rate limiters by IP or user name are good examples. What we haven't covered yet is acquiring more than one permit at a time. It turns out RateLimiter can also be used e.g. to limit network bandwidth or the amount of data being sent/received. Imagine you create a search servlet and you want to impose that no more than 1000 results are returned per second. In each request user decides how many results she wants to receive per response: it can be 500 requests each containing 2 results or 1 huge request asking for 1000 results at once. But never more than 1000 results within a second on average. Users are free to use their quota as they wish: @WebFilter(urlPatterns = Array ("/search")) class SearchServlet extends HttpServlet { val limiter = RateLimiter.create(1000) override def doGet(req: HttpServletRequest, resp: HttpServletResponse) { val resultsCount = req.getParameter("results").toInt limiter.acquire(resultsCount) //process and return results... } } By default we acquire() one permit per invocation. Non-blocking servlet would call limiter.tryAcquire(resultsCount) and check the results, you know that by now. If you are interested in rate limiting of network traffic, don't forget to check out my Tenfold increase in server throughput with Servlet 3.0 asynchronous processing. RateLimiter, due to a blocking nature, is not very well suited to write scalable upload/download servers with throttling. The last example I would like to share with you is throttling client code to avoid overloading the server we are talking to. Imagine a batch import/export process that calls some server thousands of times exchanging data. If we don't throttle the client and there is no rate limiting on the server side, server might get overloaded and crash. RateLimiter is once again very helpful: val limiter = RateLimiter.create(20) def longRunning() { for (item <- items) { limiter.acquire() server.sync(item) } } This sample is very similar to the first one. Difference being that this time we block instead of discard missing permits. Thanks to blocking, external call to server.sync(item) won't overload the 3rd-party server, calling it at most 20 times per second. Of course if you have several threads interacting with the server, they can all share the same RateLimiter. To wrap-up: RateLimiter allows you to perform certain actions not more often than with a given frequency It's a small and lightweight class (no threads involved!) You can create thousands of rate limiters (per client?) or share one among several threads We haven't covered warm-up functionality - if RateLimiter was completely idle for a long time, it will gradually increase allowed frequency over configured time up to configured maximum value instead of allowing maximum frequency from the very beginning I have a feeling that we'll go back to this class soon. I hope you'll find it useful in your next project! 1 - I am using Guava 14.0-SNAPSHOT. If 14.0 stable is not available by the time you are reading this, you must use more verbose tryAcquire(1, 0, TimeUnit.MICROSECONDS) instead of tryAcquire() and acquire(1) instead of acquire().

Continuing to discover the powerful set of Redis features, the one worth mentioning about is out of the box support of pub/sub messaging. Pub/Sub messaging is essential part of many software architectures. Some software systems demand from messaging solution to provide high-performance, scalability, queues persistence and durability, fail-over support, transactions, and many more nice-to-have features, which in Java world mostly always leads to using one of JMS implementation providers. In my previous projects I have actively used Apache ActiveMQ (now moving towards Apache ActiveMQ Apollo). Though it's a great implementation, sometimes I just needed simple queuing support and Apache ActiveMQ just looked overcomplicated for that. Alternatives? Please welcome Redis pub/sub! If you are already using Redis as key/value store, few additional lines of configuration will bring pub/sub messaging to your application in no time. Spring Data Redis project abstracts very well Redis pub/sub API and provides the model so familiar to everyone who uses Spring capabilities to integrate with JMS. As always, let's start with the POM configuration file. It's pretty small and simple, includes necessary Spring dependencies, Spring Data Redis and Jedis, great Java client for Redis. 4.0.0 com.example.spring redis 0.0.1-SNAPSHOT jar UTF-8 3.1.1.RELEASE org.springframework.data spring-data-redis 1.0.1.RELEASE cglib cglib-nodep 2.2 log4j log4j 1.2.16 redis.clients jedis 2.0.0 jar org.springframework spring-core ${spring.version} org.springframework spring-context ${spring.version} org.apache.maven.plugins maven-compiler-plugin 2.3.2 1.6 1.6 Moving on to configuring Spring context, let's understand what we need to have in order for a publisher to publish some messages and for a consumer to consume them. Knowing the respective Spring abstractions for JMS will help a lot with that. we need connection factory -> JedisConnectionFactory we need a template for publisher to publish messages -> RedisTemplate we need a message listener for consumer to consume messages -> RedisMessageListenerContainer Using Spring Java configuration, let's describe our context: package com.example.redis.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.data.redis.connection.jedis.JedisConnectionFactory; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.data.redis.listener.ChannelTopic; import org.springframework.data.redis.listener.RedisMessageListenerContainer; import org.springframework.data.redis.listener.adapter.MessageListenerAdapter; import org.springframework.data.redis.serializer.GenericToStringSerializer; import org.springframework.data.redis.serializer.StringRedisSerializer; import org.springframework.scheduling.annotation.EnableScheduling; import com.example.redis.IRedisPublisher; import com.example.redis.impl.RedisMessageListener; import com.example.redis.impl.RedisPublisherImpl; @Configuration @EnableScheduling public class AppConfig { @Bean JedisConnectionFactory jedisConnectionFactory() { return new JedisConnectionFactory(); } @Bean RedisTemplate< String, Object > redisTemplate() { final RedisTemplate< String, Object > template = new RedisTemplate< String, Object >(); template.setConnectionFactory( jedisConnectionFactory() ); template.setKeySerializer( new StringRedisSerializer() ); template.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) ); template.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) ); return template; } @Bean MessageListenerAdapter messageListener() { return new MessageListenerAdapter( new RedisMessageListener() ); } @Bean RedisMessageListenerContainer redisContainer() { final RedisMessageListenerContainer container = new RedisMessageListenerContainer(); container.setConnectionFactory( jedisConnectionFactory() ); container.addMessageListener( messageListener(), topic() ); return container; } @Bean IRedisPublisher redisPublisher() { return new RedisPublisherImpl( redisTemplate(), topic() ); } @Bean ChannelTopic topic() { return new ChannelTopic( "pubsub:queue" ); } } Very easy and straightforward. The presence of @EnableScheduling annotation is not necessary and is required only for our publisher implementation: the publisher will publish a string message every 100 ms. package com.example.redis.impl; import java.util.concurrent.atomic.AtomicLong; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.data.redis.listener.ChannelTopic; import org.springframework.scheduling.annotation.Scheduled; import com.example.redis.IRedisPublisher; public class RedisPublisherImpl implements IRedisPublisher { private final RedisTemplate< String, Object > template; private final ChannelTopic topic; private final AtomicLong counter = new AtomicLong( 0 ); public RedisPublisherImpl( final RedisTemplate< String, Object > template, final ChannelTopic topic ) { this.template = template; this.topic = topic; } @Scheduled( fixedDelay = 100 ) public void publish() { template.convertAndSend( topic.getTopic(), "Message " + counter.incrementAndGet() + ", " + Thread.currentThread().getName() ); } } And finally our message listener implementation (which just prints message on a console). package com.example.redis.impl; import org.springframework.data.redis.connection.Message; import org.springframework.data.redis.connection.MessageListener; public class RedisMessageListener implements MessageListener { @Override public void onMessage( final Message message, final byte[] pattern ) { System.out.println( "Message received: " + message.toString() ); } } Awesome, just two small classes, one configuration to wire things together and we have full pub/sub messaging support in our application! Let's run the application as standalone ... package com.example.redis; import org.springframework.context.ApplicationContext; import org.springframework.context.annotation.AnnotationConfigApplicationContext; import com.example.redis.config.AppConfig; public class RedisPubSubStarter { public static void main(String[] args) { new AnnotationConfigApplicationContext( AppConfig.class ); } } ... and see following output in a console: ... Message received: Message 1, pool-1-thread-1 Message received: Message 2, pool-1-thread-1 Message received: Message 3, pool-1-thread-1 Message received: Message 4, pool-1-thread-1 Message received: Message 5, pool-1-thread-1 Message received: Message 6, pool-1-thread-1 Message received: Message 7, pool-1-thread-1 Message received: Message 8, pool-1-thread-1 Message received: Message 9, pool-1-thread-1 Message received: Message 10, pool-1-thread-1 Message received: Message 11, pool-1-thread-1 Message received: Message 12, pool-1-thread-1 Message received: Message 13, pool-1-thread-1 Message received: Message 14, pool-1-thread-1 Message received: Message 15, pool-1-thread-1 Message received: Message 16, pool-1-thread-1 ... Great! There is much more which you could do with Redis pub/sub, excellent documentation is available for you on Redis official web site.

Spring Framework, since it’s inception, included an eventing mechanism which can be used for application-wide eventing. This eventing mechanism was developed to be used internally by Spring Framework for eventing, such as notification of context being refreshed, etc, but it can be used for application specific custom events as well. This eventing API is based on an interface named org.springframework.context.ApplicationListener, which defined one method named onApplicationEvent. Below code snippet shows a simple events listener which just logs the event information. package com.yohanliyanage.blog.springevents; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.springframework.context.ApplicationEvent; import org.springframework.context.ApplicationListener; public class MyEventListener implements ApplicationListener { private static final Log LOG = LogFactory.getLog(MyEventListener.class); public void onApplicationEvent(ApplicationEvent event) { LOG.info("Event Occurred : " + event); } } To register this event listener, all that we have to do is to add it as a Spring managed bean. If we just add it as a bean in Spring Bean Configuration XML, or if we have annotation scanning enabled, adding an annotation such as @Component, would ensure that our listener will receive events via Spring. Below XML block shows the simple bean definition in XML for registering this listener. < ?xml version="1.0" encoding="UTF-8"?> Now, to test this code, let’s write up a main method which creates the Spring Application Context. In this code, it’s assumed that the Spring bean definition file is located at META-INF/spring/application-context.xml, which is in class path. You can download the sample code here. public class Main { public static void main(String[] args) throws InterruptedException { ApplicationContext context = new ClassPathXmlApplicationContext("classpath:META-INF/spring/application-context.xml"); } } When we run this, we get the following output. 18:45:00 INFO Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@7a982589: startup date [Sun Sep 30 18:45:00 IST 2012]; root of context hierarchy 18:45:00 INFO Loading XML bean definitions from class path resource [META-INF/spring/application-context.xml] 18:45:00 INFO Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@33e228bc: defining beans [com.yohanliyanage.blog.springevents.MyEventListener#0]; root of factory hierarchy 18:45:00 INFO Event Occurred : org.springframework.context.event.ContextRefreshedEvent[source=org.springframework.context.support.ClassPathXmlApplicationContext@7a982589: startup date [Sun Sep 30 18:45:00 IST 2012]; root of context hierarchy] As highlighted above, our listener gets notified by the framework when Spring Context is refreshed during initialization. This is a framework event, and of course, there’s no magic to it. But how can we generate application specific, custom events? As you will see in the blow code block, this is also very simple and straight-forward. First, let’s create our own event implementation. For this, we just have to write a class that extends from ApplicationEvent. package com.yohanliyanage.blog.springevents; import org.springframework.context.ApplicationEvent; public class MyCustomEvent extends ApplicationEvent { private static final long serialVersionUID = -5308299518665062983L; public MyCustomEvent(Object source) { super(source); } } Next, we have to write a class which does the event publishing. In order to publish an event, we need to get a reference to ApplicationEventPublisher. This can be easily done by implementing the ApplicationEventPublisherAware, as shown in the code block below. package com.yohanliyanage.blog.springevents; import org.springframework.context.ApplicationEventPublisher; import org.springframework.context.ApplicationEventPublisherAware; public class MyEventPublisher implements ApplicationEventPublisherAware { private ApplicationEventPublisher publisher; public void setApplicationEventPublisher(ApplicationEventPublisher publisher) { this.publisher = publisher; } public void publish() { this.publisher.publishEvent(new MyCustomEvent(this)); } } Now, this bean also should be added to the Spring bean configuration, as follows. Note that you do not have to write a separate publisher class in your application. Your existing Spring beans can easily to this by just implementing the ApplicationEventPublisherAware interface. Now, let’s slightly modify the Main class to invoke the publish() method in our event publisher. public static void main(String[] args) throws InterruptedException { ApplicationContext context = new ClassPathXmlApplicationContext("classpath:META-INF/spring/application-context.xml"); MyEventPublisher publisher = context.getBean(MyEventPublisher.class); publisher.publish(); } When we run the application now, we get the following output. 19:21:18 INFO Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@7a982589: startup date [Sun Sep 30 19:21:18 IST 2012]; root of context hierarchy 19:21:18 INFO Loading XML bean definitions from class path resource [META-INF/spring/application-context.xml] 19:21:19 INFO Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2565a3c2: defining beans [com.yohanliyanage.blog.springevents.MyEventListener#0,com.yohanliyanage.blog.springevents.MyEventPublisher#0]; root of factory hierarchy 19:21:19 INFO Event Occurred : org.springframework.context.event.ContextRefreshedEvent[source=org.springframework.context.support.ClassPathXmlApplicationContext@7a982589: startup date [Sun Sep 30 19:21:18 IST 2012]; root of context hierarchy] 19:21:19 INFO Event Occurred : com.yohanliyanage.blog.springevents.MyCustomEvent[source=com.yohanliyanage.blog.springevents.MyEventPublisher@5a676437] As highlighted above, our custom event has been triggered, and our listener was able to handle that event. So far, so good. But if you have noticed, our listener gets invoked for all of the events that occurs in the application, including framework events. But in most of the cases, this is not desirable. The listener will be interested in one or more specific events. Up until Spring 3.0, this eventing mechanism did not had support for filtering events. That is, if you implement an ApplicationListener, you would end up receiving all events that occurs in the application, and you had to manually look into the ApplicationEvent object that gets passed in to your listener to identify and discard events that you are not interested in. This of course, was a hassle, and probably due to this, Spring Eventing did not get attention of most of the developers. With Spring 3.0, this API was enhanced with Generics support, to provide filtering of events. Now, the ApplicationListener interface is parameterized as follows. public interface ApplicationListener < E extends ApplicationEvent > extends EventListener The onApplicationEvent method parameter uses generic type E. With this, we can implement our listener as follows. package com.yohanliyanage.blog.springevents; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.springframework.context.ApplicationListener; public class MyEventListener implements ApplicationListener < MyCustomEvent > { private static final Log LOG = LogFactory.getLog(MyEventListener.class); public void onApplicationEvent(MyCustomEvent event) { LOG.info("Event Occurred : " + event); } } This listener implementation’s onApplicationEvent method will be called only for MyCustomEvent based events. This in turn provides the necessary event filtering, where we don’t have to write boilerplate code to discard unnecessary events. The log output is as follows. 19:29:31 INFO Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@7a982589: startup date [Sun Sep 30 19:29:31 IST 2012]; root of context hierarchy 19:29:31 INFO Loading XML bean definitions from class path resource [META-INF/spring/application-context.xml] 19:29:31 INFO Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2565a3c2: defining beans [com.yohanliyanage.blog.springevents.MyEventListener#0,com.yohanliyanage.blog.springevents.MyEventPublisher#0]; root of factory hierarchy 19:29:31 INFO Event Occurred : com.yohanliyanage.blog.springevents.MyCustomEvent[source=com.yohanliyanage.blog.springevents.MyEventPublisher@5a676437] As seen above in the output, we no longer receive the unwanted framework specific events, or any other events that we are not interested in. So in conclusion, Spring does provide a decent eventing mechanism which is quite useful for implementing eventing support in applications. While this has been around since the early days of Spring, it did not see wide adoption primarily due to it’s incapability of filtering out specific events for a listener. But with Spring 3.0, things are improved, and now it has reached a state where we can leverage it to broadcast events in our applications with ease. One thing to note is that by default, Spring Eventing is synchronous. But this can be made asynchronous by providing a custom ApplicationEventMulticaster implementation that would make use of a TaskExecutor. Download Source Code – Spring Eventing Example Project

HTML5 Color Picker (canvas) In our new tutorial we are going to create an easy but effective color picker using HTML5. I think that you have already seen different jQuery versions of colorpicker, our goal today is to create something similar, and even better. In order to make it more unique, there are 5 different colorwheels which you can use. If you are ready – let’s start. This would be a good time to test the demos and download the sources: Live Demo 1 Live Demo 2 Live Demo 3 Live Demo 4 Live Demo 5 download in package If you are ready – let’s start coding ! Step 1. HTML Our first step is the html markup: R G B RGB HEX As you see, our color picker consists of two main components: the preview element and the hidden (by default) color picker element. Once we click by preview element – we will display color picker. Step 2. JS Our next step – is javascript. Please review our result code: js/script.js $(function(){ var bCanPreview = true; // can preview // create canvas and context objects var canvas = document.getElementById('picker'); var ctx = canvas.getContext('2d'); // drawing active image var image = new Image(); image.onload = function () { ctx.drawImage(image, 0, 0, image.width, image.height); // draw the image on the canvas } // select desired colorwheel var imagesrc="images/colorwheel1.png"; switch ($(canvas).attr('var')) { case '2': imagesrc="images/colorwheel2.png"; break; case '3': imagesrc="images/colorwheel3.png"; break; case '4': imagesrc="images/colorwheel4.png"; break; case '5': imagesrc="images/colorwheel5.png"; break; } image.src = imageSrc; $('#picker').mousemove(function(e) { // mouse move handler if (bCanPreview) { // get coordinates of current position var canvasOffset = $(canvas).offset(); var canvasX = Math.floor(e.pageX - canvasOffset.left); var canvasY = Math.floor(e.pageY - canvasOffset.top); // get current pixel var imageData = ctx.getImageData(canvasX, canvasY, 1, 1); var pixel = imageData.data; // update preview color var pixelColor = "rgb("+pixel[0]+", "+pixel[1]+", "+pixel[2]+")"; $('.preview').css('backgroundColor', pixelColor); // update controls $('#rVal').val(pixel[0]); $('#gVal').val(pixel[1]); $('#bVal').val(pixel[2]); $('#rgbVal').val(pixel[0]+','+pixel[1]+','+pixel[2]); var dColor = pixel[2] + 256 * pixel[1] + 65536 * pixel[0]; $('#hexVal').val('#' + ('0000' + dColor.toString(16)).substr(-6)); } }); $('#picker').click(function(e) { // click event handler bCanPreview = !bCanPreview; }); $('.preview').click(function(e) { // preview click $('.colorpicker').fadeToggle("slow", "linear"); bCanPreview = true; }); }); As you can see – there are only 64 lines of our colorpicker, so, as usual, in the beginning we create new canvas and context objects, then – draw an color wheel on the context. As you see – there is small switch case to select desired image (of colorwheel), I decided to use a new attribute for canvas object: ‘var’. So, you can easily change this colorwheel with different ‘var’ value, example: or or or or Well, finally, we have to add event handlers to next events: mousemove (by picker), click (by picker) and click (by preview). As you remember we have to display and hide color picker when we click at Preview element. In order to achieve it – I use ‘fadeToggle’ jQuery function (which was added in version 1.4.4): $('.preview').click(function(e) { // preview click $('.colorpicker').fadeToggle("slow", "linear"); bCanPreview = true; }); When we move our mouse over the Picker object – we should refresh information about current color, and, once we click at the Picker object – we should fix current color (or – disable preview by mousemove): $('#picker').mousemove(function(e) { // mouse move handler if (bCanPreview) { // get coordinates of current position var canvasOffset = $(canvas).offset(); var canvasX = Math.floor(e.pageX - canvasOffset.left); var canvasY = Math.floor(e.pageY - canvasOffset.top); // get current pixel var imageData = ctx.getImageData(canvasX, canvasY, 1, 1); var pixel = imageData.data; // update preview color var pixelColor = "rgb("+pixel[0]+", "+pixel[1]+", "+pixel[2]+")"; $('.preview').css('backgroundColor', pixelColor); // update controls $('#rVal').val(pixel[0]); $('#gVal').val(pixel[1]); $('#bVal').val(pixel[2]); $('#rgbVal').val(pixel[0]+','+pixel[1]+','+pixel[2]); var dColor = pixel[2] + 256 * pixel[1] + 65536 * pixel[0]; $('#hexVal').val('#' + ('0000' + dColor.toString(16)).substr(-6)); } }); $('#picker').click(function(e) { // click event handler bCanPreview = !bCanPreview; }); Step 3. CSS There are CSS styles of our color picker: /* colorpicker styles */ .colorpicker { background-color: #222222; border-radius: 5px 5px 5px 5px; box-shadow: 2px 2px 2px #444444; color: #FFFFFF; font-size: 12px; position: absolute; width: 460px; } #picker { cursor: crosshair; float: left; margin: 10px; border: 0; } .controls { float: right; margin: 10px; } .controls > div { border: 1px solid #2F2F2F; margin-bottom: 5px; overflow: hidden; padding: 5px; } .controls label { float: left; } .controls > div input { background-color: #121212; border: 1px solid #2F2F2F; color: #DDDDDD; float: right; font-size: 10px; height: 14px; margin-left: 6px; text-align: center; text-transform: uppercase; width: 75px; } .preview { background: url("../images/select.png") repeat scroll center center transparent; border-radius: 3px; box-shadow: 2px 2px 2px #444444; cursor: pointer; height: 30px; width: 30px; } Live Demo 1 Live Demo 2 Live Demo 3 Live Demo 4 Live Demo 5 download in package Conclusion We have just created our own small and effective color picker with HTML5 (canvas). I hope that you like it. I will be glad to see your questions and comments. Good luck!

In Groovy we can use the drop() and take() methods to get elements from a collection or String object. Since Groovy 1.8.7 we also can use the dropWhile() and takeWhile() methods and use a closure to define a condition to stop dropping or taking elements. With the dropWhile() method we drop elements or characters until the condition in the closure is true. And the takeWhile() method returns elements from a collection or characters from a String until the condition of the closure is true. In the following example we see how we can use the methods: def s = "Groovy Rocks!" assert s.takeWhile { it != 'R' } == 'Groovy ' assert s.dropWhile { it != 'R' } == 'Rocks!' def list = 0..10 assert 0..4 == list.takeWhile { it < 5 } assert 5..10 == list.dropWhile { it < 5 } def m = [name: 'mrhaki', loves: 'Groovy', worksAt: 'JDriven'] assert [name: 'mrhaki'] == m.takeWhile { key, value -> key.length() == 4 } assert [loves: 'Groovy', worksAt: 'JDriven'] == m.dropWhile { it.key == 'name' } (Code is written with Groovy 2.0.4)

MongoDB version 2.2 was released in late August and the biggest change it brought was the addition of the Aggregation Framework. Previously the aggregations required the usage of map/reduce, which in MongoDB doesn’t perform that well, mainly because of the single-threaded Javascript-based execution. The aggregation framework steps away from the Javascript and is implemented in C++, with an aim to accelerate performance of analytics and reporting up to 80 percent compared to using MapReduce. The aim of this post is to show examples of running the MongoDB Aggregation Framework with the official MongoDB C# drivers. Aggregation Framework and Linq Even though the current version of the MongoDB C# drivers (1.6) supports Linq, the support doesn’t extend to the aggregation framework. It’s highly probable that the Linq-support will be added later on and there’s already some hints about this in the driver’s source code. But at this point the execution of the aggregations requires the usage of the BsonDocument-objects. Aggregation Framework and GUIDs If you use GUIDs in your documents, the aggregation framework doesn’t work. This is because by default the GUIDs are stored in binary format and the aggregations won’t work against documents which contain binary data.. The solution is to store the GUIDs as strings. You can force the C# drivers to make this conversion automatically by configuring the mapping. Given that your C# class has Id-property defined as a GUID, the following code tells the driver to serialize the GUID as a string: BsonClassMap.RegisterClassMap(cm => { cm.AutoMap(); cm.GetMemberMap(c => c.Id) .SetRepresentation( BsonType.String); }); The example data These examples use the following documents: > db.examples.find() { "_id" : "1", "User" : "Tom", "Country" : "Finland", "Count" : 1 } { "_id" : "2", "User" : "Tom", "Country" : "Finland", "Count" : 3 } { "_id" : "3", "User" : "Tom", "Country" : "Finland", "Count" : 2 } { "_id" : "4", "User" : "Mary", "Country" : "Sweden", "Count" : 1 } { "_id" : "5", "User" : "Mary", "Country" : "Sweden", "Count" : 7 } Example 1: Aggregation Framework Basic usage This example shows how the aggregation framework can be executed through C#. We’re not going run any calculations to the data, we’re just going to filter it by the User. To run the aggregations, you can use either the MongoDatabase.RunCommand –method or the helper MongoCollection.Aggregate. We’re going to use the latter: var coll = localDb.GetCollection("examples"); ... coll.Aggregate(pipeline); The hardest part when working with Aggregation Framework through C# is building the pipeline. The pipeline is similar concept to the piping in PowerShell. Each operation in the pipeline will make modifications to the data: the operations can for example filter, group and project the data. In C#, the pipeline is a collection of BsonDocument object. Each document represents one operation. In our first example we need to do only one operation: $match. This operator will filter out the given documents. The following BsonDocument is a pipeline operation which filters out all the documents which don’t have User-field set to “Tom”. var match = new BsonDocument { { "$match", new BsonDocument { {"User", "Tom"} } } }; To execute this operation we add it to an array and pass the array to the MongoCollection.Aggregate-method: var pipeline = new[] { match }; var result = coll.Aggregate(pipeline); The MongoCollection.Aggregate-method returns an AggregateResult-object. It’s ResultDocuments-property (IEnumarable) contains the documents which are the output of the aggregation. To check how many results there were, we can get the Count: var result = coll.Aggregate(pipeline); Console.WriteLine(result.ResultDocuments.Count()); The result documents are BsonDocument-objects. If you have a C#-class which represent the documents, you can cast the results: var matchingExamples = result.ResultDocuments .Select(BsonSerializer.Deserialize) .ToList(); foreach (var example in matchingExamples) { var message = string.Format("{0} - {1}", example.User, example.Count); Console.WriteLine(message); } Another alternative is to use C#’s dynamic type. The following extension method uses JSON.net to convert a BsonDocument into a dynamic: public static class MongoExtensions { public static dynamic ToDynamic(this BsonDocument doc) { var json = doc.ToJson(); dynamic obj = JToken.Parse(json); return obj; } } Here’s a way to convert all the result documents into dynamic objects: var matchingExamples = result.ResultDocuments .Select(x => x.ToDynamic()) .ToList(); Example 2: Multiple filters & comparison operators This example filters the data with the following criteria: User: Tom Count: >= 2 var match = new BsonDocument { { "$match", new BsonDocument { {"User", "Tom"}, {"Count", new BsonDocument { { "$gte", 2 } } } } }; The execution of this operation is identical to the first example: var pipeline = new[] { match }; var result = coll.Aggregate(pipeline); var matchingExamples = result.ResultDocuments .Select(x => x.ToDynamic()) .ToList(); Also the result are as expected: foreach (var example in matchingExamples) { var message = string.Format("{0} - {1}", example.User, example.Count); Console.WriteLine(message); } Example 3: Multiple operations In our first two examples, the pipeline was as simple as possible: It contained only one operation. This example will filter the data with the same exact criteria as the second example, but this time using two $match operations: User: Tom Count: >= 2 var match = new BsonDocument { { "$match", new BsonDocument { {"User", "Tom"} } } }; var match2 = new BsonDocument { { "$match", new BsonDocument { {"Count", new BsonDocument { { "$gte", 2 } } } } }; var pipeline = new[] { match, match2 }; The output stays the same: The first operation “match” takes all the documents from the examples collection and removes every document which doesn’t match the criteria User = Tom. The output of this operation (3 documents) then moves to the second operation “match2” of the pipeline. This operation only sees those 3 documents, not the original collection. The operation filters out these documents based on its criteria and moves the result (2 documents) forward. This is where our pipeline ends and this is also our result. Example 4: Group and sum Thus far we’ve used the aggregation framework to just filter out the data. The true strength of the framework is its ability to run calculations on the documents. This example shows how we can calculate how many documents there are in the collection, grouped by the user. This is done using the $group-operator: var group = new BsonDocument { { "$group", new BsonDocument { { "_id", new BsonDocument { { "MyUser","$User" } } }, { "Count", new BsonDocument { { "$sum", 1 } } } } } }; The grouping key (in our case the User-field) is defined with the _id. The above example states that the grouping key has one field (“MyUser”) and the value for that field comes from the document’s User-field ($User). In the $group operation the other fields are aggregate functions. This example defines the field “Count” and adds 1 to it for every document that matches the group key (_id). var pipeline = new[] { group }; var result = coll.Aggregate(pipeline); var matchingExamples = result.ResultDocuments .Select(x => x.ToDynamic()) .ToList(); foreach (var example in matchingExamples) { var message = string.Format("{0} - {1}", example._id.MyUser, example.Count); Console.WriteLine(message); } Note the format in which the results are outputted: The user’s name is accessed through _id.MyUser-property. Example 5: Group and sum by field This example is similar to example 4. But instead of calculating the amount of documents, we calculate the sum of the Count-fields by the user: var group = new BsonDocument { { "$group", new BsonDocument { { "_id", new BsonDocument { { "MyUser","$User" } } }, { "Count", new BsonDocument { { "$sum", "$Count" } } } } } }; The only change is that instead of adding 1, we add the value from the Count-field (“$Count”). Example 6: Projections This example shows how the $project operator can be used to change the format of the output. The grouping in example 5 works well, but to access the user’s name we currently have to point to the _id.MyUser-property. Let’s change this so that user’s name is available directly through UserName-property: var group = new BsonDocument { { "$group", new BsonDocument { { "_id", new BsonDocument { { "MyUser","$User" } } }, { "Count", new BsonDocument { { "$sum", "$Count" } } } } } }; var project = new BsonDocument { { "$project", new BsonDocument { {"_id", 0}, {"UserName","$_id.MyUser"}, {"Count", 1}, } } }; var pipeline = new[] { group, project }; The code removes the _id –property from the output. It adds the UserName-property, which value is accessed from field _id.MyUser. The projection operations also states that the Count-value should stay as it is. var matchingExamples = result.ResultDocuments .Select(x => x.ToDynamic()) .ToList(); foreach (var example in matchingExamples) { var message = string.Format("{0} - {1}", example.UserName, example.Count); Console.WriteLine(message); } Example 7: Group with multiple fields in the keys For this example we add a new row into our document collection, leaving us with the following: { "_id" : "1", "User" : "Tom", "Country" : "Finland", "Count" : 1 } { "_id" : "2", "User" : "Tom", "Country" : "Finland", "Count" : 3 } { "_id" : "3", "User" : "Tom", "Country" : "Finland", "Count" : 2 } { "_id" : "4", "User" : "Mary", "Country" : "Sweden", "Count" : 1 } { "_id" : "5", "User" : "Mary", "Country" : "Sweden", "Count" : 7 } { "_id" : "6", "User" : "Tom", "Country" : "England", "Count" : 3 } This example shows how you can group the data by using multiple fields in the grouping key: var group = new BsonDocument { { "$group", new BsonDocument { { "_id", new BsonDocument { { "MyUser","$User" }, { "Country","$Country" }, } }, { "Count", new BsonDocument { { "$sum", "$Count" } } } } } }; var project = new BsonDocument { { "$project", new BsonDocument { {"_id", 0}, {"UserName","$_id.MyUser"}, {"Country", "$_id.Country"}, {"Count", 1}, } } }; var pipeline = new[] { group, project }; var result = coll.Aggregate(pipeline); var matchingExamples = result.ResultDocuments .Select(x => x.ToDynamic()) .ToList(); foreach (var example in matchingExamples) { var message = string.Format("{0} - {1} - {2}", example.UserName, example.Country, example.Count); Console.WriteLine(message); } Example 8: Match, group and project This example shows how you can combine many different pipeline operations. The data is first filtered ($match) by User=Tom, then grouped by the Country (“$group”) and finally the output is formatted into a readable format ($project). Match: var match = new BsonDocument { { "$match", new BsonDocument { {"User", "Tom"} } } }; Group: var group = new BsonDocument { { "$group", new BsonDocument { { "_id", new BsonDocument { { "Country","$Country" }, } }, { "Count", new BsonDocument { { "$sum", "$Count" } } } } } }; Project: var project = new BsonDocument { { "$project", new BsonDocument { {"_id", 0}, {"Country", "$_id.Country"}, {"Count", 1}, } } }; Result: var pipeline = new[] { match, group, project }; var result = coll.Aggregate(pipeline); var matchingExamples = result.ResultDocuments .Select(x => x.ToDynamic()) .ToList(); foreach (var example in matchingExamples) { var message = string.Format("{0} - {1}", example.Country, example.Count); Console.WriteLine(message); } More There are many other interesting operators in the MongoDB Aggregation Framework, like $unwind and $sort. The usage of these operators is identical to ones we used above so it should be possible to copy-paste one of the examples and use it as a basis for these other operations. Links MongoDB C# Language Center MongoDB Aggregation Framework Easy to follow blog post about the aggregation framework

How to allow instance methods to run as JUnit BeforeClass behavior.

This is the third article in the series of "Become a Java GC Expert". In the first issue Understanding Java Garbage Collection we have learned about the processes for different GC algorithms, about how GC works, what Young and Old Generation is, what you should know about the 5 types of GC in the new JDK 7, and what the performance implications are for each of these GC types. In the second article How to Monitor Java Garbage Collection I have explained how JVM actually runs the Garbage Collection in the real time, how we can monitor GC, and which tools we can use to make this process faster and more effective. In this third article based on real cases as our examples I will show some of the best options you can use for GC tuning. I have written this article under the assumption that you have already understood the previous articles in this series. Therefore, for your further understanding, if you haven't already read the two previous articles, please do so before reading this one. Is GC Tuning Required? Or more precisely is GC tuning required for Java-based services? I should say GC tuning is not always required for all Java-based services. This means a Java-based system in operation has the following options and actions: The memory size has been specified using -Xms and –Xmx options. The -server option is included. Logs such as Timeout log are not left in the system. In other words, if you have not set the memory size and too many Timeout logs are printed, you need to perform GC tuning on your system. But, there is one thing to keep in mind: GC tuning is the last task to be done. Think about the fundamental cause of GC tuning. The Garbage Collector clears an object created in Java. The number of objects necessary to be cleared by the garbage collector as well as the number of GCs to be executed depend on the number of objects which have been created. Therefore, to control the GC performed by your system, you should, first, decrease the number of objects created. There is a saying, "many a little makes a mickle." We need to take care of small things, or they will add up and become something big which is difficult to manage. We need to use and make StringBuilder or StringBuffer a way of life instead of String. And it is better to accumulate as few logs as possible. However, we know that there are some cases we cannot help. We have seen that XML and JSON parsing use the most memory. Even though we use String as little as possible and process logs as well as we can, a huge temporary memory is used for parsing XML or JSON, some 10-100 MB. However, it is difficult not to use XML and JSON. Just understand that it takes too much memory. If application memory usage improves after repeated tunings, you can start GC tuning. I classify the purposes of GC tuning into two. One is to minimize the number of objects passed to the old area; and the other is to decrease Full GC execution time. Minimizing Number of Objects Passed to Old Area Generational GC is the GC provided by Oracle JVM, excluding the G1 GC which can be used from JDK 7 and higher versions. In other words, an object is created in the Eden area and transferred from and to the Survivor area. After that, the objects left are sent to the Old area. Some objects are created in the Eden area and directly passed to the Old area because of their large size. GC in the Old area takes relatively more time than the GC in the New area. Therefore, decreasing the number of objects passed to the Old area can decrease the full GC in frequency. Decreasing the number of objects passed to the Old area may be misunderstood as choosing to leave the object in the New area. However, this is impossible. Instead, you can adjust the size of the New area. Decreasing Full GC Time The execution time of Full GC is relatively longer than that of Minor GC. Therefore, if it takes too much time to execute Full GC (1 second or more), timeout may occur in several connected parts. If you try to decrease the Old area size to decrease Full GC execution time, OutOfMemoryError may occur or the number of Full GCs may increase. Alternatively, if you try to decrease the number of Full GC by increasing the Old area size, the execution time will be increased. Therefore, you need to set the Old area size to a "proper" value. Options Affecting the GC Performance As I have mentioned at the end of Understanding Java Garbage Collection, do not think that "Somebody's got a great performance when he used GC options. Why don't we use that option as he did?" The reason is that the size of objects created and their lifetime is different from one Web service to another. Simply consider, if a task is performed under the conditions of A, B, C, D and E, and the same task is performed under the conditions of only A and B, then which one will be done quicker? From a common-sense standpoint, the answer would be the task which is performed under conditions of A and B. Java GC options are the same. Setting several options does not enhance the speed of executing GC. Rather, it may make it slower. The basic principle of GC tuning is to apply the different GC options to two or more servers and compare them, and then add those options to the server for which the server has demonstrated enhanced performance or better GC time. Keep this in mind. The following table shows options related to memory size among the GC options that can affect performance. Table 1: JVM Options to Be Checked for GC Tuning. Classification Option Description Heap area size -Xms Heap area size when starting JVM -Xmx Maximum heap area size New area size -XX:NewRatio Ratio of New area and Old area -XX:NewSize New area size -XX:SurvivorRatio Ratio of Eden area and Survivor area I frequently use -Xms, -Xmx, and -XX:NewRatio options for GC tuning. -Xms and -Xmx option are particularly required. How you set the NewRatio option makes a significant difference on GC performance. Some people ask how to set the Perm area size? You can set the Perm area size with the -XX:PermSize and -XX:MaxPermSize options but only when OutOfMemoryError occurs and the cause is the Perm area size. Another option that may affect the GC performance is the GC type. The following table shows available options by GC type (based on JDK 6.0). Table 2: Available Options by GC Type. Classification Option Remarks Serial GC -XX:+UseSerialGC Parallel GC -XX:+UseParallelGC -XX:ParallelGCThreads=value Parallel Compacting GC -XX:+UseParallelOldGC CMS GC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=value -XX:+UseCMSInitiatingOccupancyOnly G1 -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC In JDK 6, these two options must be used together. Except G1 GC, the GC type is changed by setting the option at the first line of each GC type. The most general GC type that does not intrude is Serial GC. It is optimized for client systems. There are a lot of options that affect GC performance. But you can get significant effect by setting the options mentioned above. Remember that setting too many options does not promise enhanced GC execution time. Procedure of GC Tuning The procedure of GC tuning is similar to the general performance improvement procedure. The following is the GC tuning procedure that I use. 1. Monitoring GC status You need to monitor the GC status to check the GC status of the system in operation. Please see various GC monitoring methods in How to Monitor Java Garbage Collection. 2. Deciding whether to tune GC after analyzing the monitoring result After checking the GC status, you should analyze the monitoring result and decide whether to tune GC or not. If the analysis shows that the time taken to execute GC is just 0.1-0.3 seconds. you don't need to waste your time on tuning the GC. However, if the GC execution time is 1-3 seconds, or more than 10 seconds, GC tuning is necessary. But, if you have allocated about 10GB Java memory and it is impossible to decrease the memory size, there is no way to tune GC. Before tuning GC, you need to think about why you need to allocate large memory size. If you have allocated the memory of 1 GB or 2 GB and OutOfMemoryError occurs, you should execute heap dump to verify and remove the cause. Note: Heap dump is a file of the memory that is used to check the objects and data in the Java memory. This file can be created by using the jmap command included in the JDK. While creating the file, the Java process stops. Therefore, do not create this file while the system is operating. Search on the Internet the detailed description on heap dump. For Korean readers, see my book I published last year: The story of troubleshooting for Java developers and system operators (Sangmin Lee, Hanbit Media, 2011, 416 pages). 3. Setting GC type/memory size If you have decided on GC tuning, select the GC type and set the memory size. At this time, if you have several servers, it is important to check the difference of each GC option by setting different GC options for each server. 4. Analyzing results Start analyzing the results after collecting data for at least 24 hours after setting GC options. If you are lucky, you will find the most suitable GC options for the system. If you are not, you should analyze the logs and check how the memory has been allocated. Then you need to find the optimum options for the system by changing the GC type/memory size. 5. If the result is satisfactory, apply the option to all servers and terminate GC tuning. If the GC tuning result is satisfactory, apply the option to all the servers and terminate GC tuning. In the following section, you will see the tasks to be done in each stage. Monitoring GC Status and Analyzing Results The best way to check the GC status of the Web Application Server (WAS) in operation is to use the jstat command. I have explained the jstat command in How To Monitor Java Garbage Collection, so I will describe the data to check in this article. The following example shows a JVM for which GC tuning has not been done (however, it is not the operation server). $ jstat -gcutil 21719 1s S0 S1 E O P YGC YGCT FGC FGCT GCT 48.66 0.00 48.10 49.70 77.45 3428 172.623 3 59.050 231.673 48.66 0.00 48.10 49.70 77.45 3428 172.623 3 59.050 231.673 Here, check the values of YGC and YGCT. Divide YGCT by YGC. Then you get 0.050 seconds (50 ms). It means that it takes average 50 ms to execute GC in the Young area. With that result, you don't need to care about GC for the Young area. And now, check the values of FGCT and FGC. Divide FGCT by FGC. Then you get 19.68 seconds. It means that it takes average 19.68 seconds to execute GC. It may take 19.68 seconds to execute GC three times. Otherwise, it takes 1 second to execute GC two times and 58 seconds for once. In both cases, GC tuning is required. You can easily check GC status by using the jstat command; however, the best way to analyze GC is by generating logs with the –verbosegc option. For a detailed description on how to generate and tools to analyze logs, I have explained it the previous article. HPJMeter is my favorite among tools that are used to analyze the -verbosegc log. It is easy to use and analyze. With HPJmeter you can easily check the distribution of GC execution times and the frequency of GC occurrence. If the GC execution time meets all of the following conditions, GC tuning is not required. Minor GC is processed quickly (within 50 ms). Minor GC is not frequently executed (about 10 seconds). Full GC is processed quickly (within 1 second). Full GC is not frequently executed (once per 10 minutes). The values in parentheses are not the absolute values; they vary according to the service status. Some services may be satisfied with 0.9 seconds of Full GC processing speed, but some may not. Therefore, check the values and decide whether to execute GC tuning or not by considering each service. There is one thing you should be careful of when you check the GC status; do not check the time of Minor GC and Full GC only. You must check the number of GC executions, as well. If the New area size is too small, Minor GC will be too frequently executed (sometimes once or more per 1 second). In addition, the number of objects passed to the Old area increases, causing increased Full GC executions. Therefore, apply the –gccapacity option in the stat command to check how much the area is occupied. Setting GC Type/Memory Size Setting GC Type There are five GC types for Oracle JVM. However, if not JDK 7, one among Parallel GC, Parallel Compacting GC and CMS GC should be selected. There is no principle or rule to decide which one to select. If so, how can we select one? The most recommended way is to apply all three. However, one thing is clear - CMS GC is faster than other Parallel GCs. At this time, if so, just apply CMS GC. However, CMS GC is not always faster. Generally, Full GC of CMS GC is fast, however, when concurrent mode failure occurs, it is slower than other Parallel GCs. Concurrent mode failure Let's take a deeper look into the concurrent mode failure. The biggest difference between Parallel GC and CMS GC is the compaction task. The compaction task is to remove memory fragmentation by compacting memory in order to remove the empty space between allocated memory areas. In the Parallel GC type, the compaction is executed whenever Full GC is executed, taking too much time. However, after executing Full GC, memory can be allocated in a faster way since the next memory can be allocated sequentially. On the contrary, CMS GC does not accompany compaction. Therefore, the CMS GC is executed faster. However, when compaction is not executed, some empty spaces are generated in the memory as before executing Disk Defragmenter. Therefore, there may be no space for large objects. For example, 300 MB is left in the Old area, but some 10 MB objects cannot be sequentially saved in the area. In this case, "Concurrent mode failure" warning occurs and compaction is executed. However, if CMS GC is used, it takes a longer time to execute compaction than other Parallel GCs. And, it may cause another problem. For a more detailed description on concurrent mode failure, see Understanding CMS GC Logs, written by Oracle engineers. In conclusion, you should find the best GC type for your system. Each system requires its proper GC type, so you need to find the best GC type for your system. If you are running six servers, I recommend you to set the same options for each of two servers, add the -verbosegc option, and then analyze the result. Setting Memory Size The following shows the relationship between the memory size, the number of GC execution, and the GC execution time. Large memory size decreases the number of GC executions. increases the GC execution time. Small memory size decreases the GC execution time. increases the number of GC executions. There is no "right" answer to set the memory size to small or large. 10 GB is OK if the server resource is good and Full GC can be completed within 1 second even when the memory has been set to 10 GB. But most servers are not in the status. When the memory is set to 10 GB, it takes about 10 ~ 30 seconds to execute Full GC. Of course, the time may vary according the object size. If so, how we should set the memory size? Generally, I recommend 500 MB. But note that it does not mean that you should set the WAS memory with the –Xms500m and –Xmx500m options. Based on the current status before GC tuning, check the memory size left after Full GC. If there is about 300 MB left after Full GC, it is good to set the memory to 1 GB (300 MB (for default usage) + 500 MB (minimum for the Old area) + 200 MB (for free memory)). That means you should set the memory space with more than 500 MB for the Old area. Therefore, if you have three operation servers, set one server to 1 GB, one to 1.5 GB, and one to 2 GB, and then check the result. Theoretically, GC will be done fast in the order of 1 GB > 1.5 GB > 2 GB, so 1 GB will be the fastest to execute GC. However, it cannot be guaranteed that it takes 1 second to execute Full GC with 1 GB and 2 seconds with 2 GB. The time depends on the server performance and the object size. Therefore, the best way to create the measurement data is to set as many as possible and monitor them. You should set one more thing for setting the memory size: NewRatio. NewRatio is the ratio of the New area and the Old area. If XX:NewRatio=1, New area:Old area is 1:1. For 1 GB, New area:Old area is 500MB: 500MB. If NewRatio is 2, New area:Old area is 1:2. Therefore, as the value gets larger, the Old area size gets larger and the New area size gets smaller. It may not be an important thing, but NewRatio value significantly affects the entire GC performance. If the New area size is small, much memory is passed to the Old area, causing frequent Full GC and taking a long time to handle it. You may simply think that NewRatio 1 would be the best; however, it may not be so. When NewRatio is set to 2 or 3, the entire GC status may be better. And I have seen such cases. What is the fastest way to complete GC tuning? Comparing the results from performance tests is the fastest way to get the result. To set different options for each server and monitor the status, it is recommended to check the data after at least one or two days. However, you should prepare for giving the same load with the operation situation when you execute GC tuning through performance test. And the request ratio such as the URL that gives the load must be identical to that of the operation situation. However, giving accurate load is not easy for the professional performance tester and takes too long time for preparing. Therefore, it is more convenient and easier to apply the options to operation and wait for the result even though it takes a longer time. Analyzing GC Tuning Results After applying the GC option and setting the -verbosegc option, check whether the logs are accumulated as desired with the tail command. If the option is not exactly set and no log is accumulated, you will waste your time. If logs are accumulated as desired, check the result after collecting data for one or two days. The easiest way is to move logs to the local PC and analyze the data by using HPJMeter. In the analysis, focus on the following. The priority is determined by me. The most important item to decide the GC option is Full GC execution time. Full GC execution time Minor GC execution time Full GC execution interval Minor GC execution interval Entire Full GC execution time Entire Minor GC execution time Entire GC execution time Full GC execution times Minor GC execution timesl It is a very lucky case to find the most appropriate GC option, and in most cases, it's not. Be careful when executing GC tuning because OutOfMemoryError may occur if you try to complete GC tuning all at once. Examples of Tuning So far, we have theoretically discussed GC tuning without any examples. Now we will take a look at the examples of GC tuning. Example 1 The following example is GC tuning for Service S. For the newly developed Service S, it took too much time to execute Full GC. See the result of jstat –gcutil. S0 S1 E O P YGC YGCT FGC FGCT GCT 12.16 0.00 5.18 63.78 20.32 54 2.047 5 6.946 8.993 Information to the left Perm area is not important for the initial GC tuning. At this time, the values from the right YGC are important. The average value taken to execute Minor GC and Full GC once is calculated as below. Table 3: Average Time Taken to Execute Minor GC and Full GC for Service S. GC Type GC Execution Times GC Execution Time Average Minor GC 54 2.047 37 s Full GC 5 6.946 1,389 ms 37 ms is not bad for Minor GC. However, 1.389 seconds for Full GC means that timeout may frequently occur when GC occurs in the system of which DB Timeout is set to 1 second. In this case, the system requires GC tuning. First, you should check how the memory is used before starting GC tuning. Use the jstat –gccapacity option to check the memory usage. The result checked from this server is as follows. NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC PGCMN PGCMX PGC PC YGC FGC 212992.0 212992.0 212992.0 21248.0 21248.0 170496.0 1884160.0 1884160.0 1884160.0 1884160.0 262144.0 262144.0 262144.0 262144.0 54 5 The key values are as follows. New area usage size: 212,992 KB Old area usage size: 1,884,160 KB Therefore, the totally allocated memory size is 2 GB, excluding the Perm area, and New area:Old area is 1:9. To check the status in a more detailed way than jstat, the -verbosegc log has been added and three options were set for the three instances as shown below. No other option has been added. NewRatio=2 NewRatio=3 NewRatio=4 After one day, the GC log of the system has been checked. Fortunately, no Full GC has occurred in this system after NewRatio has been set. Why? The reason is that most of the objects created from the system are destroyed soon, so the objects are not passed to the Old area but destroyed in the New area. In this status, it is not necessary to change other options. Just select the best value for NewRatio. So, how can we determine the best value? To get it, analyze the average response time of Minor GC for each NewRatio. The average response time of Minor GC for each option is as follows: NewRatio=2: 45 ms NewRatio=3: 34 ms NewRatio=4: 30 ms We have concluded that NewRatio=4 is the best option since the GC time is the shortest even though the New area size is the smallest. After applying the GC option, the server has no Full GC. For your information, the following is the result of executing jstat –gcutil some days after the JVM of the service had started. S0 S1 E O P YGC YGCT FGC FGCT GCT 8.61 0.00 30.67 24.62 22.38 2424 30.219 0 0.000 30.219 You many think that GC has not frequently occurred since the server has few requests. However, Full GC has not been executed while Minor GC has been executed 2,424 times. Example 2 This example is for Service A. We found that the JVM had not operated for a long time (8 seconds or more) periodically in the Application Performance Manager (APM) in the company. So we executed GC tuning. We were searching for the reason and found that it took a long time to execute Full GC, so we decided to execute GC tuning. As the starting stage of GC tuning, we added the -verbosegc option and the result is as follows. Figure 1: Duration Graph before GC Tuning. The above graph, which shows the duration, is one of the graphs that the HPJMeter automatically provides after analysis. The X-axis shows the time after the JVM has started and the Y-axis shows the response time of each GC. The green dots, the CMS, indicates the Full GC result, and the blue bots, Parallel Scavenge, indicates the Minor GC result. Previous I said that CMS GC would be the fastest. But the above result show that there were some cases which took up to 15 seconds. What has caused such result? Please remember what I said before: CMS gets slower when compaction is executed. In addition, the memory of the service has been set by using –Xms1g and –Xmx4g and the memory allocated was 4 GB. So I changed the GC type from CMS GC to Parallel GC. I changed the memory size to 2 GB and then set the NewRatio to 3. The result of jstat –gcutil after a few hours is as follows. S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 30.48 3.31 26.54 37.01 226 11.131 4 11.758 22.890 The Full GC time was faster, 3 seconds per one time, compared to 15 seconds for 4 GB. However, 3 seconds is still not so fast. So I created six cases as follows. Case 1: -XX:+UseParallelGC -Xms1536m -Xmx1536m -XX:NewRatio=2 Case 2: -XX:+UseParallelGC -Xms1536m -Xmx1536m -XX:NewRatio=3 Case 3: -XX:+UseParallelGC -Xms1g -Xmx1g -XX:NewRatio=3 Case 4: -XX:+UseParallelOldGC -Xms1536m -Xmx1536m -XX:NewRatio=2 Case 5: -XX:+UseParallelOldGC -Xms1536m -Xmx1536m -XX:NewRatio=3 Case 6: -XX:+UseParallelOldGC -Xms1g -Xmx1g -XX:NewRatio=3 Which one would be the fastest? The result showed that the smaller the memory size was, the better the result was. The following figure shows the duration graph of Case 6, which showed the highest GC improvement. The slowest response time was 1.7 seconds and the average had been changed to within 1 second, showing the improved result. Figure 2: Duration Graph after Applying Case 6. With the result, I changed all GC options of the service to Case 6. However, this change causes OutOfMemoryError at night each day. It is difficult to detail the reason here, but in short, batch data processing made a lack of JVM memory. The related problems are being cleared now. It is very dangerous to analyze the GC logs accumulated for a short time and to apply the result to all servers as executing GC tuning. Keep in mind that GC tuning can be executed without failure only when you analyze the service operation as well as the GC logs. We have reviewed two GC tuning examples to see how GC tuning is executed. As I mentioned, the GC option set in the examples can be identically set for the server which has the same CPU, OS version and JDK version with the service that executes the same functions. However, do not apply the option I did to your services in operation, since they may not work for you. Conclusion I execute GC tuning based on my experiences without executing heap dump and analyzing the memory in detail. Precise memory status analysis may draw the better GC tuning results. However, that kind of analysis may be helpful when the memory is used in the constant and routine pattern. But, if the service is heavily used and there are a lot of memory usage patterns, GC tuning based on reliable previous experience may be recommendable. I have executed the performance test by setting the G1 GC option to some servers, but have not applied to any operation server yet. The G1 GC option shows a faster result than any other GC types. However, it requires to upgrade to JDK 7. In addition, stability is still not guaranteed. Nobody knows if there is any critical bug or not. So the time is not yet ripe for applying the option. After JDK 7 is stabilized (this does not mean that it is not stable) and WAS is optimized for JDK 7, enabling stable application of G1 GC may finally work as expected and some day we may not need the GC tuning. For more detail on GC tuning, search on Slideshare.com for related materials. The most recommendable material is Everything I Ever Learned About JVM Performance Tuning @Twitter, written by Attila Szegedi, a Twitter engineer. Please take the time to read it.

JUnit 4.4 added a new assertion mechanism with the method assertThat(). Have a look and consider using it in place of assertEquals(). assertThat(result, is(42)); assertThat(output, containsString("foo"));

Curator's note: This article originally appeared at WindowsAzure.com. To use this feature and other new Windows Azure capabilities, sign up for the free preview. Just as you can quickly create and deploy a web application created from the gallery, you can also deploy a website created on a workstation with traditional developer tools from Microsoft or other companies. Table of Contents Deployment Options How to: Create a Website Using the Management Portal How to: Create a Website from the Gallery How to: Delete a Website Next Steps Deployment Options Windows Azure supports deploying websites from remote computers using WebDeploy, FTP, GIT or TFS. Many development tools provide integrated support for publication using one or more of these methods and may only require that you provide the necessary credentials, site URL and hostname or URL for your chosen deployment method. Credentials and deployment URLs for all enabled deployment methods are stored in the website's publish profile, a file which can be downloaded in the Windows Azure (Preview) Management Portal from the Quick Start page or the quick glance section of the Dashboard page. If you prefer to deploy your website with a separate client application, high quality open source GIT and FTP clients are available for download on the Internet for this purpose. How to: Create a Website Using the Management Portal Follow these steps to create a website in Windows Azure. Login to the Windows Azure (Preview) Management Portal. Click the Create New icon on the bottom left of the Management Portal. Click the Web Site icon, click the Quick Create icon, enter a value for URL and then click the check mark next to create web site on the bottom right corner of the page. When the website has been created you will see the text Creation of Web Site '[SITENAME]' Completed. Click the name of the website displayed in the list of websites to open the website's Quick Start management page. On the Quick Start page you are provided with options to set up TFS or GIT publishing if you would like to deploy your finished website to Windows Azure using these methods. FTP publishing is set up by default for websites and the FTP Host name is displayed under FTP Hostname on the Quick Start and Dashboard pages. Before publishing with FTP or GIT choose the option to Reset deployment credentials on the Dashboard page. Then specify the new credentials (username and password) to authenticate against the FTP Host or the Git Repository when deploying content to the website. The Configure management page exposes several configurable application settings in the following sections: Framework: Set the version of .NET framework or PHP required by your web application. Diagnostics: Set logging options for gathering diagnostic information for your website in this section. App Settings: Specify name/value pairs that will be loaded by your web application on start up. For .NET sites, these settings will be injected into your .NET configuration AppSettings at runtime, overriding existing settings. For PHP and Node sites these settings will be available as environment variables at runtime. Connection Strings: View connection strings for linked resources. For .NET sites, these connection strings will be injected into your .NET configuration connectionStrings settings at runtime, overriding existing entries where the key equals the linked database name. For PHP and Node sites these settings will be available as environment variables at runtime. Default Documents: Add your web application's default document to this list if it is not already in the list. If your web application contains more than one of the files in the list then make sure your website's default document appears at the top of the list. How to: Create a Website from the Gallery The gallery makes available a wide range of popular web applications developed by Microsoft, third party companies, and open source software initiatives. Web applications created from the gallery do not require installation of any software other than the browser used to connect to the Windows Azure Management Portal. In this tutorial, you'll learn: How to create a new site through the gallery. How to deploy the site through the Windows Azure Portal. You'll build a Word press blog that uses a default template. The following illustration shows the completed application: Note To complete this tutorial, you need a Windows Azure account that has the Windows Azure Web Sites feature enabled. You can create a free trial account and enable preview features in just a couple of minutes. For details, see Create a Windows Azure account and enable preview features. Create a web site in the portal Login to the Windows Azure Management Portal. Click the New icon on the bottom left of the dashboard. Click the Web Site icon, and click From Gallery. Locate and click the WordPress icon in list, and then click Next. On the Configure Your App page, enter or select values for all fields: Enter a URL name of your choice Leave Create a new MySQL database selected in the Database field Select the region closest to you Then click Next. On the Create New Database page, you can specify a name for your new MySQL database or use the default name. Select the region closest to you as the hosting location. Select the box at the bottom of the screen to agree to ClearDB's usage terms for your hosted MySQL database. Then click the check to complete the site creation. After you click Complete Windows Azure will initiate build and deploy operations. While the web site is being built and deployed the status of these operations is displayed at the bottom of the Web Sites page. After all operations are performed, A final status message when the site has been successfully deployed. Launch and manage your WordPress site Click on your new site from the Web Sites page to open the dashboard for the site. On the Dashboard management page, scroll down and click the link on the left under Site Url to open the site’s welcome page. Enter appropriate configuration information required by WordPress and click Install WordPress to finalize configuration and open the web site’s login page. Login to the new WordPress web site by entering the username and password that you specified on the Welcome page. You'll have a new WordPress site that looks similar to the site below. How to: Delete a Website Websites are deleted using the Delete icon in the Windows Azure Management Portal. The Delete icon is available in the Windows Azure Portal when you click Web Sites to list all of your websites and at the bottom of each of the website management pages. Next Steps For more information about Websites, see the following: Walkthrough: Troubleshooting a Website on Windows Azure

Tony Hoare introduced null references in ALGOL W back in 1965 “simply because it was so easy to implement”. After many years he regretted his decision calling it "my billion dollar mistake". Unfortunately the vast majority of the languages created in the last decades have been built with the same wrong design decision so language designers and software engineers started to look for workarounds to avoid the infamous NullPointerException. Functional languages like Haskell or Scala structurally resolve this problem by wrapping the nullable values in an Option/Maybe monad. Other imperative languages like Groovy introduced a null-safe dereferencing operator (?. operator) to safely navigate values that could be potentially null. A similar feature has been proposed (and then discarded) as part of the project Coin in Java 7. Honestly I don't miss a null safe dereferencing operator in Java even because I can imagine that the majority of developers would start abusing it "just in case". Moreover, since the upcoming Java 8 will have lambda expressions, it will be straightforward to implement an Option monad that, as I hope to show in the remaining part of the post, is a far more powerful and flexible construct. I don't want to delve in category theory and explain what a monad is, even because there are already tons of very goodarticlesdoing this. My purpose is to quickly implement an Option monad using the Java 8 lambda expression syntax and then show how to use it with a very practical example. In Scala, a monad M is any class having the following 3 methods: def map[B](f: A => B): M[B] def flatMap[B](f: A => M[B]): M[B] def filter(p: A => Boolean): M[A] In particular you can think to an Option monad as a wrapper around a, possibly absent, value. So an Option of a generic type A could be define as it follows: import java.util.functions.Predicate; public abstract class Option { public static final None NONE = new None(); public abstract Option map(Func1 f); public abstract Option flatMap(Func1> f); public abstract Option filter(Predicate predicate); public abstract A getOrElse(A def); public static Some some(A value) { return new Some(value); } public static None none() { return NONE; } public static Option asOption(A value) { if (value == null) return none(); else return some(value); } } I also added some convenient factory methods for the Some and None concrete implementations of Option that I will implement later. Here Predicate is a single method interface defined in the new java.util.functions package: public interface Predicate { boolean test(T t); } that is used to determine if the input object matches a given criteria, while Func1 is another single method interface: public interface Func1 { R apply(A1 arg1); } that I defined to represent a more generic function of one argument of type A1 returning a result of type R. The abstract class Option has then two concrete implementations, one representing the absence of a value (something that we are used to wrongly model with the infamous null reference): public class None extends Option { None() { } @Override public Option map(Func1 f) { return NONE; } @Override public Option flatMap(Func1> f) { return NONE; } @Override public Option filter(Predicate predicate) { return NONE; } @Override public A getOrElse(A def) { return def; } } and the other wrapping an actually existing value: public class Some extends Option { private final A value; Some(A value) { this.value = value; } @Override public Option map(Func1 f) { return some(f.apply(value)); } @Override public Option flatMap(Func1> f) { return f.apply(value); } @Override public Option filter(Predicate predicate) { if (predicate.test(value)) return this; else return None.NONE; } @Override public A getOrElse(A def) { return value; } } Now, to try to put the Option at work with a concrete example, let's suppose we have a Map representing a set of named parameters with the corresponding values. We want to develop the method int readPositiveIntParam(Map params, String name) { // TODO ... } that, if the value associated with a given key is a String representing a positive integer returns that integer, but returns zero in all other case. In other words we want the following test to pass: @Test public void testMap() { Map param = new HashMap(); param.put("a", "5"); param.put("b", "true"); param.put("c", "-3"); // the value of the key "a" is a String representing a positive int so return it assertEquals(5, readPositiveIntParam(param, "a")); // returns zero since the value of the key "b" is not an int assertEquals(0, readPositiveIntParam(param, "b")); // returns zero since the value of the key "c" is an int but it is negative assertEquals(0, readPositiveIntParam(param, "c")); // returns zero since there is no key "d" in the map assertEquals(0, readPositiveIntParam(param, "d")); } If we couldn't rely on our Option we should accomplish this task with something similar to this: int readPositiveIntParam(Map params, String name) { String value = params.get(name); if (value == null) return 0; int i = 0; try { i = Integer.parseInt(value); } catch (NumberFormatException nfe) { } if (i < 0) return 0; return i; } too many conditional branches and returning points, isn't it? Using the Option monad we can achieve the same result with a single fluent statement: int readPositiveIntParam(Map params, String name) { return asOption(params.get(name)) .flatMap(FunctionUtils::stringToInt) .filter(i -> i > 0) .getOrElse(0); } where we used an helper static method FunctionUtils.stringToInt() as a function literal, with the :: syntax also introduced in Java 8, defined as: import static Option.*; public class FunctionUtils { public static Option stringToInt(String s) { try { return some(Integer.parseInt(s)); } catch (NumberFormatException nfe) { return none(); } } } This methods tries to convert a String in an int and, if it can't, it returns the None Option. Note that we could also define this behavior inline, while invoking the flatMap() method, using an anonymous lambda expression, but my advice is to develop a small library of utility functions, as I started doing here, in order to leverage the grater reusability allowed by functional programming. I think the comparison of the two readPositiveIntParam methods I provided illustrates well how the extensive use of the Option monad can finally allow us to write completely NullPointerException free software and, more in general, how a bigger employment of functional programming can dramatically reduce its cyclomatic complexity.

SOAP headers can be added to a Web service request in different ways, if you use Apache CXF. The way I prefer is the one I’ve mentioned here – as it doesn’t require changes to wsdl or method signatures and it’s much faster as it doesn’t break streaming and the memory overhead is less. The headers in the list are streamed at the appropriate time to the wire according to the databinding object found in the Header object. About SOAP headers, Like any good messaging protocol, SOAP defines the concept of a message header. It is an optional part of any SOAP message. If it exists, the header contains application-specific information (like authentication, payment, etc) about the SOAP message i.e. information about the message, or about the context in which the message is sent, or basically whatever the creator of the message thought was a good idea to put there instead of the actual body of the message. If the Header element is present, it must be the first child element of the Envelope element. /** * @author Singaram Subramanian * */ /* Create a ClientProxyFactoryBean reference and assign it an instance of JaxWsProxyFactoryBean, a factory for creating JAX-WS proxies. This class provides access to the internal properties used to set-up proxies. Using it provides more control than the standard JAX-WS APIs. */ ClientProxyFactoryBean factory = new JaxWsProxyFactoryBean(); factory.setServiceClass(singz.ws.cxf.sample.SampleServiceInterface.class); // Set the web service endpoint URL here factory.setAddress("http://xxx.xxx.com/services/SampleService/v1"); SampleServiceInterface serviceClient = (SampleServiceInterface) factory.create(); // Get the underlying Client object from the proxy object of service interface Client proxy = ClientProxy.getClient(serviceClient); // Creating SOAP headers to the web service request // Create a list for holding all SOAP headers List headersList = new ArrayList(); Header testSoapHeader1 = new Header(new QName("uri:singz.ws.sample", "soapheader1"), "SOAP Header Message 1", new JAXBDataBinding(String.class)); Header testSoapHeader2 = new Header(new QName("uri:singz.ws.sample", "soapheader2"), "SOAP Header Message 2", new JAXBDataBinding(String.class)); headersList.add(testSoapHeader1); headersList.add(testSoapHeader2); // Add SOAP headers to the web service request proxy.getRequestContext().put(Header.HEADER_LIST, headersList); More on this @ http://cxf.apache.org/faq.html#FAQ-HowcanIaddsoapheaderstotherequest%2Fresponse%3F

Exception Testing Why test exception flows? Just like with all of your code, test coverage writes a contract between your code and the business functionality that the code is supposed to produce leaving you with a living documentation of the code along with the added ability to stress the functionality early and often. I won't go into the many benefits of testing instead I will focus on just Exception Testing. There are many ways to test an exception flow thrown from a piece of code. Lets say that you have a guarded method that requires an argument to be not null. How would you test that condition? How do you keep JUnit from reporting a failure when the exception is thrown? This blog covers a few different methods culminating with JUnit's ExpectedException implemented with JUnit's @Rule functionality. The "old" way In a not so distant past the process to test an exception required a dense amount of boilerplate code in which you would start a try/catch block, report a failure if your code did not produce the expected behavior and then catch the exception looking for the specific type. Here is an example: public class MyObjTest { @Test public void getNameWithNullValue() { try { MyObj obj = new MyObj(); myObj.setName(null); fail("This should have thrown an exception"); } catch (IllegalArgumentException e) { assertThat(e.getMessage().equals("Name must not be null")); } } } As you can see from this old example, many of the lines in the test case are just to support the lack of functionality present to specifically test exception handling. One good point to make for the try/catch method is the ability to test the specific message and any custom fields on the expected exception. We will explore this a bit further down with JUnit's ExpectedException and @Rule annotation. JUnit adds expected exceptions JUnit responded back to the users need for exception handling by adding a @Test annotation field "expected". The intention is that the entire test case will pass if the type of exception thrown matched the exception class present in the annotation. public class MyObjTest { @Test(expected = IllegalArgumentException.class) public void getNameWithNullValue() { MyObj obj = new MyObj(); myObj.setName(null); } } As you can see from the newer example, there is quite a bit less boiler plate code and the test is very concise, however, there are a few flaws. The main flaw is that the test condition is too broad. Suppose you have two variables in a signature and both cannot be null, then how do you know which variable the IllegalArgumentException was thrown for? What happens when you have extended a Throwable and need to check for the presence of a field? Keep these in mind as you read further, solutions will follow. JUnit @Rule and ExpectedException If you look at the previous example you might see that you are expecting an IllegalArgumentException to be thrown, but what if you have a custom exception? What if you want to make sure that the message contains a specific error code or message? This is where JUnit really excelled by providing a JUnit @Rule object specifically tailored to exception testing. If you are unfamiliar with JUnit @Rule, read the docs here. ExpectedException JUnit provides a JUnit class ExpectedException intended to be used as a @Rule. The ExpectedException allows for your test to declare that an exception is expected and gives you some basic built in functionality to clearly express the expected behavior. Unlike the @Test(expected) annotation feature, ExpectedException class allows you to test for specific error messages and custom fields via the Hamcrest matchers library. An example of JUnit's ExpectedException import org.junit.rules.ExpectedException; public class MyObjTest { @Rule public ExpectedException thrown = ExpectedException.none(); @Test public void getNameWithNullValue() { thrown.expect(IllegalArgumentException.class); thrown.expectMessage("Name must not be null"); MyObj obj = new MyObj(); obj.setName(null); } } As I eluded to above, the framework allows you to test for specific messages ensuring that the exception being thrown is the case that the test is specifically looking for. This is very helpful when the nullability of multiple arguments is in question. Custom Fields Arguably the most useful feature of the ExpectedException framework is the ability to use Hamcrest matchers to test your custom/extended exceptions. For example, you have a custom/extended exception that is to be thrown in a method and inside the exception has an "errorCode". How do you test that functionality without introducing the boiler plate code from the try/catch block listed above? How about a custom Matcher! This code is available at: https://github.com/mike-ensor/custom-exception-testing Solution: First the test case import org.junit.rules.ExpectedException; public class MyObjTest { @Rule public ExpectedException thrown = ExpectedException.none(); @Test public void someMethodThatThrowsCustomException() { thrown.expect(CustomException.class); thrown.expect(CustomMatcher.hasCode("110501")); MyObj obj = new MyObj(); obj.methodThatThrowsCustomException(); } } Solution: Custom matcher import com.thepixlounge.exceptions.CustomException; import org.hamcrest.Description; import org.hamcrest.TypeSafeMatcher; public class CustomMatcher extends TypeSafeMatcher { public static BusinessMatcher hasCode(String item) { return new BusinessMatcher(item); } private String foundErrorCode; private final String expectedErrorCode; private CustomMatcher(String expectedErrorCode) { this.expectedErrorCode = expectedErrorCode; } @Override protected boolean matchesSafely(final CustomException exception) { foundErrorCode = exception.getErrorCode(); return foundErrorCode.equalsIgnoreCase(expectedErrorCode); } @Override public void describeTo(Description description) { description.appendValue(foundErrorCode) .appendText(" was not found instead of ") .appendValue(expectedErrorCode); } } NOTE: Please visit https://github.com/mike-ensor/custom-exception-testing to get a copy of a working Hamcrest Matcher, JUnit @Rule and ExpectedException. And there you have it, a quick overview of different ways to test Exceptions thrown by your code along with the ability to test for specific messages and fields from within custom exception classes. Please be specific with your test cases and try to target the exact case you have setup for your test, remember, tests can save you from introducing side-effect bugs!

For my own interest I started exploring Spring REST Services with GWT. It took some time to figure it out and then I came accross RestyGWT. With the help of RestyGWT, I managed to integrate GWT with Spring REST Services. My idea was to keep the GWT way of creating the service and serviceAsync and yet make a REST call. Below are the steps which will help achieve above. My preffered development environment is Eclipse, so as a prerequisite you must have Eclipse with Maven support installed. Lets Begin, Create a Maven project, goto File-->New-->Other. In the wizard type "Maven". Select Maven Project and click on Next. In the "Select project name and location" page of the wizard, make sure that "Create a simple project (skip archetype selection)" option is checked, hit "Next" to continue with default values. In the "Enter group id for the artifact" page of the wizard, enter values for group id and artifactid. Select the packaging as "war", hit "Finish" to exit the wizard and to create your project. Modify the POM file to add required dependencies as below 4.0.0 com.sagar.restgwt RestGWT 0.0.1-SNAPSHOT war com.google.gwt gwt-servlet ${gwt.version} runtime com.google.gwt gwt-user ${gwt.version} provided org.fusesource.restygwt restygwt 1.2 javax.ws.rs jsr311-api 1.1 provided org.codehaus.jackson jackson-mapper-asl 1.4.1 org.springframework spring-core ${org.springframework.version} org.springframework spring-web ${org.springframework.version} org.springframework spring-webmvc ${org.springframework.version} . 2.5.0-rc1 1.6 3.1.1.RELEASE UTF-8 ${project.build.directory}\${project.build.finalName} restgwt ${webappDirectory}/WEB-INF/classes org.codehaus.mojo gwt-maven-plugin 2.5.0-rc1 compile test generateAsync ${webappDirectory} org.apache.maven.plugins maven-compiler-plugin ${java-version} ${java-version} true "Update Project Configuration" by Right clicking on your project-->Maven. The below steps assumes that your are aware of the GWT project structure. Create your GWT module. This can be done by installing GWT plugin for Eclipse. Once the GWT module is ready update your .gwt.xml with the below given content. In the client package create the service to make REST Call. With this approach we dont have to create the ServiceAsync interface. We will be creating our service interface by extending the "RestService", provided by RestyGWT. Code Snippet: InfoService.Java [ A service interface to make REST call. ] @Path("/service") public interface InfoService extends RestService { public static class Util { private static InfoService instance; public static InfoService getService() { if (instance == null) { instance = GWT.create(InfoService.class); } Resource resource = new Resource(GWT.getModuleBaseURL() + "service"); ((RestServiceProxy) instance).setResource(resource); return instance; } } @GET @Path("/loadInfo") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) public void getInfo(MethodCallback callback); } OrderConfirmation.java [ Model which will be returned as a response. ] public class OrderConfirmation { public String message; public Long ready_time; /** * Example of how to create an instance of a JsonEncoderDecoder for a data * transfer object. */ public interface OrderConfirmationJED extends JsonEncoderDecoder { } @Override public String toString() { if (GWT.isClient()) { OrderConfirmationJED jed = GWT.create(OrderConfirmationJED.class); return jed.encode(this).toString(); } return super.toString(); } } RestGWT.java [ GWT module entrypoint to see things running. ] public class RestGWT implements EntryPoint { public void onModuleLoad() { Button button = new Button("Click Me"); button.addClickHandler(new ClickHandler() { @Override public void onClick(ClickEvent event) { InfoService.Util.getService().getInfo(new MethodCallback() { @Override public void onSuccess(Method method, OrderConfirmation response) { RootPanel.get().add(new Label(response.toString())); } @Override public void onFailure(Method method, Throwable exception) { GWT.log("Error"); } }); } }); RootPanel.get().add(button); } } Create the Spring managed controller as below, this should be presnt in the "server" package as per GWT project structure. Also you can notice that we don't have to implement our service interface. RestGWTController.java [ Spring managed controller. ] @Controller public class RestGWTController { @RequestMapping(value = "/loadInfo", method = RequestMethod.GET, headers = "Accept=application/json") public @ResponseBody OrderConfirmation handleRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { GreetingServiceEndpoint endpoint = greetingService.getGreetingServiceEndpointPort(); OrderConfirmation confirmation = new OrderConfirmation(); confirmation.message = endpoint.sayHello(); confirmation.ready_time = System.currentTimeMillis() + 1000 * 60 * 30; return confirmation; } } Settings required for Spring src/main/webapp/WEB-INF/web.xml Rest GWT This is web-project for RestGWT contextConfigLocation /WEB-INF/applicationContext.xml org.springframework.web.context.ContextLoaderListener Spring MVC Dispatcher Servlet org.springframework.web.servlet.DispatcherServlet contextConfigLocation /WEB-INF/classes/action-servlet.xml 1 Spring MVC Dispatcher Servlet /restgwt/service/* RestGWT.html src/main/webapp/WEB-INF/applicationCotext.xml src/main/webapp/WEB-INF/classes/action-servlet.xml Now all configuration is completed. To build the application right click on your project → Run As → Maven Install. This will create the war file in your project's target/restgwt folder. To test this approach we are going to deploy our web application to an Apache Tomcat 7 server. To launch the application point your browser to the following address http://localhost:8080/restgwt/ Enjoy Coding Nilabh

If you look around the web for examples of using Spring 3.1’s built in caching then you’ll usually bump into Spring’s SimpleCacheManager, which the Guys at Spring say is “Useful for testing or simple caching declarations”. I actually prefer to think of SimpleCacheManager as lightweight rather than simple; useful in those situations where you want a small in memory cache on a per JVM basis. If the Guys at Spring were running a supermarket then SimpleCacheManagerwould be in their own brand ‘basics’ product range. If, on the other hand, you need a heavy duty cache, one that’s scalable, persistent and distributed, then Spring also comes with a built in ehCache wrapper. The good news is that swapping between Spring's caching implementations is easy. In theory it’s all a matter of configuration and, to prove the theory correct, I took the sample code from my Caching and @Cacheable blog and ran it using an EhCache implementation. The configuration steps are similar to those described in my last blog Caching and Config in that you still need to specify: ...in your Spring config file to switch caching on. You also need to define a bean with an id of cacheManager, only this time you reference Spring’s EhCacheCacheManager class instead of SimpleCacheManager. The example above demonstrates an EhCacheCacheManager configuration. Notice that it references a second bean with an id of 'ehcache'. This is configured as follows: "ehcache" has two properties: configLocation and shared. 'configLocation' is an optional attribute that’s used to specify the location of an ehcache configuration file. In my test code I used the following example file: ...which creates two caches: a default cache and one named “employee”. If this file is missing then the EhCacheManagerFactoryBean simply picks up a default ehcache config file: ehcache-failsafe.xml, which is located in ehcache’s ehcache-core jar file. The other EhCacheManagerFactoryBean attribute is 'shared'. This is supposed to be optional as the documentation states that it defines "whether the EHCache CacheManager should be shared (as a singleton at the VM level) or independent (typically local within the application). Default is 'false', creating an independent instance.” However, if this is set to false then you’ll get the following exception: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cache.interceptor.CacheInterceptor#0': Cannot resolve reference to bean 'cacheManager' while setting bean property 'cacheManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheManager' defined in class path resource [ehcache-example.xml]: Cannot resolve reference to bean 'ehcache' while setting bean property 'cacheManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehcache' defined in class path resource [ehcache-example.xml]: Invocation of init method failed; nested exception is net.sf.ehcache.CacheException: Another unnamed CacheManager already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following: 1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary 2. Shutdown the earlier cacheManager before creating new one with same name. The source of the existing CacheManager is: InputStreamConfigurationSource [stream=java.io.BufferedInputStream@424c414] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) ... stack trace shortened for clarity at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cacheManager' defined in class path resource [ehcache-example.xml]: Cannot resolve reference to bean 'ehcache' while setting bean property 'cacheManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehcache' defined in class path resource [ehcache-example.xml]: Invocation of init method failed; nested exception is net.sf.ehcache.CacheException: Another unnamed CacheManager already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following: 1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary 2. Shutdown the earlier cacheManager before creating new one with same name. The source of the existing CacheManager is: InputStreamConfigurationSource [stream=java.io.BufferedInputStream@424c414] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) ... stack trace shortened for clarity at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322) ... 38 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ehcache' defined in class path resource [ehcache-example.xml]: Invocation of init method failed; nested exception is net.sf.ehcache.CacheException: Another unnamed CacheManager already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following: 1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary 2. Shutdown the earlier cacheManager before creating new one with same name. The source of the existing CacheManager is: InputStreamConfigurationSource [stream=java.io.BufferedInputStream@424c414] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322) ... 48 more Caused by: net.sf.ehcache.CacheException: Another unnamed CacheManager already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following: 1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary 2. Shutdown the earlier cacheManager before creating new one with same name. The source of the existing CacheManager is: InputStreamConfigurationSource [stream=java.io.BufferedInputStream@424c414] at net.sf.ehcache.CacheManager.assertNoCacheManagerExistsWithSameName(CacheManager.java:521) at net.sf.ehcache.CacheManager.init(CacheManager.java:371) at net.sf.ehcache.CacheManager.(CacheManager.java:339) at org.springframework.cache.ehcache.EhCacheManagerFactoryBean.afterPropertiesSet(EhCacheManagerFactoryBean.java:104) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452) ... 55 more ...when you try to run a bunch of unit tests. I think that this comes down to a simple bug Spring’s the ehcache manager factory as it’s trying to create multiple cache instances using new() rather than using, as the exception states, “one of the CacheManager.create() static factory methods" which allows it to reuse same CacheManager with same name. Hence, my first JUnit test works okay, but all others fail. The offending line of code is: this.cacheManager = (this.shared ? CacheManager.create() : new CacheManager()); My full XML config file is listed below for completeness: In using ehcache, the only other configuration details to consider are the Maven dependencies. These are pretty straight forward as the Guys at Ehcache have combined all the various ehcache jars into one Maven POM module. This POM module can be added to your project's POM file using the XML below: net.sf.ehcache ehcache 2.6.0 pom test Finally, the ehcache Jar files are available from both the Maven Central and Sourceforge repositories: sourceforge http://oss.sonatype.org/content/groups/sourceforge/ true true

there are many different web api standards that are turning the web browser into a complete application platform. with websockets we get nice asynchronous communication, various standards allow us access to sensors in laptops and mobile devices and we can even determine how full the battery is. one of the standards i'm really interested in is webrtc. with webrtc we can get real-time audio and video communication between browsers without needing plugins or additional tools. a couple of months ago i wrote about how you can use webrtc to access the webcam and use it for face recognition . at that time, none of the browser allowed you to access the microphone. a couple of months later though, and both the developer version of firefox and developer version of chrome, allow you to access the microphone! so let's see what we can do with this. most of the examples i've seen so far focus on processing the input directly, within the browser, using the web audio api . you get synthesizers, audio visualizations, spectrometers etc. what was missing, however, was a means of recording the audio data and storing it for further processing at the server side. in this article i'll show you just that. i'm going to show you how you can create the following (you might need to enlarge it to read the response from the server): in this screencast you can see the following: a simple html page that access your microphone the speech is recorded and using websockets is sent to a backend the backend combines the audio data and sends it to google's speech to text api the result from this api call is returned to the browser and all this is done without any plugins in the browser! so what's involved to accomplish all this. allowing access to your microphone the first thing you need to do is make sure you've got an up to date version of chrome. i use the dev build, and am currently on this version: since this is still an experimental feature we need to enable this using the chrome flags. make sure the "web audio input" flag is enabled. with this configuration out of the way we can start to access our microphone. access the audio stream from the microphone this is actually very easy: function callback(stream) { var context = new webkitaudiocontext(); var mediastreamsource = context.createmediastreamsource(stream); ... } $(document).ready(function() { navigator.webkitgetusermedia({audio:true}, callback); ... } as you can see i use the webkit prefix functions directly, you could, of course, also use a shim so it is browser independent. what happens in the code above is rather straightforward. we ask, using getusermedia, for access to the microphone. if this is successful our callback gets called with the audio stream as its parameter. in this callback we use the web audio specification to create a mediastreamsource from our microphone. with this mediastreamsource we can do all the nice web audio tricks you can see here . but we don't want that, we want to record the stream and send it to a backend server for further processing. in future versions this will probably be possible directly from the webrtc api, at this time, however, this isn't possible yet. luckily, though, we can use a feature from the web audio api to get access to the raw data. with the javascriptaudionode we can create a custom node, which we can use to access the raw data (which is pcm encoded). before i started my own work on this i searched around a bit and came across the recoder.js project from here: https://github.com/mattdiamond/recorderjs . matt created a recorder that can record the output from web audio nodes, and that's exactly what i needed. all i needed to do now was connect the stream we just created to the recorder library: function callback(stream) { var context = new webkitaudiocontext(); var mediastreamsource = context.createmediastreamsource(stream); rec = new recorder(mediastreamsource); } with this code, we create a recorder from our stream. this recorder provides the following functions: record: start recording from the input stop: stop recording clear: clear the current recording exportwav: export the data as a wav file connect the recorder to the buttons i've created a simple webpage with an output for the text and two buttons to control the recording: the 'record' button starts the recording, and once you hit the 'export' button the recording stops, and is sent to the backend for processing. record button: $('#record').click(function() { rec.record(); ws.send("start"); $("#message").text("click export to stop recording and analyze the input"); // export a wav every second, so we can send it using websockets intervalkey = setinterval(function() { rec.exportwav(function(blob) { rec.clear(); ws.send(blob); }); }, 1000); }); this function (using jquery to connect it to the button) when clicked starts the recording. it also uses a websocket (ws), see further down on how to setup the websocket, to indicate to the backend server to expect a new recording (more on this later). finally when the button is clicked an interval is created that passes the data to the backend, encoded as wav file, every second. we do this to avoid sending too large chunks of data to the backend and improve performance. export button: $('#export').click(function() { // first send the stop command rec.stop(); ws.send("stop"); clearinterval(intervalkey); ws.send("analyze"); $("#message").text(""); }); the export button, bad naming i think when i'm writing this, stops the recording, the interval and informs the backend server that it can send the received data to the google api for further processing. connecting the frontend to the backend to connect the webapplication to the backend server we use websockets. in the previous code fragments you've already seen how they are used. we create them with the following: var ws = new websocket("ws://127.0.0.1:9999"); ws.onopen = function () { console.log("openened connection to websocket"); }; ws.onmessage = function(e) { var jsonresponse = jquery.parsejson(e.data ); console.log(jsonresponse); if (jsonresponse.hypotheses.length > 0) { var bestmatch = jsonresponse.hypotheses[0].utterance; $("#outputtext").text(bestmatch); } } we create a connection, and when we receive a message from the backend we just assume it contains the response to our speech analysis. and that's it for the complete front end of the application. we use getusermedia to access the microphone, use the web audio api to get access to the raw data and communicate with websockets with the backend server. the backend server our backend server needs to do a couple of things. it first needs to combine the incoming chunks to a single audio file, next it needs to convert this to a format google apis expect, which is flac. finally we make a call to the google api and return the response. i've used jetty as the websocket server for this example. if you want to know the details about setting this up, look at the facedetection example. in this article i'll only show the code to process the incoming messages. first step, combine the incoming data the data we receive is encoded as wav (thanks to the recorder.js library we don't have to do this ourselves). in our backend we thus receive sound fragments with a length of one second. we can't just concatenate these together, since wav files have a header that tells how long the fragment is (amongst other things), so we have to combine them, and rewrite the header. lets first look at the code (ugly code, but works good enough for now :) public void onmessage(byte[] data, int offset, int length) { if (currentcommand.equals("start")) { try { // the temporary file that contains our captured audio stream file f = new file("out.wav"); // if the file already exists we append it. if (f.exists()) { log.info("adding received block to existing file."); // two clips are used to concat the data audioinputstream clip1 = audiosystem.getaudioinputstream(f); audioinputstream clip2 = audiosystem.getaudioinputstream(new bytearrayinputstream(data)); // use a sequenceinput to cat them together audioinputstream appendedfiles = new audioinputstream( new sequenceinputstream(clip1, clip2), clip1.getformat(), clip1.getframelength() + clip2.getframelength()); // write out the output to a temporary file audiosystem.write(appendedfiles, audiofileformat.type.wave, new file("out2.wav")); // rename the files and delete the old one file f1 = new file("out.wav"); file f2 = new file("out2.wav"); f1.delete(); f2.renameto(new file("out.wav")); } else { log.info("starting new recording."); fileoutputstream fout = new fileoutputstream("out.wav",true); fout.write(data); fout.close(); } } catch (exception e) { ...} } } this method gets called for each chunk of audio we receive from the browser. what we do here is the following: first, we check whether we have a temp audio file, if not we create it if the file exists we use java's audiosystem to create an audio sequence this sequence is then written to another file the original is deleted and the new one is renamed. we repeat this for each chunk so at this point we have a wav file that keeps on growing for each added chunk. now before we convert this, lets look at the code we use to control the backend. public void onmessage(string data) { if (data.startswith("start")) { // before we start we cleanup anything left over cleanup(); currentcommand = "start"; } else if (data.startswith("stop")) { currentcommand = "stop"; } else if (data.startswith("clear")) { // just remove the current recording cleanup(); } else if (data.startswith("analyze")) { // convert to flac ... // send the request to the google speech to text service ... } } the previous method responded to binary websockets messages. the one shown above responds to string messages. we use this to control, from the browser, what the backend should do. let's look at the analyze command, since that is the interesting one. when this command is issued from the frontend the backend needs to convert the wav file to flac and send it to the google service. convert to flac for the conversion to flac we need an external library since java standard has no support for this. i used the javaflacencoder from here for this. // get an encoder flac_fileencoder flacencoder = new flac_fileencoder(); // point to the input file file inputfile = new file("out.wav"); file outputfile = new file("out2.flac"); // encode the file log.info("start encoding wav file to flac."); flacencoder.encode(inputfile, outputfile); log.info("finished encoding wav file to flac."); easy as that. now we got a flac file that we can send to google for analysis. send to google for analysis a couple of weeks ago i ran across an article that explained how someone analyzed chrome and found out about an undocumented google api you can use for speech to text. if you post a flac file to this url: https://www.google.com/speech-api/v1/recognize?xjerr=1&client=chromium&l... you receive a response like this: { "status": 0, "id": "ae466ffa24a1213f5611f32a17d5a42b-1", "hypotheses": [ { "utterance": "the quick brown fox", "confidence": 0.857393 }] } to do this from java code, using httpclient, you do the following: // send the request to the google speech to text service log.info("sending file to google for speech2text"); httpclient client = new defaulthttpclient(); httppost p = new httppost(url); p.addheader("content-type", "audio/x-flac; rate=44100"); p.setentity(new fileentity(outputfile, "audio/x-flac; rate=44100")); httpresponse response = client.execute(p); f (response.getstatusline().getstatuscode() == 200) { log.info("received valid response, sending back to browser."); string result = new string(ioutils.tobytearray(response.getentity().getcontent())); this.connection.sendmessage(result); } and that are all the steps that are needed.

Explore SQL query optimization and normalization.

As you may have seen from my previous tutorials and case studies, Java Heap Space OutOfMemoryError problems can be complex to pinpoint and resolve. One of the common problems I have observed from Java EE production systems is OutOfMemoryError: unable to create new native thread; error thrown when the HotSpot JVM is unable to further create a new Java thread. This article will revisit this HotSpot VM error and provide you with recommendations and resolution strategies. If you are not familiar with the HotSpot JVM, I first recommend that you look at a high level view of its internal HotSpot JVM memory spaces. This knowledge is important in order for you to understand OutOfMemoryError problems related to the native (C-Heap) memory space. OutOfMemoryError: unable to create new native thread – what is it? Let’s start with a basic explanation. This HotSpot JVM error is thrown when the internal JVM native code is unable to create a new Java thread. More precisely, it means that the JVM native code was unable to create a new “native” thread from the OS (Solaris, Linux, MAC, Windows...). We can clearly see this logic from the OpenJDK 1.6 and 1.7 implementations as per below: Unfortunately at this point you won’t get more detail than this error, with no indication of why the JVM is unable to create a new thread from the OS… HotSpot JVM: 32-bit or 64-bit? Before you go any further in the analysis, one fundamental fact that you must determine from your Java or Java EE environment is which version of HotSpot VM you are using e.g. 32-bit or 64-bit. Why is it so important? What you will learn shortly is that this JVM problem is very often related to native memory depletion; either at the JVM process or OS level. For now please keep in mind that: A 32-bit JVM process is in theory allowed to grow up to 4 GB (even much lower on some older 32-bit Windows versions). For a 32-bit JVM process, the C-Heap is in a race with the Java Heap and PermGen space e.g. C-Heap capacity = 2-4 GB – Java Heap size (-Xms, -Xmx) – PermGen size (-XX:MaxPermSize) A 64-bit JVM process is in theory allowed to use most of the OS virtual memory available or up to 16 EB (16 million TB) As you can see, if you allocate a large Java Heap (2 GB+) for a 32-bit JVM process, the native memory space capacity will be reduced automatically, opening the door for JVM native memory allocation failures. For a 64-bit JVM process, your main concern, from a JVM C-Heap perspective, is the capacity and availability of the OS physical, virtual and swap memory. OK great but how does native memory affect Java threads creation? Now back to our primary problem. Another fundamental JVM aspect to understand is that Java threads created from the JVM requires native memory from the OS. You should now start to understand the source of your problem… The high level thread creation process is as per below: A new Java thread is requested from the Java program & JDK The JVM native code then attempt to create a new native thread from the OS The OS then attempts to create a new native thread as per attributes which include the thread stack size. Native memory is then allocated (reserved) from the OS to the Java process native memory space; assuming the process has enough address space (e.g. 32-bit process) to honour the request The OS will refuse any further native thread & memory allocation if the 32-bit Java process size has depleted its memory address space e.g. 2 GB, 3 GB or 4 GB process size limit The OS will also refuse any further Thread & native memory allocation if the virtual memory of the OS is depleted (including Solaris swap space depletion since thread access to the stack can generate a SIGBUS error, crashing the JVM * http://bugs.sun.com/view_bug.do?bug_id=6302804 In summary: Java threads creation require native memory available from the OS; for both 32-bit & 64-bit JVM processes For a 32-bit JVM, Java thread creation also requires memory available from the C-Heap or process address space Problem diagnostic Now that you understand native memory and JVM thread creation a little better, is it now time to look at your problem. As a starting point, I suggest that your follow the analysis approach below: Determine if you are using HotSpot 32-bit or 64-bit JVM When problem is observed, take a JVM Thread Dump and determine how many Threads are active Monitor closely the Java process size utilization before and during the OOM problem replication Monitor closely the OS virtual memory utilization before and during the OOM problem replication; including the swap memory space utilization if using Solaris OS Proper data gathering as per above will allow you to collect the proper data points, allowing you to perform the first level of investigation. The next step will be to look at the possible problem patterns and determine which one is applicable for your problem case. Problem pattern #1 – C-Heap depletion (32-bit JVM) From my experience, OutOfMemoryError: unable to create new native thread is quite common for 32-bit JVM processes. This problem is often observed when too many threads are created vs. C-Heap capacity. JVM Thread Dump analysis and Java process size monitoring will allow you to determine if this is the cause. Problem pattern #2 – OS virtual memory depletion (64-bit JVM) In this scenario, the OS virtual memory is fully depleted. This could be due to a few 64-bit JVM processes taking lot memory e.g. 10 GB+ and / or other high memory footprint rogue processes. Again, Java process size & OS virtual memory monitoring will allow you to determine if this is the cause. Problem pattern #3 – OS virtual memory depletion (32-bit JVM) The third scenario is less frequent but can still be observed. The diagnostic can be a bit more complex but the key analysis point will be to determine which processes are causing a full OS virtual memory depletion. Your 32-bit JVM processes could be either the source or the victim such as rogue processes using most of the OS virtual memory and preventing your 32-bit JVM processes to reserve more native memory for its thread creation process. Please note that this problem can also manifest itself as a full JVM crash (as per below sample) when running out of OS virtual memory or swap space on Solaris. # # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space? # # Internal Error (allocation.cpp:166), pid=2290, tid=27 # Error: ChunkPool::allocate # # JRE version: 6.0_24-b07 # Java VM: Java HotSpot(TM) Server VM (19.1-b02 mixed mode solaris-sparc ) # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x003fa800): JavaThread "CompilerThread1" daemon [_thread_in_native, id=27, stack(0x65380000,0x65400000)] Stack: [0x65380000,0x65400000], sp=0x653fd758, free space=501k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) ……………… Native memory depletion: symptom or root cause? You now understand your problem and know which problem pattern you are dealing with. You are now ready to provide recommendations to address the problem…are you? Your work is not done yet, please keep in mind that this JVM OOM event is often just a “symptom” of the actual root cause of the problem. The root cause is typically much deeper so before providing recommendations to your client I recommend that you really perform deeper analysis. The last thing you want to do is to simply address and mask the symptoms. Solutions such as increasing OS physical / virtual memory or upgrading all your JVM processes to 64-bit should only be considered once you have a good view on the root cause and production environment capacity requirements. The next fundamental question to answer is how many threads were active at the time of the OutOfMemoryError? In my experience with Java EE production systems, the most common root cause is actually the application and / or Java EE container attempting to create too many threads at a given time when facing non happy paths such as thread stuck in a remote IO call, thread race conditions etc. In this scenario, the Java EE container can start creating too many threads when attempting to honour incoming client requests, leading to increase pressure point on the C-Heap and native memory allocation. Bottom line, before blaming the JVM, please perform your due diligence and determine if you are dealing with an application or Java EE container thread tuning problem as the root cause. Once you understand and address the root cause (source of thread creations), you can then work on tuning your JVM and OS memory capacity in order to make it more fault tolerant and better “survive” these sudden thread surge scenarios. Recommendations: First perform a JVM Thread Dump analysis and determine the source of all the active threads vs. an established baseline. Determine what is causing your Java application or Java EE container to create so many threads at the time of the failure Please ensure that your monitoring tools closely monitor both your Java VM processes size & OS virtual memory. This crucial data will be required in order to perform a full root cause analysis Do not assume that you are dealing with an OS memory capacity problem. Look at all running processes and determine if your JVM processes are actually the source of the problem or victim of other processes consuming all the virtual memory Revisit your Java EE container thread configuration & JVM thread stack size. Determine if the Java EE container is allowed to create more threads than your JVM process and / or OS can handle Determine if the Java Heap size of your 32-bit JVM is too large, preventing the JVM to create enough threads to fulfill your client requests. In this scenario, you will have to consider reducing your Java Heap size (if possible), vertical scaling or upgrade to a 64-bit JVM Capacity planning analysis to the rescue As you may have seen from my past article on the Top 10 Causes of Java EE Enterprise Performance Problems, lack of capacity planning analysis is often the source of the problem. Any comprehensive load and performance testing exercise should also properly determine the Java EE container threads, JVM & OS native memory requirement for your production environment; including impact measurements of "non-happy" paths. This approach will allow your production environment to stay away from this type of problem and lead to better system scalability and stability in the long run. Please provide any comment and share your experience with JVM native thread troubleshooting.