recently some pwc tech supremos wrote an article: agile coding in enterprise it: code small and local . subsections: moving away from the monolith why microservices? msa: a think-small approach for rapid development thinking the msa way: minimalism is a must where msa makes sense in msa, integration is the problem, not the solution conclusion msa is short for microservices architecture(s), in the above article. the article posits that microservices is the antidote to monoliths. it doesn’t mention cookie cutter scaling at all, which is another antidote to monoliths, with the right build infrastructure and devops. here’s a view of hypothetical architecture a company could deploy if they were doing microservices: w is web server. p and q don’t stand for anything in particular. here’s the same solution as cookie-cutter scaling, and the alternate (historical) choice of monolith to the right of it: the cookie cutter approach will often leverage components that are dependency injected into each other, and though monoliths might be the same today, pre 2004 they were probably hairballs of singletons (the design patten, not the springframework idiom). continuous delivery, agile? here’s one excerpt that confuses me: " … makes no sense to design and develop software over an 18-month process to accommodate all possible use cases when those use cases can change unexpectedly and the life span of code modules might be less than 18 months…. as i recall, the 18 month-delay problem was solved previously. agile methodologies principally, and continuous delivery/deployment in more recent times. it does not matter whether you’re compiling a monolith, a cookie-cutter solution, old soa services, or microservices, the 18-month fear isn’t real if you’re doing agile and/or cd. agile and cd were increasing the release cadence, and allowing the organization to pivot faster before microservices. it doesn’t matter whether you’ve got a monolith, something cookie-cutter scaled, or soa (micro or not), you’re going to be able to benefit from agile practices and devops setup that facilitates cd. in something like 30 thoughtworks client engagements since 2002, i have not seen the 18-month process at all. in fact i last encountered it in 1997 on an as/400 project, which was the last time i saw a waterfall process being championed. build(s) and trunk elsewhere there is a suggestion: “each microservice [has] its own build, to avoid trunk conflict”. that isn’t unique to microservices, of course. component based systems today also have a multiple build file (module) structure in a source tree. hopefully “trunk” mentioned is alluding to trunk based development, as i would recommend. build technologies this is a expansion on the above, and you can skip this paragraph if you want. hierarchical build systems like maven has allow you to have one build file per module (whether that’s a service or a simple jar destined for the classpath of a bigger thing). buck has a build grammar that allows for a build to grow/shrink/change based on what is being built (from implicitly shared source). maven is for the java ecosystem, while buck promises to be multi-language. both are doing multi-module builds for the sake of a composed or servicified deployment. both maven and buck are presently competing to draw the most reduced set of compile/test/deploy operations for the changes since last build for a hierarchy of modules. anyway, what is it we are striving for? what we want is to develop cheaply, and to deploy smoothly and often, without defect. we want the ability to deploy without large permanent or temporary headcount overseeing or participating in deployment. aside from development costs, and support/operation, deployment costs are a potentially big factor in total cost of ownership. what i like about cookie-cutter is the uniformity of the deployable things. the team size for deployment of such a thing doesn’t grow with the numbers of nodes that binary is being deployed to. at least, if you’re able to automate the deployment to those nodes, and have a strategy for handling the users connected to the stack at redeployment time somehow (sessions or stateless). the uniformity of the deployment is a cheapener, i think. when you have a number of dissimilar services, you might be able to minimize release personnel if you’re only doing one service. if more than one service is being updated in a particular deployment, you’re going to have to concentrate to make sure you don’t experience a multiplier effect for the participants. it is possible of course, to keep the headcount small, but the practice needed beforehand is bigger, which in turn allows for some calmness around the actual deployment. if we’ve stepped away from the project management office thinking that suggests three buggy releases a year (which is more usual than 18 month schedules of old), then we can employ continuous deployment to further eliminate personnel costs around going live. this is something that microservices does well at, but because the most adept proponents design forwards & backwards compatibility into the permutations most likely to co-exist in production. it is at least much quicker to redeploy and bounce one small service, n times than the the cookie-cutter uniform deployment.

Originally written by Sébastien Deluze on the SpringSource blog Spring Jackson support has been improved lately to be more flexible and powerful. This blog post gives you an update about the most useful Jackson related features available in Spring Framework 4.x and Spring Boot. All the code samples are coming from this spring-jackson-demo sample application, feel free to have a look at the code. JSON Views It can sometimes be useful to filter contextually objects serialized to the HTTP response body. In order to provide such capabilities, Spring MVC now has builtin support for Jackson’s Serialization Views. The following example illustrates how to use @JsonView to filter fields depending on the context of serialization - e.g. getting a "summary" view when dealing with collections, and getting a full representation when dealing with a single resource: public class View { interface Summary {} } public class User { @JsonView(View.Summary.class) private Long id; @JsonView(View.Summary.class) private String firstname; @JsonView(View.Summary.class) private String lastname; private String email; private String address; private String postalCode; private String city; private String country; } public class Message { @JsonView(View.Summary.class) private Long id; @JsonView(View.Summary.class) private LocalDate created; @JsonView(View.Summary.class) private String title; @JsonView(View.Summary.class) private User author; private List recipients; private String body; } Thanks to Spring MVC @JsonView support, it is possible to choose, on a per handler method basis, which field should be serialized: @RestController public class MessageController { @Autowired private MessageService messageService; @JsonView(View.Summary.class) @RequestMapping("/") public List getAllMessages() { return messageService.getAll(); } @RequestMapping("/{id}") public Message getMessage(@PathVariable Long id) { return messageService.get(id); } } In this example, if all messages are retrieved, only the most important fields are serialized thanks to the getAllMessages() method annotated with@JsonView(View.Summary.class): [ { "id" : 1, "created" : "2014-11-14", "title" : "Info", "author" : { "id" : 1, "firstname" : "Brian", "lastname" : "Clozel" } }, { "id" : 2, "created" : "2014-11-14", "title" : "Warning", "author" : { "id" : 2, "firstname" : "Stéphane", "lastname" : "Nicoll" } }, { "id" : 3, "created" : "2014-11-14", "title" : "Alert", "author" : { "id" : 3, "firstname" : "Rossen", "lastname" : "Stoyanchev" } } ] In Spring MVC default configuration, MapperFeature.DEFAULT_VIEW_INCLUSION is set tofalse. That means that when enabling a JSON View, non annotated fields or properties likebody or recipients are not serialized. When a specific Message is retrieved using the getMessage() handler method (no JSON View specified), all fields are serialized as expected: { "id" : 1, "created" : "2014-11-14", "title" : "Info", "body" : "This is an information message", "author" : { "id" : 1, "firstname" : "Brian", "lastname" : "Clozel", "email" : "[email protected]", "address" : "1 Jaures street", "postalCode" : "69003", "city" : "Lyon", "country" : "France" }, "recipients" : [ { "id" : 2, "firstname" : "Stéphane", "lastname" : "Nicoll", "email" : "[email protected]", "address" : "42 Obama street", "postalCode" : "1000", "city" : "Brussel", "country" : "Belgium" }, { "id" : 3, "firstname" : "Rossen", "lastname" : "Stoyanchev", "email" : "[email protected]", "address" : "3 Warren street", "postalCode" : "10011", "city" : "New York", "country" : "USA" } ] } Only one class or interface can be specified with the @JsonView annotation, but you can use inheritance to represent JSON View hierarchies (if a field is part of a JSON View, it will be also part of parent view). For example, this handler method will serialize fields annotated with@JsonView(View.Summary.class) and @JsonView(View.SummaryWithRecipients.class): public class View { interface Summary {} interface SummaryWithRecipients extends Summary {} } public class Message { @JsonView(View.Summary.class) private Long id; @JsonView(View.Summary.class) private LocalDate created; @JsonView(View.Summary.class) private String title; @JsonView(View.Summary.class) private User author; @JsonView(View.SummaryWithRecipients.class) private List recipients; private String body; } @RestController public class MessageController { @Autowired private MessageService messageService; @JsonView(View.SummaryWithRecipients.class) @RequestMapping("/with-recipients") public List getAllMessagesWithRecipients() { return messageService.getAll(); } } JSON Views could also be specified when using RestTemplate HTTP client orMappingJackson2JsonView by wrapping the value to serialize in a MappingJacksonValue as shown in this code sample. JSONP As described in the reference documentation, you can enable JSONP for @ResponseBody andResponseEntity methods by declaring an @ControllerAdvice bean that extendsAbstractJsonpResponseBodyAdvice as shown below: @ControllerAdvice public class JsonpAdvice extends AbstractJsonpResponseBodyAdvice { public JsonpAdvice() { super("callback"); } } With such @ControllerAdvice bean registered, it will be possible to request the JSON webservice from another domain using a In this example, the received payload would be: parseResponse({ "id" : 1, "created" : "2014-11-14", ... }); JSONP is also supported and automatically enabled when using MappingJackson2JsonViewwith a request that has a query parameter named jsonp or callback. The JSONP query parameter name(s) could be customized through the jsonpParameterNames property. XML support Since 2.0 release, Jackson provides first class support for some other data formats than JSON. Spring Framework and Spring Boot provide builtin support for Jackson based XML serialization/deserialization. As soon as you include the jackson-dataformat-xml dependency to your project, it is automatically used instead of JAXB2. Using Jackson XML extension has several advantages over JAXB2: Both Jackson and JAXB annotations are recognized JSON View are supported, allowing you to build easily REST Webservices with the same filtered output for both XML and JSON data formats No need to annotate your class with @XmlRootElement, each class serializable in JSON will serializable in XML You usually also want to make sure that the XML library in use is Woodstox since: It is faster than Stax implementation provided with the JDK It avoids some known issues like adding unnecessary namespace prefixes Some features like pretty print don't work without it In order to use it, simply add the latest woodstox-core-asl dependency available to your project. Customizing the Jackson ObjectMapper Prior to Spring Framework 4.1.1, Jackson HttpMessageConverters were usingObjectMapper default configuration. In order to provide a better and easily customizable default configuration, a new Jackson2ObjectMapperBuilder has been introduced. It is the JavaConfig equivalent of the well known Jackson2ObjectMapperFactoryBean used in XML configuration. Jackson2ObjectMapperBuilder provides a nice API to customize various Jackson settings while retaining Spring Framework provided default ones. It also allows to createObjectMapper and XmlMapper instances based on the same configuration. Both Jackson2ObjectMapperBuilder and Jackson2ObjectMapperFactoryBean define a better Jackson default configuration. For example, theDeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES property set to false, in order to allow deserialization of JSON objects with unmapped properties. Jackson support for Java 8 Date & Time API data types is automatically registered when Java 8 is used and jackson-datatype-jsr310 is on the classpath. Joda-Time support is registered as well when jackson-datatype-joda is part of your project dependencies. These classes also allow you to register easily Jackson mixins, modules, serializers or even property naming strategy like PropertyNamingStrategy.CAMEL_CASE_TO_LOWER_CASE_WITH_UNDERSCORES if you want to have your userName java property translated to user_name in JSON. With Spring Boot As described in the Spring Boot reference documentation, there are various ways tocustomize the Jackson ObjectMapper. You can for example enable/disable Jackson features easily by adding properties likespring.jackson.serialization.indent_output=true to application.properties. As an alternative, in the upcoming 1.2 release Spring Boot also allows to customize the Jackson configuration (JSON and XML) used by Spring MVC HttpMessageConverters by declaring a Jackson2ObjectMapperBuilder @Bean: @Bean public Jackson2ObjectMapperBuilder jacksonBuilder() { Jackson2ObjectMapperBuilder builder = new Jackson2ObjectMapperBuilder(); builder.indentOutput(true).dateFormat(new SimpleDateFormat("yyyy-MM-dd")); return builder; } This is useful if you want to use advanced Jackson configuration not exposed through regular configuration keys. Without Spring Boot In a plain Spring Framework application, you can also use Jackson2ObjectMapperBuilder to customize the XML and JSON HttpMessageConverters as shown bellow: @Configuration @EnableWebMvc public class WebConfiguration extends WebMvcConfigurerAdapter { @Override public void configureMessageConverters(List> converters) { Jackson2ObjectMapperBuilder builder = new Jackson2ObjectMapperBuilder(); builder.indentOutput(true).dateFormat(new SimpleDateFormat("yyyy-MM-dd")); converters.add(new MappingJackson2HttpMessageConverter(builder.build())); converters.add(new MappingJackson2XmlHttpMessageConverter(builder.createXmlMapper(true).build())); } } More to come With the upcoming Spring Framework 4.1.3 release, thanks to the addition of a Spring context aware HandlerInstantiator (see SPR-10768 for more details), you will be able to autowire Jackson handlers (serializers, deserializers, type and type id resolvers). This will allow you to build, for example, a custom deserializer that will replace a field containing only a reference in the JSON payload by the full Entity retrieved from the database.

both high availability (ha) and disaster recovery (dr) have been essential it topics. fundamentally ha is about fault tolerance relevant to the availability of an examined subject like application, database, vms, etc. while dr roots on the ability to resume operations in the aftermath of a catastrophic event. a fundamental difference of these two is that ha expects no down time and no data loss, while dr does. they are different issues and should be addressed separately. background for many it shops, either ha or dr has been a high risk and high cost item. both are essential to business continuity, while traditionally tough technical problems to solve with very significant and long-term commitments on resources. not only they are technically challenging, but a continual cost-cutting which has become an it standard practice in the past two decades makes purchasing hardware/software and constructing either ha or dr solution on premises further distant from it’s financial and technical realties. sense of urgency too often, the technical challenges and resource commitments overwhelm it and turn ha and dr into academic discussions, or symbolic items on a project checklist. at the same time, information is rapidly exploding as internet, mobility and social-network are becoming integral in our daily lives and businesses. there are progressively more data to process and store. for many businesses, the needs for ha and dr is urgent for better managing risks. and continual availability and on-demand recoverability of it are becoming increasingly critical. this is the reality, now the good news is that the recent introduction of cloud computing has fundamentally changed how an ha or dr solution can be implemented. microsoft azure is a vivid example of ha and dr solutions with significantly reduced the required financial commitment and involved technical complexities. the traditional approach by establishing redundancy and acquiring a physical dr site with long-term resources and financial commitments is now largely replaced with consumable services which can be configured in minutes by mouse-clicking and with a manageable cost structure based on usage. ha and dr have become it solutions which are financially realistic and technically feasible for businesses in all sizes. ha, redundancy, and microsoft azure lrs ha is to eliminate a single point of failure of an examined component, an application for example. it denotes a strategy to employ redundancy such that a target application can and will continue being available without downtime while experiencing a failure of hosting hardware or software. there are various and well-developed ha solutions like a hyper-v host cluster using redundant hardware to eliminate a single point of failure of hosting os or hardware, and an application cluster for eliminating a single point of failure by running the application in multiple vm instances with a synchronous state. although ha implementations may vary, the fundamental principle nevertheless remains the same. ha expects neither downtime nor data loss while experiencing an outage of a target hardware or software. ha has become dramatically simple in microsoft azure. basically, all data written to disk in microsoft azure are kept at least in the so-called lrs, locally redundant storage. lrs replicates a transaction synchronously to three different storage nodes across fault domains and upgrade domains within the same region for durability. in layman’s terms, microsoft azure by default maintains at least three copies of user data to achieve ha. dr, replication, and microsoft azure grs dr is about having a plan and backups in place to resume operations in the aftermath of a catastrophic event. unplanned outage is assumed in a dr scenario, therefore some data loss is also expected. notice that ha and dr are different business problems and addressed differently. while both ha and dr are based on applying redundancy, i.e. a source and replicas, or multiple identical nodes of an examines component like application instance, databases, or vms, there are however differences between the two. a dr solution generally employs replicas or backups, are implemented with asynchronous processes, and expects an outage of a source and with some data loss in transit while the outage occurs. while ha requires a logical representation with a real-time integrity using synchronous processes across all participating nodes, expects neither downtime nor data loss while experiencing an outage of a participating node. for a critical workload, one approach of dr is to establish geo-replication to address an outage of an entire geographic area caused by a natural disaster, for example. the concern is that a catastrophic event may impact an entire geographic area causing a datacenter where a mission critical application is being hosted becomes unavailable for an extended period of time. in microsoft azure, geo redundant storage or grs is the default and an optional setting, as shown above, while configuring a storage account. grs will queue a transaction committed to lrs as an asynchronous replication to a secondary region, a few hundreds miles away from the primary region where a storage account is originated. at the secondary region, data is also stored in lrs, i.e. made durable by replicating it to three storage nodes. specifically, a microsoft azure storage account configured with grs essentially maintains three replicas locally for high availability, and replicates the content and maintains three replicas at a secondary datacenter a few hundreds miles away for dr. so all are six copies, three locally and three remotely. all these are configured by one, yes one mouse click from a dropdown list while creating a storage account. the above is a conceptual model illustrated a data flow of grs. grs replication has little performance impact on an application since application data are committed to lrs in real-time while replication to grs is queued, i.e. asynchronously. a write to lrs is synchronous and in real-time, once committed, the changes are expected within 15 minutes to be asynchronously replicated to the secondary site. for a ra-grs storage account, in addition to one primary endpoint for read/write operations as it is in a grs, there is also one secondary endpoint as read only becomes available as shown below. the cost implications of grs or ra-grs include the additional storage and the transmission costs for egress traffic, as applicable, of the secondary datacenter. ingress traffic is free . and microsoft azure storage sla offers 99.9% availability and a cost calculator is also available. microsoft azure recovery services so far, much is about backing up or replicating data. to successfully restore, a dr plan must be put in place and ensure its availability upon a dr scenario in progress. either placing a dr plan at a primary site where the source is or a secondary site where a replica stays has some issues and concerns. keeping a dr plan at the source site where all the resources are in place and on-the-job trainings seems logical. or does it? dr is assuming a catastrophic event over an extended geographic areas where the source site is experiencing an outage. in such case, keeping a dr plan in the source site defeats the purpose. maintaining a dr plan at the secondary site is the choice then. in a dr scenario, a recovery site is to be brought on line within a expected period of time according to a dr plan, and having the dr plan right there and then at a recovery site makes all the sense. or does it? this decision introduces a number of requirements including the physical readiness, the timeliness, and the financial implications on securing and maintaining a dr plan at a remote physical facility. for a vmm server running on system center 2012 sp1 or later, an idea, reliable and straightforward way is to use azure recovery services to maintain a dr plan as shown below. and for any backup needs, using cloud as a backup site makes backing up and restoring data an anytime anywhere operation. azure site recovery vault this service essentially acts as the director of a dr process. it orchestrates and manages the protection and failover of vms in clouds managed by virtual machine manager 2012 sp1 or later. a noticeable advantage is the ability to test a recovery configuration, exercise a proactive failover and recovery, and automate recovery in the event of a site outage. the sla of site recovery services is 99.9% availability to ensure a configured dr plan is always in place with expected updates. this is a dr solution that it can implement, simulate, verify, bring online and be absolutely confident with the readiness. azure backup vault this is a reliable, scalable and inexpensive data protection solution with zero capital investment and extremely low operational expense. like other secure communication with microsoft azure, you will first upload a public certificate to microsoft azure. then download the backup agent to register a target server with the backup vault. then select what to be backed up. both microsoft azure backup sla (99.9% availability) and cost calculator are available for better assessing the solution. closing thoughts form an application’s view, ha is an on-going event while dr is an anticipation. ha and dr are different business problems and should be addressed differently. nevertheless, microsoft azure provides a single platform to gracefully address ha with lrs, dr with grs, and dr orchestration with recovery services, and all with published sla s and a predictable cost structure . going forward, it pros can now include ha and dr as a reliable, scalable and relatively inexpensive proposition by employing microsoft azure as a solution platform. call to action register at microsoft virtual academy, http://aka.ms/mva1 , and train yourself on microsoft azure by taking the track of courses. go to http://aka.ms/azure200 and acquire a free trial subscription and assess microsoft azure for ha and dr solutions. review my recommended content at http://aka.ms/recommended .

Originally written by Artem Bilan on the SpringSource blog. Dear Spring Community! Recently we published the Spring Integration Java DSL: Line by line tutorial, which uses Java 8 Lambdas extensively. We received some feedback that this is good introduction to the DSL, but a similar tutorial is needed for those users, who can't move to the Java 8 or aren't yet familiar with Lambdas, but wish to take advantage So, to help those Spring Integration users who want to moved from XML configuration to Java & Annotation configuration, we provide this line-by-line tutorial to demonstrate that, even without Lambdas, we gain a lot from Spring Integration Java DSL usage. Although, most will agree that the lambda syntax provides for a more succinct definition. We analyse here the same Cafe Demo sample, but using the pre Java 8 variant for configuration. Many options are the same, so we just copy/paste their description here to achieve a complete picture. Since this Spring Integration Java DSL configuration is quite different to the Java 8 lambda style, it will be useful for all users to get a knowlage how we can achieve the same result with a rich variety of options provided by the Spring Integration Java DSL. The source code for our application is placed in a single class, which is a Boot application; significant lines are annotated with a number corresponding to the comments, which follow: @SpringBootApplication // 1 @IntegrationComponentScan // 2 public class Application { public static void main(String[] args) throws Exception { ConfigurableApplicationContext ctx = SpringApplication.run(Application.class, args); // 3 Cafe cafe = ctx.getBean(Cafe.class); // 4 for (int i = 1; i <= 100; i++) { // 5 Order order = new Order(i); order.addItem(DrinkType.LATTE, 2, false); order.addItem(DrinkType.MOCHA, 3, true); cafe.placeOrder(order); } System.out.println("Hit 'Enter' to terminate"); // 6 System.in.read(); ctx.close(); } @MessagingGateway // 7 public interface Cafe { @Gateway(requestChannel = "orders.input") // 8 void placeOrder(Order order); // 9 } private final AtomicInteger hotDrinkCounter = new AtomicInteger(); private final AtomicInteger coldDrinkCounter = new AtomicInteger(); // 10 @Autowired private CafeAggregator cafeAggregator; // 11 @Bean(name = PollerMetadata.DEFAULT_POLLER) public PollerMetadata poller() { // 12 return Pollers.fixedDelay(1000).get(); } @Bean @SuppressWarnings("unchecked") public IntegrationFlow orders() { // 13 return IntegrationFlows.from("orders.input") // 14 .split("payload.items", (Consumer) null) // 15 .channel(MessageChannels.executor(Executors.newCachedThreadPool()))// 16 .route("payload.iced", // 17 new Consumer>() { // 18 @Override public void accept(RouterSpec spec) { spec.channelMapping("true", "iced") .channelMapping("false", "hot"); // 19 } }) .get(); // 20 } @Bean public IntegrationFlow icedFlow() { // 21 return IntegrationFlows.from(MessageChannels.queue("iced", 10)) // 22 .handle(new GenericHandler() { // 23 @Override public Object handle(OrderItem payload, Map headers) { Uninterruptibles.sleepUninterruptibly(1, TimeUnit.SECONDS); System.out.println(Thread.currentThread().getName() + " prepared cold drink #" + coldDrinkCounter.incrementAndGet() + " for order #" + payload.getOrderNumber() + ": " + payload); return payload; // 24 } }) .channel("output") // 25 .get(); } @Bean public IntegrationFlow hotFlow() { // 26 return IntegrationFlows.from(MessageChannels.queue("hot", 10)) .handle(new GenericHandler() { @Override public Object handle(OrderItem payload, Map headers) { Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS); // 27 System.out.println(Thread.currentThread().getName() + " prepared hot drink #" + hotDrinkCounter.incrementAndGet() + " for order #" + payload.getOrderNumber() + ": " + payload); return payload; } }) .channel("output") .get(); } @Bean public IntegrationFlow resultFlow() { // 28 return IntegrationFlows.from("output") // 29 .transform(new GenericTransformer() { // 30 @Override public Drink transform(OrderItem orderItem) { return new Drink(orderItem.getOrderNumber(), orderItem.getDrinkType(), orderItem.isIced(), orderItem.getShots()); // 31 } }) .aggregate(new Consumer() { // 32 @Override public void accept(AggregatorSpec aggregatorSpec) { aggregatorSpec.processor(cafeAggregator, null); // 33 } }, null) .handle(CharacterStreamWritingMessageHandler.stdout()) // 34 .get(); } @Component public static class CafeAggregator { // 35 @Aggregator // 36 public Delivery output(List drinks) { return new Delivery(drinks); } @CorrelationStrategy // 37 public Integer correlation(Drink drink) { return drink.getOrderNumber(); } } } Examining the code line by line... 1. @SpringBootApplication This new meta-annotation from Spring Boot 1.2. Includes @Configuration and@EnableAutoConfiguration. Since we are in a Spring Integration application and Spring Boot has auto-configuration for it, the @EnableIntegration is automatically applied, to initialize the Spring Integration infrastructure including an environment for the Java DSL -DslIntegrationConfigurationInitializer, which is picked up by theIntegrationConfigurationBeanFactoryPostProcessor from /META-INF/spring.factories. 2. @IntegrationComponentScan The Spring Integration analogue of @ComponentScan to scan components based on interfaces, (the Spring Framework's @ComponentScan only looks at classes). Spring Integration supports the discovery of interfaces annotated with @MessagingGateway (see #7 below). 3. ConfigurableApplicationContext ctx = SpringApplication.run(Application.class, args); The main method of our class is designed to start the Spring Boot application using the configuration from this class and starts an ApplicationContext via Spring Boot. In addition, it delegates command line arguments to the Spring Boot. For example you can specify --debug to see logs for the boot auto-configuration report. 4. Cafe cafe = ctx.getBean(Cafe.class); Since we already have an ApplicationContext we can start to interact with application. AndCafe is that entry point - in EIP terms a gateway. Gateways are simply interfaces and the application does not interact with the Messaging API; it simply deals with the domain (see #7 below). 5. for (int i = 1; i <= 100; i++) { To demonstrate the cafe "work" we intiate 100 orders with two drinks - one hot and one iced. And send the Order to the Cafe gateway. 6. System.out.println("Hit 'Enter' to terminate"); Typically Spring Integration application are asynchronous, hence to avoid early exit from themain Thread we block the main method until some end-user interaction through the command line. Non daemon threads will keep the application open but System.read()provides us with a mechanism to close the application cleanly. 7. @MessagingGateway The annotation to mark a business interface to indicate it is a gateway between the end-application and integration layer. It is an analogue of component from Spring Integration XML configuration. Spring Integration creates a Proxy for this interface and populates it as a bean in the application context. The purpose of this Proxy is to wrap parameters in a Message object and send it to the MessageChannel according to the provided options. 8. @Gateway(requestChannel = "orders.input") The method level annotation to distinct business logic by methods as well as by the target integration flows. In this sample we use a requestChannel reference of orders.input, which is a MessageChannel bean name of our IntegrationFlow input channel (see below #14). 9. void placeOrder(Order order); The interface method is a central point to interact from end-application with the integration layer. This method has a void return type. It means that our integration flow is one-wayand we just send messages to the integration flow, but don't wait for a reply. 10. private AtomicInteger hotDrinkCounter = new AtomicInteger(); private AtomicInteger coldDrinkCounter = new AtomicInteger(); Two counters to gather the information how our cafe works with drinks. 11. @Autowired private CafeAggregator cafeAggregator; The POJO for the Aggregator logic (see #33 and #35 below). Since it is a Spring bean, we can simply inject it even to the current @Configuration and use in any place below, e.g. from the .aggregate() EIP-method. 12. @Bean(name = PollerMetadata.DEFAULT_POLLER) public PollerMetadata poller() { The default poller bean. It is a analogue of component from Spring Integration XML configuration. Required for endpoints where the inputChannelis a PollableChannel. In this case, it is necessary for the two Cafe queues - hot and iced (see below #18). Here we use the Pollers factory from the DSL project and use its method-chain fluent API to build the poller metadata. Note that Pollers can be used directly from an IntegrationFlow definition, if a specific poller (rather than the default poller) is needed for an endpoint. 13. @Bean public IntegrationFlow orders() { The IntegrationFlow bean definition. It is the central component of the Spring Integration Java DSL, although it does not play any role at runtime, just during the bean registration phase. All other code below registers Spring Integration components (MessageChannel,MessageHandler, EventDrivenConsumer, MessageProducer, MessageSource etc.) in theIntegrationFlow object, which is parsed by the IntegrationFlowBeanPostProcessor to process those components and register them as beans in the application context as necessary (some elements, such as channels may already exist). 14. return IntegrationFlows.from("orders.input") The IntegrationFlows is the main factory class to start the IntegrationFlow. It provides a number of overloaded .from() methods to allow starting a flow from aSourcePollingChannelAdapter for a MessageSource implementations, e.g.JdbcPollingChannelAdapter; from a MessageProducer, e.g.WebSocketInboundChannelAdapter; or simply a MessageChannel. All ".from()" options have several convenient variants to configure the appropriate component for the start of theIntegrationFlow. Here we use just a channel name, which is converted to aDirectChannel bean definition during the bean definition phase while parsing theIntegrationFlow. In the Java 8 variant, we used here a Lambda definition - and thisMessageChannel has been implicitly created with the bean name based on theIntegrationFlow bean name. 15. .split("payload.items", (Consumer) null) Since our integration flow accepts messages through the orders.input channel, we are ready to consume and process them. The first EIP-method in our scenario is .split(). We know that the message payload from orders.input channel is an Order domain object, so we can simply use here a Spring (SpEL) Expression to return Collection. So, this performs the split EI pattern, and we send each collection entry as a separate message to the next channel. In the background, the .split() method registers aExpressionEvaluatingSplitter MessageHandler implementation and anEventDrivenConsumer for that MessageHandler, wiring in the orders.input channel as the inputChannel. The second argument for the .split() EIP-method is for an endpointConfigurer to customize options like autoStartup, requiresReply, adviceChain etc. We use herenull to show that we rely on the default options for the endpoint. Many of EIP-methods provide overloaded versions with and without endpointConfigurer. Currently.split(String expression) EIP-method without the endpointConfigurer argument is not available; this will be addressed in a future release. 16. .channel(MessageChannels.executor(Executors.newCachedThreadPool())) The .channel() EIP-method allows the specification of concrete MessageChannels between endpoints, as it is done via output-channel/input-channel attributes pair with Spring Integration XML configuration. By default, endpoints in the DSL integration flow definition are wired with DirectChannels, which get bean names based on theIntegrationFlow bean name and index in the flow chain. In this case we select a specificMessageChannel implementation from the Channels factory class; the selected channel here is an ExecutorChannel, to allow distribution of messages from the splitter to separate Threads, to process them in parallel in the downstream flow. 17. .route("payload.iced", The next EIP-method in our scenario is .route(), to send hot/iced order items to different Cafe kitchens. We again use here a SpEL expression to get the routingKey from the incoming message. In the Java 8 variant, we used a method-reference Lambda expression, but for pre Java 8 style we must use SpEL or an inline interface implementation. Many anonymous classes in a flow can make the flow difficult to read so we prefer SpEL in most cases. 18. new Consumer>() { The second argument of .route() EIP-method is a functional interface Consumer to specify ExpressionEvaluatingRouter options using a RouterSpec Builder. Since we don't have any choice with pre Java 8, we just provide here an inline implementation for this interface. 19. spec.channelMapping("true", "iced") .channelMapping("false", "hot"); With the Consumer>#accept()implementation we can provide desired AbstractMappingMessageRouter options. One of them is channelMappings, when we specify the routing logic by the result of router expresion and the target MessageChannel for the apropriate result. In this case iced andhot are MessageChannel names for IntegrationFlows below. 20. .get(); This finalizes the flow. Any IntegrationFlows.from() method returns anIntegrationFlowBuilder instance and this get() method extracts an IntegrationFlowobject from the IntegrationFlowBuilder configuration. Everything starting from the.from() and up to the method before the .get() is an IntegrationFlow definition. All defined components are stored in the IntegrationFlow and processed by theIntegrationFlowBeanPostProcessor during the bean creation phase. 21. @Bean public IntegrationFlow icedFlow() { This is the second IntegrationFlow bean definition - for iced drinks. Here we demonstrate that several IntegrationFlows can be wired together to create a single complex application. Note: it isn't recommended to inject one IntegrationFlow to another; it might cause unexpected behaviour. Since they provide Integration components for the bean registration and MessageChannels one of them, the best way to wire and inject is viaMessageChannel or @MessagingGateway interfaces. 22. return IntegrationFlows.from(MessageChannels.queue("iced", 10)) The iced IntegrationFlow starts from a QueueChannel that has a capacity of 10messages; it is registered as a bean with the name iced. As you remember we use this name as one of the route mappings (see above #19). In our sample, we use here a restricted QueueChannel to reflect the Cafe kitchen busy state from real life. And here is a place where we need that global poller for the next endpoint which is listening on this channel. 23. .handle(new GenericHandler() { The .handle() EIP-method of the iced flow demonstrates the concrete Cafe kitchen work. Since we can't minimize the code with something like Java 8 Lambda expression, we provide here an inline implementation for the GenericHandler functional interface with the expected payload type as the generic argument. With the Java 8 example, we distribute this.handle() between several subscriber subflows for a PublishSubscribeChannel. However in this case, the logic is all implemented in the one method. 24. Uninterruptibles.sleepUninterruptibly(1, TimeUnit.SECONDS); System.out.println(Thread.currentThread().getName() + " prepared cold drink #" + coldDrinkCounter.incrementAndGet() + " for order #" + payload.getOrderNumber() + ": " + payload); return payload; The business logic implementation for the current .handle() EIP-component. WithUninterruptibles.sleepUninterruptibly(1, TimeUnit.SECONDS); we just block the current Thread for some timeout to demonstrate how quickly the Cafe kitchen prepares a drink. After that we just report to STDOUT that the drink is ready and return the currentOrderItem from the GenericHandler for the next endpoint in our IntegrationFlow. In the background, the DSL framework registers a ServiceActivatingHandler for theMethodInvokingMessageProcessor to invoke the GenericHandler#handle at runtime. In addition, the framework registers a PollingConsumer endpoint for the QueueChannelabove. This endpoint relies on the default poller to poll messages from the queue. Of course, we always can use a specific poller for any concrete endpoint. In that case, we would have to provide a second endpointConfigurer argument to the .handle() EIP-method. 25. .channel("output") Since it is not the end of our Cafe scenario, we send the result of the current flow to theoutput channel using the convenient EIP-method .channel() and the name of theMessageChannel bean (see below #29). This is the logical end of the current iced drink subflow, so we use the .get() method to return the IntegrationFlow. Flows that end with a reply-producing handler that don't have a final .channel() will return the reply to the message replyChannel header. 26. @Bean public IntegrationFlow hotFlow() { The IntegrationFlow definition for hot drinks. It is similar to the previous iced drinks flow, but with specific hot business logic. It starts from the hot QueueChannel which is mapped from the router above. 27. Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS); The sleepUninterruptibly for hot drinks. Right, we need more time to boil the water! 28. @Bean public IntegrationFlow resultFlow() { One more IntegrationFlow bean definition to prepare the Delivery for the Cafe client based on the Drinks. 29. return IntegrationFlows.from("output") The resultFlow starts from the DirectChannel, which is created during the bean definition phase with this provided name. You should remember that we use the outputchannel name from the Cafe kitchens flows in the last .channel() in those definitions. 30. .transform(new GenericTransformer() { The .transform() EIP-method is for the appropriate pattern implementation and expects some object to convert one payload to another. In our sample we use an inline implementation of the GenericTransformer functional interface to convert OrderItem to Drink and we specify that using generic arguments. In the background, the DSL framework registers aMessageTransformingHandler and an EventDrivenConsumer endpoint with default options to consume messages from the output MessageChannel. 31. public Drink transform(OrderItem orderItem) { return new Drink(orderItem.getOrderNumber(), orderItem.getDrinkType(), orderItem.isIced(), orderItem.getShots()); } The business-specific GenericTransformer#transform() implementation to demonstrate how we benefit from Java Generics to transform one payload to another. Note: Spring Integration uses ConversionService before any method invocation and if you provide some specific Converter implementation, some domain payload can be converted to another automatically, when the framework has an appropriate registered Converter. 32. .aggregate(new Consumer() { The .aggregate() EIP-method provides options to configure anAggregatingMessageHandler and its endpoint, similar to what we can do with the component when using Spring Integration XML configuration. Of course, with the Java DSL we have more power to configure the aggregator in place, without any other extra beans. However we demonstrate here an aggregator configuration with annotations (see below #35). From the Cafe business logic perspective we compose the Delivery for the initial Order, since we .split() the original order to the OrderItems near the beginning. 33. public void accept(AggregatorSpec aggregatorSpec) { aggregatorSpec.processor(cafeAggregator, null); } An inline implementation of the Consumer for the AggregatorSpec. Using theaggregatorSpec Builder we can provide desired options for the aggregator component, which will be registered as an AggregatingMessageHandler bean. Here we just provide theprocessor as a reference to the autowired (see #11 above) CafeAggregator component (see #35 below). The second argument of the .processor() option is methodName. Since we are relying on the aggregator annotation configuration for the POJO, we don't need to provide the method here and the framework will determine the correct POJO methods in the background. 34. .handle(CharacterStreamWritingMessageHandler.stdout()) It is the end of our flow - the Delivery is delivered to the client! We just print here the message payload to STDOUT using out-of-the-boxCharacterStreamWritingMessageHandler from Spring Integration Core. This is a case to show how existing components from Spring Integration Core (and its modules) can be used from the Java DSL. 35. @Component public static class CafeAggregator { The bean to specify the business logic for the aggregator above. This bean is picked up by the @ComponentScan, which is a part of the @SpringBootApplication meta-annotation (see above #1). So, this component becomes a bean and we can automatically wire (@Autowired) it to other components in the application context (see #11 above). 36. @Aggregator public Delivery output(List drinks) { return new Delivery(drinks); } The POJO-specific MessageGroupProcessor to build the output payload based on the payloads from aggregated messages. Since we mark this method with the @Aggregatorannotation, the target AggregatingMessageHandler can extract this method for theMethodInvokingMessageGroupProcessor. 37. @CorrelationStrategy public Integer correlation(Drink drink) { return drink.getOrderNumber(); } The POJO-specific CorrelationStrategy to extract the custom correlationKey from each inbound aggregator message. Since we mark this method with @CorrelationStrategyannotation the target AggregatingMessageHandler can extract this method for theMethodInvokingCorrelationStrategy. There is a similar self-explained@ReleaseStrategy annotation, but we rely in our Cafe sample just on the defaultSequenceSizeReleaseStrategy, which is based on the sequenceDetails message header populated by the splitter from the beginning of our integration flow. Well, we have finished describing the Cafe Demo sample based on the Spring Integration Java DSL when Java Lambda support is not available. Compare it with XML sample and also seeLambda support tutorial to get more information regarding Spring Integration. As you can see, using the DSL without lambdas is a little more verbose because you need to provide boilerplate code for inline anonymous implementations of functional interfaces. However, we believe it is important to support the use of the DSL for users who can't yet move to Java 8. Many of the DSL benefits (fluent API, compile-time validation etc) are available for all users. The use of lambdas continues the Spring Framework tradition of reducing or eliminating boilerplate code, so we encourage users to try Java 8 and lambdas and to encourage their organizations to consider allowing the use of Java 8 for Spring Integration applications. In addition see the Reference Manual for more information. As always, we look forward to your comments and feedback (StackOverflow (spring-integration tag), Spring JIRA, GitHub) and we very much welcome contributions! Thank you for your time and patience to read this!

When I needed to do prototyping, proof of concept or play with some new technology in free time, starting new project was always a little annoying barrier with Maven. Have to say that setting up Maven project is not hard and you can use Maven Archetypes. But Archetypes are often out of date. Who wants to play with old technologies? So I always end up wiring in dependencies I wanted to play with. Not very productive spent time. But than Spring Boot came to my way. I fell in love. In last few months I created at least 50 small playground projects, prototypes with Spring Boot. Also incorporated it at work. It’s just perfect for prototyping, learning, microservices, web, batch, enterprise, message flow or command line applications. You have to be dinosaur or be blind not to evaluate Spring Boot for your next Spring project. And when you finish evaluate it, you will go for it. I promise. I feel a need to highlight how easy is Black Box Testing of Spring Boot microservice. Black Box Testing refers to testing without any poking with application artifact. Such testing can be called also integration testing. You can also perform performance or stress testing way I am going to demonstrate. Spring Boot Microservice is usually web application with embedded Tomcat. So it is executed as JAR from command line. There is possibility to convert Spring Boot project into WAR artifact, that can be hosted on shared Servlet container. But we don’t want that now. It’s better when microservice has its own little embedded container. I used existing Spring’s REST service guide as testing target. Focus is mostly on testing project, so it is handy to use this “Hello World” REST application as example. I expect these two common tools are set up and installed on your machine: Maven 3 Git So we’ll need to download source code and install JAR artifact into our local repository. I am going to use command line to download and install the microservice. Let’s go to some directory where we download source code. Use these commands: git clone [email protected]:spring-guides/gs-rest-service.git cd gs-rest-service/complete mvn clean install If everything went OK, Spring Boot microservice JAR artifact is now installed in our local Maven repository. In serious Java development, it would be rather installed into shared repository (e.g. Artifactory, Nexus,… ). When our microservice is installed, we can focus on testing project. It is also Maven and Spring Boot based. Black box testing will be achieved by downloading the artifact from Maven repository (doesn’t matter if it is local or remote). Maven-dependency-plugin can help us this way: org.apache.maven.plugins maven-dependency-plugin copy-dependencies compile copy-dependencies gs-rest-service true It downloads microservice artifact into target/dependency directory by default. As you can see, it’s hooked to compile phase of Maven lifecycle, so that downloaded artifact is available during test phase. Artifact version is stripped from version information. We use latest version. It makes usage of JAR artifact easier during testing. Readers skilled with Maven may notice missing plugin version. Spring Boot driven project is inherited from parent Maven project called spring-boot-starter-parent. It contains versions of main Maven plugins. This is one of the Spring Boot’s opinionated aspects. I like it, because it provides stable dependencies matrix. You can change the version if you need. When we have artifact in our file system, we can start testing. We need to be able to execute JAR file from command line. I used standard JavaProcessBuilder this way: public class ProcessExecutor { public Process execute(String jarName) throws IOException { Process p = null; ProcessBuilder pb = new ProcessBuilder("java", "-jar", jarName); pb.directory(new File("target/dependency")); File log = new File("log"); pb.redirectErrorStream(true); pb.redirectOutput(Redirect.appendTo(log)); p = pb.start(); return p; } } This class executes given process JAR based on given file name. Location is hard-coded to target/dependency directory, where maven-dependency-plugin located our artifact. Standard and error outputs are redirected to file. Next class needed for testing is DTO (Data transfer object). It is simple POJO that will be used for deserialization from JSON. I use Lombok project to reduce boilerplate code needed for getters, setters, hashCode and equals. @Data @AllArgsConstructor @NoArgsConstructor public class Greeting { private long id; private String content; } Test itself looks like this: public class BlackBoxTest { private static final String RESOURCE_URL = "http://localhost:8080/greeting"; @Test public void contextLoads() throws InterruptedException, IOException { Process process = null; Greeting actualGreeting = null; try { process = new ProcessExecutor().execute("gs-rest-service.jar"); RestTemplate restTemplate = new RestTemplate(); waitForStart(restTemplate); actualGreeting = restTemplate.getForObject(RESOURCE_URL, Greeting.class); } finally { process.destroyForcibly(); } Assert.assertEquals(new Greeting(2L, "Hello, World!"), actualGreeting); } private void waitForStart(RestTemplate restTemplate) { while (true) { try { Thread.sleep(500); restTemplate.getForObject(RESOURCE_URL, String.class); return; } catch (Throwable throwable) { // ignoring errors } } } } It executes Spring Boot microservice process first and wait unit it starts. To verify if microservice is started, it sends HTTP request to URL where it’s expected. The service is ready for testing after first successful response. Microservice should send simple greeting JSON response for HTTP GET request. Deserialization from JSON into our Greeting DTO is verified at the end of the test. Source code is shared on Github.

[This article was written by Yaron Parasol.] Docker containers were created to help enable the fast, and reliable deployment of application components or tiers, by creating a container that holds a self-contained ready to deploy parts of applications, with the middleware and the app business logic needed to run them successfully. For example, a Spring application within a Tomcat container. By design, Docker is purposely an isolated self-contained part of the application, typically one tier or even one node in a tier. However, an application is typically multi-tier in its architecture and that means you have tiers with dependencies between them, where the nature of the dependencies can be anything from network connections and remote API invocations, to exchange of messages between application tiers. And hence an app is a set of different containers with specific configurations. This is why you need a way to glue the pieces of your app together. While, Docker has a basic solution for connecting containers using a Docker bridge, this solution is not always the preferred one, especially when deploying the container across different hosts and you need to take care of real network settings. Docker orchestration with TOSCA + Cloudify. Check it out. Go So, what role does the orchestrator play? The orchestrator will take care of two things: The timing of container creation - as containers need to be created by order of dependencies and Container configuration in order to allow containers to communicate with one another - and for that the orchestrator needs to pass runtime properties between containers. As a side note here: With Docker you need a special tweak here, as you typically don’t touch config files inside a container, you keep the container intact, so there is an interesting workaround for cases that this is required. One method to do this is by using a YAML-based orchestration plan to orchestrate the deployment of apps and post-deployment automation processes, which is the approach Cloudify employs. Based on TOSCA (topology and orchestration standard of cloud apps), this orchestration plan describes the components and their lifecycle, and the relationships between components, especially when it comes to complex topologies. This includes, what’s connected to what, what’s hosted on what, and other such considerations. TOSCA is able to describe the infrastructure, as well as, the middleware tier, and app layers on top of these. Cloudify basically takes this TOSCA orchestration plan (dubbed blueprints in Cloudify speak) and materializes these using workflows that traverse the graph of components, or this plan of components and issues commands to agents. These then create the app components and glue them together. The agents use extensions called plugins that are adaptors between the Cloudify configuration and the various infrastructure as a service (IaaS) and automation tools’ APIs. In our case, we created a plugin to interface with the Docker API. Introducing the Docker Cloudify Plugin The Cloudify-Docker plugin is quite straightforward, it installs the Docker API endpoint/server on the machine and then uses the Docker-Py binding to create, configure, and remove containers. TOSCA lifecycle events are: Create - installation of the app components Configure - configuration of the component Start - startup/running the component There is also stop & delete - for shutdown and removal We started by using the create - to create the container, we did not implement configure at the beginning, and start to run the application. But then we realized that for containers with dependencies we need to have runtime properties, such as IP import of the counterpart container in order to create the container for example. When we create an app server container, we need the port and IP of the database container. So, we pushed the creation of the container to the configure event, and used a TOSCA relationship pre-configure hook, to get the dependent container’s info at runtime. The way to expose the runtime info to the container with the dependencies is by setting them as environment variables. 01.interfaces: 02. cloudify.interfaces.lifecycle: 03. configure: 04. implementation: docker.docker_plugin.tasks.configure 05. inputs: 06. container_config: 07. command: mongod--rest--httpinterface --smallfiles 08. image: dockerfile/mongodb 09. start: 10. implementation: docker.docker_plugin.tasks.run 11. inputs: 12. container_start: 13. port_bindings: 14. 27017: 27017 15. 28017: 28017 Nodecellar Example I’d like to explain how this works by using our Nodecellar app as an example. The Nodecellar app is composed of two hosts that, in this case, Cloudify didn’t create but just SSHed into and then installed agents on. On one we have the MongoD container, with a MongoD process. On the other we have the Nodecellar container with NodeJS and the Nodecellar app within it. The Nodecellar container needs a connection to the MongoD container to run the app queries when the app starts. Ultimately, an orchestrator should not be limited to software deployment, the whole idea behind Docker Is to allow for agility, so we’d also like to use Docker in situations of auto-scale out and auto-heal, CD. In our next post we’ll show exactly that - how Cloudify can be used with Docker for post-deployment scenarios.

Originally authored by Artem Bilan on the SpringSource blog Dear Spring Community! Just after the Spring Integration Java DSL 1.0 GA release announcement I want to introduce the Spring Integration Java DSL to you as a line by line tutorial based on the classic Cafe Demo integration sample. We describe here Spring Boot support, Spring Framework Java and Annotation configuration, the IntegrationFlow feature and pay tribute to Java 8 Lambdasupport which was an inspiration for the DSL style. Of course, it is all backed by the Spring Integration Core project. But, before we launch into the description of the Cafe demonstration app here's a shorter example just to get started... @Configuration @EnableAutoConfiguration @IntegrationComponentScan public class Start { public static void main(String[] args) throws InterruptedException { ConfigurableApplicationContext ctx = SpringApplication.run(Start.class, args); List strings = Arrays.asList("foo", "bar"); System.out.println(ctx.getBean(Upcase.class).upcase(strings)); ctx.close(); } @MessagingGateway public interface Upcase { @Gateway(requestChannel = "upcase.input") Collection upcase(Collection strings); } @Bean public IntegrationFlow upcase() { return f -> f .split() // 1 .transform(String::toUpperCase) // 2 .aggregate(); // 3 } } We will leave the description of the infrastructure (annotations etc) to the main cafe flow description. Here, we want you to concentrate on the last @Bean, the IntegrationFlow as well as the gateway method which sends messages to that flow. In the main method we send a collection of strings to the gateway and print the results to STDOUT. The flow first splits the collection into individual Strings (1); each string is then transformed to upper case (2) and finally we re-aggregate them back into a collection (3) Since that's the end of the flow, the framework returns the result of the aggregation back to the gateway and the new payload becomes the return value from the gateway method. The equivalent XML configuration might be... or... Cafe Demo The purpose of the Cafe Demo application is to demonstrate how Enterprise Integration Patterns (EIP) can be used to reflect the order-delivery scenario in a real life cafe. With this application, we handle several drink orders - hot and iced. After running the application we can see in the standard output (System.out.println) how cold drinks are prepared quicker than hot. However the delivery for the whole order is postponed until the hot drink is ready. To reflect the domain model we have several classes: Order, OrderItem, Drink andDelivery. They all are mentioned in the integration scenario, but we won't analyze them here, because they are simple enough. The source code for our application is placed only in a single class; significant lines are annotated with a number corresponding to the comments, which follow: @SpringBootApplication // 1 @IntegrationComponentScan // 2 public class Application { public static void main(String[] args) throws Exception { ConfigurableApplicationContext ctx = SpringApplication.run(Application.class, args);// 3 Cafe cafe = ctx.getBean(Cafe.class); // 4 for (int i = 1; i <= 100; i++) { // 5 Order order = new Order(i); order.addItem(DrinkType.LATTE, 2, false); //hot order.addItem(DrinkType.MOCHA, 3, true); //iced cafe.placeOrder(order); } System.out.println("Hit 'Enter' to terminate"); // 6 System.in.read(); ctx.close(); } @MessagingGateway // 7 public interface Cafe { @Gateway(requestChannel = "orders.input") // 8 void placeOrder(Order order); // 9 } private AtomicInteger hotDrinkCounter = new AtomicInteger(); private AtomicInteger coldDrinkCounter = new AtomicInteger(); // 10 @Bean(name = PollerMetadata.DEFAULT_POLLER) public PollerMetadata poller() { // 11 return Pollers.fixedDelay(1000).get(); } @Bean public IntegrationFlow orders() { // 12 return f -> f // 13 .split(Order.class, Order::getItems) // 14 .channel(c -> c.executor(Executors.newCachedThreadPool()))// 15 .route(OrderItem::isIced, mapping -> mapping // 16 .subFlowMapping("true", sf -> sf // 17 .channel(c -> c.queue(10)) // 18 .publishSubscribeChannel(c -> c // 19 .subscribe(s -> // 20 s.handle(m -> sleepUninterruptibly(1, TimeUnit.SECONDS)))// 21 .subscribe(sub -> sub // 22 .transform(item -> Thread.currentThread().getName() + " prepared cold drink #" + this.coldDrinkCounter.incrementAndGet() + " for order #" + item.getOrderNumber() + ": " + item) // 23 .handle(m -> System.out.println(m.getPayload())))))// 24 .subFlowMapping("false", sf -> sf // 25 .channel(c -> c.queue(10)) .publishSubscribeChannel(c -> c .subscribe(s -> s.handle(m -> sleepUninterruptibly(5, TimeUnit.SECONDS)))// 26 .subscribe(sub -> sub .transform(item -> Thread.currentThread().getName() + " prepared hot drink #" + this.hotDrinkCounter.incrementAndGet() + " for order #" + item.getOrderNumber() + ": " + item) .handle(m -> System.out.println(m.getPayload())))))) .transform(orderItem -> new Drink(orderItem.getOrderNumber(), orderItem.getDrinkType(), orderItem.isIced(), orderItem.getShots())) // 27 .aggregate(aggregator -> aggregator // 28 .outputProcessor(group -> // 29 new Delivery(group.getMessages() .stream() .map(message -> (Drink) message.getPayload()) .collect(Collectors.toList()))) // 30 .correlationStrategy(m -> ((Drink) m.getPayload()).getOrderNumber()), null) // 31 .handle(CharacterStreamWritingMessageHandler.stdout()); // 32 } } Examining the code line by line... 1. @SpringBootApplication This new meta-annotation from Spring Boot 1.2. Includes @Configuration and@EnableAutoConfiguration. Since we are in a Spring Integration application and Spring Boot has auto-configuration for it, the @EnableIntegration is automatically applied, to initialize the Spring Integration infrastructure including an environment for the Java DSL -DslIntegrationConfigurationInitializer, which is picked up by theIntegrationConfigurationBeanFactoryPostProcessor from /META-INF/spring.factories. 2. @IntegrationComponentScan The Spring Integration analogue of @ComponentScan to scan components based on interfaces, (the Spring Framework's @ComponentScan only looks at classes). Spring Integration supports the discovery of interfaces annotated with @MessagingGateway (see #7 below). 3. ConfigurableApplicationContext ctx = SpringApplication.run(Application.class, args); The main method of our class is designed to start the Spring Boot application using the configuration from this class and starts an ApplicationContext via Spring Boot. In addition, it delegates command line arguments to the Spring Boot. For example you can specify --debug to see logs for the boot auto-configuration report. 4. Cafe cafe = ctx.getBean(Cafe.class); Since we already have an ApplicationContext we can start to interact with application. AndCafe is that entry point - in EIP terms a gateway. Gateways are simply interfaces and the application does not interact with the Messaging API; it simply deals with the domain (see #7 below). 5. for (int i = 1; i <= 100; i++) { To demonstrate the cafe "work" we intiate 100 orders with two drinks - one hot and one iced. And send the Order to the Cafe gateway. 6. System.out.println("Hit 'Enter' to terminate"); Typically Spring Integration application are asynchronous, hence to avoid early exit from themain Thread we block the main method until some end-user interaction through the command line. Non daemon threads will keep the application open but System.read()provides us with a mechanism to close the application cleanly. 7. @MessagingGateway The annotation to mark a business interface to indicate it is a gateway between the end-application and integration layer. It is an analogue of component from Spring Integration XML configuration. Spring Integration creates a Proxy for this interface and populates it as a bean in the application context. The purpose of this Proxy is to wrap parameters in a Message object and send it to the MessageChannel according to the provided options. 8. @Gateway(requestChannel = "orders.input") The method level annotation to distinct business logic by methods as well as by the target integration flows. In this sample we use a requestChannel reference of orders.input, which is a MessageChannel bean name of our IntegrationFlow input channel (see below #13). 9. void placeOrder(Order order); The interface method is a central point to interact from end-application with the integration layer. This method has a void return type. It means that our integration flow is one-wayand we just send messages to the integration flow, but don't wait for a reply. 10. private AtomicInteger hotDrinkCounter = new AtomicInteger(); private AtomicInteger coldDrinkCounter = new AtomicInteger(); Two counters to gather the information how our cafe works with drinks. 11. @Bean(name = PollerMetadata.DEFAULT_POLLER) public PollerMetadata poller() { The default poller bean. It is a analogue of component from Spring Integration XML configuration. Required for endpoints where the inputChannelis a PollableChannel. In this case, it is necessary for the two Cafe queues - hot and iced (see below #18). Here we use the Pollers factory from the DSL project and use its method-chain fluent API to build the poller metadata. Note that Pollers can be used directly from an IntegrationFlow definition, if a specific poller (rather than the default poller) is needed for an endpoint. 12. @Bean public IntegrationFlow orders() { The IntegrationFlow bean definition. It is the central component of the Spring Integration Java DSL, although it does not play any role at runtime, just during the bean registration phase. All other code below registers Spring Integration components (MessageChannel,MessageHandler, EventDrivenConsumer, MessageProducer, MessageSource etc.) in theIntegrationFlow object, which is parsed by the IntegrationFlowBeanPostProcessor to process those components and register them as beans in the application context as necessary (some elements, such as channels may already exist). 13. return f -> f The IntegrationFlow is a Consumer functional interface, so we can minimize our code and concentrate just only on the integration scenario requirements. Its Lambda acceptsIntegrationFlowDefinition as an argument. This class offers a comprehensive set of methods which can be composed to the chain. We call these EIP-methods, because they provide implementations for EI patterns and populate components from Spring Integration Core. During the bean registration phase, the IntegrationFlowBeanPostProcessor converts this inline (Lambda) IntegrationFlow to a StandardIntegrationFlow and processes its components. The same we can achieve using IntegrationFlows factory (e.g.IntegrationFlow.from("channelX"). ... .get()), but we find the Lambda definition more elegant. An IntegrationFlow definition using a Lambda populates DirectChannel as an inputChannel of the flow and it is registered in the application context as a bean with the name orders.input in this our sample (flow bean name + ".input"). That's why we use that name for the Cafe gateway. 14. .split(Order.class, Order::getItems) Since our integration flow accepts message through the orders.input channel, we are ready to consume and process them. The first EIP-method in our scenario is .split(). We know that the message payload from orders.input channel is an Order domain object, so we can simply use its type here and use the Java 8 method-reference feature. The first parameter is a type of message payload we expect, and the second is a method reference to the getItems() method, which returns Collection. So, this performs thesplit EI pattern, when we send each collection entry as a separate message to the next channel. In the background, the .split() method registers a MethodInvokingSplitterMessageHandler implementation and the EventDrivenConsumer for thatMessageHandler, and wiring in the orders.input channel as the inputChannel. 15. .channel(c -> c.executor(Executors.newCachedThreadPool())) The .channel() EIP-method allows the specification of concrete MessageChannels between endpoints, as it is done via output-channel/input-channel attributes pair with Spring Integration XML configuration. By default, endpoints in the DSL integration flow definition are wired with DirectChannels, which get the bean names based on theIntegrationFlow bean name and index in the flow chain. In this case we use anotherLambda expression, which selects a specific MessageChannel implementation from itsChannels factory and configures it with the fluent API. The current channel here is anExecutorChannel, to allow to distribute messages from the splitter to separateThreads, to process them in parallel in the downstream flow. 16. .route(OrderItem::isIced, mapping -> mapping The next EIP-method in our scenario is .route(), to send hot/iced order items to different Cafe kitchens. We again use here a method reference (isIced()) to get theroutingKey from the incoming message. The second Lambda parameter represents arouter mapping - something similar to sub-element for the component from Spring Integration XML configuration. However since we are using Java we can go a bit further with its Lambda support! The Spring Integration Java DSL introduced thesubflow definition for routers in addition to traditional channel mapping. Each subflow is executed depending on the routing and, if the subflow produces a result, it is passed to the next element in the flow definition after the router. 17. .subFlowMapping("true", sf -> sf Specifies the integration flow for the current router's mappingKey. We have in this samples two subflows - hot and iced. The subflow is the same IntegrationFlow functional interface, therefore we can use its Lambda exactly the same as we do on the top levelIntegrationFlow definition. The subflows don't have any runtime dependency with its parent, it's just a logical relationship. 18. .channel(c -> c.queue(10)) We already know that a Lambda definition for the IntegrationFlow starts from[FLOW_BEAN_NAME].input DirectChannel, so it may be a question "how does it work here if we specify .channel() again?". The DSL takes care of such a case and wires those two channels with a BridgeHandler and endpoint. In our sample, we use here a restrictedQueueChannel to reflect the Cafe kitchen busy state from real life. And here is a place where we need that global poller for the next endpoint which is listening on this channel. 19. .publishSubscribeChannel(c -> c The .publishSubscribeChannel() EIP-method is a variant of the .channel() for aMessageChannels.publishSubscribe(), but with the .subscribe() option when we can specify subflow as a subscriber to the channel. Right, subflow one more time! So, subflows can be specified to any depth. Independently of the presence .subscribe() subflows, the next endpoint in the parent flow is also a subscriber to this .publishSubscribeChannel(). Since we are in the .route() subflow already, the last subscriber is an implicit BridgeHandlerwhich just pops the message to the top level - to a similar implicit BridgeHandler to pop message to the next .transform() endpoint in the main flow. And one more note about this current position of our flow: the previous EIP-method is .channel(c -> c.queue(10)) and this one is for MessageChannel too. So, they are again tied with an implicit BridgeHandleras well. In a real application we could avoid this .publishSubscribeChannel() just with the single .handle() for the Cafe kitchen, but our goal here to cover DSL features as much as possible. That's why we distribute the kitchen work to several subflows for the samePublishSubscribeChannel. 20. .subscribe(s -> The .subscribe() method accepts an IntegrationFlow as parameter, which can be specified as Lambda to configure subscriber as subflow. We use here several subflow subscribers to avoid multi-line Lambdas and cover some DSL as we as Spring Integration capabilities. 21. s.handle(m -> sleepUninterruptibly(1, TimeUnit.SECONDS))) Here we use a simple .handle() EIP-method just to block the current Thread for some timeout to demonstrate how quickly the Cafe kitchen prepares a drink. Here we use Google Guava Uninterruptibles.sleepUninterruptibly, to avoid using a try...catch block within the Lambda expression, although you can do that and your Lambda will be multi-line. Or you can move that code to a separate method and use it here as method reference. Since we don't use any Executor on the .publishSubscribeChannel() all subscribers will beperformed sequentially on the same Thread; in our case it is one of TaskScheduler's Threads from poller on the previous QueueChannel. That's why this sleep blocks all downstream process and allows to demonstrate the busy state for that restricted to 10QueueChannel. 22. .subscribe(sub -> sub The next subflow subscriber which will be performed only after that sleep with 1 second foriced drink. We use here one more subflow because .handle() of previous one is one-way with the nature of the Lambda for MessageHandler. That's why, to go ahead with process of our whole flow, we have several subscribers: some of subflows finish after their work and don't return anything to the parent flow. 23. .transform(item -> Thread.currentThread().getName() + " prepared cold drink #" + this.coldDrinkCounter.incrementAndGet() + " for order #" + item.getOrderNumber() + ": " + item) The transformer in the current subscriber subflow is to convert the OrderItem to the friendly STDOUT message for the next .handle. Here we see the use of generics with the Lambda expression. This is implemented using the GenericTransformer functional interface. 24. .handle(m -> System.out.println(m.getPayload()))))) The .handle() here just to demonstrate how to use Lambda expression to print thepayload to STDOUT. It is a signal that our drink is ready. After that the final (implicit) subscriber to the PublishSubscribeChannel just sends the message with the OrderItemto the .transform() in the main flow. 25. .subFlowMapping("false", sf -> sf The .subFlowMapping() for the hot drinks. Actually it is similar to the previous iceddrinks subflow, but with specific hot business logic. 26. s.handle(m -> sleepUninterruptibly(5, TimeUnit.SECONDS))) The sleepUninterruptibly for hot drinks. Right, we need more time to boil the water! 27. .transform(orderItem -> new Drink(orderItem.getOrderNumber(), orderItem.getDrinkType(), orderItem.isIced(), orderItem.getShots())) The main OrderItem to Drink transformer, which is performed when the .route()subflow returns its result after the Cafe kitchen subscribers have finished preparing the drink. 28. .aggregate(aggregator -> aggregator The .aggregate() EIP-method provides similar options to configure anAggregatingMessageHandler and its endpoint, like we can do with the component when using Spring Integration XML configuration. Of course, with the Java DSL we have more power to configure the aggregator just in place, without any other extra beans. And Lambdas come to the rescue again! From the Cafe business logic perspective we compose theDelivery for the initial Order, since we .split() the original order to the OrderItems near the beginning. 29. .outputProcessor(group -> The .outputProcessor() of the AggregatorSpec allows us to emit a custom result after aggregator completes the group. It's an analogue of ref/method from the component or the @Aggregator annotation on a POJO method. Our goal here to compose aDelivery for all Drinks. 30. new Delivery(group.getMessages() .stream() .map(message -> (Drink) message.getPayload()) .collect(Collectors.toList()))) As you see we use here the Java 8 Stream feature for Collection. We iterate over messages from the released MessageGroup and convert (map) each of them to its Drinkpayload. The result of the Stream (.collect()) (a list of Drinks) is passed to theDelivery constructor. The Message with this new Delivery payload is sent to the next endpoint in our Cafe scenario. 31. .correlationStrategy(m -> ((Drink) m.getPayload()).getOrderNumber()), null) The .correlationStrategy() Lambda demonstrates how we can customize an aggregator behaviour. Of course, we can rely here just only on a built-in SequenceDetails from Spring Integration, which is populated by default from .split() in the beginning of our flow to each split message, but the Lambda sample for the CorrelationStrategy is included for illustration. (With XML, we could have used a correlation-expression or a customCorrelationStrategy). The second argument in this line for the .aggregate() EIP-method is for the endpointConfigurer to customize options like autoStartup,requiresReply, adviceChain etc. We use here null to show that we rely on the default options for the endpoint. Many of EIP-methods provide overloaded versions with and withoutendpointConfigurer, but .aggregate() requires an endpoint argument, to avoid an explicit cast for the AggregatorSpec Lambda argument. 32. .handle(CharacterStreamWritingMessageHandler.stdout()); It is the end of our flow - the Delivery is delivered to the client! We just print here the message payload to STDOUT using out-of-the-boxCharacterStreamWritingMessageHandler from Spring Integration Core. This is a case to show how existing components from Spring Integration Core (and its modules) can be used from the Java DSL. Well, we have finished describing the Cafe Demo sample based on the Spring Integration Java DSL. Compare it with XML sample to get more information regarding Spring Integration. This is not an overall tutorial to the DSL stuff. We don't review here theendpointConfigurer options, Transformers factory, the IntegrationComponentSpechierarchy, the NamespaceFactories, how we can specify several IntegrationFlow beans and wire them to a single application etc., see the Reference Manual for more information. At least this line-by-line tutorial should show you Spring Integration Java DSL basics and its seamless fusion between Spring Framework Java & Annotation configuration, Spring Integration foundation and Java 8 Lambda support! Also see the si4demo to see the evolution of Spring Integration including the Java DSL, as shown at the 2014 SpringOne/2GX Conference. (Video should be available soon). As always, we look forward to your comments and feedback (StackOverflow (spring-integration tag), Spring JIRA, GitHub) and we very much welcome contributions! P.S. Even if this tutorial is fully based on the Java 8 Lambda support, we don't want to miss pre Java 8 users, we are going to provide similar non-Lambda blog post. Stay tuned!

Here’s something that stumped me for a while today. I’ve got the following Linq query in my repository (this is using the ORM from DevExpress, XPO, but the basic idea is the same) internal virtual IOrderedQueryable GetMyData(string keyVal) { return (from MyEntity ent in new XPQuery(Context) where ent.Key == keyVal orderby ent.SortCol select end); } The problem I was having was in mocking the return value from this method. One cannot create an interface so I could not create a list of items to return from the mocked method. I finally hit on this magic combination of linq queries that lets me return a set built by hand for the mock. var emptyLst = new List(); var lst = (from d in emptyLst select d).AsQueryable().OrderBy(x => x.Key ); _mockRepo.Setup(r => r.MyMockedEvent).Returns(lst); This seems to work like a charm

I’m a huge Vaadin fan and I’ve created a Github workshop I can demo at conferences. A common issue with such kind of workshops is that attendees have to prepare their workstations in advance… and there’s always a significant part of them that comes with not everything ready. At this point, two options are available to the speaker: either wait for each of the attendee to finish the preparation – too bad for the people who took the time at home to do that, or start anyway – and lose the not-ready part. Given the current buzz around Docker, I thought that could be a very good way to make the workshop preparation quicker – only one step, and hasslefree – no problem regarding the quirks of your operation system. The required steps I ask the attendees are the following: Install Git Install Java, Maven and Tomcat Clone the git repo Build the project (to prepare the Maven repository) Deploy the built webapp Start Tomcat These should directly be automated into Docker. As I wasted much time getting this to work, here’s the tale of my journey in achieving this (be warned, it’s quite long). If you’ve got similar use-cases, I hope it will be useful in you getting things done faster. Starting with Docker The first step was to get to know the basics about Docker. Fortunately, I had the chance to attend a Docker workshop by David Gageot at Duchess Swiss. This included both Docker installation and basics of Dockerfile. I assume readers have likewise a basic understanding of Docker. For those who don’t, I guess browsing the Docker’s official documentation is a nice idea: Installation Dockerfile reference Building my first Dockerfile The Docker image can be built with the following command ran into the directory of the Dockerfile: $ docker build -t vaadinworkshop . The first issues one can encounter when playing with Docker the first time, is to get the following error message: Get http:///var/run/docker.sock/v1.14/containers/json: dial unix /var/run/docker.sock: no such file or directory The reason is because one didn’t export the required environment variables displayed by the boot2docker information message. If you lost the exact data, no worry, just use the shellinit boot2docker parameter: $ boot2docker shellinit Writing /Users/i303869/.docker/boot2docker-vm/ca.pem: Writing /Users/i303869/.docker/boot2docker-vm/cert.pem: Writing /Users/i303869/.docker/boot2docker-vm/key.pem: export DOCKER_HOST=tcp://192.168.59.103:2376 export DOCKER_CERT_PATH=/Users/i303869/.docker/boot2docker-vm Copy-paste the export lines above will solve the issue. These can also be set in one’s .bashrc script as it seems these values seldom change. Next in line is the following error: Get http://192.168.59.103:2376/v1.14/containers/json: malformed HTTP response "x15x03x01x00x02x02" This error message seems to be because of a mismatch between versions of the client and the server. It seems it is because of a bug on Mac OSX when upgrading. For a long term solution, reinstall Docker from scratch; for a quick fix, use the --tls flag with the docker command. As it is quite cumbersome to type it everything, one can alias it: $ alias docker="docker --tls" My last mistake when building the image comes from building the Dockerfile from a not empty directory. Docker sends every file it finds in the directory of the Dockerfile to the Docker container for build: $ docker --tls build -t vaadinworkshop . Sending build context to Docker daemon Too many kB Fix: do not try this at home and start from a directory container the Dockerfile only. Starting from scratch Dockerfiles describe images – images are built as a layered list of instructions. Docker images are designed around single inheritance: one image has to be set a single parent. An image requiring no parent starts from scratch, but Docker provides 4 base official distributions: busybox, debian, ubuntu and centos (operating systems are generally a good start). Whatever you want to achieve, it is necessary to choose the right parent. Given the requirements I set for myself (Java, Maven, Tomcat and Git), I tried to find the right starting image. Many Dockerfiles are already available online on the Docker hub. The browsing app is quite good, but to be really honest, the search can really be improved. My intention was to use the image that matched the most of my requirements, then fill the gap. I could find no image providing Git, but I thought the dgageot/maven Dockerfile would be a nice starting point. The problem is that the base image is a busybox and provides no installer out-of-the-box (apt-get, yum, whatever). For this reason, David uses a lot of curl to get Java 8 and Maven in his Dockerfiles. I foolishly thought I could use a different flavor of busybox that provides the opkg installer. After a while, I accumulated many problems, resolving one heading to another. In the end, I finally decided to use the OS I was most comfortable with and to install everything myself: FROM ubuntu:utopic Scripting Java installation Installing git, maven and tomcat packages is very straightforward (if you don’t forget to use the non-interactive options) with RUN and apt-get: RUN apt-get update && \ apt-get install -y --force-yes git maven tomcat8 Java doesn’t fall into this nice pattern, as Oracle wants you to accept the license. Nice people did however publish it to a third-party repo. Steps are the following: Add the needed package repository Configure the system to automatically accept the license Configure the system to add un-certified packages Update the list of repositories At last, install the package Also add a package for Java 8 system configuration. RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && \ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && \ apt-get install -y --force-yes oracle-java8-installer oracle-java8-set-default Building the sources Getting the workshop’s sources and building them is quite straightforward with the following instructions: RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN mvn package The drawback of this approach is that Maven will start from a fresh repository, and thus download the Internet the first time it is launched. At first, I wanted to mount a volume from the host to the container to share the ~/.m2/repository folder to avoid this, but I noticed this could only be done at runtime through the -v option as the VOLUME instruction cannot point to a host directory. Starting the image The simplest command to start the created Docker image is the following: $ docker run -p 8080:8080 Do not forget the port forwarding from the container to the host, 8080 for the standard HTTP port. Also, note that it’s not necessary to run the container as a daemon (with the -d option). The added value of that is that the standard output of the CMD (see below) will be redirected to the host. When running as a daemon and wanting to check the logs, one has to execute bash in the container, which requires a sequence of cumbersome manipulations. Configuring and launching Tomcat Tomcat can be launched when starting the container by just adding the following instruction to the Dockerfile: CMD ["catalina.sh", "run"] However, trying to start the container at this point will result in the following error: Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina start SEVERE: Cannot start server. Server instance is not configured. I have no idea why, but it seems Tomcat 8 on Ubuntu is not configured in any meaningful way. Everything is available but we need some symbolic links here and there as well as creating the temp directory. This translates into the following instruction in the Dockerfile: RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && \ ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && \ ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && \ ln -s /etc/tomcat8 $CATALINA_HOME/conf && \ mkdir $CATALINA_HOME/temp The final trick is to connect the exploded webapp folder created by Maven to Tomcat’s webapps folder, which it looks for deployments: RUN mkdir $CATALINA_HOME/webapps && \ ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop At this point, the Holy Grail is not far away, you just have to browse the URL… if only we knew what the IP was. Since running on Mac, there’s an additional VM beside the host and the container that’s involved. To get this IP, type: $ boot2docker ip The VM's Host only interface IP address is: 192.168.59.103 Now, browsing http://192.168.59.103:8080/vaadinworkshop/ will bring us to the familiar workshop screen: Developing from there Everything works fine but didn’t we just forget about one important thing, like how workshop attendees are supposed to work on the sources? Easy enough, just mount the volume when starting the container: docker run -v /Users//vaadin7-workshop:/vaadin7-workshop -p 8080:8080 vaadinworkshop Note that the host volume must be part of /Users and if on OSX, it must use boot2docker v. 1.3+. Unfortunately, it seems now is the showstopper, as mounting an empty directory from the host to the container will not make the container’s directory available from the host. On the contrary, it will empty the container’s directory given that the host’s directory doesn’t exist… It seems there’s an issue in Docker on Mac. The installation of JHipster runs into the same problem, and proposes to use the Samba Docker folder sharing project. I’m afraid I was too lazy to go further at this point. However, this taught me much about Docker, its usages and use-cases (as well as OSX integration limitations). For those who are interested, you’ll find below the Docker file. Happy Docker! FROM ubuntu:utopic MAINTAINER Nicolas Frankel # Config to get to install Java 8 w/o interaction RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && apt-get install -y --force-yes git oracle-java8-installer oracle-java8-set-default maven tomcat8 RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN git checkout v7.2-1 RUN mvn package ENV JAVA_HOME /usr/lib/jvm/java-8-oracle ENV CATALINA_HOME /usr/share/tomcat8 ENV PATH $PATH:$CATALINA_HOME/bin # Configure Tomcat 8 directories RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && ln -s /etc/tomcat8 $CATALINA_HOME/conf && mkdir $CATALINA_HOME/temp && mkdir $CATALINA_HOME/webapps && ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop VOLUME ["/vaadin7-workshop"] CMD ["catalina.sh", "run"] # docker build -t vaadinworkshop . # docker run -v ~/vaadin7-workshop training/webapp -p 8080:8080 vaadinworkshop

[This article was written by Artem Bilan.] Dear Spring community, As we promised in the Release Candidate blog post, we are pleased to announce that the Spring Integration Java DSL 1.0 GA is now available. As usual, use the Release Repository with Maven or Gradle, or download a distribution archive, to give it a spin. See the project home page for more information. First of all, we are glad to share with you that on Nov 12, 2014, DZone research recognized Spring Integration as the leader in the ESB / Integration framework space, leading with 42% marketshare, in a publication of their recent survey results. And the report is the most popular DZone Guide in November, with more than 12 000 downloads already! Don't miss it: very exciting. We hope the release of the Spring Integration Java DSL adds more excitement!. Many thanks to all contributors, including several who are new to the community. The release includes just a few bug fixes, since the release candidate, and a lot of JavaDocs! Not specifically related to the the release, I want to present here some resources on the matter. We are observing many valuable DSL questions on Stack Overflow. Josh Long's tech tip showing how we can use together Spring Boot, REST, Spring Integration 4.1 WebSocket support and Spring Integration Java DSL plus Java 8 features. The Jdbc Splitter implementation in the project tests. My gist to demonstrate how we can use Reactor Streams together with the Spring Integration Java DSL. Dave Syer has started to use Spring Integration Java DSL in the Spring Cloud Bus project. Don't miss the si4demo to see the evolution of Spring Integration including the Java DSL, as shown at the 2014 SpringOne/2GX Conference. (Video should be available soon). Especial thanks to Biju Kunjummen who has done some nice articles on DZone to introduce Spring Integration Java DSL: https://dzone.com/articles/spring-integration-java-dsl, https://dzone.com/articles/spring-integration-java-dsl-0. And of course, with the latest Spring XD, we can build Modules based on @Configuration including Spring Integration Java DSL IntegrationFlow definitions. Just after this announcement I'm going to publish a DSL Tutorial to explain concepts and features using the Java DSL version of the Cafe Demo sample as material. As always, we look forward to your comments and feedback (StackOverflow (spring-integration tag), Spring JIRA, GitHub) and we very much welcome contributions!

there is currently a strong trend for microservice based architectures and frequent discussions comparing them to monoliths. there is much advice about breaking-up monoliths into microservices and also some amusing fights between proponents of the two paradigms - see the great microservices vs monolithic melee . the term 'monolith' is increasingly being used as a generic insult in the same way that 'legacy' is! however, i believe that there is a great deal of misunderstanding about exactly what a 'monolith' is and those discussing it are often talking about completely different things. a monolith can be considered an architectural style or a software development pattern (or anti-pattern if you view it negatively). styles and patterns usually fit into different viewtypes (a viewtype is a set, or category, of views that can be easily reconciled with each other [clements et al., 2010]) and some basic viewtypes we can discuss are: module - the code units and their relation to each other at compile time. allocation - the mapping of the software onto its environment. runtime - the static structure of the software elements and how they interact at runtime. a monolith could refer to any of the basic viewtypes above. module monolith if you have a module monolith then all of the code for a system is in a single codebase that is compiled together and produces a single artifact. the code may still be well structured (classes and packages that are coherent and decoupled at a source level rather than a big-ball-of-mud) but it is not split into separate modules for compilation. conversely a non-monolithic module design may have code split into multiple modules or libraries that can be compiled separately, stored in repositories and referenced when required. there are advantages and disadvantages to both but this tells you very little about how the code is used - it is primarily done for development management. allocation monolith for an allocation monolith, all of the code is shipped/deployed at the same time. in other words once the compiled code is 'ready for release' then a single version is shipped to all nodes. all running components have the same version of the software running at any point in time. this is independent of whether the module structure is a monolith. you may have compiled the entire codebase at once before deployment or you may have created a set of deployment artifacts from multiple sources and versions. either way this version for the system is deployed everywhere at once (often by stopping the entire system, rolling out the software and then restarting). a non-monolithic allocation would involve deploying different versions to individual nodes at different times. this is again independent of the module structure as different versions of a module monolith could be deployed individually. runtime monolith a runtime monolith will have a single application or process performing the work for the system (although the system may have multiple, external dependencies). many systems have traditionally been written like this (especially line-of-business systems such as payroll, accounts payable, cms etc). whether the runtime is a monolith is independent of whether the system code is a module monolith or not. a runtime monolith often implies an allocation monolith if there is only one main node/component to be deployed (although this is not the case if a new version of software is rolled out across regions, with separate users, over a period of time). note that my examples above are slightly forced for the viewtypes and it won't be as hard-and-fast in the real world. conclusion be very carefully when arguing about 'microservices vs monoliths'. a direct comparison is only possible when discussing the runtime viewtype and properties. you should also not assume that moving away from a module or allocation monolith will magically enable a microservice architecture (although it will probably help). if you are moving to a microservice architecture then i'd advise you to consider all these viewtypes and align your boundaries across them i.e. don't just code, build and distribute a monolith that exposes subsets of itself on different nodes.

In the modern world of microservices it's important to provide strict and polyglot clients for your service. It's better if your API is self-documented. One of the best tools for it is Apache Thrift. I want to explain how to use it with my favorite platform for microservices - Spring Boot. All project source code is available on GitHub: https://github.com/bsideup/spring-boot-thrift Project skeleton I will use Gradle to build our application. First, we need our main build.gradle file: buildscript { repositories { jcenter() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.1.8.RELEASE") } } allprojects { repositories { jcenter() } apply plugin:'base' apply plugin: 'idea' } subprojects { apply plugin: 'java' } Nothing special for a Spring Boot project. Then we need a gradle file for thrift protocol modules (we will reuse it in next part): import org.gradle.internal.os.OperatingSystem repositories { ivy { artifactPattern "http://dl.bintray.com/bsideup/thirdparty/[artifact]-[revision](-[classifier]).[ext]" } } buildscript { repositories { jcenter() } dependencies { classpath "ru.trylogic.gradle.plugins:gradle-thrift-plugin:0.1.1" } } apply plugin: ru.trylogic.gradle.thrift.plugins.ThriftPlugin task generateThrift(type : ru.trylogic.gradle.thrift.tasks.ThriftCompileTask) { generator = 'java:beans,hashcode' destinationDir = file("generated-src/main/java") } sourceSets { main { java { srcDir generateThrift.destinationDir } } } clean { delete generateThrift.destinationDir } idea { module { sourceDirs += [file('src/main/thrift'), generateThrift.destinationDir] } } compileJava.dependsOn generateThrift dependencies { def thriftVersion = '0.9.1'; Map platformMapping = [ (OperatingSystem.WINDOWS) : 'win', (OperatingSystem.MAC_OS) : 'osx' ].withDefault { 'nix' } thrift "org.apache.thrift:thrift:$thriftVersion:${platformMapping.get(OperatingSystem.current())}@bin" compile "org.apache.thrift:libthrift:$thriftVersion" compile 'org.slf4j:slf4j-api:1.7.7' } We're using my Thrift plugin for Gradle. Thrift will generate source to the "generated-src/main/java" directory. By default, Thrift uses slf4j v1.5.8, while Spring Boot uses v1.7.7. It will cause an error in runtime when you will run your application, that's why we have to force a slf4j api dependency. Calculator service Let's start with a simple calculator service. It will have 2 modules: protocol and app.We will start with protocol. Your project should look as follows: calculator/ protocol/ src/ main/ thrift/ calculator.thrift build.gradle build.gradle settings.gradle thrift.gradle Where calculator/protocol/build.gradle contains only one line: apply from: rootProject.file('thrift.gradle') Don't forget to put these lines to settings.gradle, otherwise your modules will not be visible to Gradle: include 'calculator:protocol' include 'calculator:app' Calculator protocol Even if you're not familiar with Thrift, its protocol description file (calculator/protocol/src/main/thrift/calculator.thrift) should be very clear to you: namespace cpp com.example.calculator namespace d com.example.calculator namespace java com.example.calculator namespace php com.example.calculator namespace perl com.example.calculator namespace as3 com.example.calculator enum TOperation { ADD = 1, SUBTRACT = 2, MULTIPLY = 3, DIVIDE = 4 } exception TDivisionByZeroException { } service TCalculatorService { i32 calculate(1:i32 num1, 2:i32 num2, 3:TOperation op) throws (1:TDivisionByZeroException divisionByZero); } Here we define TCalculatorService with only one method - calculate. It can throw an exception of type TDivisionByZeroException. Note how many languages we're supporting out of the box (in this example we will use only Java as a target, though) Now run ./gradlew generateThrift, you will get generated Java protocol source in the calculator/protocol/generated-src/main/java/ folder. Calculator application Next, we need to create the service application itself. Just create calculator/app/ folder with the following structure: src/ main/ java/ com/ example/ calculator/ handler/ CalculatorServiceHandler.java service/ CalculatorService.java CalculatorApplication.java build.gradle Our build.gradle file for app module should look like this: apply plugin: 'spring-boot' dependencies { compile project(':calculator:protocol') compile 'org.springframework.boot:spring-boot-starter-web' testCompile 'org.springframework.boot:spring-boot-starter-test' } Here we have a dependency on protocol and typical starters for Spring Boot web app. CalculatorApplication is our main class. In this example I will configure Spring in the same file, but in your apps you should use another config class instead. package com.example.calculator; import com.example.calculator.handler.CalculatorServiceHandler; import org.apache.thrift.protocol.*; import org.apache.thrift.server.TServlet; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.context.annotation.*; import javax.servlet.Servlet; @Configuration @EnableAutoConfiguration @ComponentScan public class CalculatorApplication { public static void main(String[] args) { SpringApplication.run(CalculatorApplication.class, args); } @Bean public TProtocolFactory tProtocolFactory() { //We will use binary protocol, but it's possible to use JSON and few others as well return new TBinaryProtocol.Factory(); } @Bean public Servlet calculator(TProtocolFactory protocolFactory, CalculatorServiceHandler handler) { return new TServlet(new TCalculatorService.Processor(handler), protocolFactory); } } You may ask why Thrift servlet bean is called "calculator". In Spring Boot, it will register your servlet bean in context of the bean name and our servlet will be available at /calculator/. After that we need a Thrift handler class: package com.example.calculator.handler; import com.example.calculator.*; import com.example.calculator.service.CalculatorService; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class CalculatorServiceHandler implements TCalculatorService.Iface { @Autowired CalculatorService calculatorService; @Override public int calculate(int num1, int num2, TOperation op) throws TException { switch(op) { case ADD: return calculatorService.add(num1, num2); case SUBTRACT: return calculatorService.subtract(num1, num2); case MULTIPLY: return calculatorService.multiply(num1, num2); case DIVIDE: try { return calculatorService.divide(num1, num2); } catch(IllegalArgumentException e) { throw new TDivisionByZeroException(); } default: throw new TException("Unknown operation " + op); } } } In this example I want to show you that Thrift handler can be a normal Spring bean and you can inject dependencies in it. Now we need to implement CalculatorService itself: package com.example.calculator.service; import org.springframework.stereotype.Component; @Component public class CalculatorService { public int add(int num1, int num2) { return num1 + num2; } public int subtract(int num1, int num2) { return num1 - num2; } public int multiply(int num1, int num2) { return num1 * num2; } public int divide(int num1, int num2) { if(num2 == 0) { throw new IllegalArgumentException("num2 must not be zero"); } return num1 / num2; } } That's it. Well... almost. We still need to test our service somehow. And it should be an integration test. Usually, even if your application is providing JSON REST API, you still have to implement a client for it. Thrift will do it for you. We don't have to care about it. Also, it will support different protocols. Let's use a generated client in our test: package com.example.calculator; import org.apache.thrift.protocol.*; import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.TTransport; import org.junit.*; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.*; import org.springframework.boot.test.IntegrationTest; import org.springframework.boot.test.SpringApplicationConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.web.WebAppConfiguration; import static org.junit.Assert.*; @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = CalculatorApplication.class) @WebAppConfiguration @IntegrationTest("server.port:0") public class CalculatorApplicationTest { @Autowired protected TProtocolFactory protocolFactory; @Value("${local.server.port}") protected int port; protected TCalculatorService.Client client; @Before public void setUp() throws Exception { TTransport transport = new THttpClient("http://localhost:" + port + "/calculator/"); TProtocol protocol = protocolFactory.getProtocol(transport); client = new TCalculatorService.Client(protocol); } @Test public void testAdd() throws Exception { assertEquals(5, client.calculate(2, 3, TOperation.ADD)); } @Test public void testSubtract() throws Exception { assertEquals(3, client.calculate(5, 2, TOperation.SUBTRACT)); } @Test public void testMultiply() throws Exception { assertEquals(10, client.calculate(5, 2, TOperation.MULTIPLY)); } @Test public void testDivide() throws Exception { assertEquals(2, client.calculate(10, 5, TOperation.DIVIDE)); } @Test(expected = TDivisionByZeroException.class) public void testDivisionByZero() throws Exception { client.calculate(10, 0, TOperation.DIVIDE); } } This test will run your Spring Boot application, bind it to a random port and test it. All client-server communications will be performed in the same way real world clients are. Note how easy to use our service is from the client side. We're just calling methods and catching exceptions.

[This article was written by Barak Merimovich.] We have discussed OpenStack networking extensively in previous posts. In this post, I’d like to dive into a more advanced OpenStack networking scenario. Many cloud images are not configured to automatically bring up all network cards that are available. They will usually only have a single network card configured. To correctly set up a host in the cloud with multiple network cards, log on to the machine and bring up the additional interfaces. echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 Networks in the cloud A complex network architecture is a mainstay of modern IaaS clouds. Understanding how to configure your cloud-based networks, and hosts, is critical to getting your application working in the cloud. This is especially true with Cloudify, the open source cloud orchestration platform I work on. The cloud, like the world, used to be flat It was not that long a time ago that most IaaS providers only supported flat networks – all of your hosts were in one large network. Separation between services running in the cloud was enforced in software or with firewalls/security-groups. But technically, all of the hosts were connected to the same network and visible to each other. The flat network model is simple, and therefore easy to reason and understand. It was a good choice for the early days of the IaaS cloud and no doubt helped with getting applications into the cloud in the first place. It was one of the things that made EC2 so easy to use for anyone just starting out with the ‘cloud’. This model is in fact still available on Amazon Web Services under the title ‘EC2-Classic’. And for many applications, a flat network is good enough. But as cloud adoption increases, more complex applications are moving into the clouds, and issues like network separation, security, SLA and broadcast domains make more complex networks models a must. Software Defined Networks (SDN) fill that gap. They are now a staple of most major IaaS clouds. AWS has AWS-VPC, OpenStack has the Neutron project and there are many other implementations. Working with SDN requires knowing a bit more about how information moves around between your cloud resources. In this post I am going to discuss how to set up a host in the cloud so it will play nice with complex networks. I’ll be using OpenStack, but the concepts are similar for other cloud infrastructures. Openstack configuration I am going to start with an empty tenant, only the public network is available. First, lets set up out networks and router: neutron router-create demo-router neutron net-create demo-network-1 neutron net-create demo-network-2 neutron subnet-create --name demo-subnet-1 demo-network-1 10.0.0.0/24 neutron subnet-create --name demo-subnet-2 demo-network-2 10.0.1.0/24 neutron router-interface-add demo-router demo-subnet-1 neutron router-interface-add demo-router demo-subnet-2 neutron router-gateway-set demo-router public Note the network IDs: neutron net-list | id | name | subnets | | 2c33efe2-6204-4125-9716-3bc525630016 | demo-network-1 | 928dafa0-83ef-459c-b20d-71d8ea596fa2 10.0.0.0/24 | | aa30627e-c181-4a4b-89bf-5dd7c26c244e | demo-network-2 | 26d573f7-7953-4a54-825b-ed7bbc0661c7 10.0.1.0/24 | | e502de8d-929a-4ee0-bd18-efa297875cf6 | public | d40dab51-a729-452c-9ee6-b9ad08d10808 | We’ll start with a standard Ubuntu cloud image: glance image-create --name "Ubuntu 12.04 Standard" --location "http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img" --disk-format qcow2 --container-format bare Create the keypair and security group: nova keypair-add demo-keypair > demo-keypair.pem chmod 400 demo-keypair.pem nova secgroup-create demo-security-group "Security group for demo" nova secgroup-add-rule demo-security-group tcp 22 22 0.0.0.0/0 Let’s spin up an instance connected to both our networks: nova boot -flavor m1.small --image "Ubuntu 12.04 Standard" --nic net-id=2c33efe2-6204-4125-9716-3bc525630016 --nic net-id=aa30627e-c181-4a4b-89bf-5dd7c26c244e --security-groups demo-security-group --key-name demo-keypair demo-vm And set up floating IPs for the first network: nova list | ID | Name | Status | Task State | Power State | Networks | 2b17588b-8980-4489-9a04-6539a159dc3c | demo-vm | ACTIVE | None | Running | demo-network-1=10.0.0.2; demo-network-2=10.0.1.2 | neutron floatingip-create public neutron floatingip-list | id | fixed_ip_address | floating_ip_address | port_id | | 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 | | 192.168.15.42 | | neutron port-list | id | name | mac_address | fixed_ips | | 1ccfd334-7328-4b22-b93e-24a0888276ab | | fa:16:3e:14:39:39 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.3"} | | a482c4f6-fa74-476e-b1ce-cd8dd0c70815 | | fa:16:3e:18:92:79 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.2"} | | b23d7836-30c5-4bff-b873-15c87ba051f6 | | fa:16:3e:3a:28:40 | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.3"} | | d421b447-2adf-406f-876b-142238683344 | | fa:16:3e:9d:fc:7f | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.2"} | | dcf8696b-cc80-4b48-b09c-61c0f8ab02ac | | fa:16:3e:5b:39:fb | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.1"} | | f6a1666e-495a-4d3f-afa3-754b3cb3cfc0 | | fa:16:3e:8a:1b:fb | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.1"} | neutron floatingip-associate 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 d421b447-2adf-406f-876b-142238683344 Note how we matched the VM’s IP to its port, and associated the floating IP to the port. I wish there was an easier way to do this from the CLI… If everything worked correctly, you should have the following setup: Let’s make sure ssh works correctly: ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] ifconfig eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:230 errors:0 dropped:0 overruns:0 frame:0 TX packets:224 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:46297 (46.2 KB) TX bytes:31130 (31.1 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Huh?! The VM only has one working network interface! Where is my second NIC? Was there a configuration problem with the OpenStack network setup? The answer is here: ssh -i demo-keypair.pem [email protected] ifconfig -a eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:324 errors:0 dropped:0 overruns:0 frame:0 TX packets:332 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:69973 (69.9 KB) TX bytes:47218 (47.2 KB) eth1 Link encap:Ethernet HWaddr fa:16:3e:29:6d:22 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) The second NIC exists, but is not running. The issue is not with the OpenStack network configuration – it’s with the image. The image itself should be configured to work correctly with multiple NICs. All we have to do is bring up the NIC. So we ssh into the instance: ssh -i demo-keypair.pem [email protected] And run the following commands: echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 The second NIC should now be running: ifconfig eth1 eth1 Link encap:Ethernet HWaddr fa:16:3e:18:92:79 inet addr:10.0.1.2 Bcast:10.0.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe18:9279/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:45 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15376 (15.3 KB) TX bytes:3960 (3.9 KB) And there you go – your VM can access both networks. This issue can make life complicated when setting up a complex, or even a not very complex, application. When will this issue hurt you? Well, imagine a scenario where you have a web server and a database server. The web server is connected to both Network1 and Network2, and the database server is only connected to Network2. Network1 is connected to the external world over a router, and Network 2 is completely internal, adding another layer of security to the critical database server. So what happens if the web server only has one network card? If only the NIC for Network1 is up, the web server can’t access the database. If only the NIC for Network2 is up, the web server can’t be reached from the external world. Even worse, if this web server is accessed via a floating IP, this IP will also not work, so you won’t be able to access the web server and fix the issue. Tricky. In conclusion The above commands will bring up your additional network card. You will of-course need to repeat this process for each additional network card, and for each VM. You can use a start-up script (a.k.a. user-data script) or system service to run these commands, but there are better ways. I’ll discuss how to automate the network setup in a follow-up post. This was originally posted at Barak's blog Head in the Clouds, find it here.

The last couple of weeks I've been playing around with docker and kubernetes. If you are not familiar with kubernetes let's just say for now that its an open source container cluster management implementation, which I find really really awesome. One of the first things I wanted to try out was running an Apache ZooKeeper ensemble inside kubernetes and I thought that it would be nice to share the experience. For my experiments I used Docker v. 1.3.0 and Openshift V3, which I built from source and includes Kubernetes. ZooKeeper on Docker Managing a ZooKeeper ensemble is definitely not a trivial task. You usually need to configure an odd number of servers and all of the servers need to be aware of each other. This is a PITA on its own, but it gets even more painful when you are working with something as static as docker images. The main difficulty could be expressed as: "How can you create multiple containers out of the same image and have them point to each other?" One approach would be to use docker volumes and provide the configuration externally. This would mean that you have created the configuration for each container, stored it somewhere in the docker host and then pass the configuration to each container as a volume at creation time. I've never tried that myself, I can't tell if its a good or bad practice, I can see some benefits, but I can also see that this is something I am not really excited about. It could look like this: docker run -p 2181:2181 -v /path/to/my/conf:/opt/zookeeper/conf my/zookeeper An other approach would be to pass all the required information as environment variables to the container at creation time and then create a wrapper script which will read the environment variables, modify the configuration files accordingly, launch zookeeper. This is definitely easier to use, but its not that flexible to perform other types of tuning without rebuilding the image itself. Last but not least one could combine the two approaches into one and do something like: Make it possible to provide the base configuration externally using volumes. Use env and scripting to just configure the ensemble. There are plenty of images out there that take one or the other approach. I am more fond of the environment variables approach and since I needed something that would follow some of the kubernetes conventions in terms of naming, I decided to hack an image of my own using the env variables way. Creating a custom image for ZooKeeper I will just focus on the configuration that is required for the ensemble. In order to configure a ZooKeeper ensemble, for each server one has to assign a numeric id and then add in its configuration an entry per zookeeper server, that contains the ip of the server, the peer port of the server and the election port. The server id is added in a file called myid under the dataDir. The rest of the configuration looks like: server.1=server1.example.com:2888:3888 server.2=server2.example.com:2888:3888 server.3=server3.example.com:2888:3888 ... server.current=[bind address]:[peer binding port]:[election biding port]Note that if the server id is X the server.X entry needs to contain the bind ip and ports and not the connection ip and ports. So what we actually need to pass to the container as environment variables are the following: The server id. For each server in the ensemble: The hostname or ip The peer port The election port If these are set, then the script that updates the configuration could look like: if [ ! -z "$SERVER_ID" ]; then echo "$SERVER_ID" > /opt/zookeeper/data/myid #Find the servers exposed in env. for i in `echo {1..15}`;do HOST=`envValue ZK_PEER_${i}_SERVICE_HOST` PEER=`envValue ZK_PEER_${i}_SERVICE_PORT` ELECTION=`envValue ZK_ELECTION_${i}_SERVICE_PORT` if [ "$SERVER_ID" = "$i" ];then echo "server.$i=0.0.0.0:2888:3888" >> conf/zoo.cfg elif [ -z "$HOST" ] || [ -z "$PEER" ] || [ -z "$ELECTION" ] ; then #if a server is not fully defined stop the loop here. break else echo "server.$i=$HOST:$PEER:$ELECTION" >> conf/zoo.cfg fi done fi For simplicity the function that read the keys and values from env are excluded. The complete image and helping scripts to launch zookeeper ensembles of variables size can be found in the fabric8io repository. ZooKeeper on Kubernetes The docker image above, can be used directly with docker, provided that you take care of the environment variables. Now I am going to describe how this image can be used with kubernetes. But first a little rambling... What I really like about using kubernetes with ZooKeeper, is that kubernetes will recreate the container, if it dies or the health check fails. For ZooKeeper this also means that if a container that hosts an ensemble server dies, it will get replaced by a new one. This guarantees that there will be constantly a quorum of ZooKeeper servers. I also like that you don't need to worry about the connection string that the clients will use, if containers come and go. You can use kubernetes services to load balance across all the available servers and you can even expose that outside of kubernetes. Creating a Kubernetes confing for ZooKeeper I'll try to explain how you can create 3 ZooKeeper Server Ensemble in Kubernetes. What we need is 3 docker containers all running ZooKeeper with the right environment variables: { "image": "fabric8/zookeeper", "name": "zookeeper-server-1", "env": [ { "name": "ZK_SERVER_ID", "value": "1" } ], "ports": [ { "name": "zookeeper-client-port", "containerPort": 2181, "protocol": "TCP" }, { "name": "zookeeper-peer-port", "containerPort": 2888, "protocol": "TCP" }, { "name": "zookeeper-election-port", "containerPort": 3888, "protocol": "TCP" } ] } The env needs to specify all the parameters discussed previously. So we need to add along with the ZK_SERVER_ID, the following: ZK_PEER_1_SERVICE_HOST ZK_PEER_1_SERVICE_PORT ZK_ELECTION_1_SERVICE_PORT ZK_PEER_2_SERVICE_HOST ZK_PEER_2_SERVICE_PORT ZK_ELECTION_2_SERVICE_PORT ZK_PEER_3_SERVICE_HOST ZK_PEER_3_SERVICE_PORT ZK_ELECTION_3_SERVICE_PORT An alternative approach could be instead of adding all these manual configuration, to expose peer and election as kubernetes services. I tend to favor the later approach as it can make things simpler when working with multiple hosts. It's also a nice exercise for learning kubernetes. So how do we configure those services? To configure them we need to know: the name of the port the kubernetes pod the provide the service The name of the port is already defined in the previous snippet. So we just need to find out how to select the pod. For this use case, it make sense to have a different pod for each zookeeper server container. So we just need to have a label for each pod, the designates that its a zookeeper server pod and also a label that designates the zookeeper server id. "labels": { "name": "zookeeper-pod", "server": 1 } Something like the above could work. Now we are ready to define the service. I will just show how we can expose the peer port of server with id 1, as a service. The rest can be done in a similar fashion: { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-peer-1", "kind": "Service", "port": 2888, "containerPort": "zookeeper-peer-port", "selector": { "name": "zookeeper-pod", "server": 1 } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } I hope you found it useful. There is definitely room for improvement so feel free to leave comments.

This article explains how errors are handled when using the messaging system with Spring Integration and how to handle route and redirect to specific channel.

This technical tip shows how to read Excel worksheet cells values in multiple threads simultaneously inside android applications. Often you need to read worksheet cells values in multiple threads simultaneously. To do so, set Worksheet.getCells().setMultiThreadReading() to true. If you do not set this property you might get the wrong cell values. Setting it to true, you always get the correct values. To achieve this task first create a workbook and adds a worksheet. Populates the worksheet with some string values then create two threads that simultaneously read values from random cells. If the values read are correct, then nothing happens. But if the values read are incorrect, then a message box shows up in the LogCat window. If you comment this line: testWorkbook.getWorksheets().get(0).getCells().setMultiThreadReading(true); the following message will show up in LogCat window: if (s.equals("R" + row + "C" + col)!=true) { System.out.println("This message box will show up when cells read values are incorrect."); } Otherwise, the program run without showing any message which means all values read from cells are correct. public class ThreadProc implements Runnable { boolean isRunning = true; Workbook testWorkbook; Random r = new Random(); public ThreadProc(Workbook workbook) { this.testWorkbook = workbook; } public int randomNext(int Low, int High) { int R = r.nextInt(High-Low) + Low; return R; } public void kill() { this.isRunning = false; } public void run(){ while(this.isRunning) { int row = randomNext(0, 100); int col = randomNext(0, 10); String s = testWorkbook.getWorksheets().get(0).getCells().get(row, col).getStringValue(); if (s.equals("R" + row + "C" + col)!=true) { System.out.println("This message box will show up when cells read values are incorrect."); } } } } //........MainActivity.java........ //................................. import java.io.File; import java.util.Random; import android.app.Activity; import android.os.Bundle; import android.os.Environment; import android.view.Menu; import com.aspose.cells.CellsHelper; import com.aspose.cells.IWarningCallback; import com.aspose.cells.WarningInfo; import com.aspose.cells.WarningType; import com.aspose.cells.Workbook; public class MainActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); try{ Workbook testWorkbook = new Workbook(); testWorkbook.getWorksheets().clear(); testWorkbook.getWorksheets().add("Sheet1"); for (int row = 0; row < 100; row++) for (int col = 0; col < 10; col++) testWorkbook.getWorksheets().get(0).getCells().get(row, col).setValue("R" + row + "C" + col); //Commenting this line will show a pop-up message testWorkbook.getWorksheets().get(0).getCells().setMultiThreadReading(true); ThreadProc tp = new ThreadProc(testWorkbook); Thread myThread1 = new Thread(tp); myThread1.start(); Thread myThread2 = new Thread(tp); myThread2.start(); Thread.currentThread().sleep(5*1000); tp.kill(); } catch (Exception e) { e.printStackTrace(); } } ......... }

In Parts 1 and 2 we have covered a number of common issues people run into when managing a sharded MongoDB cluster. In this final post of the series we will cover a subtle, but important distinction in terms of balancing a sharded cluster as well as an interesting limitation that can be worked around relatively easily, but is nonetheless surprising when it comes up. 6. Chunk balancing != data balancing != traffic balancing The balancer in a sharded cluster cares about just one thing: Are chunks for a given collection evenly balanced across all shards? If they are not, then it will take steps to rectify that imbalance. This all sounds perfectly logical, and even with extra complexity like tagging involved the logic is pretty straight forward. If we assume that all chunks are equal, then we can rest assured that our data is being evenly balanced across all the shards in our cluster and rest easy at night. Although that is sometimes, perhaps even frequently, the case it is not always true - chunks are not always equal. There can be massive “jumbo” chunks that exceed the maximum chunk size (64MiB), completely empty chunks and everything in between. Let’s use an example from our first pitfall, the monotonically increasing shard key. For our example, we have picked just such a key to shard on (date), and up until this point we have had just one shard and had not sharded the collection. We are about to add a second shard to our cluster and so we enable sharding on the collection and do the necessary admin work to add the new shard into the cluster. Once the collection is enabled for sharding, the first shard contains all the newly minted chunks. Let’s represent them in a simplified table of 10 chunks. This is not representative of a real data set, but it will do for illustrative purposes: Table 1 - Initial Chunk Layout Now we add our second shard. The balancer will kick in and attempt to distribute the chunks evenly. It will do this by moving the lowest range chunks to the new shard until the counts are identical. Once it is finished balancing, our table now looks like this: Table 2 - Balanced Chunk Layout That looks pretty good at the moment, but lets imagine that more recent chunks are more likely to have more activity (updates say) than older chunks. Adding the traffic share estimates for each chunk shows that shard1 is taking far more traffic (72%) than shard2 (28%) despite the chunks seeming balanced overall based on the approximate size. Hence, chunk balancing is not equal to traffic balancing. Using that same example, let’s add another wrinkle - periodic deletion of old data. Every 3 months we run a job to delete any data older than 12 months. Let’s look at the impact of that on our table after we run it for the first time (assuming the first run happens on July 1st 2015). Table 3 - Post-Delete Chunk Layout The distribution of data is now completely skewed toward shard1 - shard2 is in fact empty! However, the balancer is completely unaware of this imbalance - the chunk count has remained the same the entire time, and as far as it is concerned the system is in a steady state. With no data on shard2, our traffic imbalance as seen above will be even worse, and we have essentially negated the benefit of having a second shard for this collection. Possible Mitigation Strategies If data and traffic balance are important, select an appropriate shard key Move chunks manually to address the imbalances - swap “hot” chunks for “cool” chunks, empty chunks for larger chunks 7. Waiting too long to shard a collection (collection too large) This is not very common, but when it falls on your shoulders, it can be quite challenging to solve. There is a maximum data size for a collection when when it is initially split which is a function of the chunk size and data size as noted on the limits page. If your collection contains less than 256GiB of data, then there will be no issue. If the collection size exceeds 256GiB but is less than 400GiB, then MongoDB may be able to do an initial split without any special measures being taken. Otherwise, with larger initial data sizes and the default settings, the initial split will fail. It is worth noting that once split the collection may grow as needed and without any real limitations as long as you can continue to add shards as data size grows. Possible Mitigation Strategies Since the limit is dictated by the chunk size and the data size, and assuming there is not much to be done about the data size, then the remaining variable is the chunk size. This is adjustable (default is 64MiB) and can be raised in order to let a large collection split initially and then reduced once that has been completed. The required chunk size increase will depend on the actual data size. However, this is relatively easy to work out - simply divide your data size by 256GB and then multiply that figure by 64MiB (and round up if it is not a nice even number). As an example, let’s consider a 4TiB collection: 4TiB divided by 256GiB = 16 64MiB x 16 = 1024MiB Hence, set the max chunk size to 1024MiB, then perform the initial sharding of the collection, and then finally reduce the chunk size back to 64MiB using the same procedure. . Thanks for reading through the Sharding Pitfall series! If you want to learn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.

By Adam Comerford, Senior Solutions Engineer In Part I we discussed important considerations when picking a shard key. In this post we will go through some recommendations when running a sharded cluster at scale. Scalability is one of the core benefits of sharding in MongoDB but this can give you a false sense of security; even with that flexibility, you still have to make smart decisions about how and when you deploy resources. In this post, we will cover a couple of common mistakes that people tend to make when it comes to running a sharded cluster. 3. Waiting too long to add a new shard (overloaded) You sharded your database and scaled horizontally for a reason, perhaps it was to add more memory or disk capacity. Whatever the reason, if your application usage grows over time so (generally) does your database utilization. Eventually, your current sharded cluster will pass a certain point, let’s call it 80% utilized (as a nice round estimate), such that it becomes problematic to add another shard. Why? Well, adding a new shard to a cluster is not free, and it is not instantaneous. It consumes resources and (initially) accepts very little traffic. Essentially, at the start of its existence, a newly added shard costs you capacity instead of adding capacity. The length of time it will stay in this state will depend on the balancer and how long it takes for a significant portion of “busy/active” chunks to move onto the new shard. It can often be easier to visualize this process, so let’s make up some hypothetical numbers and set the bar relatively low. Our imaginary existing cluster will be a set of 2 shards, with 2000 chunks (500 considered “active”) and to that we need to add a 3rd shard. This 3rd shard will eventually store one third of the active chunks (and total chunks). The question is, when does this shard stop adding overhead overall and instead become an asset? In reality, this will vary from cluster to cluster and have a lot of dependencies and variables - in other words you need to have good metrics about your cluster, particularly your load bottleneck. Therefore we will once again use our imaginations and go with a relatively low bar: when 5% of active chunks—that is, those chunks seeing most traffic—have migrated to the new shard, you should expect a net gain in performance. In our imaginary system we have evaluated our load levels, the expected impact of migrations and have determine that once that 5% threshold of active chunks has been migrated to the new shard it can be considered a net gain for the overall system. Once all chunks have been balanced, then the migration overhead disappears, but initially this will be an expected trade off. This chart shows how long it would take for new shards to reach net positive contribution in your cluster (the dotted line implies net gain): In this fabricated example, it takes almost 2 hours for the new shard to attain a viable level of active chunks and be considered a net gain for the overall system. Although these numbers are fictional, these numbers are based on setups we have seen in real systems with moderate load. From there it is relatively easy to imagine this set of migrations taking even longer on an overloaded set of shards, and taking far longer for our newly added shard to cross the threshold and become a net gain. As such it is best to be proactive and add capacity before it becomes a necessity. Possible Mitigation Strategies Manual balancing of targeted “hot” chunks (chunk that is being accessed more than others) to move activity to the new shard more quickly Add the shard at low traffic time so that there is less competition for resources Disable balancing on some collections, prioritise balancing busy collections first 4. Under-provisioning Config Servers Provisioning enough resources without being wasteful is always tricky, and all the more so in a complicated distributed system like a MongoDB sharded cluster. Everyone wants to use their hardware, virtual instances, virtual machines, containers and the like in the most efficient way possible, and get the best bang for their buck. Hence it is only natural to take a look at the various pieces of a distributed cluster and look for lower utilized pieces that could be put on less expensive resources. The most common pitfall here with MongoDB are the config servers, which are often neglected when stress testing a cluster. In testing environments and smaller deployments (unless specific measures are taken to stress them) they are relatively lightly loaded and usually identified as candidates for lesser instances/hardware. The problem is that these are critical pieces of infrastructure. They may not be heavily loaded all the time, but when they do see load and struggle to service requests, that can impact all queries (reads, writes, authentication) and add latency to all requests made of the cluster in question. In particular, the first config server in the list supplied to your mongos processes is vital. This is the config server that all mongos processes will default to read from when fetching or refreshing their view of the data distribution in your cluster. Similarly, this is the server that will be hit when attempting to authenticate a user. If it is under-provisioned and cannot service queries, or if it has problems with networking (packet loss, congestion), then the effects will be significant. Possible Mitigation Strategies Ensure the config servers are load tested, slightly over-provisioned (the first config server in particular) If using virtual machines or cloud based instances, investigate increasing available resources Turning off the balancer, disabling chunk splitting will reduce the chances of high read traffic to the config servers (no migrations, no meta data refresh) but this is only a temporary fix unless you have a perfect write distribution and may not eliminate issues completely. 5. Using the count() command on sharded collections This pitfall is very common, and it seems to hit somewhat randomly in terms of how long someone has been running a sharded environment. At some point, a question will arise along the lines of: “How are we tracking/verifying/checking how many documents we have in each collection on each shard, how balanced are they and do they agree with ?” Hopefully no one is actually constructing questions this way in your organization, but you get the basic idea. The most obvious way to do a quick check on this type of thing is to count the documents and see if the numbers make sense and/or agree with counts elsewhere. That thinking naturally leads people to the count command and they proceed to use it to gather figures for their documents and collections. Unfortunately, on a busy, mature sharded cluster, the results will very rarely be what is expected. The reason for this is that the count command as implemented today has several optimizations in place to make it faster to run in general and those speed optimizations essentially bypass a key piece of the sharding functionality needed to return accurate results in this case. This is a known bug and is being tracked in SERVER-3645, but does not stop people from consistently hitting this issue. The nature of the issue means that count will report documents in the results that it should not, for example: Documents that are being deleted as part of a chunk migrations Documents that have been left behind from previous chunk migrations (also known as orphans) Documents currently being copied as part of an in-flight chunk migration A regular query (rather than a count) will have its results filtered by the respective primary and not suffer from the same problem. Hence, if you were to manually count the results from a query client-side you would get an accurate result. This quirk of sharded environments will eventually be fixed, but for now it will inevitably crop up from time to time in all active sharded clusters used by a large team. Possible Mitigation Strategies Do counts on the client side, or use targeted, range based queries (with a primary read preference) to count instead Use cleanUpOrphaned and disable the balancer (make sure it has finished current round) when performing counts across the cluster If you want tolearn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.

in this article i explained how spring built-in transformers works for while transforming object message to map message. sometimes the messages need to be transformed before they can be consumed to achieve a business purpose. for example, a producer uses a plain xml as its payload to produce a message, while a consumer is interested in java object or types like plain text ,name-value pairs, or json model. spring integration provides endpoints such as service activators, channel adapters, message bridges, gateways, transformers, filters, and routers. in this example how transformers endpoint transform object message to map message. references: spring integration spring with jms spring with junit mockrunner sts high level view spring-mockrunner.xml in spring-mockrunner.xml file, i defined mockqueue, mockqueueconnectionfactory for inbound queue, and outbound queue for quick testing purpose. inboundqueue is where you will publish object message from objecttomaptransformertest.java class. outboundqueue where this queue expecting mapmessage type object and this queue is listing mapmessagelistener.java class. for more information mockrunner works please check my previous article mockrunner with spring jms . pom.xml 4.0.0 org.springframework.samples spring-int-jms-basic 0.0.1-snapshot 1.6 utf-8 utf-8 3.2.3.release 1.0.13 1.7.5 4.11 org.springframework spring-context ${spring-framework.version} org.springframework spring-tx ${spring-framework.version} org.springframework.integration spring-integration-core 2.2.4.release org.springframework.integration spring-integration-jmx 2.2.4.release org.springframework.integration spring-integration-jms 2.2.4.release org.slf4j slf4j-api ${slf4j.version} compile ch.qos.logback logback-classic ${logback.version} runtime org.springframework spring-test ${spring-framework.version} test junit junit ${junit.version} test com.mockrunner mockrunner-jms 1.0.3 javax.jms jms 1.1 org.codehaus.jackson jackson-mapper-asl 1.9.3 compile spring-int-jms.xml the endpoint is configured to connect to a jms server, fetch the messages,and publish them onto a local channel i.e inputchannel. where as connection-factory, and destination referred mockqueueconnectionfactory, and mockqueue(inboundqueue) beans from spring-mockrunner.xml file. inputchannel and outputchannel defined as queue channel objecttomaptransformer: object-to-map-transformer element that takes the payload from the input channel original here mockrunner-in-queue object message and emits a name-value paired map object onto the output channel i.e outputchannel and outboundjmsadapter bean fetch this message and publish to queue i.e mockrunner-out-queue. inboundjmsadapter : inbound-channel-adapter bean is responsible for receiving messages from a jms server here it is reading from mock queue name mockrunner-in-queue see objecttomaptransformertest.java class. outboundjmsadapter : outbound-channel-adapter bean is responsible to fetch messages from the channel i.e outputchannel and publish them to jms queue or topic. in this outbounjmsadapter reading message outputchannel as mapmessage and publish to outboundqueue(mockrunner-out-queue). mapmessagelistener.java package com.spijb.listener; import javax.jms.jmsexception; import javax.jms.mapmessage; import javax.jms.session; import org.slf4j.logger; import org.slf4j.loggerfactory; import org.springframework.jms.listener.sessionawaremessagelistener; public class mapmessagelistener implements sessionawaremessagelistener { private static final logger log = loggerfactory.getlogger(mapmessagelistener.class); @override public void onmessage(mapmessage message, session session) throws jmsexception { log.info("message received \r\n"+message); } } it is plain mapmessagelistener class to print received message from queue. department.java package com.spijb.domain; import java.io.serializable; public class department implements serializable{ private static final long serialversionuid = 1l; private final integer deptno; private final string name; private final string location; public department() { deptno=10; name="sales"; location="tx"; } public department(integer dno,string name,string loc) { this.deptno=dno; this.name=name; this.location=loc; } public integer getdeptno() { return deptno; } public string getname() { return name; } public string getlocation() { return location; } @override public string tostring() { return this.deptno+"-> "+this.name+"->"+this.location; } } domain object to send as a message, by default constructor assign deptno 10 , name as sales, location as tx also provide parameter constructor. spring junit class objecttomaptransformertest.java package com.spijb.invoker; import javax.jms.jmsexception; import javax.jms.message; import javax.jms.objectmessage; import javax.jms.session; import org.junit.test; import org.junit.runner.runwith; import org.springframework.beans.factory.annotation.autowired; import org.springframework.jms.core.jmstemplate; import org.springframework.jms.core.messagecreator; import org.springframework.test.context.contextconfiguration; import org.springframework.test.context.junit4.springjunit4classrunner; import com.mockrunner.mock.jms.mockqueue; import com.spijb.domain.department; @runwith(springjunit4classrunner.class) @contextconfiguration({"classpath:spring-mockrunner.xml","classpath:spring-int-jms.xml"}) public class objecttomaptransformertest { @autowired private jmstemplate jmstemplate; @autowired private mockqueue inboundqueue; @test public void shouldsendmessage() throws interruptedexception { final department defaultdepartment = new department(); jmstemplate.send(inboundqueue,new messagecreator() { @override public message createmessage(session session) throws jmsexception { objectmessage objectmessage = session.createobjectmessage(); objectmessage.setobject(defaultdepartment); return objectmessage; } }); thread.sleep(5000); } } spring with junit class where you can send message to inputchannel i.e inboundqueue using mockrunner. output : info: started inboundjmsadapter oct 06, 2014 1:24:25 pm org.springframework.integration.endpoint.abstractendpoint start info: started org.springframework.integration.config.consumerendpointfactorybean#1 13:24:26.882 [org.springframework.jms.listener.defaultmessagelistenercontainer#0-1] info c.spijb.listener.mapmessagelistener - message received com.mockrunner.mock.jms.mockmapmessage: {location=tx, name=sales, deptno=10} oct 06, 2014 1:24:30 pm org.springframework.context.support.abstractapplicationcontext doclose info: closing org.springframework.context.support.genericapplicationcontext@5840979b: startup date [mon oct 06 13:24:25 cdt 2014]; root of context hierarchy oct 06, 2014 1:24:30 pm org.springframework.context.support.defaultlifecycleprocessor$lifecyclegroup stop info: stopping beans in phase 2147483647 in the above highlighted one is output as map.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.