If you are using Amazon Web Services(AWS), you are probably aware how to access and use resources like SNS, SQS, S3 using key and secret. With the aws-java-sdk that is straight forward: AmazonSNSClient snsClient = new AmazonSNSClient( new BasicAWSCredentials("your key", "your secret")) One of the difficulties with this approach is storing the key/secret securely especially when there are different set of these for different environments. Using java property files, combined with maven or spring profiles might help a little bit to externalize the key/secret out of your source code, but still doesn't solve the issue of securely accessing these resources. Amazon has another service to help you in this occasion. No, no, this is not one more service to pay for in order to use the previous services. It is a free service, actually it is a feature of the amazon account. AWS Identity and Access Management (IAM) lets you securely control access to AWS services and resources for your users, you can manage users and groups and define permissions for AWS resources. One interesting functionality of IAM is the ability to assign roles to EC2 instances. The idea is you create roles with sets of permissions and you launch an EC2 instance by assigning the role to the instance. And when you deploy an application on that instance, the application doesn't need to have access key and secret in order to access other amazon resource. The application will use the role credentials to sign the requests. This has a number of benefits like a centralized place to control all the instances credentials, reduced risk with auto refreshing credentials and so on. Here is a short video demonstrating how to assign roles to an EC2 instance: Once you have role based security enabled for an instance, to access other resources from that instances you have to create and AwsClient using the chained credential provider: AmazonSNSClient snsClient = new AmazonSNSClient( new DefaultAWSCredentialsProviderChain()) The provider will search your system properties, environment properties and finally call instance metadata API to retrieve the role credentials in chain of responsibility fashion. It will also refresh the credentials in the background periodically depending on its expiration period. And finally, if you want to use role based security from Camel applications running on Amazon, all you have to do is create an instance of the client with configured chained credentials object and don't specify any key or secret: from("direct:start") .to("aws-sns://MyTopic?amazonSNSClient=#snsClient");

What would you expect from someone who is OFBiz and Camel committer? To integrate them for fun? Fine, here it is. In addition to being fun, I believe this integration will be of real benefit for the OFBiz community, because despite the fact of being a complete ERP software, OFBiz lacks the ability to easily integrate with external systems. The goal of this project is instead of reinventing the wheel and trying to integrate OFBiz with each system separately, integrate it with Camel and let Camel do what it does best: connect your application with every possible protocol and system out there. Quick OFBiz introduction The Apache Open For Business Project is an open source, enterprise automation software. It consist mainly from two parts: A full-stack framework for rapid business application development. It has Entity Engine for the data layer (imagine something like iBATIS and Hibernate combined). It is the same entity engine that powers millions of Attlasian Jira instances. But don't get me wrong, it is not meant for usage outside of OFBiz framework, so use it only as OFBiz data layer. Service Engine - this might be hard to grasp for someone only with DDD background, but OFBiz doesn't have any domain objects. Instead for the service layer it uses SOA and has thousands of services that contains the business logic. A service is an atomic bit of isolated business logic, usually reading and updating the database. If you need you can make services triggering each other using ECAs(event-condition-action) which is kind of rule engine that allows define pre/post conditions for triggering other service calls when a service is executed. The service itself can be written written in java, groovy or simple language (an XML DSL for simple database manipulation) and usually requires authentication, authorisation and finally executed in a transaction. UI widgets - an XML DSL which allows you easily create complex pages with tables, forms and trees. And the really great thing about this framework is that 'The whole is greater than the sum of its parts' - all of the above layers works together amazingly: if you have an entity definition (a table) in your data layer, you can use it in your service layer during your service interface definition or its implementation. It takes one line of code(a long one) to create a service which has as input parameters the table columns and return the primary key as result of the service. Then if you are creating a screen with tables or forms, you can base it on your entity definitions or service definitions. It is again only few lines of code to create a form with fields mapping to a service or entity fields. Out of the box business applications. These are vertical applications for managing the full life cycle of a business domain like ordering, accounting, manufacturing and many more in a horizontally integrated manner. So creating an order from order or ecommerce application will interact with facility to check whether a product is available, and after the order is created will create accounting transaction in accounting application. Before the order is shipped from the facility, it will create invoices and once the invoice is paid, it will close the order. You get the idea. Camel in 30 seconds Apache Camel is an integration framework based on known Enterprise Integration Patterns(EIP). Camel can also be presented as consisting of two artifacts: The routing framework which can be defined using java, scala, xml DSL with all the EIPs like Pipe, Filter, Router, Splitter, Aggregator, Throttler, Normalizer and many more. Components and transformers ie all the different connectors to more than 100 different applications and protocols like: AMQP, AWS, web services, REST, MongoDB, Twitter, Websocket, you name it. If you can imagine a tool, that enables you to consume data from one system, then manipulate the data (transform, filter, split, aggregate) and send it to other systems, using a declarative, concise, English-like DSL without any boilerplate code - that's Apache Camel. Let OFBiz talk to all of the systems Camel do The main interaction point with OFBiz are either by using the Entity Engine for direct data manipulation or by calling services through Service Engine. The latter is preferred because it ensures that the user executing the service is authorised to do so, the operation is transactional to ensure data integrity, and also all the business rules are satisfied (there might be other services that have to be executed with ECA rules). So if we can create an OFBiz endpoint in Camel and execute OFBiz services from Camel messages, that would allow OFBiz to receive notifications from Camel endpoints. What about the other way around - making OFBiz notify Camel endpoints? The ideal way would be to have an OFBiz service that sends the IN parameters to Camel endpoints as message body and headers and return the reply message as OFBiz service response. If you are wondering: why is it so great, what is an endpoint, where is the real world, who is gonna win Euro2012... have a look at the complete list of available Camel components, and you will find out the answer. Running Camel in OFBiz container I've started an experimental ofbiz-camel project on github which allows you to do all of the above. It demonstrates how to poll files from a directory using Camel and create notes in OFBiz with the content of the file using createNote service. The project also has an OFBiz service, that enables sending messages from OFBiz to Camel. For example using that service it is possible to send a message to Camel file://data endpoint, and Camel will create a file in the data folder using the service parameters. The integration between OFBiz and Camel is achieved by running Camel in an OFBiz container as part of the OFBiz framework. This makes quite tight integration, but ensures that there will not be any http, rmi or any other overhead in between. It is still WIP and may change totally. Running Camel and OFBiz separately Another approach is KISS: run Camel and OFBiz as they are - separate applications, and let them interact with RMI, WS* or something else. This doesn't require any much coding, but only configuring both systems to talk to each other. I've created a simple demo camel-ofbiz-rmi which demonstrates how to listen for tweets with a specific keyword and store them in OFBiz as notes by calling createNote service using RMI. It uses Camel's twitter and rmi components and requires only configuration. Notice that this example demonstrates only one way interaction: from Camel to OFBiz. In order to invoke a Camel endpoint from OFBiz you can you have to write some RMI, WS* or other code. PS: I'm looking forward to hear your real world integration requirements for OFBiz.

Way 1. Simple method call Object A calls a method on object B. This is clearly the simplest type of communication between two objects but is also the way which results in the highest coupling. Object A’s class has a dependency upon object B’s class. Wherever you try to take object A’s class, object B’s class (and all of its dependencies) are coming with it. Way 2. Decouple the callee from the caller Object A’s class declares an interface and calls a method on that interface. Object B’s class implements that interface. This is a step in the right direction as object A’s class has no dependency on object B’s class. However, something else has to create object B and introduce it to object A for it to call. So we have created the need for an additional class which has a dependency upon object B’s class. We have also created a dependency from B to A. However, these can be a small price to pay if we are serious about taking object A’s class off to other projects. Way 3. Use an Adaptor Object A’s class declares an interface and calls a method on that interface. An adaptor class implements the interface and wraps object B, forwarding calls to it. This frees up object B’s class from being dependent on object A’s class. Now we are getting closer to some real decoupling. This is particularly useful if object B’s class is a third-party class which we have no control over. Way 4. Dependency Injection Dependency injection is used to find, create and call object B. This amounts to deferring until runtime how object A will talk to object B. This way certainly feels to have the lowest coupling, but in reality just shifts the coupling problem into the wiring realm. At least before we could rely on the compiler to ensure that there was a concrete object on the other end of each call – and furthermore we had the convenience of using the development tools to help us unpick the interaction between objects. Way 5. Chain of command pattern The chain of command pattern is used to allow object A to effectively say “does anyone know how to handle this call?”. Object B, which is listening out for these cries for help, picks up the message and figures out for itself if it is able to respond. This approach does mean that object A has to be ready for the outcome that nobody is able to respond, however it buys us great flexibility in how the responder is implemented. Chain of command – way 5 – is the decoupling winner and here's an example to help explain why. Let object A be a raster image file viewer, with responsibilities for allowing the user to pick the file to open, and zoom in and out on the image as it is displayed. Let object B be a loader which has the responsibility of opening a gif file and returning an array of colored pixels. Our aim is to avoid tying object A's class to object B's class because object B's class uses a third party library. Additionally, object A doesn't want to know about how the image file is interpreted, or even if it is a gif, jpg, png or whatever. In this example object B, or more likely a wrapper of object B, will declare a method which equips it to respond to any requests to open an image file. The method will respond with an array of pixels if the file is of a format it recognizes, or respond with null if it does not recognize the format. The framework then simply asks handlers in turn until one provides a non-null response. With this framework in place we are now free to slide in more image loaders with the addition of just one more handler class. And furthermore, on the source end of the call, we can add other classes to not just view the images, but print them, edit them or manipulate them in any other way we choose. In conclusion, we can see that decoupling can be achieved and yield flexibility, but this does not mean it is appropriate for every call from one object to another. The best thing to do is start with straight method calls, but keep cohesion in mind. Then if at a later stage it becomes necessary to swap in and out different objects it won't be too hard to extract an interface and put in place a decoupling mechanism.

If you are using Visual Studio plugin for Git, but you have also configured Git with MSys git, probably you could be surprised by some Visual Studio behavior.

This Post explains Topics in Active MQ (Message Broker) with Subscribing and Publishing. For this we will write two java clients. As we did for wso2 Message Broker TopicSubscriber.java to Subcribe for messages TopicPublisher.java to to Publish the messages Let's Start. [1] Get Active MQ from http://activemq.apache.org/download.html [1.1] Start Active MQ from \bin\activemq.bat You can see the started server form http://localhost:8161/admin/ [2] Create Porject "Client" on IDE that you preferred [3] Add activemq-all-5.7.0.jar to lib Dir in the project (activemq-all-5.7.0.jar can be found in root folder) [4] Creat class "TopicSubscriber.java" to Subcribe for messages package simple; import java.util.Properties; import javax.jms.JMSException; import javax.jms.Message; import javax.jms.MessageListener; import javax.jms.Session; import javax.jms.TextMessage; import javax.jms.Topic; import javax.jms.TopicConnection; import javax.jms.TopicConnectionFactory; import javax.jms.TopicSession; import javax.naming.InitialContext; import javax.naming.NamingException; public class TopicSubscriber { private String topicName = "news.sport"; private String initialContextFactory = "" +"org.apache.activemq.jndi.ActiveMQInitialContextFactory"; private String connectionString = "tcp://" +"localhost:61616"; private boolean messageReceived = false; public static void main(String[] args) { TopicSubscriber subscriber = new TopicSubscriber(); subscriber.subscribeWithTopicLookup(); } public void subscribeWithTopicLookup() { Properties properties = new Properties(); TopicConnection topicConnection = null; properties.put("java.naming.factory.initial", initialContextFactory); properties.put("connectionfactory.QueueConnectionFactory", connectionString); properties.put("topic." + topicName, topicName); try { InitialContext ctx = new InitialContext(properties); TopicConnectionFactory topicConnectionFactory = (TopicConnectionFactory) ctx .lookup("QueueConnectionFactory"); topicConnection = topicConnectionFactory.createTopicConnection(); System.out .println("Create Topic Connection for Topic " + topicName); while (!messageReceived) { try { TopicSession topicSession = topicConnection .createTopicSession(false, Session.AUTO_ACKNOWLEDGE); Topic topic = (Topic) ctx.lookup(topicName); // start the connection topicConnection.start(); // create a topic subscriber javax.jms.TopicSubscriber topicSubscriber = topicSession .createSubscriber(topic); TestMessageListener messageListener = new TestMessageListener(); topicSubscriber.setMessageListener(messageListener); Thread.sleep(5000); topicSubscriber.close(); topicSession.close(); } catch (JMSException e) { e.printStackTrace(); } catch (NamingException e) { e.printStackTrace(); } catch (InterruptedException e) { e.printStackTrace(); } } } catch (NamingException e) { throw new RuntimeException("Error in initial context lookup", e); } catch (JMSException e) { throw new RuntimeException("Error in JMS operations", e); } finally { if (topicConnection != null) { try { topicConnection.close(); } catch (JMSException e) { throw new RuntimeException( "Error in closing topic connection", e); } } } } public class TestMessageListener implements MessageListener { public void onMessage(Message message) { try { System.out.println("Got the Message : " + ((TextMessage) message).getText()); messageReceived = true; } catch (JMSException e) { e.printStackTrace(); } } } } [5] Creat class "TopicPublisher.java" to to Publish the messages package simple; import javax.jms.*; import javax.naming.InitialContext; import javax.naming.NamingException; import java.util.Properties; public class TopicPublisher { private String topicName = "news.sport"; private String initialContextFactory = "org.apache.activemq" +".jndi.ActiveMQInitialContextFactory"; private String connectionString = "tcp://localhost:61616"; public static void main(String[] args) { TopicPublisher publisher = new TopicPublisher(); publisher.publishWithTopicLookup(); } public void publishWithTopicLookup() { Properties properties = new Properties(); TopicConnection topicConnection = null; properties.put("java.naming.factory.initial", initialContextFactory); properties.put("connectionfactory.QueueConnectionFactory", connectionString); properties.put("topic." + topicName, topicName); try { // initialize // the required connection factories InitialContext ctx = new InitialContext(properties); TopicConnectionFactory topicConnectionFactory = (TopicConnectionFactory) ctx .lookup("QueueConnectionFactory"); topicConnection = topicConnectionFactory.createTopicConnection(); try { TopicSession topicSession = topicConnection.createTopicSession( false, Session.AUTO_ACKNOWLEDGE); // create or use the topic System.out.println("Use the Topic " + topicName); Topic topic = (Topic) ctx.lookup(topicName); javax.jms.TopicPublisher topicPublisher = topicSession .createPublisher(topic); String msg = "Hi, I am Test Message"; TextMessage textMessage = topicSession.createTextMessage(msg); topicPublisher.publish(textMessage); System.out.println("Publishing message " +textMessage); topicPublisher.close(); topicSession.close(); Thread.sleep(20); } catch (InterruptedException e) { e.printStackTrace(); } } catch (JMSException e) { throw new RuntimeException("Error in JMS operations", e); } catch (NamingException e) { throw new RuntimeException("Error in initial context lookup", e); } } } [6] Firstly Run "TopicSubscriber.java" and then run "TopicPublisher.java" Here is out put from both TopicSubscriber:: Create Topic Connection for Topic news.sport Got the Message : Hi, I am Test Message TopicPublisher:: Use the Topic news.sport Publishing message ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId = ID:Madhuka-THINK-51683-1359787878456-1:1:1:1:1, originalDestination = null, originalTransactionId = null, producerId = null, destination = topic://news.sport, transactionId = null, expiration = 0, timestamp = 1359787878729, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false, text = Hi, I am Test Message} [More] Here is full message that we have send to TopicSubscriber. We can get that any parameter in above. Here is sample to get TimeStamp and ID from JMS message. public class TestMessageListener implements MessageListener { public void onMessage(Message message) { try { System.out.println("Got the Message TimeStamp: " + message.getJMSTimestamp()); System.out.println("Got the Message JMS ID : " + message.getJMSMessageID()); messageReceived = true; } catch (JMSException e) { e.printStackTrace(); } } } Now go to ActiveMQ server at http://localhost:8161/admin/ See that Topic and message count for that topic. Now you time to check more in 'Active MQ'

Let’s say you are writing an application in Ruby. You are probably talking to every API under the sun and are happily writing tests to make sure your code isn’t failing. Because you don’t want to rely on 3rd parties or an internet connection to make your tests pass or fail you mock everything with let’s say, Webmock. This also makes your tests much much faster. After all even the fastest internet is much slower than the processor talking to its memory. If you’re too lazy to mock out every API under the sun, you might use VCR to record requests and play them back later. The main advantage being, you don’t have to worry about meticulously reimplementing everything, and you can nuke the recordings at any time to make sure your code still works against the real API. Life is good. Enter Javascript, stage left Then Javascript becomes more and more prominent. Suddenly your application’s logic is shifting from backend to browser and before you know it, most of your tests are pretty irrelevant. You’re fine for a while with Capybara or Cucumber. Launch a headless browser, click around the site from the comfort of RSpec, make sure users see what they’re supposed to. Balance restored. Then you add a payment form. Or something. Suddenly your frontend is talking to an API. In case of Stripe or Balanced it’s even a feature. A great benefit for the user. jQuery(function($) { $('#payment-form').submit(function(event) { var $form = $(this); // Disable the submit button to prevent repeated clicks $form.find('button').prop('disabled', true); Stripe.createToken($form, stripeResponseHandler); // Prevent the form from submitting with the default action return false; }); }); Well that sucks, you’re suddenly back to square one. Your tests take minutes to execute. Your tests fail without an internet connection. Your tests rely on some 3rd party service being up. Your tests suck. Who wants to code when running ~5 tests takes 3 minutes? Nobody. Enter puffing-billy, stage right The problem is that neither Webmock nor VCR can handle requests originating in a browser because they happen in a different thread and they can’t mess around with those. Luckily, a year ago Olly Smith, created puffing-billy. The idea was great – spin up a web proxy, tell your headless browser to use it, when your code makes a request it will go through the proxy, which will try to use a Webmock to handle it, otherwise pass it on to the vast internet. But who wants to mock everything out manually? Over the past few weeks I set upon the task of fixing this problem and restoring sanity to my life. Good tests are transparent to the application and I’ll be damned if I use any of the suggested solutions on the internet like “Well you just put a switch in your code that knows if you’re in a test and then doesn’t talk to Stripe” Screw that. This morning I submitted a pull request to puffing-billy. I added the ability for puffing-billy to behave like it was VCR, but for your browser. When a request is made, it gets cached. The cache is then persisted between sessions, and requests are played back to the browser as needed. It’s not as sophisticated as VCR just yet, but it gets the job done and my test runtime has gone from 3 minutes to just under a minute. That’s a big deal in my book! The caching even understands that some URL’s are needlessly different on every request (social buttons, analytics etc.) so you can configure it to normalize those requests to a single recording that is played back every time. Your tests don’t really rely on gAnalytics working right? And the best thing is, you don’t even have to change your tests. You add something like this in your spec_helper.rb: Billy.configure do |c| c.cache = true c.ignore_params = ["http://www.google-analytics.com/__utm.gif", "http://b.siftscience.com/i.gif", "https://r.twimg.com/jot", "http://p.twitter.com/t.gif", "http://p.twitter.com/f.gif", "http://www.facebook.com/plugins/like.php", "https://www.facebook.com/dialog/oauth", "http://cdn.api.twitter.com/1/urls/count.json"] c.persist_cache = true c.cache_path = 'spec/req_cache/' end # need to call this because of a race condition between persist_cache # being set and the proxy being loaded for the first time Billy.proxy.restore_cache Capybara.javascript_driver = :poltergeist_billy A test for the payment form looks the same as usual: scenario "physical product" do product = start_buying build(:product, :physical, user: @seller, active: true) VCR.use_cassette('Balanced/purchase_with_cc') do within '#new_order' do fill_in 'order_email', with: Faker::Internet.safe_email fill_in_address fill_in_card click_on 'Buy Now' end page.should have_css('#receipt', :visible => true) end validate_receipt product, @seller end Puffing-billy will transparently cache every requests the browser makes and VCR records any requests made by your backend logic. It’s pretty sweet. What do you guys think? I only have 20 days of Ruby experience and the internet has told me it really wants something like this, but I couldn’t find anyone who’s already made it.

With unit tests you can check that your code behaviours just as you expect it to. When writing your unit tests you shouldn't need to worry about if any other area of the application is working correctly. The benefits of unit testing are: Decouples your code Write more modular classes Functions are smaller and more focused Your functions are more defensive Quality of code becomes higher You will find it easier to reuse code. When writing unit tests you just need to test this one method of your application, if your method relies on another class/variable there should be a way you can inject this into the method. This is where dependency injection in your code comes in handy, it will allow you to inject objects into your classes to change the output of the class. There are a few things you need to do to make a method unit testable, methods will need an input from a parameter or a class variable and it will need a return or set a class variable in the method. If the method hasn't got these things then the method can not be unit testable. If there isn't a return of the method then there is no way in knowing how the method performs. Dependency Injection Dependency injection is when your object has a dependency on another object. The simplest form to understand what dependency injection is to think of a setter method. A setter method will take one parameter and set a class variable from this parameter. This is using code injection to pass in a parameter to be used as the class variable value. public function setValue( $val ) { $this->val = $val; } Without dependency injection this method will look like this. public function setValue() { $this->val = 10; } For unit testing you need to be aware of any classes that your class is dependent on. For example if you have a login class that will connect to a database. class login { private $db = false; public function __construct() { $this->db = new Database(); } public function loginUser( $user, $password ) { $this->db->checkLogin( $user, $password ); } } This login class has a dependency of the class Database in the constructor, which means that we can't unit test this correctly. If we want to unit test this then the database class has to be development and tested. If the database class is broken and we try to unit test the loginUser() method the test will always fail and we won't know that it's the database class which is broke or the loginUser() method that is broke. If the database class is finished development, tested and data is in the database then we can use this for the loginUser() function. But now our tests are dependent on data being correct in the database. If we pass in a username and password it must be in the database for our test to pass. Our code could be correct but if the data isn't there then our unit tests will fail. This isn't correct use of unit tests and is more suited to be an integration test. To fix this problem we can use dependency injection to pass in a database connector which will set the database class variable. There are 2 ways we can inject a variable into a class, it can either be in the constructor of the class or by using a setter method. I tend to use constructor for all required dependences and use the setter method if there is a default value for the class variable. class login { private $db = false; public function __construct( $db ) { $this->db = $db; } public function loginUser( $user, $password ) { $this->db->checkLogin( $user, $password ); } } Now this class isn't dependant on a certain database class we can pass in the database class by using the parameter on the login class constructor. We can unit test this loginUser() method by first setting the $this->db class variable. We don't want to rely on a real database as the data can change so we can either create a test harness database class or you can mock the database class. A test harness class will allow you to create your database class and hardcode any data that you need. In the example above we can create a method checkLogin(), in our test harness we can then hardcode a successful login username and password to make the loginUser() method pass. Or you can use a PHP mocking framework to mock a class/method/return value. Both methods have their benefits but mocking is normally quicker to code, but there are times when you want to hardcode certain variables in a class. Mocking Objects In TDD With PHP Mocking objects in test driven development allows you create objects to act as a certain class, if your test depends on another method to return a value, you can mock this method and make it return any value you want. In the example we used above you can mock the database class and choose what value we are expecting back from the checkLogin() method. When mocking a method you can choose what you want to return from this method, therefore we can write tests to see what will happen when checkLogin() returns TRUE and then we can write another test to see what happens when checkLogin() returns FALSE. Mocking objects means that you can run your unit tests without depending on another class returning the values you are expecting, ao you can test just your code in this one method. Here are some of the most popular PHP mocking frameworks: Mocking with PHPUnit - http://www.phpunit.de/manual/3.0/en/mock-objects.html Mocking with Phake - http://phake.digitalsandwich.com/docs/html/ Mocking with Mockery - https://github.com/padraic/mockery Mocking with Enchane PHP - https://github.com/Enhance-PHP/Enhance-PHP Mocking with FBMock - https://github.com/facebook/FBMock Dependency Injection With Interfaces If we are going to pass in a database connector in a constructor of the login class, then this database connector will always have to have a method of checkLogin(). This is why we should code our dependences by using interfaces to make sure that we are always passing in the correct type of class. class login { private $db = false; public function __construct( IDatabase $db ) { $this->db = $db; } } class database implements IDatabase { public function checkLogin( $username, $password ) { // check the login credentials } } interface IDatabase { public function checkLogin( $username, $password ); } This will make sure that the class we pass into the constructor is a type of IDatabase, so if our database class doesn't implement IDatabase then the code will fail and therefore our unit tests will fail. This means whatever we pass into the constructor we know that this class will be able to run the methods it needs for the unit tests to run.

When creating some Class Library you should pay attention to the visibility of its members and have a clear vision of what you’d like to expose to its users and what on the other side should be hidden. When writing unit tests against such assemblies however, you obviously want to test everything, from the internal members to the externally exposes parts. Assume you have a class library named Base containing the following classes and interfaces ICommandHandler - which is a publicly exposed interface CommandHandler - basically its concrete implementation IUndoRedoStack - an interface that is used only internally UndoRedoStack - its concrete implementation Note that I’m programming exclusively against interaces as that’s a major requirement for being able to create the necessary isolation for testing each component on its own. Testing Classes with Internal Visibility Lets take a closer look at the CommandHandler class class CommandHandler : ICommandHandler { //... } When creating a test for the CommandHandler class you would proceed as follows [TestClass] public class CommandHandlerTest { private CommandHandler commandHandler; [TestInitialize] public void Setup() { commandHandler = new CommandHandler(); } [TestCleanup] public void Teardown() { commandHandler = null; } [TestMethod] publiic void ShouldExecuteAGivenCommand() { //the test content } } When you execute such test, it won’t compile however. A best practice is to place the tests in a separate DLL (I usually name it like Base.UnitTests if the tested assembly is called Base) and as such, CommandHandler won’t be visible as it has been defined to only have internal visibility. In a previous blog post I already explained on how to overcome this issue, namely by specifying the InternalsVisibleTo attribute in the tested assembly. Check out that blog post for more details. Mocking Interfaces with Internal Visibility using Moq Now, CommandHandler has a dependency on IUndoRedoStack class CommandHandler : ICommandHandler { public CommandHandler(IUndoRedoStack undoRedoStack) { //... } } The CommandHandler has an Execute(command) method and suppose we’d like to test the fact that when calling it with a given ICommand object, that specific object gets added to the undoRedoStack. We would write [TestInitialize] public void Setup() { mockUndoRedo = new Mock

the most important part of building an integration with an api is actually writing the code that will connect with the web service and invoke its methods. i'll show you why using the façade pattern to decouple calls from your existing code is a good idea and help you identify what kind of problems you might be able to prevent. so, first things first, what is the façade pattern? a façade is an object that provides simple access to complex - or external - functionality. it might be used to group together several methods into a single one, to abstract a very complex method into several simple calls or, more generically, to decouple two pieces of code where there's a strong dependency of one over the other. what happens when you develop api calls inside your code and, suddenly, the api is upgraded and some of its methods or parameters change? you'll have to change your application code to handle those changes. also, by changing your internal application code, you might have to change the way some of your objects behave. it is easy to overlook every instance and can require you to doublecheck multiple lines of code. there's a better way to keep api calls up-to-date. by writing a façade with the single responsibility of interacting with the external web service, you can defend your code from external changes. now, whenever the api changes, all you have to do is update your façade. your internal application code will remain untouched. from a test driven development point-of-view, using a facade offers a big advantage. you're now able to write simple tests against the façade without affecting your internal code test results. by using this strategy, you'll be able to know immediately whenever an api is not working as you expected and make the necessary changes to the façade. this is the approach we follow at cloudwork when building integrations between any third-party apis. api façades act as tight compartments that protect the rest of the application from external changes and simplify the way we interact with different web services. this article is cross-posted at using facades to decouple api integrations .

Introduction The IT buzzword of 2012 is without a doubt Big Data. It’s new and here to stay, and for a good reason. Big data is data that exceeds the processing capacity of conventional database systems. Great examples are CERN with the Large Hadron Collider, whose experiments generate 25 petabytes of data annually, or Walmart, which handles more than one million customer transaction every hour. Problems These vast amounts of data leave us with two problems. Problem 1: To gain value from this data, one must choose an alternative way to process it. The value of big data to an organization falls into two categories: analytical use, and enabling new products. Big data analytics can reveal insights hidden previously by data too costly to process, such as peer influence among customers, revealed by analyzing shoppers’ transactions, social and geographical data. Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data, in contrast to the somewhat static nature of running predetermined reports. Problem 2: The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. Remember the CERN case where the LHC produces over 25 Petabytes of data annually? No “classic” database architecture or setup is capable of holding these amounts of data. Solutions Fortunately, both problems can be solved by implementing the correct infrastructure and rethinking data storage. There are two critical factors in Big Data environments: size and speed. We already discussed the vast amounts of data and desire to be able to access and process the data fast. The latter is the main differentiator from more traditional data warehouses. Just imagine what you can do when you can access all your data real-time. Enter big data. A common Big Data implementation is an in-memory data grid that lives in a distributed cluster, ensuring both speed, by storing data in-memory, and capacity by using scalability features provided by a cluster. As a bonus, availability is ensured by using a distributed cluster. As for the data storage, there are typically two kinds: in-memory databases and in-memory data grids. But first some background. It is not a new attempt to use main memory as a storage area instead of a disk. In our daily lives there are numerous examples of main memory databases (MMDB), as they perform much faster than disk-based databases. An every day example is a mobile phone. When you SMS or call someone most mobile service providers use MMDB to get the information on your contact as soon as possible. The same applies to your phone. When someone calls you, the caller details are looked up in the contacts application, usually providing a name and sometimes a picture. In memory data grids In Memory Data Grid (IMDG) is the same as MMDB in that it stores data in main memory, but it has a totally different architecture. The features of IMDG can be summarized as follows: Data is distributed and stored on multiple servers. Each server operates in the active mode. A data model is usually object-oriented (serialized) and non-relational. According to the necessity, you often need to add or reduce servers. No traditional database features such as tables. In other words, IMDG is designed to store data in main memory, ensure scalability and store an object itself. These days, there are many IMDG products, both commercial and open source. Some of the most commonly used products are: Hazelcast (http://www.hazelcast.com) JBoss Infinispan (http://www.jboss.org/infinispan) GridGain DataGrid (http://www.gridgain.com/features/in-memory-data-grid/) VMware Gemfire (http://www.vmware.com/nl/products/application-platform/vfabric-gemfire/overview.html) Oracle Coherence (http://www.oracle.com/technetwork/middleware/coherence/overview/index.html) Gigaspaces XAP (http://www.gigaspaces.com/datagrid) Terracotta Enterprise Suite (http://terracotta.org/products/enterprise-suite) Why Memory? The main reasons for using main memory for data storage are once again the two main themes of Big Data: speed and capacity. The processing performance of main memory is 800 times faster than an HDD and up to 40 times faster than an. Moreover, the latest x86 server supports main memory of hundreds of GB per server. It is said that the limit of a traditional processing database’s (OLTP) data capacity is approximately 1 TB and that the OLTP processing data capacity would not increment well. If servers using main memory of 1 TB or larger become more commonly used, you will be able to conduct operations with the entire data placed in main memory, at least in the field of OLTP. IMDG Architecture To use main memory as a storage area, two weak points should be overcome: Limited capacity: involves data that exceeds the maximum capacity of the main memory of the server Reliability: involves data loss in case of a (system) failure. IMDG overcomes the limit of capacity by ensuring horizontal scalability using a distributed architecture, and resolves the issue of reliability through a replication system as part of the grid (or a distributed cluster). Now let’s discuss how an IMDG actually works. First of all, it is important to understand that an IMDG is not the same as an in-memory database, also referred to as MMDB (main memory databases). Typical examples of MMDBs are Oracle TimesTen or Sap Hana. MMDBs are full database products that simply reside in memory. As a result of being a full-blown database, they also carry the weight and overhead of database management features. IMDG is different. No tables, indexes, triggers, stored procedures, process managers etc. Just plain storage. The data model used in IMDG is key-value pairs. A key-value pair is a list with only two parts: a key and a value. The key can be used for storing and retrieving the values in the list. A key can be compared to the index or primary key of a table in a database. Note that IMDG are closely tied to development environments such as Java as the key-value pairs are represented by the structures provided by such a programming environment. Most IMDGs are written in Java, and can only be used within other Java applications. Therefore, the values of key-value pairs can be anything supported by Java, ranging from simple data types such as a string or number, to complex objects. This overcomes the two important hurdles: as you can store complex Java objects as value, there’s no need to translate these objects into a relational datamodel (which is the case in more traditional applications using a database for storage). Furthermore, the seeming limitation of being able to store only one value per key, is actually no limitation at all. Large memory sizes Most of the products introduced above use Java as an implementation language. Java reserves and uses a part of the RAM (internal memory) for dynamic memory allocation. This reserved memory space is called the Java heap. All runtime objects created by a Java application are stored in heap. Using large amounts of data causes two problems. Size limitation: By default, the heap size is 128 MB, but for current business applications, this limit is reached easily. Once the heap is “full”, no new objects can be created and the Java application will show some nasty errors. Performance: It is possible to increase the size of the heap, but this introduces some new problems. When a heap reaches a size of more than 4 gigabytes, Java will have serious issues with memory managements, causing your application to slow down or even freeze. Java has a feature called Garbage Collector, which periodically scans the heap and checks each object if it is still valid and being used. If not, the garbage collector removes the object and defragments the newly available space. The problem is, the larger the heap size, the more work to do for the garbage collector, resulting in performance degradation. Imagine a large bank has a Java application that manages customers, accounts and transactions. We have seen that an IMDG allows the application to store and access all data very quickly by caching it in memory, instead of storing the data in relatively slow databases. Let’s assume the combined data has a size of 40 gigabytes. Storing it in heap is simply not possible, considering the performance penalties of Java’s memory management capabilities. The graph below illustrates the garbage collection pause time when placing cached data in heap: Terracotta’s BigMemory product has a method to overcome these limitations. The method is to use an off-heap memory (direct buffer). Data will not be stored in Java’s heap, but directly in the available internal memory (RAM). Since, this is not subject to Java’s garbage collector, there are no performance penalties. The differences on performance are significant, as can be seen in the graph below: Using off-heap storage has some major benefits: You can use all the available memory on your machine, not just the memory that is allocated to the heap (usually less that 512 Mb). This allows you to store more data in a in-memory data grid, greatly speeding up your application. The heap can be relieved by storing data in native memory, speeding up Java applications as less heap space has to be garbage collected. Clustering, fail over and high availability So far, we have seen IMDG features that are applicable to a single server. However, the real power of IMDG lies in it’s networking and clustering capabilities, providing features as data replication, data synchronization between clients, fail over and high availability. To achieve this, a cluster of servers (or server array) acts a backbone of the infrastructure. Applications (that still can have their own IMDG or off-heap cache) that are connected to the cluster can share, replicate and backup their data with either the cluster or other applications. The graph below depicts a typical setup using Terracotta's BigMemory: The caches on the application servers are usually referred to as “level 1” cache, while the data cache on the server array is referred to as “level 2” cache. There are many different scenarios possible for storing, clustering, synchronizing and replicating data. Covering all these topics goes far beyond the scope of this article. For more information, consult the technical documentation of the product of your choice. Conclusion Big Data brings us some new challenges. First of all, storing and accessing vast amounts of data makes us rethink traditional methods and technologies. Next, there’s the question what to do with all the available data. The potential value for marketing, financial and other businesses is huge. In order to facilitate Big Data, in-memory data grids are considered the best option. IMDGs with off-heap storage are even more powerful, allowing data centric enterprise application to overcome certain limits of the Java platform, such as memory and performance constraints. As the amount of data that (large) companies produce and store, grows exponentially, databases will hit a limit. Accessing your data without a performance penalty simply will not be possible. The answer to this is using an IMDG.

Need help dealing with Maven's non-resolvable parent problem? Check out this post to learn how.

n this post i’m going to cover how to do job chaining in quartz versus obsidian scheduler . both are java job schedulers, but they have different approaches so i thought i’d highlight them here and give some guidance to users using both options. it’s very common when using a job scheduler to need to chain one job to another. chaining in this case refers to executing a specific job after a certain job completes (or maybe even fails). often we want to do this conditionally, or pass on data to the target job so it can receive it as input from the original job. we’ll start with demonstrating how to do this in quartz, which will take a fair bit of work. obsidian will come after since it’s so simple. chaining in quartz quartz is the most popular job scheduler out there, but unfortunately it doesn’t provide any way to give you chaining without you writing some code. quartz is a low-level library at heart, and it doesn’t try to solve these types of problems for you, which in my mind is unfortunate since it puts the onus on developers. but despite this, many teams still end up using quartz, so hopefully this is useful to some of you. i’m going to outline probably the most basic way to perform chaining. it will allow a job to chain to another, passing on its jobdatamap (for state). this is simpler than using listeners, which would require extra configuration, but if you want to take a look, check out this listener for a starting point. sample code this will rely on an abstract class that will provided basic flow and chaining functionality to any subclasses. it acts as a very simple template class. first, let’s create the abstract class that gives us chaining behaviour: import static org.quartz.jobbuilder.newjob; import static org.quartz.triggerbuilder.newtrigger; import org.quartz.*; import org.quartz.impl.*; public abstract class chainablejob implements job { private static final string chain_job_class = "chainedjobclass"; private static final string chain_job_name = "chainedjobname"; private static final string chain_job_group = "chainedjobgroup"; @override public void execute(jobexecutioncontext context) throws jobexecutionexception { // execute actual job code doexecute(context); // if chainjob() was called, chain the target job, passing on the jobdatamap if (context.getjobdetail().getjobdatamap().get(chain_job_class) != null) { try { chain(context); } catch (schedulerexception e) { e.printstacktrace(); } } } // actually schedule the chained job to run now private void chain(jobexecutioncontext context) throws schedulerexception { jobdatamap map = context.getjobdetail().getjobdatamap(); @suppresswarnings("unchecked") class jobclass = (class) map.remove(chain_job_class); string jobname = (string) map.remove(chain_job_name); string jobgroup = (string) map.remove(chain_job_group); jobdetail jobdetail = newjob(jobclass) .withidentity(jobname, jobgroup) .usingjobdata(map) .build(); trigger trigger = newtrigger() .withidentity(jobname + "trigger", jobgroup + "trigger") .startnow() .build(); system.out.println("chaining " + jobname); stdschedulerfactory.getdefaultscheduler().schedulejob(jobdetail, trigger); } protected abstract void doexecute(jobexecutioncontext context) throws jobexecutionexception; // trigger job chain (invocation waits for job completion) protected void chainjob(jobexecutioncontext context, class jobclass, string jobname, string jobgroup) { jobdatamap map = context.getjobdetail().getjobdatamap(); map.put(chain_job_class, jobclass); map.put(chain_job_name, jobname); map.put(chain_job_group, jobgroup); } } there’s a fair bit of code here, but it’s nothing too complicated. we create the basic flow for job chaining by creating an abstract class which calls a doexecute() method in the child class, then chains the job if it was requested by calling chainjob() . so how do we use it? check out the job below. it actually chains to itself to demonstrate that you can chain any job and that it can be conditional. in this case, we will chain the job to another instance of the same class if it hasn’t already been chained, and we get a true value from new random().nextboolean() . import java.util.*; import org.quartz.*; public class testjob extends chainablejob { @override protected void doexecute(jobexecutioncontext context) throws jobexecutionexception { jobdatamap map = context.getjobdetail().getjobdatamap(); system.out.println("executing " + context.getjobdetail().getkey().getname() + " with " + new linkedhashmap(map)); boolean alreadychained = map.get("jobvalue") != null; if (!alreadychained) { map.put("jobtime", new date().tostring()); map.put("jobvalue", new random().nextlong()); } if (!alreadychained && new random().nextboolean()) { chainjob(context, testjob.class, "secondjob", "secondjobgroup"); } } } the call to chainjob() at the end will result in the automatic job chaining behaviour in the parent class. note that this isn’t called immediately, but only executes after the job completes its doexecute() method. here’s a simple harness that demonstrates everything together: import org.quartz.*; import org.quartz.impl.*; public class test { public static void main(string[] args) throws exception { // start up scheduler stdschedulerfactory.getdefaultscheduler().start(); jobdetail job = jobbuilder.newjob(testjob.class) .withidentity("firstjob", "firstjobgroup").build(); // trigger our source job to triggers another trigger trigger = triggerbuilder.newtrigger() .withidentity("firstjobtrigger", "firstjobbtriggergroup") .startnow() .withschedule( simpleschedulebuilder.simpleschedule().withintervalinseconds(1) .repeatforever()).build(); stdschedulerfactory.getdefaultscheduler().schedulejob(job, trigger); thread.sleep(5000); // let job run a few times stdschedulerfactory.getdefaultscheduler().shutdown(); } } sample output executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=5420204983304142728, jobtime=sat mar 02 15:19:29 pst 2013} executing firstjob with {} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=-2361712834083016932, jobtime=sat mar 02 15:19:31 pst 2013} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=7080718769449337795, jobtime=sat mar 02 15:19:32 pst 2013} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=7235143258790440677, jobtime=sat mar 02 15:19:33 pst 2013} executing firstjob with {} deficiencies well, we’re up and chaining, but there are some problems with this approach: it doesn’t integrate with a container like spring to use configured jobs. more code would be required. it forces you to know up front which jobs you want to chain, and write code for it. configuration is fixed, unless, once again, you write more code. no real-time changes (unless you write more code). a fair bit of code to maintain , and high likelihood you will have to expand it for more functionality. the theme here is that it’s doable, but it’s up to you to do the work to make it happen. obsidian avoids these problems by making chaining configurable, instead of it being a feature of the job itself. read on to find out how. chaining in obsidian in contrast to quartz, chaining in obsidian requires no code and no up-front knowledge of which jobs will chain or how you might want to chain them later. chaining is a form of configuration, and like all job-related configuration in obsidian, you can make live changes at any time without a build or any code at all. job configuration can use a native rest api or the web ui that’s included with obsidian. the following chaining features are available for free: no code and no redeploy to add or remove chains. you can chain specific configurations of job classes. you can chain only on certain states, including failure. chain conditionally based on source job saved state (equivalent to quartz’s jobdatamap), including multiple conditions. regexp/equals/greater than, etc. chain only when matching a schedule. check out the feature and ui documentation to find out more. now that we know what’s possible, let’s see an example. once you have your jobs configured , just create a new chain using the ui. rest api support will be here shortly but as of 1.5.1 chaining isn’t included in the api. if you need to script this right now, we can provide pointers . in the ui, it looks like the following: easy, huh? all configuration is stored in a database, so it’s easy to replicate it in various environments or to automate it via scripting. as a bonus, obsidian tracks and shows you all chaining state including what job triggered a chained job. it will even tell you why a job chain didn’t fire, whether it’s because the job status didn’t match, or one of your conditions didn’t. conclusion that summarizes how you can go about chaining in quartz and obsidian. quartz definitely has a minimalist approach, but that leaves developers with a lot of work to do. meanwhile, obsidian provides rich functionality out of the box to keep developers working on their own rich functionality, instead of the plumbing that so often seems to consume their time. if you have any suggestions or feature requests for obsidian, drop us a note by leaving a comment or by contacting us .

I've already written about my experiments with Paratest. Paratest is a PHPUnit wrapper that allows you to run tests written for PHPUnit in parallel, making us of multiple processes running on the same machine. In a world where cycle time is an important metric, trading resources to get the test suite to finish earlier is a net gain; especially when you're stepping on unstable stones and run the suite very often. Rationale Our clock speeds aren't going to increase anymore; you should already know that the direction in which the CPU architectures are heading is to provide multiple cores instead. A PHPUnit process executed serially can get to 100% CPU, while executing two processes can theoretically double your performance, in essence halving the time taken for running your test suite. Moreover, while unit tests do not touch external resources, integration and end-to-end ones are relying on the response time of a database, network calls (even towards the same machine), files to be written, and so on. This means the CPU is not being fully utilized, because a percentage of the time is spent waiting for these resources to become available or to respond. The presence of an high iowait makes room for a number of processes greater than the number of cores available. In some cases, the resources you are using do not depend on the local machine, but are provided remotely in unlimited number. Facebook runs tests in parallel on multiple machines, but for an example available to any wallet check out SauceLabs: providing to you many different Selenium Servers from their cloud, so that you can run tests on N different browser instances at the same time. The thing you have to keep in mind is that tests must be totally independent from each other to be executed. Clearing the whole database before a test or counting the rows of an entire table aren't a suitable option when parallelizing, as tests would interfere with each other. Recent contributions Paratest delegates to a Runner object the execution of a test suite, and collects log files to aggregate the results into a single count of passed and failed tests (among with other statistics such as assertions, errors and time). The default runner creates a new php process for each test class (or even test method with the --functional configuration). This is fine for large scale tests that take in the order of tens of seconds to be run, but there is indeed a slow down with unit tests. This decrease in performance is noticeable whenever there is a sizeable PHPUnit bootstrap file, which is executed again for each of the tests. For this reason, I contributed a new Runner, named WrapperRunner. This Runner executes only a fixed number of processes (which I called Workers, with common terminology) and that take PHPUnit commands on their standard input: $ bin/phpunit-wrapper phpunit tests/MyTest.php ... phpunit tests/OtherTest.php ... Since the bootstrap file is require_once()d into the process, it is only executed once for each Worker, so if you have 1000 tests executed in 5 processes it will be executed 5 times instead of 1000. The tests are capable of reusing the same bootstrap because they previously ran serially after a single execution of it: if they are capable of running in parallel at all, they should share the bootstrap flawlessly. I will probably use Gearman in the future to manage workers as Behat does, but for now this is a very easy-to-install solution: only PHP code. With this contribution, Paratest gets back to a comparable time to PHPUnit for unit tests, while before it was doubling the (short) time required. The more cores we get, the more this time could go down. The WrapperRunner is currently an open PR. Another contribution from Dimitris Baltas and that I ported also to the new WrapperRunner regards providing an environment variable named TEST_TOKEN. If tests are using a shared resource such as a database table, they cannot easily run in parallel if they are working on the same data or reading globally, for example counting the documents in a Mongo collection The TEST_TOKEN shuld be used to map each process to a different instance of the shared resource: for example, a set of schemas database1, database2, ..., database5. The token is guaranteed to be given to a single test at any time in both Runners. This pull request is open too and will be merged into Paratest soon.

It didn’t take long. A few months after we released an open source continuous integration tool (Anthill) in 2001, we were asked, “It’s great that I have the build setup, now how do I deploy to the test lab?” That email started a clear transformation in our thinking. Lesson 1: Builds generally exist to be tested or released to customers.The corollary is: Continuous integration is not about build, it is about quality and checking quality generally requires a deployment or six. In 2005, we updated our AnthillPro 2.5 with a shiny new deployment capability. It worked ok, but with that generation of tool being so build oriented, it was never a clean fit. Lesson 2: Deployments are a serious challenge, and can not be bolted on to a build tool. This lesson has been reinforced over the years as we’ve watched the results of various tools tacking on deployments. In 2006, we released AnthillPro 3. Oh boy, what a change. Now deployments and other stuff you do to builds days and weeks later were their own thing. Demonstrating a stunning lack of marketing savvy, we called them “non-originating workflows” (processes that don’t make builds) and declared a new type of tool, an “Application Lifecycle Automation Server”. Today, you’d call it a Continuous Delivery tool (thank you Jez and Dave for the better name). To my knowledge, AnthillPro was the first tool to really take a build lifecycle or build pipeline seriously and had the ambition to manage from continuous integration build through to production release quickly and with a solid audit trail. That brings us to Lesson 3: When you come from the development side of the house as either an engineer or a vendor, you have a lot to learn about audit and security. Around the same time we also learned a great deal about the Dev / Ops gap. While some of our customers implemented the tool as intended, most of the time it was used either by a dev organization doing continuous integration and deployment to the early test environments or a production support / release management group that only did the “official” builds and focused on the production release. The failure of the developer initiated efforts to get to production provided a key insight. Lesson 4: When it comes to deployment automation, start with production, and work your way to the lower environments, treating them as a simpler case of the hard problem. We also learned about the limitations of a build pipeline approach in complex applications. Lesson 5: Deployments of complex systems often require a coordinated release of many builds. The need to start at production style deployments (and not care as much about the build) and the emphasis on deployment time dependencies focused our efforts on a deployment centric tool, uDeploy to compliment the pipeline approach in AnthillPro. Production and scaled usage come together to make the deployment tool a critical piece of infrastructure. If you lose a datacenter, can you recover the tool and use it to recover other applications into a backup datacenter?Lesson 6: Your release and deployment systems need to be configured with high availability and recoverability in mind. For the better part of the last decade, we’ve been trying to streamline deployments and kill off the release weekend. Many of customers have eliminated or shrunk those big events down to a more manageable size. Big releases though still can require the coordination of dozens of people across various teams. One time events like configuring the deployment tool with the password for a new database in each environment, or routine manual steps like putting eyeballs on the new production system before putting it out to the public still need to be managed. Lesson 7: Deployment is still just one part of release. With that in mind, we’ve added uRelease to the portfolio. Different people in the application delivery value chain need different tools. Developers need continuous integration and great testing. Change management needs great auditing around production deployments. Testers need their environments to be not broken, updated regularly and close enough to production to make their testing worthwhile. The patterns and needs overlap. Everyone needs deployments that work. Everyone benefits from understanding the lifecycle of a build (or other version) as well as what is in a given environment. Everyone benefits when release planning meetings are shorter. Lesson 8: No single tool is enough. An integrated toolchain that exchanges information is required. So there you go. Twelve years of tooling, eight with deployments. Summed up in six over-simplified lessons. I’m sure we’ll be updating this next year. Lesson 1: Builds generally exist to be tested or released to customers. Lesson 2: Deployments are a serious challenge, and can not be bolted on to a build tool. Lesson 3: When you come from the development side of the house as either an engineer or a vendor, you have a lot to learn about audit and security. Lesson 4: When it comes to deployment automation, start with production, and work your way to the lower environments, treating them as a simpler case of the hard problem. Lesson 5: Deployments of complex systems often require a coordinated release of many builds. Lesson 6: Release and deployment systems need to be configured with high availability and recoverability in mind. Lesson 7: Deployment is still just one part of release Lesson 8: No single tool does “DevOps”. An integrated toolchain that exchanges information is required. What are the biggest lessons you’ve learned about deployments in the last few years? Share in the comments below!

talend has several connectors to integrate sap systems. however, this guide is no introduction to talend’s sap components. instead, this guide helps to understand different alternatives to integrate sap systems with talend set up a local sap system configure talend studio for using sap components use talend’s sap wizard run a first talend job which connects to sap all further required information and example use cases for talend’s sap components should be available in the talend component guide at www.help.talend.com . if that’s not the case, please create a jira documentation ticket ( https://jira.talendforge.org/browse/doct )! now let’s take a look at different alternatives for integration of sap systems with talend. alternatives for sap integration three protocols exist for communication between sap and external programs: dynamic information and action gateway (diag): e.g. used by sap gui remote function call (rfc): a function call with input and output parameters (like a java interface) hypertext transfer protocol (http): internet standard the following alternatives are available for integrating sap systems using some of these protocols. file sap supports the direct import of files (call-transaction-program, batch-input, direct input). files have to be in a specific format to be imported. transformation and integration can be realized with talend’s various file components such as tfileinputdelimited. rfc remote function call is the proprietary sap ag interface for communication between a sap system and other sap or third-party compatible system over tcp/ip or cpi-c connections. remote function calls may be associated with sap software and abap programming, and provide a way for an external program (written in languages such as php, asp, java, or c, c++) to use data returned from the server. data transactions are not limited to getting data from the server, but can insert data into server records as well. sap can act as the client or server in an rfc call. a remote function call (rfc) is the call or remote execution of a remote function module in an external system. in the sap system, these functions are provided by the rfc interface system. the rfc interface system enables function calls between two sap systems, or between a sap system and an external system. tsapinput and tsapoutput are talend’s components to use rfcs. business application programming interface (bapi) a bapi is an object-oriented view on most data and transactions of a sap system (called “business objects”). object types of the business objects are stored in the business object repository (bor). bapis are always implemented as rfcs and therefore can be called the same way. additionally, they have the following characteristics (compared to rfcs): stable interface no view layer no exceptions, instead export parameter: “return” most business objects offer the following standard bapis: getlist getdetail change creationfromdata tsapinput and tsapoutput are talend’s components to use bapis. application link enabling (ale) application link enabling (ale) is used for asynchronous messaging between different systems via “intermediate documents (idoc)”. idoc is a sap document format for business transaction data transfers. it is used to realize distributed business processes. idoc is similar to xml in purpose, but differs in syntax. both serve the purpose of data exchange and automation in computer systems, but the idoc technology takes a different approach. while xml allows having some metadata about the document itself, an idoc is obligated to have information at its header like its creator, creation time, etc. while xml has a tag-like tree structure containing data and meta-data, idocs use a table with the data and meta-data. idocs also have a session that explains all the processes which the document passed or will pass, allowing one to debug and trace the status of the document. an idoc consists of control record (it contains the type of idoc, port of the partner, release of sap r/3 which produced the idoc, etc.) data records of different types. the number and type of segments is mostly fixed for each idoc type, but there is some flexibility (for example an sd order can have any number of items). status records containing messages such as 'idoc created', 'the recipient exists', 'idoc was successfully passed to the port', 'could not book the invoice because...' different idoc types are available to handle different types of messages. for example, the idoc format orders01 may be used for both purchase orders and order confirmations. tsapidocinput and tsapidocoutput are talend’s components to use ale / idoc. bapis can also be called asynchronously via ale. all new idocs are even based on bapis. soap web services sap supports soap web services. not just sap as java, but also sap as abap! integration can be realized with talend’s esb / web service components such as tesbrequest, tesbresponse, or tesbconsumer. installation of sap server and client installation can take about 6 to 8 hours, but it is an “all in one installation”, i.e. you can install it overnight. steps for installation: get yourself a windows 7 64 bit laptop or vm with 8+ gb ram and 50+gb free disc space get a sap community account (for free, just register): http://scn.sap.com/welcome download sap netweaver (software downloads --> sap netweaver main releases: http://www.sdn.sap.com/irj/scn/nw-downloads download current version of sap netweaver application server abap 64-bit trial install sap server: follow installation guide – a html website included in the download in root of extracted download folder (start.htm --> there click on “installation” link) install sap gui (rich client frontend): start.htm --> there click on “install sap gui” link and follow instructions download the sap jco for the operating system on which your connector is running. the sap jco is available for download from sap's website at http://service.sap.com/connectors . you must have an sapnet account to access the sap jco (if you do not already have one, contact your local sap basis administrator). usage of sap server hint: you have to use a windows user which has a password (as you need to enter windows credentials when stopping sap). if you have a windows user without a password (for instance if you use windows within a vm on your mac), sap cannot process these credentials (i.e. it cannot process an empty password field) --> change your windows password before starting sap start the management console (windows startmenu --> programs --> sap management console) start and stop the sap server (right click on “nsp” --> start / stop) default user: sap* (sap system super user) password: the one which you entered at installation of sap netweaver, e.g. admin123 usage of sap client a sap client should be used to get information about the sap system (functions, data, etc.) similarly to using e.g. mysql workbench to get information from a mysql database. sap gui (view layer) communicates with sap as abap (business logic layer). the application server communicates with the relational database (db layer). different clients are available for sap: sap gui windows sap gui java web browser external rfc-program for local development demos, sap gui windows is probably the best alternative. start sap gui windows by: clicking shortcut “windows start menu --> sap frontend --> sap logon” entering username and password clicking logon sap transactions in sap, you call sap programs via sap transaction codes. important transactions codes are for example: bapi: bapi explorer, view all sap bapi's se16: data browser, view/add table data se38: program editor here is a list of several other important transaction codes: http://www.sapdev.co.uk/tcodes/tcodes.htm installation of demo data the sap installation includes some demo data. as most people do not want to install “real” sap modules such as sap fi, sap crm or sap bi on their local system, this demo data is perfect for demos using talend’s sap connectors. to install the flight demo on a local sap system, you just have to open the abap editor (transaction: se38) and execute the program sapbc_data_generator. this program generates example data within the flight tables and does some further initializations. here is a good tutorial with more information and how to test the flight application: http://help.sap.com/saphelp_erp60_sp/helpdata/de/db/7c623cf568896be10000000a11405a/content.htm configuration of talend studio to use sap components talend’s sap components are already included in the studio. however, two further steps are required to be able to use them: copy sapjco3.dll to the directory c:/windows/system32 sap java connector jar must be added copy sapjco3.jar to the directory “talend/studio/lib/java” (re-) start talend studio check if sap library is added successfully open view “talend modules” (eclipse --> windows --> show view --> talend --> modules) sort by column “context” look for “tsap*” contexts and check if sapjco3.jar has status “installed” usage of sap components with talend studio this section describes how to use talend’s sap components and the sap wizard in general (using one specific example for calling a bapi). detailed descriptions of all sap components (for using bapis, rfcs, idocs, bw, etc.) are available in the documentation talend_components_rg_x.y.z.pdf at www.help.talend.com . connection to a sap system a connection to a sap system can be done “built-in” or via “metadata --> sap connections” (the latter only in enterprise version). using the latter has several advantages: reuse connection configuration quick check if connection to sap works wizards for retrieving functions from sap (instead of handwriting without wizard) quick test with test parameters if function works before finishing development lifecycle for a sap job development lifecycle for sap job: create connection (if not existing yet) right click on metadata --> sap connection create sap connection follow wizard sap jco version: 3 client: “001” userid: “sap*” password: “admin123” --> as you defined it while installation language: “en” hostname: “localhost” system number: “00” retrieve function (bapi / rfc) right click on created connection click on “retrieve sap function” enter search filter (e.g. bapi_fl*) click on “search” select and double click on your function (e.g bapi_flcust_getlist) you see all input, output and table parameters for this sap function click on “test in” --> here you see parameters in more detail: you now have to define which input and output parameters you want to use --> remove all other by selecting them and clicking “remove” button hint: if you do not remove an input parameter, you usually have to enter a value for it! select the output type - can be a single (single record), a table (list of records), or a structure output hint: difference between table and structure in sap: http://www.sapfans.com/forums/viewtopic.php?f=12&t=119794 if you want to do a quick test: enter values for input parameters (if there are any for your function call), then click “launch” button in this example, there is only an optional input parameter max_rows you should see data in the output fields in this example, you see the record with custname “sap ag” and street “neurottstr. 16” click “finish” button under “metadata --> sap connections --> “your connection” --> sap functions: there you can now see your function (in this example: bapi_flcust_getlist) create sap job drag&drop the created function into a job (without the wizard, you also can enter all data by hand) tsapinput component is proposed automatically. click ok to add it to your job go to “initialize input” and add parameter values in this example, there is just the parameter “max_rows” hint: the parameter value can be changed from a hardcoded value to a variable, of course (just click control space on your keyboard to get access to all available variables via code completion in your studio) go to the tsapinput component and add the desired output mapping (i.e. which values you want to process further with other components scroll to the bottom to “outputs” add the correct table / structure name (in this example: "customer_list") click on mapping (which is empty and has to be filled) click on “mapping”, then click on “…” add the wanted output columns of your sap function add the same names at the column “schema xpathqueries” (do not forget the double quotes here!) click “ok” button connect the tsapinput component to a tlogrowcomponent and synchronize the schema hint: always try out if this works before adding further logic to your job! run and test your job (you will see five rows logged (as you have configured max_rows = 5 that's it. now enjoy talend's sap components :-) best regards, kai wähner (twitter: @kaiwaehner) content from my blog: http://www.kai-waehner.de/blog/2013/03/03/sap-integration-with-talend-components-connectors-bapi-rfc-idoc-bw-soap/

In continuation of my earlier blogs on Introduction to Spring MVC and Testing DAO layer in Spring MVC, in this blog I will demonstrate how to test Service layer in Spring MVC. The objective of this demo is 2 fold, to build the Service layer using TDD and increase the code coverage during JUnit testing of Service layer. For people in hurry, get the latest code from Github and run the below command mvn clean test -Dtest=com.example.bookstore.service.AccountServiceTest Since in my earlier blog, we have already tested the DAO layer, in this blog we only need to focus on testing service layer. We need to mock the DAO layer so that we can control the behavior in Service layer and cover various scenarios. Mockito is a good framework which is used to mock a method and return known data and assert that in the JUnit. As a first step we define the AccountServiceTestContextConfiguration class with AccountServiceTest class. If you notice there are 2 beans defined in that class and we marked the as a @Configuration which shows that it is a Spring Context class. In the JUnit test we @Autowired AccountService class. And AccountServiceImpl @Autowired the AccountRepository class. When creating the Bean in the configuration file we also stubbed the AccountRepository class using Mockito, @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration public class AccountServiceTest { @Configuration static class AccountServiceTestContextConfiguration { @Bean public AccountService accountService() { return new AccountServiceImpl(); } @Bean public AccountRepository accountRepository() { return Mockito.mock(AccountRepository.class); } } //We Autowired the AccountService bean so that it is injected from the configuration @Autowired private AccountService accountService; @Autowired private AccountRepository accountRepository; During the setup of the JUnit we use Mockito mock findByUsername method to return a predefined account object as below @Before public void setup() { Account account = new AccountBuilder() { { address("Herve", "4650", "Rue de la gare", "1", null, "Belgium"); credentials("john", "secret"); name("John", "Doe"); } }.build(true); Mockito.when(accountRepository.findByUsername("john")).thenReturn(account); } Now we write the tests as below and test both the positive and negative scenarios, @Test(expected = AuthenticationException.class) public void testLoginFailure() throws AuthenticationException { accountService.login("john", "fail"); } @Test() public void testLoginSuccess() throws AuthenticationException { Account account = accountService.login("john", "secret"); assertEquals("John", account.getFirstName()); assertEquals("Doe", account.getLastName()); } } Finally we verify if the findByUsername method is called only once successfully as below in the teardown, @After public void verify() { Mockito.verify(accountRepository, VerificationModeFactory.times(1)).findByUsername(Mockito.anyString()); // This is allowed here: using container injected mocks Mockito.reset(accountRepository); } AccountService class looks as below, @Service @Transactional(readOnly = true) public class AccountServiceImpl implements AccountService { @Autowired private AccountRepository accountRepository; @Override public Account login(String username, String password) throws AuthenticationException { Account account = this.accountRepository.findByUsername(username, password); } else { throw new AuthenticationException("Wrong username/password", "invalid.username"); } return account; } } I hope this blog helped you. In my next blog, I will demo how to build a controller JUnit test. Reference: Pro Spring MVC: With Web Flow by by Marten Deinum, Koen Serneels

In continuation of my blog JUnit testing of Spring MVC application – Introduction, in this blog, I will show how to design and implement DAO layer for the Bookstore Spring MVC web application using Test Driven development. For people in hurry, get the latest code from Github and run the below command mvn clean test -Dtest=com.example.bookstore.repository.JpaBookRepositoryTest As a part of TDD, Write a basic CRUD (create, read, update, delete) operations on a Book DAO class com.example.bookstore.repository.JpaBookRepository. Don’t have the database wiring yet in this DAO class. Once we build the JUnit tests, we use JPA as a persistence layer. We also use H2 as a inmemory database for testing purpose. Create Book POJO class Create the JUnit test as below, public class JpaBookRepositoryTest { @Test public void testFindById() { Book book = bookRepository.findById(this.book.getId()); assertEquals(this.book.getAuthor(), book.getAuthor()); assertEquals(this.book.getDescription(), book.getDescription()); assertEquals(this.book.getIsbn(), book.getIsbn()); } @Test public void testFindByCategory() { List books = bookRepository.findByCategory(category); assertEquals(1, books.size()); for (Book book : books) { assertEquals(this.book.getCategory().getId(), category.getId()); assertEquals(this.book.getAuthor(), book.getAuthor()); assertEquals(this.book.getDescription(), book.getDescription()); assertEquals(this.book.getIsbn(), book.getIsbn()); } } @Test @Rollback(true) public void testStoreBook() { Book book = new BookBuilder() { { description("Something"); author("JohnDoe"); title("John Doe's life"); isbn("1234567890123"); category(category); } }.build(); bookRepository.storeBook(book); Book book1 = bookRepository.findById(book.getId()); assertEquals(book1.getAuthor(), book.getAuthor()); assertEquals(book1.getDescription(), book.getDescription()); assertEquals(book1.getIsbn(), book.getIsbn()); } } If you notice since the JpaBookRepository is only a skeleton class without implementation, all the tests will fail. As a next step, we need to create a Configuration and wire a datasource, and for the test purpose we will be using H2 database. And we also need to wire this back to JUnit test as below, @Configuration public class InfrastructureContextConfiguration { @Autowired private DataSource dataSource; //some more configurations.. @Bean public DataSource dataSource() { EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder(); builder.setType(EmbeddedDatabaseType.H2); return builder.build(); } } //JUnit test wiring is as below @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(classes = { InfrastructureContextConfiguration.class, TestDataContextConfiguration.class }) @Transactional public class JpaBookRepositoryTest { //the test methods } Next step is to setup and teardown sample data in the JUnit test case as below, public class JpaBookRepositoryTest { @PersistenceContext private EntityManager entityManager; private Book book; private Category category; @Before public void setupData() { EntityBuilderManager.setEntityManager(entityManager); category = new CategoryBuilder() { { name("Evolution"); } }.build(); book = new BookBuilder() { { description("Richard Dawkins' brilliant reformulation of the theory of natural selection"); author("Richard Dawkins"); title("The Selfish Gene: 30th Anniversary Edition"); isbn("9780199291151"); category(category); } }.build(); } @After public void tearDown() { EntityBuilderManager.clearEntityManager(); } } Once we do the wiring, we need to implement the com.example.bookstore.repository.JpaBookRepository and use JPA to do the CRUD on the database and run the tests. The tests will succeed. Finally if you run Cobertura for this example from STS, we will get over 90% of line coverage for com.example.bookstore.repository.JpaBookRepository. In case you want to try few exercises you can implement repository for Account and User. I hope this blog helped you. In my next blog I will talk about Mochito and Implementing the Service layer.

In order to receive JMS messages, Spring provides the concept of message listener containers. These are beans that can be tied to receive messages that arrive at certain destinations. This post will examine the different ways in which containers can be configured. A simple example is below where the DefaultMessageListenerContianer has been configured to watch one queue (the property jms.queue.name) and has a reference to a myMessageListener bean which implements the MessageListener interface (ie onMessage): This is all very well but means that the myMessageListener bean will have to handle the JMS Message object and process accordingly depending upon the type of javax.jms.Message and its payload. For example: if (message instanceof MapMessage) { // cast, get object, do something } An alternative is to use a MessageListenerAdapter. This class abstracts away the above processing and leaves your code to deal with just the message's payload. For example: The delegate is a reference to a myMessageReceiverDelegate bean which has one or more methods called processMessage. It does not need to implement the MessageListener interface. This method can be overload to handle different payload types. Spring behind the scenes will determine which gets called. For example: public void processMessage(final HashMap message) { // do something } public void processMessage(final String message) { // do something } For the given approach though, only one queue can be tied to the container. Another approach is to tie many listeners (therefore many queues) to the one container, The below Spring XML, using the jms namespace, shows how two listeners for different queues can be tied to one container: The myMessageReceiverDelegate bean is treated as an adapter delegate, therefore does not need to implement the MessageListener interface. Each listener can have a different delegate but for the above example, all messages arriving at the two queues are processed by the one receiver bean ie myMessageReceiverDelegate. If there is a need to check the message type and extract the payload, then the listener can use a class which implements the MessageListener interface (eg the myMessageListener bean used in the first example). The onMessage method will then be called when messages arrive at the specified destination:

For people in hurry get the code from Github. In continuation of my earlier blog on spring-test-mvc junit testing Spring Security layer with InMemoryDaoImpl, in this blog I will discuss how to use achieve method level access control. Please follow the steps in this blog to setup spring-test-mvc and run the below test case. mvn test -Dtest=com.example.springsecurity.web.controllers.SecurityControllerTest The JUnit test case looks as below, @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(loader = WebContextLoader.class, value = { "classpath:/META-INF/spring/services.xml", "classpath:/META-INF/spring/security.xml", "classpath:/META-INF/spring/mvc-config.xml" }) public class SecurityControllerTest { @Autowired CalendarService calendarService; @Test public void testMyEvents() throws Exception { Authentication auth = new UsernamePasswordAuthenticationToken("[email protected]", "user1"); SecurityContext securityContext = SecurityContextHolder.getContext(); securityContext.setAuthentication(auth); calendarService.findForUser(0); SecurityContextHolder.clearContext(); } @Test(expected = AuthenticationCredentialsNotFoundException.class) public void testForbiddenEvents() throws Exception { calendarService.findForUser(0); } } @Test(expected=AccessDeniedException.class) public void testWrongUserEvents() throws Exception { Authentication auth = new UsernamePasswordAuthenticationToken("[email protected]", "user2"); SecurityContext securityContext = SecurityContextHolder.getContext(); securityContext.setAuthentication(auth); calendarService.findForUser(0); SecurityContextHolder.clearContext(); } If you notice, if the user did not login or if the user is trying to access another users information it will throw an exception. The interface access control is as below, public interface CalendarService { @PreAuthorize("hasRole('ROLE_ADMIN') or principal.id == #userId") List findForUser(int userId); } The PreAuthorize only works on interface so that any implementation that implements this interface has this access control. I hope this blog helps you.

Want to check out what life is like as a developer? Check out this post for a series of GIFs demonstrating the dev life.