A couple of weeks ago I was trying to write a test around some OAuth code that we have on an internal application and I was using Jersey Client to send the various requests. I initially started with the following code: Client = Client.create(); ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class ); But when I ran the test I was getting the following exception: com.sun.jersey.api.client.ClientHandlerException: java.net.ProtocolException: Server redirected too many times (20) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) at com.sun.jersey.api.client.WebResource.get(WebResource.java:191) at com.neotechnology.testlab.manager.webapp.AuthenticationIntegrationTest.shouldRedirectToGitHubForAuthentication(AuthenticationIntegrationTest.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at com.neotechnology.kirkaldy.testing.Resources$1.evaluate(Resources.java:84) at com.neotechnology.kirkaldy.testing.FailureOutput$2.evaluate(FailureOutput.java:37) at org.junit.rules.RunRules.evaluate(RunRules.java:18) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.junit.runner.JUnitCore.run(JUnitCore.java:157) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63) Caused by: java.net.ProtocolException: Server redirected too many times (20) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1446) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:249) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) ... 28 more If we check the traffic going across port 59680 we can see what’s going wrong: $ sudo ngrep -d lo0 port 59680 interface: lo0 (127.0.0.0/255.0.0.0) filter: (ip) and ( port 59680 ) ##### T 127.0.0.1:59704 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## T 127.0.0.1:59680 -> 127.0.0.1:59704 [AP] HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=mdyw3a4fmqc1b6p53birm4dd;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59679/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length : 0..Server: Jetty(8.1.8.v20121106).... ########### T 127.0.0.1:59707 -> 127.0.0.1:59680 [AP] GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## T 127.0.0.1:59680 -> 127.0.0.1:59707 [AP] HTTP/1.1 302 Found..Cache-Control: no-cache..Set-Cookie: JSESSIONID=8gggez0ns9ftiex4314mbgz9;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106).... ########### T 127.0.0.1:59713 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Java/1.6.0_45..Host: localhost:59680..Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2..Connection: keep-alive.... ## The response we receive includes a direction to the client to store a cookie but we can see on the next request that the cookie hasn’t been included. I came across this post, which had a few suggestions on how to get around the problem, but the only approach that worked for me was to use jersey-apache-client for which I added the following dependency: com.sun.jersey.contribs jersey-apache-client 1.13 jar I then change my client code to read like this: ApacheHttpClientConfig config = new DefaultApacheHttpClientConfig(); config.getProperties().put(ApacheHttpClientConfig.PROPERTY_HANDLE_COOKIES, true); ApacheHttpClient client = ApacheHttpClient.create( config ); client.setFollowRedirects(true); client.getClientHandler().getHttpClient().getParams().setBooleanParameter( HttpClientParams.ALLOW_CIRCULAR_REDIRECTS, true ); ClientResponse response = client.resource( "http://localhost:59680" ).get( ClientResponse.class ); If we run that and watch the output using ngrep we can see that it now handles cookies correctly: $ sudo ngrep -d lo0 port 59680 Password: interface: lo0 (127.0.0.0/255.0.0.0) filter: (ip) and ( port 59680 ) ##### T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 302 Found..Set-Cookie: JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a;Path=/..Expires: Thu, 01 Jan 1970 00:00:00 GMT..Location: http://localhost:60322/authorize?client_id=basic-client&state=the-state&scope=user%2Crepo..Content-Length : 0..Server: Jetty(8.1.8.v20121106).... ## T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET /auth/callback?code=timey-wimey&state=the-state HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 302 Found..Cache-Control: no-cache..Location: http://localhost:59680/..Content-Length: 0..Server: Jetty(8.1.8.v20121106).... ## T 127.0.0.1:60372 -> 127.0.0.1:59680 [AP] GET / HTTP/1.1..User-Agent: Jakarta Commons-HttpClient/3.1..Host: localhost:59680..Cookie: $Version=0; JSESSIONID=vn8zzf9ep3x4mtw66ydm0n6a; $Path=/.... ## T 127.0.0.1:59680 -> 127.0.0.1:60372 [AP] HTTP/1.1 200 OK..Vary: Accept-Encoding..Accept-Ranges: bytes..Content-Type: text/html..Content-Length: 2439..Last-Modified: Tue, 23 Jul 2013 10:48:15 GMT..Server: Jetty(8.1.8.v20121106)....... . . . . . . . . . . . ....

introduction openstack is one of the most popular open source cloud computing projects to provide infrastructure as a service solution. its key components are compute (nova), networking (neutron, formerly known as quantum), storage (object and block storage, swift and cinder, respectively), openstack dashboard (horizon), identity service (keystone) and image service (glance). there are other official incubated projects like metering (celiometer) and orchestration and service definition (heat). savanna is a hadoop as a service for openstack introduced by mirantis . it is still in an early phase (version .02 was released in summer 2013) and according to its roadmap version 1.0 is targeted for official openstack incubation. in principle, heat also could be used for hadoop cluster provisioning but savanna is especially tuned for providing hadoop-specific api functionality while heat is meant to be used for generic purposes. savanna architecture savanna is integrated with the core openstack components such as keystone, nova, glance, swift and horizon. it has a rest api that supports the hadoop cluster provisioning steps. savanna api is implemented as a wsgi server that, by default, listens to port 8386. in addition, savanna can also be integrated with horizon, the openstack dashboard to create a hadoop cluster from the management console. savanna also comes with a vanilla plugin that deploys a hadoop cluster image. the standard out-of-the-box vanilla plugin supports hadoop 1.1.2 version. installing savanna the simplest option to try out savanna is to use devstack in a virtual machine. i was using an ubuntu 12.04 virtual instance in my tests. in that environment we need to execute the following commands to install devstack and savanna api: $ sudo apt-get install git-core $ git clone https://github.com/openstack-dev/devstack.git $ vi localrc # edit localrc admin_password=nova mysql_password=nova rabbit_password=nova service_password=$admin_password service_token=nova # enable swift enabled_services+=,swift swift_hash=66a3d6b56c1f479c8b4e70ab5c2000f5 swift_replicas=1 swift_data_dir=$dest/data # force checkout prerequsites # force_prereq=1 # keystone is now configured by default to use pki as the token format which produces huge tokens. # set uuid as keystone token format which is much shorter and easier to work with. keystone_token_format=uuid # change the floating_range to whatever ips vm is working in. # in nat mode it is subnet vmware fusion provides, in bridged mode it is your local network. floating_range=192.168.55.224/27 # enable auto assignment of floating ips. by default savanna expects this setting to be enabled extra_opts=(auto_assign_floating_ip=true) # enable logging screen_logdir=$dest/logs/screen $ ./stack.sh # this will take a while to execute $ sudo apt-get install python-setuptools python-virtualenv python-dev $ virtualenv savanna-venv $ savanna-venv/bin/pip install savanna $ mkdir savanna-venv/etc $ cp savanna-venv/share/savanna/savanna.conf.sample savanna-venv/etc/savanna.conf # to start savanna api: $ savanna-venv/bin/python savanna-venv/bin/savanna-api --config-file savanna-venv/etc/savanna.conf to install savanna ui integrated with horizon, we need to run the following commands: $ sudo pip install savanna-dashboard $ cd /opt/stack/horizon/openstack-dashboard $ vi settings.py horizon_config = { 'dashboards': ('nova', 'syspanel', 'settings', 'savanna'), installed_apps = ( 'savannadashboard', .... $ cd /opt/stack/horizon/openstack-dashboard/local $ vi local_settings.py savanna_url = 'http://localhost:8386/v1.0' $ sudo service apache2 restart provisioning a hadoop cluster as a first step, we need to configure keystone-related environment variables to get the authentication token: ubuntu@ip-10-59-33-68:~$ vi .bashrc $ export os_auth_url=http://127.0.0.1:5000/v2.0/ $ export os_tenant_name=admin $ export os_username=admin $ export os_password=nova ubuntu@ip-10-59-33-68:~$ source .bashrc ubuntu@ip-10-59-33-68:~$ ubuntu@ip-10-59-33-68:~$ env | grep os os_password=nova os_auth_url=http://127.0.0.1:5000/v2.0/ os_username=admin os_tenant_name=admin ubuntu@ip-10-59-33-68:~$ keystone token-get +-----------+----------------------------------+ | property | value | +-----------+----------------------------------+ | expires | 2013-08-09t20:31:12z | | id | bdb582c836e3474f979c5aa8f844c000 | | tenant_id | 2f46e214984f4990b9c39d9c6222f572 | | user_id | 077311b0a8304c8e86dc0dc168a67091 | +-----------+----------------------------------+ $ export auth_token="bdb582c836e3474f979c5aa8f844c000" $ export tenant_id="2f46e214984f4990b9c39d9c6222f572" then we need to create the glance image that we want to use for our hadoop cluster. in our example we have used mirantis's vanilla image but we can also build our own image: $ wget http://savanna-files.mirantis.com/savanna-0.2-vanilla-1.1.2-ubuntu-12.10.qcow2 $ glance image-create --name=savanna-0.2-vanilla-hadoop-ubuntu.qcow2 --disk-format=qcow2 --container-format=bare < ./savanna-0.2-vanilla-1.1.2-ubuntu-12.10.qcow2 ubuntu@ip-10-59-33-68:~/devstack$ glance image-list +--------------------------------------+-----------------------------------------+-------------+------------------+-----------+--------+ | id | name | disk format | container format | size | status | +--------------------------------------+-----------------------------------------+-------------+------------------+-----------+--------+ | d0d64f5c-9c15-4e7b-ad4c-13859eafa7b8 | cirros-0.3.1-x86_64-uec | ami | ami | 25165824 | active | | fee679ee-e0c0-447e-8ebd-028050b54af9 | cirros-0.3.1-x86_64-uec-kernel | aki | aki | 4955792 | active | | 1e52089b-930a-4dfc-b707-89b568d92e7e | cirros-0.3.1-x86_64-uec-ramdisk | ari | ari | 3714968 | active | | d28051e2-9ddd-45f0-9edc-8923db46fdf9 | savanna-0.2-vanilla-hadoop-ubuntu.qcow2 | qcow2 | bare | 551699456 | active | +--------------------------------------+-----------------------------------------+-------------+------------------+-----------+--------+ $ export image_id=d28051e2-9ddd-45f0-9edc-8923db46fdf9 then we have installed httpie , an open source http client that can be used to send rest requests to savanna api: $ sudo pip install httpie from now on we will use httpie to send savanna commands. we need to register the image with savanna: $ export savanna_url="http://localhost:8386/v1.0/$tenant_id" $ http post $savanna_url/images/$image_id x-auth-token:$auth_token username=ubuntu http/1.1 202 accepted content-length: 411 content-type: application/json date: thu, 08 aug 2013 21:28:07 gmt { "image": { "os-ext-img-size:size": 551699456, "created": "2013-08-08t21:05:55z", "description": "none", "id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9", "metadata": { "_savanna_description": "none", "_savanna_username": "ubuntu" }, "mindisk": 0, "minram": 0, "name": "savanna-0.2-vanilla-hadoop-ubuntu.qcow2", "progress": 100, "status": "active", "tags": [], "updated": "2013-08-08t21:28:07z", "username": "ubuntu" } } $ http $savanna_url/images/$image_id/tag x-auth-token:$auth_token tags:='["vanilla", "1.1.2", "ubuntu"]' http/1.1 202 accepted content-length: 532 content-type: application/json date: thu, 08 aug 2013 21:29:25 gmt { "image": { "os-ext-img-size:size": 551699456, "created": "2013-08-08t21:05:55z", "description": "none", "id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9", "metadata": { "_savanna_description": "none", "_savanna_tag_1.1.2": "true", "_savanna_tag_ubuntu": "true", "_savanna_tag_vanilla": "true", "_savanna_username": "ubuntu" }, "mindisk": 0, "minram": 0, "name": "savanna-0.2-vanilla-hadoop-ubuntu.qcow2", "progress": 100, "status": "active", "tags": [ "vanilla", "ubuntu", "1.1.2" ], "updated": "2013-08-08t21:29:25z", "username": "ubuntu" } } then we need to create a nodegroup templates (json files) that will be sent to savanna. there is one template for the master nodes ( namenode , jobtracker ) and another template for the worker nodes such as datanode and tasktracker . the hadoop version is 1.1.2. $ vi ng_master_template_create.json { "name": "test-master-tmpl", "flavor_id": "2", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "node_processes": ["jobtracker", "namenode"] } $ vi ng_worker_template_create.json { "name": "test-worker-tmpl", "flavor_id": "2", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "node_processes": ["tasktracker", "datanode"] } $ http $savanna_url/node-group-templates x-auth-token:$auth_token < ng_master_template_create.json http/1.1 202 accepted content-length: 387 content-type: application/json date: thu, 08 aug 2013 21:58:00 gmt { "node_group_template": { "created": "2013-08-08t21:58:00", "flavor_id": "2", "hadoop_version": "1.1.2", "id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "name": "test-master-tmpl", "node_configs": {}, "node_processes": [ "jobtracker", "namenode" ], "plugin_name": "vanilla", "updated": "2013-08-08t21:58:00", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } } $ http $savanna_url/node-group-templates x-auth-token:$auth_token < ng_worker_template_create.json http/1.1 202 accepted content-length: 388 content-type: application/json date: thu, 08 aug 2013 21:59:41 gmt { "node_group_template": { "created": "2013-08-08t21:59:41", "flavor_id": "2", "hadoop_version": "1.1.2", "id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "name": "test-worker-tmpl", "node_configs": {}, "node_processes": [ "tasktracker", "datanode" ], "plugin_name": "vanilla", "updated": "2013-08-08t21:59:41", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } } the next step is to define the cluster template: $ vi cluster_template_create.json { "name": "demo-cluster-template", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "node_groups": [ { "name": "master", "node_group_template_id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "count": 1 }, { "name": "workers", "node_group_template_id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "count": 2 } ] } $ http $savanna_url/cluster-templates x-auth-token:$auth_token < cluster_template_create.json http/1.1 202 accepted content-length: 815 content-type: application/json date: fri, 09 aug 2013 07:04:24 gmt { "cluster_template": { "anti_affinity": [], "cluster_configs": {}, "created": "2013-08-09t07:04:24", "hadoop_version": "1.1.2", "id": "{ "name": "cluster-1", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "cluster_template_id" : "64c4117b-acee-4da7-937b-cb964f0471a9", "user_keypair_id": "stack", "default_image_id": "3f9fc974-b484-4756-82a4-bff9e116919b" }", "name": "demo-cluster-template", "node_groups": [ { "count": 1, "flavor_id": "2", "name": "master", "node_configs": {}, "node_group_template_id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "node_processes": [ "jobtracker", "namenode" ], "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 }, { "count": 2, "flavor_id": "2", "name": "workers", "node_configs": {}, "node_group_template_id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "node_processes": [ "tasktracker", "datanode" ], "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } ], "plugin_name": "vanilla", "updated": "2013-08-09t07:04:24" } } now we are ready to create the hadoop cluster: $ vi cluster_create.json { "name": "cluster-1", "plugin_name": "vanilla", "hadoop_version": "1.1.2", "cluster_template_id" : "64c4117b-acee-4da7-937b-cb964f0471a9", "user_keypair_id": "savanna", "default_image_id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9" } $ http $savanna_url/clusters x-auth-token:$auth_token < cluster_create.json http/1.1 202 accepted content-length: 1153 content-type: application/json date: fri, 09 aug 2013 07:28:14 gmt { "cluster": { "anti_affinity": [], "cluster_configs": {}, "cluster_template_id": "64c4117b-acee-4da7-937b-cb964f0471a9", "created": "2013-08-09t07:28:14", "default_image_id": "d28051e2-9ddd-45f0-9edc-8923db46fdf9", "hadoop_version": "1.1.2", "id": "d919f1db-522f-45ab-aadd-c078ba3bb4e3", "info": {}, "name": "cluster-1", "node_groups": [ { "count": 1, "created": "2013-08-09t07:28:14", "flavor_id": "2", "instances": [], "name": "master", "node_configs": {}, "node_group_template_id": "b3a79c88-b6fb-43d2-9a56-310218c66f7c", "node_processes": [ "jobtracker", "namenode" ], "updated": "2013-08-09t07:28:14", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 }, { "count": 2, "created": "2013-08-09t07:28:14", "flavor_id": "2", "instances": [], "name": "workers", "node_configs": {}, "node_group_template_id": "773b2cfb-1e05-46f4-923f-13edc7d6aac6", "node_processes": [ "tasktracker", "datanode" ], "updated": "2013-08-09t07:28:14", "volume_mount_prefix": "/volumes/disk", "volumes_per_node": 0, "volumes_size": 10 } ], "plugin_name": "vanilla", "status": "validating", "updated": "2013-08-09t07:28:14", "user_keypair_id": "savanna" } } after a while we can run the nova command to check if the instances are created and running: $ nova list +--------------------------------------+-----------------------+--------+------------+-------------+----------------------------------+ | id | name | status | task state | power state | networks | +--------------------------------------+-----------------------+--------+------------+-------------+----------------------------------+ | 1a9f43bf-cddb-4556-877b-cc993730da88 | cluster-1-master-001 | active | none | running | private=10.0.0.2, 192.168.55.227 | | bb55f881-1f96-4669-a94a-58cbf4d88f39 | cluster-1-workers-001 | active | none | running | private=10.0.0.3, 192.168.55.226 | | 012a24e2-fa33-49f3-b051-9ee2690864df | cluster-1-workers-002 | active | none | running | private=10.0.0.4, 192.168.55.225 | +--------------------------------------+-----------------------+--------+------------+-------------+----------------------------------+ now we can log in to the hadoop master instance and run the required hadoop commands: $ ssh -i savanna.pem [email protected] $ sudo chmod 777 /usr/share/hadoop $ sudo su hadoop $ cd /usr/share/hadoop $ hadoop jar hadoop-example-1.1.2.jar pi 10 100 savanna ui via horizon in order to create nodegroup templates, cluster templates and the cluster itself we used a command line tool -- httpie -- to send rest api calls. the same functionality is also available via horizon, the standard openstack dashboard. first we need to register the image with savanna: then we need to create the nodegroup templates: after that we have to create the cluster template: and finally we have to create the cluster:

One of the five essential attributes of cloud computing (see The 5-3-2 Principle of Cloud Computing) is resource pooling, which is an important differentiator separating the thought process of traditional IT from that of a service-based, cloud computing approach. Resource pooling in the context of cloud computing and from a service provider’s viewpoint denotes a set of strategies and a methodical way of managing resources. For a user, resource pooling institutes an abstraction for presenting and consuming resources in a consistent and transparent fashion. This article presents these key concepts derived from resource pooling: Resource Pools Virtualization in the Context of Cloud Computing Standardization, Automation, and Optimization Fabric Cloud Closing Thoughts Resource Pools Ultimately, data center resources can be logically placed into three categories. They are: compute, networks, and storage. For many, this grouping may appear trivial. It is, however, a foundation upon which some cloud computing methodologies are developed, products designed, and solutions formulated. Compute This is a collection of all CPU capabilities. Essentially all data center servers, either for supporting or actually running a workload, are all part of this compute group. Compute pool represents the total capacity for executing code and running instances. The process to construct a compute pool is to first inventory all servers and identify virtualization candidates followed by implementing server virtualization. It is never too early to introduce a system management solution to facilitate the processes, which in my view is a strategic investment and a critical component for all cloud initiatives. Networks The physical and logical artifacts putting in place to connect resources, segment, and isolate resources from layer three and below, etc., are gathered in the network pool. Networking enables resources becoming visible and hence possibly manageable. In the age of instant gratification, networks and mobility are redefining the security and system administration boundaries, and play a direct and impactful role in user productivity and customer satisfaction. Networking in cloud computing is more than just remote access, but empowerment for a user to self-serve and consume resources anytime, anywhere, with any device. BYOD and consumerization of IT are various expressions of these concepts. Storage This has long been a very specialized and sometimes mysterious part of IT. An enterprise storage solution is frequently characterized as a high-cost item with a significant financial and contractual commitment, specialized hardware, proprietary API and software, a dependency on direct vendor support, etc. In cloud computing, storage has become even more noticeable since the ability to grow and shrink based on demands, i.e. elasticity, demands an enterprise-level, massive, reliable, and resilient storage solution at a global scale. While enterprise IT is consolidating resources and transforming the existing establishment into a cloud computing environment, how to leverage existing storage devices from various vendors and integrate them with the next generation storage solutions is among the highest priorities for modernizing a data center. Virtualization in the Context of Cloud Computing In the last decade, virtualization has proved its value and accelerated the realization of cloud computing. Then, virtualization was mainly server virtualization, which in an over-simplified statement means hosting multiple server instances with the same hardware while each instance runs transparently and in insolation, as if each consumes the entire hardware and is the only instance running. Much of the customer expectations, business needs, and methodologies has since evolved. Now, we should validate virtualization in the context of cloud computing to fully address the innovations rapidly changing how IT conducts business and delivers services. As discussed below, in the context of cloud computing, consumable resources are delivered in some virtualized form. Various virtualization layers collectively construct and form the so-called fabric. Server Virtualization The concept of server virtualization remains: running multiple server instances with the same hardware while each instance runs transparently and in isolation, as if each instance is the only instance running and consuming the entire server hardware. In addition to virtualizing and consolidating servers, server virtualization also signifies the practices of standardizing server deployment switching away from physical boxes to VMs. Server virtualization is for packaging, delivering, and consuming a compute pool. There are a few important considerations of virtualizing servers. IT needs the ability to identify and manage bare metal such that the entire resource life-cycle management from commencing to decommissioning can be standardized and automated. To fundamentally reduce the support and training cost while increasing productivity, a consistent platform with tools applicable across physical, virtual, on-premises, and off-premises deployments is essential. The last thing IT wants is one set of tools for physical resources and another for those virtualized, one set of tools for on-premises deployment and another for those deployed to a service provider, and one set of tools for development and another for deploying applications. The requirement is one methodology for all, one skill set for all, and one set of tools for all. This advantage is obvious when developing applications and deploying Windows Server 2012 R2 on premises or off premises to Windows Azure. The Active Directory security model can work across sites, System Center can manage resources deployed off premises to Windows Azure, and Visual Studio can publish applications across platforms. Windows infrastructure architecture, security, and deployment models are all directly applicable. Network Virtualization The similar idea of server virtualization applies here. Network virtualization is the ability to run multiple networks on the same network device while each network runs transparently and in isolation, as if each network is the only network running and consuming the entire network hardware. Conceptually, since each network instance is running in isolation, one tenant’s 192.168.x network is not aware of another tenant’s identical192.168.x network running with the same network device. Network virtualization provides the translation between physical network characteristics and the representation of and a resource identity in a virtualized network. Consequently, above the network virtualization layer, various tenants while running in isolation can have identical network configurations. A great example of network virtualization is Windows Azure virtual networking. At any given time, there can be multiple Windows Azure subscribers all allocating the same 192.168.x address space with an identical subnet scheme (192.168.1.x/16) for deploying VMs. Those VMs belonging to one subscriber will however not be aware of or visible to those deployed by others, despite the fact that the network configuration, IP scheme, and IP address assignments may all be identical. Network virtualization in Windows Azure isolates on subscriber from the others such that each subscriber operates as if the subscription is the only one employing a 192.168.x address space. Storage Virtualization I believe this is where the next wave of drastic cost reduction of IT post-server virtualization happens. Historically, storage has been a high cost item in any IT budget in each and every aspects including hardware, software, staffing, maintenance, SLA, etc. Since the introduction of Windows Server 2012, there is a clear direction where storage virtualization is built into OS and becoming a commodity. New capabilities like Storage Pool, Hyper-V over SMB, Scale-Out Fire Share, etc., are now part of Windows Server OS and are making storage virtualization part of server administration routines and easily manageable with tools and utilities like PowerShell, which is familiar to many IT professionals. The concept of storage virtualization remains consistent with the idea of logically separating a computing object from its hardware, in this case the storage capacity. Storage virtualization is the ability to integrate multiple and heterogeneous storage devices, aggregate the storage capacities, and present/manage as one logical storage device with a continuous storage space. JBOD is a technology to realize this concept. Standardization, Automation and Optimization Each of the three resource pools has an abstraction to logically present itself with characteristics and work patterns. A compute pool is a collection of physical (virtualization and infrastructure) hosts and VMs. A virtualization host hosts VMs that run workloads deployed by service owners and consumed by authorized users. A network pool encompasses network resources including physical devices, logical switches, address spaces, and site configurations. Network virtualization as enabled/defined in configurations can identify and translate a logical/virtual IP address into a physical one, such that tenants with the same network hardware can implement an identical network scheme without a concern. A storage pool is based on storage virtualization which is a concept of presenting an aggregated storage capacity as one continuous storage space as if provided from one logical storage device. In other words, the three resource pools are wrapped with server virtualization, network virtualization, and storage virtualization, respectively. Each virtualization presents a set of methodologies on which work patterns are derived and common practices are developed. These virtualization layers provides opportunities to standardize, automate, and optimize deployments and considerably facilitates the adoption of cloud computing. Standardization Virtualizing resources decouples the dependency between instances and the underlying hardware. This offers an opportunity to simplify and standardize the logical representation of a resource. For instance, a VM is defined and deployed with a VM template that provides a level of consistency with a standardized configuration. Automation Once VM characteristics are identified and standardized, we can now generate an instance by providing only instance-based information or information that depends on run-time, such as the VM machine name, which must be validated at run-time to prevent duplicated names. This requirement for providing only minimal information at deployment can be significantly simplify and streamline operations for automation. And with automation, resources can then be deployed, instantiated, relocated, taken off-line, brought back online, or removed rapidly and automatically based on set criteria. Standardization and automation are essential mechanisms so that workload can be scaled on demand, i.e., become elastic. Optimization Standardization provides a set of common criteria. Automation executes operations based on set criteria with volumes, consistency, and expediency. With standardization and automation, instances can be instantiated with consistency, efficiency, and predictability. In other words, resources can be operated in bulk with consistency and predictability. The next logical step is then to optimize the usage based on SLA. The presented progression is what resource pooling and virtualizations can provide and facilitate. These methodologies are now built into products and solutions. Windows Server 2012 R2 and System Center 2012 and later integrate server virtualization, network virtualization, and storage virtualization into one consistent solution platform with standardization, automation, and optimization for building and managing clouds. Fabric This is a significant abstraction in cloud computing. Fabric implies accessibility and discoverability, and denotes the ability to discover, identify, and manage a resource. Conceptually, fabric is an umbrella term encompassing all the underlying infrastructure supporting a cloud computing environment. At the same time, a fabric controller represents the system management solution which manages, i.e. owns, fabric. In cloud architecture, fabric consists of the three resource pools: compute, networks, and storage. Compute provides the computing capabilities, executes code, and runs instances. Networks glues the resources based on requirements. Storage is where VMs, configurations, data, and resources are kept. Fabric shields the physical complexities of the three resource pools presented with server virtualization, network virtualization, and storage virtualization. All operations are eventually directed by the fabric controller of a data center. Above fabric, there are logical views of consumable resources including VMs, virtual networks, and logical storage drives. By deploying VMs, configuring virtual networks, or acquiring storage, a user consumes resources. Under fabric, there are virtualization and infrastructure hosts, Active Directory, DNS, clusters, load balancers, address pools, network sites, library shares, storage arrays, topology, racks, cables, etc., all under the fabric controller’s command to collectively present and support fabric. For a service provider, building a cloud computing environment is essentially establishing a fabric controller and constructing fabric. Namely, instituting a comprehensive management solution, building the three resource pools, and integrating server virtualization, network virtualization, and storage virtualization to form fabric. From a user’s point of view, how and where a resource is physically provided is not a concern, but the accessibility, readiness, scalability, and fulfillment of SLA are. Cloud This is a well-defined term and we should not be confused with it. (see NIST SP 800-145 and the 5-3-2 Principle of Cloud Computing) We need to be very clear on: what a cloud must exhibit (the five essential attributes), how to consume it (with SaaS, PaaS, or IaaS), and the model a service is deployed in (like private cloud, public cloud, and hybrid cloud). Cloud is a concept, a state, a set of capabilities such that a business can be delivered as a service, i.e. available on demand. The architecture of a cloud computing environment is presented with three resource pools: compute, networks, and storage. Each is an abstraction provided by a virtualization layer. Server virtualization presents a compute pool with VMs that supply the computing, i.e. CPUs, and power to execute code and run instances. Network virtualization offers a network pool and is the mechanism that allows multiple tenants with identical network configurations on the same virtualization host while connecting, segmenting, isolating network traffic with virtual NICs, logical switches, address space, network sites, IP pools, etc. Storage virtualization provides a logical storage device with the capacity to appear continuous and aggregated with a pool of storage devices behind the scene. The three resource pools together constitute the fabric (of a cloud) while the three virtualization layers collectively form the abstraction, such that while the underlying physical infrastructure may be intricate, the user experience above fabric remains logical and consistent. Deploying a VM, configuring a virtual network, or acquiring storage is transparent with virtualization regardless of where the VM actually resides, how the virtual network is physically wired, or what devices in the aggregate the requested storage is provided with. Closing Thoughts Cloud is a very consumer-focused approach. It is about a customer’s ability and control based on SLA in getting resources when needed and with scale, and equally important releasing resources when no longer required. It is not about products and technologies. It is about servicing, consuming, and strengthening the bottom line.

Hi! Some time ago I have written an article about Mockito and using RETURNS_DEEP_STUBS when working with JAXB. Quite recently we have faced a similliar issue with deeply nesetd JAXB and the awesome testing framework written in Groovy called Spock. Natively Spock does not support creating deep stubs or spies so we needed to create a workaround for it and this article will show you how to do it. Project structure We will be working on the same data structure as in the RETURNS_DEEP_STUBS when working with JAXB article so the project structure will be quite simillar: As you can see the main difference is such that the tests are present in the /test/groovy/ folder instead of /test/java/ folder. Extended Spock Specification In order to use Spock as a testing framework you have to create Groovy test scripts that extend the Spock Specification class. The details of how to use Spock are available here. In this project I have created an abstract class that extends Specification and adds two additional methods for creating nested Test Doubles (I don't remember if I haven't seen a prototype of this approach somewhere on the internet). ExtendedSpockSpecification.groovy package com.blogspot.toomuchcoding.spock; import spock.lang.Specification /** * Created with IntelliJ IDEA. * User: MGrzejszczak * Date: 14.06.13 * Time: 15:26 */ abstract class ExtendedSpockSpecification extends Specification { /** * The method creates nested structure of spies for all the elements present in the property parameter. Those spies are set on the input object. * * @param object - object on which you want to create nested spies * @param property - field accessors delimited by a dot - JavaBean convention * @return Spy of the last object from the property path */ protected def createNestedSpies(object, String property) { def lastObject = object property.tokenize('.').inject object, { obj, prop -> if (obj[prop] == null) { def foundProp = obj.metaClass.properties.find { it.name == prop } obj[prop] = Spy(foundProp.type) } lastObject = obj[prop] } lastObject } /** * The method creates nested structure of mocks for all the elements present in the property parameter. Those mocks are set on the input object. * * @param object - object on which you want to create nested mocks * @param property - field accessors delimited by a dot - JavaBean convention * @return Mock of the last object from the property path */ protected def createNestedMocks(object, String property) { def lastObject = object property.tokenize('.').inject object, { obj, prop -> def foundProp = obj.metaClass.properties.find { it.name == prop } def mockedProp = Mock(foundProp.type) lastObject."${prop}" >> mockedProp lastObject = mockedProp } lastObject } } These two methods work in a very simillar manner. Assuming that the method's argument property looks as follows: "a.b.c.d" then the methods tokenize the string by "." and iterate over the array -["a","b","c","d"]. We then iterate over the properties of the Meta Class to find the one whose name is equal to prop (for example "a"). If that is the case we then use Spock's Mock/Spy method to create a Test Double of a given class (type). Finally we have to bind the mocked nested element to its parent. For the Spy it's easy since we set the value on the parent (lastObject = obj[prop]). For the mocks however we need to use the overloaded >> operator to record the behavior for our mock - that's why dynamically call the property that is present in the prop variable (lastObject."${prop}" >> mockedProp). Then we return from the closure the mocked/spied instance and we repeat the process for it Class to be tested Let's take a look at the class to be tested: PlayerServiceImpl.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } The test class And now the test class: PlayerServiceImplWrittenUsingSpockTest.groovy package com.blogspot.toomuchcoding.service import com.blogspot.toomuchcoding.model.* import com.blogspot.toomuchcoding.spock.ExtendedSpockSpecification /** * User: mgrzejszczak * Date: 14.06.13 * Time: 16:06 */ class PlayerServiceImplWrittenUsingSpockTest extends ExtendedSpockSpecification { public static final String COUNTRY_CODE_ENG = "ENG"; PlayerServiceImpl objectUnderTest def setup(){ objectUnderTest = new PlayerServiceImpl() } def "should return true if country code is the same when creating nested structures using groovy"() { given: PlayerDetails playerDetails = new PlayerDetails( clubDetails: new ClubDetails( country: new CountryDetails( countryCode: new CountryCodeDetails( countryCode: CountryCodeType.ENG ) ) ) ) when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return true if country code is the same when creating nested structures using spock mocks - requires CGLIB for non interface types"() { given: PlayerDetails playerDetails = Mock() ClubDetails clubDetails = Mock() CountryDetails countryDetails = Mock() CountryCodeDetails countryCodeDetails = Mock() countryCodeDetails.countryCode >> CountryCodeType.ENG countryDetails.countryCode >> countryCodeDetails clubDetails.country >> countryDetails playerDetails.clubDetails >> clubDetails when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return true if country code is the same using ExtendedSpockSpecification's createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.ENG when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return false if country code is not the same using ExtendedSpockSpecification createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.PL when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } def "should return true if country code is the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.ENG when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return false if country code is not the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.PL when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } } Let's move through the test methods one by one. First I present the code and then have a quick description of the presented snippet. def "should return true if country code is the same when creating nested structures using groovy"() { given: PlayerDetails playerDetails = new PlayerDetails( clubDetails: new ClubDetails( country: new CountryDetails( countryCode: new CountryCodeDetails( countryCode: CountryCodeType.ENG ) ) ) ) when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you could find the approach of creating nested structures by using the Groovy feature of passing properties to be set in the constructor. def "should return true if country code is the same when creating nested structures using spock mocks - requires CGLIB for non interface types"() { given: PlayerDetails playerDetails = Mock() ClubDetails clubDetails = Mock() CountryDetails countryDetails = Mock() CountryCodeDetails countryCodeDetails = Mock() countryCodeDetails.countryCode >> CountryCodeType.ENG countryDetails.countryCode >> countryCodeDetails clubDetails.country >> countryDetails playerDetails.clubDetails >> clubDetails when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you can find a test that creates mocks using Spock - mind you that you need CGLIB as a dependency when you are mocking non interface types. def "should return true if country code is the same using ExtendedSpockSpecification's createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.ENG when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you have an example of creating nested mocks using the createNestedMocks method. def "should return false if country code is not the same using ExtendedSpockSpecification createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.PL when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } An example showing that creating nested mocks using the createNestedMocks method really works - should return false for improper country code. def "should return true if country code is the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.ENG when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you have an example of creating nested spies using the createNestedSpies method. def "should return false if country code is not the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.PL when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } An example showing that creating nested spies using the createNestedSpies method really works - should return false for improper country code. Summary In this post I have shown you how you can create nested mocks and spies using Spock. It can be useful especially when you are working with nested structures such as JAXB. Still you have to bear in mind that those structures to some extend violate the Law of Demeter. If you check my previous article about Mockito you would see that: We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. And in case of this example the idea is the same - we are talking about structures so we don't violate the Law of Demeter. Advantages With a single method you can mock/spy nested elements Code cleaner than creating tons of objects that you then have to manually set Disadvantages Your IDE won't help you with providing the property names since the properties are presented as Strings You have to set Test Doubles only in the Specification context (and sometimes you want to go outside this scope) Sources As usual the sources are available at BitBucket and GitHub.

We’re all works in progress, honey. And believe me when I tell you that I’ve had to work harder than most. ― Susan Elizabeth Phillips, "Ain't She Sweet" It's pretty well understood that limiting Work In Progress - or WIP as it is often abbreviated - is a good thing. Ideally, WIP should be limited to one item in progress at a time. Having multiple pieces of inventory on-hand is a form of waste, since each incurs a handling cost, and any work done on one of them will depreciate while another is being worked on. In theory at least, restricting WIP to one item at a time will reduce this waste and get value out of the door as quickly as possible. This principle of Single Piece Flow (SPF) is central to Lean-Kanban ways of working, especially in a manufacturing context. In a software context the accepted WIP limits tend to be rather higher. It is often limited to one item per developer, such as by allowing each developer only one avatar to place on an item, and it can be reduced further if pair-programming is in use. As such, software teams might not often achieve SPF but the value of limiting WIP as far as possible is still understood. There are however problems in interpreting limited WIP in Scrum. This is because a Scrum board will often take the form of a task board ... not a Kanban board. In other words, the work being limited by Scrum teams is not always a user story or similar requirement. It is often a task. Task-limited WIP allows developers to progress tasks from any user story in any order. They could potentially limit themselves to one or two tasks from a story, complete them, then move on to a task from a different story and maybe a task from a third. In effect multiple stories - perhaps even the entire Sprint Backlog of stories - can be in progress before so much as one story gets completed. None of this breaks Scrum rules. There's nothing to stop a team, in Sprint Planning, from organizing the Sprint Backlog into any number of tasks which can be progressed in any order they choose, and from delivering all of the user stories in one go at the end of the Sprint. The Sprint Goal can of course be met by this approach, and there should still be a nice task burn-down to show the associated technical risks being managed. The problem is that it defers approval of each user story to the end of the Sprint (i.e. the Sprint Review), when it is best-practice to get continual sign-off by a Product Owner throughout the iteration. On-going inspection allows the business risks of delivery to be managed well, and not just the technical risks. This is an issue that all Scrum teams must consider when they formulate a Sprint Plan. Is it important to limit WIP in terms of user stories rather than tasks, and thereby facilitate early approval of those stories by a Product Owner? Or would this compromise the team's principle of incremental delivery ... and amount to Lean-Kanban by the back door?

Remembrance and reflection, how allied; What thin partitions sense from thought divide. - Alexander Pope Retrospectives, and why you need them A couple of months ago we looked at how to conduct a Sprint Review. We saw that a Review considers what work was done, and distinguished it from a Sprint Retrospective which reflects upon how work is being done. The distinction between the two can appear to be rather academic, and slurring a Review and a Retrospective together is a mistake that is often made by immature teams. After all, both take a reflective view of a Sprint that has just finished, and both can be said to fulfill a remit of historical inquiry. Yet while the separation of concerns might seem to be a narrow one, it is nonetheless quite precise, and there is great value to be had in maintaining the appropriate focus. A Review looks candidly at what has been achieved, and soberly at what remains to be achieved, with regard to product completion. A Retrospective on the other hand is an opportunity for the Scrum Team to inspect and adapt their actual implementation of the Scrum process. The rationale behind this inspection is methodological but it is in no sense abstract. It is grounded firmly in the desire to achieve worthwhile and practical reform. Perhaps there are certain working practices which the team can make more efficient, or which can otherwise be improved upon. If so, a Retrospective presents the ideal opportunity for those improvements to be discussed and brought into action. Failing to inspect and adapt in this manner will condemn a team to perpetual infancy and the repetition of past mistakes. Sprint Retrospectives help keep a Scrum team on the road to continual improvement. When these sessions are done well, team members will not be afraid to challenge the status quo, and will do so in a constructive and informed manner. The result will be an improved delivery of value – in fact, the sort of productivity gain that might well be identified in the Sprint Review we considered earlier. In this article we’ll switch our attention fully to Retrospectives, and examine the matter of how they should be conducted. Setting up a Retrospective As any event manager will tell you, the key to a successful gig lies in the preparation. Okay…I’ll concede that a holding a Retrospective isn’t as mammoth an undertaking as hosting the Thinking Digital conference, nor can it be said to demand the organizational skills of Bruce Springsteen’s road manager. Nevertheless it’s still important to get a few ducks in a row. Let’s start by lining them up and giving them some admittedly rather unimaginative names: Why, Who, Where, When, and What. We’ve just covered the issue of why a Retrospective needs to be held…that duck’s down. Let’s pop the rest. Who should attend a Sprint Retrospective? The invitation list for a Sprint Retrospective should be simple and uncontroversial. According to the Scrum Guide all Scrum Team members are expected to attend. That’s the Developers, the Scrum Master (who may facilitate the session), and the Product Owner. No others are expected. In fact, it would be quite irregular to extend the invitation to other people, even if they consider themselves to be important players or stakeholders. That’s because it is the Scrum Team who are responsible for the way they have implemented the Scrum Framework. Only they are in a position to inspect and adapt their very own ways of working. For this reason, all members have a duty to be present, to contribute, and to help make each Retrospective a success. Some teams exclude the Product Owner from this activity, arguing that if he or she was present, the team would not be able to have an open and frank discussion. This is a common issue and we’ll return to it later. For now though, just take it as read that a good Retrospective must include all Scrum Team members, and will give each a voice in molding the processes and working environment that they collectively own. Where should a Retrospective be held? Let’s answer this one with another question. If all of the Scrum Team members are co-located, and if they have the necessary equipment to hand (such as their Scrum board, plus a whiteboard for notes), why not hold the retrospective in situ? In other words, why not just hold the session at the team’s desks? Well, although this might sound like a capital idea, there can be problems. Perhaps it would create too much of a disturbance and annoy other teams within earshot? Then again, perhaps the physical layout of the working area is simply not conducive to holding a meeting. Perhaps the team is not entirely co-located in the first place. Any one of these things can tip the balance in favour of booking an actual meeting room, and getting everyone to decamp there for a Sprint Retrospective. If so, remember to book such a room in advance…if possible as a recurring appointment for the anticipated duration of the project. Make sure it has sufficient capacity and the resources needed. When should a Retrospective happen? The glib answer is to say that a Retrospective should happen “at the end of each Sprint”. A more useful answer would say whether or not it should precede or follow the Sprint Review. In my experience it is generally better to do the Review first, because that helps to establish a context within which the Retrospective can happen. The next thing to consider is how long to allow for the session. As with all Scrum events, a Sprint Retrospective is time-boxed. This means that it isn’t allowed to exceed a set length. The rules of Scrum are exact: for a one month Sprint the limit for a Retrospective is 3 hours, which is reduced to one-and-a-half hours for a two week Sprint. You should adjust this value by the same ratio if needed. Note that if a Retrospective finishes before the time-box expires, that’s fine and dandy. You aren’t obliged to use all of the available time. The rule is simply that the time-box must never be exceeded. Scrum is not a philosophy in which matters are allowed to drag on. What topics should the Retrospective cover? This is the biggest duck in the row, and it’ll take a few pings to knock it down. What we have to do is to establish a suitable agenda for a Sprint Retrospective. We have to formulate it in such a way that the inspection of the team’s Scrum implementation does indeed happen. We also have to make sure that any recommendations for its adaptation are elicited, agreed, and turned into achievable action items. The Scrum Guide provides us with something of a starting point. It isn’t much, but I reckon that if you look at it through a beer glass with your head sideways and one eye closed, you can just about discern a notional agenda for holding a Sprint Retrospective. A notional agenda The Scrum Guide is sparing in the advice it gives on how to conduct a Retrospective. We are told that a Scrum Team must: Inspect how the last Sprint went with regards to people, relationships, process, and tools; Identify and order the major items that went well and potential improvements; and, Create a plan for implementing improvements to the way the Scrum Team does its work…[including]…ways to increase product quality by adapting the Definition of “Done” as appropriate. Yes, I know that’s not much to go on, but each of these items is clearly significant. They seem to address the very rubric of agile practice; we can recognize in them a succinct appeal to the three legs of Transparency, Inspection, and Adaptation. In them, we can see not only a notional agenda, but also how critical a Sprint Retrospective is to the Scrum process. A Retrospective is arguably the most important time-boxed event that any agile process can have. If we want to turn these points into a more formal agenda for the session, we’ll have to make sure that each of them is addressed carefully. Towards a canonical format Scrum has been around for well over a decade now, and a fairly standard agenda for conducting a Sprint Retrospective has emerged. Here’s what it looks like. Set the scene. Ways to do this can include any or all of the following: Sketching out a timeline of significant events that occurred in the Sprint, so its historical context can be established Holding the Sprint Review shortly beforehand, so the project context is fresh in attendees’ minds Declaring the Prime Directive in order to define a professional context of mutual respect and openness Assess prior action items. Unless this is the first sprint, there will have been an earlier retrospective in which some improvements will have been proposed. Look back over each of them. Have they been followed through? In short, has the process actually been adapted following that earlier inspection? If any action items remain undone, make a note of them. They’ll have to be considered when determining actions for the future. Set up a Retrospective Board. This can be a whiteboard, or even a large sheet of paper stuck to a wall. Divide it into four quadrants and label each in the following manner. The precise terminology does tend to vary a bit. There can be subtle and not-so-subtle differences in meaning (consider the difference between “good points” and “things to continue doing”). Be aware of these differences, as they will shape the responses and ultimately the results. “What went well” (or “good points”, or “things to continue doing”) “What didn’t go well” (or “bad points”, or “things to stop doing”) “Ideas for improvement” (or things to “start doing”) “Shout-outs” (i.e. recognition of noteworthy individual contributions) Storm the Board. There are several ways in which this can be done. Here are some of the more common ones: Sticky notes. This method is fairly democratic in that each attendee gets a clear say by putting sticky notes on a board. Assertive individuals are therefore less able to dominate others. However, it can be disjointed as attention shifts from one person’s topics to another person’s. As such, it can be hard to develop a line of thought for group discussion. Here’s the process: Blocks of notes are distributed to the attendees. They are given a small time-box (5 or 10 minutes) to jot down their ideas…good points, bad points, improvements, and shouts. Each attendee should write one point per sticky note. There is no limit to the number of points they can make. After the time is up, attendees take it in turn to put their notes on the board and in the relevant quadrants As an attendee puts their sticky note on the board, they briefly state what the point is to the rest of the team Once the last attendee has finished, duplicate points will be identified by the group and removed. Facilitator-as-arbitrator. In this approach a facilitator will act as a scribe for the group, and write their ideas on the board. Group discussion of ideas is encouraged, and the facilitator can arbitrate in the event of disagreement. The downside is that it can favor the more assertive type of individual who ends up doing most of the talking. Here’s how it’s done: The facilitator stands in front of the board with a marker pen Any attendee who has a suggestion to make will make it – a good point, bad point, idea, or shout-out The facilitator writes each suggestion into the appropriate quadrant, disallowing any duplicates. The group discuss the merits of each suggestion The facilitator will erase, keep, or revise each suggestion according to group opinion Hybrid. This uses a mix of techniques, such as a facilitated session for identifying good points and bad points, and a sticky-note approach in order to elicit ideas for improvement. Changing the techniques used in a Retrospective every now and then can help keep the sessions fresh, and is certainly a good idea if you reckon they are getting a bit stale. Propose actions. I have five rules that I apply when “storming the board” with a team: For every bad point there must be an idea for improvement. In other words, for everything that people are being asked to stop doing an alternative and better course of action must be proposed. This rule helps to keep attendees focused on the need to adapt the process constructively, and not to use the session for mere complaint. If you have been storming for “good points” rather than for things to “start doing”, make sure that each of those points is matched with an idea for further improvement. It isn’t enough to look back appreciatively whenever something positive has happened. Your challenge is to translate that observation into an even bigger future win. Re-assess undone action items from the previous Retrospective. If any remain undone, ask if they are worth bringing forward. Ask why they weren’t implemented, with a view to finding out what really needs to happen to expedite them. If these outstanding actions are impractical, or are no longer relevant, jettison them and concentrate on those improvements which are valuable and achievable. Ask the “Five Whys”. For each action item you produce, you need to be sure that you have understood the root cause and that the action will be appropriate. A shallow retrospective is no retrospective at all. It has to be deep and probing. Improve the Definition of Done. The Scrum Guide doesn’t provide much advice about holding Retrospectives, but it is quite clear about the need to revisit the Definition of Done. This is something that many teams, including some quite experienced ones, forget or otherwise fail to do. So be careful to identify any room for improvement in the team’s understanding of what “done” means, and what it should take for work to be considered potentially releasable. Vote. It’s quite possible that the list of proposed actions will be extensive. In aggregate they could amount to too much change if all were to be implemented in the forthcoming Sprint. You can resolve this by getting team members to vote on action items, so that only the most important ones are taken forward. For example, if the team can take forward five items, allow each attendee to vote for five of them. The most popular can then be actioned. Other observations Here are some other things to consider when conducting a Sprint Retrospective. Decide whether or not to precede it with the Scrum “Prime Directive”. This is an assertion which is meant to be said, in earnest, before each and every Retrospective. It isn’t mentioned in the Scrum Guide, but it is widely recognized and some teams do choose to recite it. “Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand” We considered the significance of this assertion in an earlier article on Agile Teamwork in Practice, so I’m not going to say much more about it here. However, Martin Fowler has expressed his thoughts on the Prime Directive, and I suggest you read his opinion piece in full. All I’ll add is that I am in agreement with his observations and that I share his sense of revulsion. Determine what to do about Product Owner representation. According to the Scrum Primer the Product Owner may attend a Sprint Retrospective. Only “Development Team” members are actually required to be there. Yet according to the Scrum Guide, all “Scrum Team” members must attend. The Scrum Team is a wider group than the Development team and includes the Product Owner. The reason for this discrepancy probably lies in the interpretation of process ownership. If we see the Development Team as owning the process through which iterative and incremental value will be delivered to a Product Owner, then the PO would not indeed have a say in the adaptation of that process. He or she would merely be a consumer of its outputs, and would therefore be a stakeholder in a Sprint Review but not in a Sprint Retrospective. However, if we view the process as a more collaborative one, in which the Development Team works with the Product Owner to deliver potentially releasable increments of value every Sprint, then the PO would indeed be a stakeholder in how that process is managed, and must therefore attend. It’s therefore important to determine what relationship the Development Team has, or should have, with the Product Owner. It’s unquestionably best if a Product Owner is on-side as a team player, and can handle root cause analysis and the exposure of potentially uncomfortable truths. Whether or not that is the case though is only something that the team can decide. Remember they’re human. Bring snacks and drinks to keep attendees refreshed, and allow enough time for breaks – at least 10 minutes every hour. Consider wrapping up the session with a “touchy feely graph” of some sort, which captures the mood and confidence of the team. Allow everyone to mark a dot or cross on a chart to show how positive or negative they feel about things, and then see how the mood changes…hopefully for the better…from one Sprint to the next. Conclusion A Sprint Retrospective is arguably the most important event that a team can hold. It provides the means to inspect and adapt the team’s actual implementation of the Scrum framework. In this article we’ve looked at how to create an agenda for the session and how to facilitate it, and at the issues of when and where it should be held, and who should attend. Those who cannot remember the past are condemned to repeat it. - George Santayana

Jim and I have been doing a bit of work over the last week which involved calling neo4j’s HA status URI to check whether or not an instance was a master/slave and we’ve been using jersey-client. The code looked roughly like this: class Neo4jInstance { private Client httpClient; private URI hostname; public Neo4jInstance(Client httpClient, URI hostname) { this.httpClient = httpClient; this.hostname = hostname; } public Boolean isSlave() { String slaveURI = hostname.toString() + ":7474/db/manage/server/ha/slave"; ClientResponse response = httpClient.resource(slaveURI).accept(TEXT_PLAIN).get(ClientResponse.class); return Boolean.parseBoolean(response.getEntity(String.class)); } } While writing some tests against this code we wanted to stub out the actual calls to the HA slave URI so we could simulate both conditions and a brief search suggested that mockito was the way to go. We ended up with a test that looked like this: @Test public void shouldIndicateInstanceIsSlave() { Client client = mock( Client.class ); WebResource webResource = mock( WebResource.class ); WebResource.Builder builder = mock( WebResource.Builder.class ); ClientResponse clientResponse = mock( ClientResponse.class ); when( builder.get( ClientResponse.class ) ).thenReturn( clientResponse ); when( clientResponse.getEntity( String.class ) ).thenReturn( "true" ); when( webResource.accept( anyString() ) ).thenReturn( builder ); when( client.resource( anyString() ) ).thenReturn( webResource ); Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave(); assertTrue(isSlave); } which is pretty gnarly but does the job. I thought there must be a better way so I continued searching and eventually came across this post on the mailing list which suggested creating a custom ClientHandler and stubbing out requests/responses there. I had a go at doing that and wrapped it with a little DSL that only covers our very specific use case: private static ClientBuilder client() { return new ClientBuilder(); } static class ClientBuilder { private String uri; private int statusCode; private String content; public ClientBuilder requestFor(String uri) { this.uri = uri; return this; } public ClientBuilder returns(int statusCode) { this.statusCode = statusCode; return this; } public Client create() { return new Client() { public ClientResponse handle(ClientRequest request) throws ClientHandlerException { if (request.getURI().toString().equals(uri)) { InBoundHeaders headers = new InBoundHeaders(); headers.put("Content-Type", asList("text/plain")); return createDummyResponse(headers); } throw new RuntimeException("No stub defined for " + request.getURI()); } }; } private ClientResponse createDummyResponse(InBoundHeaders headers) { return new ClientResponse(statusCode, headers, new ByteArrayInputStream(content.getBytes()), messageBodyWorkers()); } private MessageBodyWorkers messageBodyWorkers() { return new MessageBodyWorkers() { public Map> getReaders(MediaType mediaType) { return null; } public Map> getWriters(MediaType mediaType) { return null; } public String readersToString(Map> mediaTypeListMap) { return null; } public String writersToString(Map> mediaTypeListMap) { return null; } public MessageBodyReader getMessageBodyReader(Class tClass, Type type, Annotation[] annotations, MediaType mediaType) { return (MessageBodyReader) new StringProvider(); } public MessageBodyWriter getMessageBodyWriter(Class tClass, Type type, Annotation[] annotations, MediaType mediaType) { return null; } public List getMessageBodyWriterMediaTypes(Class tClass, Type type, Annotation[] annotations) { return null; } public MediaType getMessageBodyWriterMediaType(Class tClass, Type type, Annotation[] annotations, List mediaTypes) { return null; } }; } public ClientBuilder content(String content) { this.content = content; return this; } } If we change our test to use this code it now looks like this: @Test public void shouldIndicateInstanceIsSlave() { Client client = client().requestFor("http://localhost:7474/db/manage/server/ha/slave"). returns(200). content("true"). create(); Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave(); assertTrue(isSlave); } Is there a better way? In Ruby I’ve used WebMock to achieve this and Ashok pointed me towards WebStub which looks nice except I’d need to pass in the hostname + port rather than constructing that in the code.

I recently wanted to attach an EBS volume to an existing EC2 instance that I had running and since it was for a one off tasks (famous last words) I decided to configure it manually. I created the EBS volume through the AWS console and one thing that initially caught me out is that the EC2 instance and EBS volume need to be in the same region and zone. Therefore if I create my EC2 instance in ‘eu-west-1b’ then I need to create my EBS volume in ‘eu-west-1b’ as well otherwise I won’t be able to attach it to that instance. I attached the device as /dev/sdf although the UI gives the following warning: Linux Devices: /dev/sdf through /dev/sdp Note: Newer linux kernels may rename your devices to /dev/xvdf through /dev/xvdp internally, even when the device name entered here (and shown in the details) is /dev/sdf through /dev/sdp. After attaching the EBS volume to the EC2 instance my next step was to SSH onto my EC2 instance and make the EBS volume available. The first step is to create a file system on the volume: $ sudo mkfs -t ext3 /dev/sdf mke2fs 1.42 (29-Nov-2011) Could not stat /dev/sdf --- No such file or directory The device apparently does not exist; did you specify it correctly? It turns out that warning was handy and the device has in fact been renamed. We can confirm this by callingfdisk: $ sudo fdisk -l Disk /dev/xvda1: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvda1 doesn't contain a valid partition table Disk /dev/xvdf: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvdf doesn't contain a valid partition table /dev/xvdf is the one we’re interested in so I re-ran the previous command: $ sudo mkfs -t ext3 /dev/xvdf mke2fs 1.42 (29-Nov-2011) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 3276800 inodes, 13107200 blocks 655360 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 400 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done Once I’d done that I needed to create a mount point for the volume and I thought the best place was probably a directory under /mnt: $ sudo mkdir /mnt/ebs The final step is to mount the volume: $ sudo mount /dev/xvdf /mnt/ebs And if we run df we can see that it’s ready to go: $ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.9G 883M 6.7G 12% / udev 288M 8.0K 288M 1% /dev tmpfs 119M 164K 118M 1% /run none 5.0M 0 5.0M 0% /run/lock none 296M 0 296M 0% /run/shm /dev/xvdf 50G 180M 47G 1% /mnt/ebs

Just about every 6 months or so an article appears cursing Maven, attracting both proponents as opponents to Maven and Ant. While it’s real fun to watch (I really get a laugh when people start to advocate the return to Ant), most of the time it’s always the same arguments. Maven lacks flexibility, the plugin system sucks (when will people learn to use plugin versions…), you can’t use scripting and the all time favorite: the release plugin sucks. Well, I am a Maven addict and I’m happy to say: yes, I agree, the release plugin sucks. Big time. But here’s something you may have forgotten: you don’t need it! Even more: you shouldn’t use it. The Maven release plugin tries to make releasing software a breeze. That’s where the plugin authors got it wrong to start with. Releases are not something done on a whim. They are carefully planned and orchestrated actions, preceded by countless rules and followed by more rules. Assuming you can bundle all that in a simple mvn release:release is just plain naive. Even Maven’s most fierce supporters agree on this. The Maven release plugin just tries to do too much stuff at once: build your software, tag it, build it again, deploy it, build the site (triggering yet another build in the process) and deploy the site. And whilst doing that, running the tests x times. Most of the time, you’re making candidate releases, so building the complete documentation is a complete waste of time. Now, if you break down the release plugin into sensible steps, you’ll really save yourself a whole lot of trouble. I use these steps to release something. As a sidenote: I use git and git-flow standards (as described here). Assume the POM’s version’s currently on 1.0-SNAPSHOT. Announce the release process Very important. As I said, you don’t release on a whim. Make sure everyone on your team knows a release is pending and has all their stuff pushed to the development branch that needs to be included. Branch the development branch into a release branch. Following git-flow rules, I make a release branch 1.0. Update the POM version of the development branch. Update the version to the next release version. For example mvn versions:set -DnewVersion=2.0-SNAPSHOT. Commit and push. Now you can put resources developing towards the next release version. Update the POM version of the release branch. Update the version to the standard CR version. For example mvn versions:set -DnewVersion=1.0.CR-SNAPSHOT. Commit and push. Run tests on the release branch. Run all the tests. If one or more fail, fix them first. Create a candidate release from the release branch. Use the Maven version plugin to update your POM’s versions. For example mvn versions:set -DnewVersion=1.0.CR1. Commit and push. Make a tag on git. Use the Maven version plugin to update your POM’s versions back to the standard CR version. For example mvn versions:set -DnewVersion=1.0.CR-SNAPSHOT. Commit and push. Checkout the new tag. Do a deployment build (mvn clean deploy). Since you’ve just run your tests and fixed any failing ones, this shouldn’t fail. Put deployment on QA environment. Iterate until QA gives a green light on the candidate release. Fix bugs. Fix bugs reported on the CR releases on the release branch. Merge into development branch on regular intervals (or even better, continuous). Run tests continuously, making bug reports on failures and fixing them as you go. Create a candidate release. Use the Maven version plugin to update your POM’s versions. For example mvn versions:set -DnewVersion=1.0.CRx. Commit and push. Make a tag on git. Use the Maven version plugin to update your POM’s versions back to the standard CR version. For example mvn versions:set -DnewVersion=1.0.CR-SNAPSHOT. Commit and push. Checkout the new tag. Do a deployment build (mvn clean deploy). Since you’ve run your tests continuously, this shouldn’t fail. Put deployment on QA environment. Once QA has signed off on the release, create a final release. Check whether there are no new commits since the last release tag (if there are, slap developers as they have done stuff that wasn’t needed or asked for). Use the Maven version plugin to update your POM’s versions. For example mvn versions:set -DnewVersion=1.0. Commit and push. Tag the release branch. Merge into the master branch. Checkout the master branch. Do a deployment build (mvn clean deploy). Start production release and deployment process (in most companies, not a small feat). This can involve building the site and doing other stuff, some not even Maven related. There’s no way in hell Maven can automate this process and if you try, you’ll bump into the many pitfalls the release plugin has to offer. The release plugin is just a combination of the versions, scm, deploy and site plugin that seriously violates the single responsibility principle. The release plugin is one of the reasons Maven has gotten a bad reputation with some people. It’s long due for an overhaul, but if you ask me, they should just remove it altogether. Releasing software is a process, not a single command on the command line. The process I just described isn’t perfect in any way, but it works and I avoid using the release plugin as it just does too much stuff. Have fun bashing Maven, but please, keep it clean :) .

When you have a piece of code that often fails and must be retried, this Java 7/8 library provides rich and unobtrusive API with fast and scalable solution to this problem: ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); RetryExecutor executor = new AsyncRetryExecutor(scheduler). retryOn(SocketException.class). withExponentialBackoff(500, 2). //500ms times 2 after each retry withMaxDelay(10_000). //10 seconds withUniformJitter(). //add between +/- 100 ms randomly withMaxRetries(20); You can now run arbitrary block of code and the library will retry it for you in case it throws SocketException: final CompletableFuture future = executor.getWithRetry(() -> new Socket("localhost", 8080) ); future.thenAccept(socket -> System.out.println("Connected! " + socket) ); Please look carefully! getWithRetry() does not block. It returns CompletableFuture immediately and invokes given function asynchronously. You can listen for that Future or even for multiple futures at once and do other work in the meantime. So what this code does is: trying to connect to localhost:8080 and if it fails with SocketException it will retry after 500 milliseconds (with some random jitter), doubling delay after each retry, but not above 10 seconds. Equivalent but more concise syntax: executor. getWithRetry(() -> new Socket("localhost", 8080)). thenAccept(socket -> System.out.println("Connected! " + socket)); This is a sample output that you might expect: TRACE | Retry 0 failed after 3ms, scheduled next retry in 508ms (Sun Jul 21 21:01:12 CEST 2013) java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0-ea] //... TRACE | Retry 1 failed after 0ms, scheduled next retry in 934ms (Sun Jul 21 21:01:13 CEST 2013) java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0-ea] //... TRACE | Retry 2 failed after 0ms, scheduled next retry in 1919ms (Sun Jul 21 21:01:15 CEST 2013) java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0-ea] //... TRACE | Successful after 2 retries, took 0ms and returned: Socket[addr=localhost/127.0.0.1,port=8080,localport=46332] Connected! Socket[addr=localhost/127.0.0.1,port=8080,localport=46332] Imagine you connect to two different systems, one is slow, second unreliable and fails often: CompletableFuture stringFuture = executor.getWithRetry(ctx -> unreliable()); CompletableFuture intFuture = executor.getWithRetry(ctx -> slow()); stringFuture.thenAcceptBoth(intFuture, (String s, Integer i) -> { //both done after some retries }); thenAcceptBoth() callback is executed asynchronously when both slow and unreliable systems finally reply without any failure. Similarly (using CompletableFuture.acceptEither()) you can call two or more unreliable servers asynchronously at the same time and be notified when the first one succeeds after some number of retries. I can’t emphasize this enough - retries are executed asynchronously and effectively use thread pool, rather than sleeping blindly. Rationale Often we are forced to retry given piece of code because it failed and we must try again, typically with a small delay to spare CPU. This requirement is quite common and there are few ready-made generic implementations with retry support in Spring Batch through RetryTemplate class being best known. But there are few other, quite similar approaches ([1], [2]). All of these attempts (and I bet many of you implemented similar tool yourself!) suffer the same issue - they are blocking, thus wasting a lot of resources and not scaling well. This is not bad per se because it makes programming model much simpler - the library takes care of retrying and you simply have to wait for return value longer than usual. But not only it creates leaky abstraction (method that is typically very fast suddenly becomes slow due to retries and delay), but also wastes valuable threads since such facility will spend most of the time sleeping between retries. Therefore Async-Retry utility was created, targeting Java 8 (with Java 7 backport existing) and addressing issues above. The main abstraction is RetryExecutor that provides simple API: public interface RetryExecutor { CompletableFuture doWithRetry(RetryRunnable action); CompletableFuture getWithRetry(Callable task); CompletableFuture getWithRetry(RetryCallable task); CompletableFuture getFutureWithRetry(RetryCallable> task); } Don’t worry about RetryRunnable and RetryCallable - they allow checked exceptions for your convenience and most of the time we will use lambda expressions anyway. Please note that it returns CompletableFuture. We no longer pretend that calling faulty method is fast. If the library encounters an exception it will retry our block of code with preconfigured backoff delays. The invocation time will sky-rocket from milliseconds to several seconds. CompletableFuture clearly indicates that. Moreover it’s not a dumb java.util.concurrent.Future we all know - CompletableFuture in Java 8 is very powerful and most importantly - non-blocking by default. If you need blocking result after all, just call .get() on Future object. Basic API The API is very simple. You provide a block of code and the library will run it multiple times until it returns normally rather than throwing an exception. It may also put configurable delays between retries: RetryExecutor executor = //... executor.getWithRetry(() -> new Socket("localhost", 8080)); Returned CompletableFuture will be resolved once connecting to localhost:8080 succeeds. Optionally we can consume RetryContext to get extra context like which retry is currently being executed: executor. getWithRetry(ctx -> new Socket("localhost", 8080 + ctx.getRetryCount())). thenAccept(System.out::println); This code is more clever than it looks. During first execution ctx.getRetryCount() returns 0, therefore we try to connect to localhost:8080. If this fails, next retry will try localhost:8081 (8080 + 1) and so on. And if you realize that all of this happens asynchronously you can scan ports of several machines and be notified about first responding port on each host: Arrays.asList("host-one", "host-two", "host-three"). stream(). forEach(host -> executor. getWithRetry(ctx -> new Socket(host, 8080 + ctx.getRetryCount())). thenAccept(System.out::println) ); For each host RetryExecutor will attempt to connect to port 8080 and retry with higher ports. getFutureWithRetry() requires special attention. I you want to retry method that already returns CompletableFuture: e.g. result of asynchronous HTTP call: private CompletableFuture asyncHttp(URL url) { /*...*/} //... final CompletableFuture> response = executor.getWithRetry(ctx -> asyncHttp(new URL("http://example.com"))); Passing asyncHttp() to getWithRetry() will yield CompletableFuture>. Not only it’s awkward to work with, but also broken. The library will barely call asyncHttp() and retry only if it fails, but not if returned CompletableFuture fails. The solution is simple: final CompletableFuture response = executor.getFutureWithRetry(ctx -> asyncHttp(new URL("http://example.com"))); In this case RetryExecutor will understand that whatever was returned from asyncHttp() is the actually just a Future and will (asynchronously) wait for result or failure. This library is much more powerful, so let’s dive into: Configuration Options In general there are two important factors you can configure: RetryPolicy that controls whether next retry attempt should be made and Backoff - that optionally adds delay between subsequent retry attempts. By default RetryExecutor repeats user task infinitely on every Throwable and adds 1 second delay between retry attempts. Creating an Instance of RetryExecutor Default implementation of RetryExecutor is AsyncRetryExecutor which you can create directly: ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); RetryExecutor executor = new AsyncRetryExecutor(scheduler); //... scheduler.shutdownNow(); The only required dependency is standard ScheduledExecutorService from JDK. One thread is enough in many cases but if you want to concurrently handle retries of hundreds or more tasks, consider increasing the pool size. Notice that the AsyncRetryExecutor does not take care of shutting down the ScheduledExecutorService. This is a conscious design decision which will be explained later. AsyncRetryExecutor has few other constructors but most of the time altering the behaviour of this class is most convenient with calling chained with*() methods. You will see plenty of examples written this way. Later on we will simply use executor reference without defining it. Assume it’s of RetryExecutor type. Retrying Policy Exceptions By default every Throwable (except special AbortRetryException) thrown from user task causes retry. Obviously this is configurable. For example in JPA you may want to retry a transaction that failed due to OptimisticLockException - but every other exception should fail immediately: executor. retryOn(OptimisticLockException.class). withNoDelay(). getWithRetry(ctx -> dao.optimistic()); Where dao.optimistic() may throw OptimisticLockException. In that case you probably don’t want any delay between retries, more on that later. If you don’t like the default of retrying on every Throwable, just restrict that using retryOn(): executor.retryOn(Exception.class) Of course the opposite might also be desired - to abort retrying and fail immediately in case of certain exception being thrown rather than retrying. It’s that simple: executor. abortOn(NullPointerException.class). abortOn(IllegalArgumentException.class). getWithRetry(ctx -> dao.optimistic()); Clearly you don’t want to retry NullPointerException or IllegalArgumentException as they indicate programming bug rather than transient failure. And finally you can combine both retry and abort policies. User code will retry in case of any retryOn() exception (or subclass) unless it should abortOn() specified exception. For example we want to retry every IOException or SQLException but abort if FileNotFoundException or java.sql.DataTruncation is encountered (order is irrelevant): executor. retryOn(IOException.class). abortIf(FileNotFoundException.class). retryOn(SQLException.class). abortIf(DataTruncation.class). getWithRetry(ctx -> dao.load(42)); If this is not enough you can provide custom predicate that will be invoked on each failure: executor. abortIf(throwable -> throwable instanceof SQLException && throwable.getMessage().contains("ORA-00911") ); Max Number of Retries Another way of interrupting retrying “loop” (remember that this process is asynchronous, there is no blocking loop) is by specifying maximum number of retries: executor.withMaxRetries(5) In rare cases you may want to disable retries and barely take advantage from asynchronous execution. In that case try: executor.dontRetry() Delays Between Retries (backoff) Retrying immediately after failure is sometimes desired (see OptimisticLockException example) but in most cases it’s a bad idea. If you can’t connect to external system, waiting a little bit before next attempt sounds reasonably. You save CPU, bandwidth and other server’s resources. But there are quite a few options to consider: should we retry with constant intervals or increase delay after each failure? should there be a lower and upper limit on waiting time? should we add random “jitter” to delay times to spread retries of many tasks in time? This library answers all these questions. Fixed Interval Between Retries By default each retry is preceded by 1 second waiting time. So if initial attempt fails, first retry will be executed after 1 second. Of course we can change that default, e.g. to 200 milliseconds: executor.withFixedBackoff(200) If we are already here, by default backoff is applied after executing user task. If user task itself consumes some time, retries will be less frequent. For example with retry delay of 200ms and average time it takes before user task fails at about 50ms RetryExecutor will retry about 4 times per second (50ms + 200ms). However if you want to keep retry frequency at more predictable level you can use fixedRate flag: executor. withFixedBackoff(200). withFixedRate() This is similar to “fixed rate” vs. “fixed delay” approaches in ScheduledExecutorService. BTW don’t expect RetryExecutor to be very precise, it does it’s best but it heavily depends on aforementioned ScheduledExecutorService accuracy. Exponentially Growing Intervals Between Retries It’s probably an active research subject, but in general you may wish to expand retry delay over time, assuming that if the user task fails several times we should try less frequently. For example let’s say we start with 100ms delay until first retry attempt is made but if that one fails as well, we should wait two times more (200ms). And later 400ms, 800ms… You get the idea: executor.withExponentialBackoff(100, 2) This is an exponential function that can grow very fast. Thus it’s useful to set maximum backoff time at some reasonable level, e.g. 10 seconds: executor. withExponentialBackoff(100, 2). withMaxDelay(10_000) //10 seconds Random Jitter One phenomena often observed during major outages is that systems tend to synchronize. Imagine a busy system that suddenly stops responding. Hundreds or thousands of requests fail and are retried. It depends on your backoff but by default all these requests will retry exactly after one second producing huge wave of traffic at one point in time. Finally such failures are propagated to other systems that, in turn, synchronize as well. To avoid this problem it’s useful to spread retries over time, flattening the load. A simple solution is to add random jitter to delay time so that not all request are scheduled for retry at the exact same time. You have choice between uniform jitter (random value from -100ms to 100ms): executor.withUniformJitter(100) //ms …and proportional jitter, multiplying delay time by random factor, by default between 0.9 and 1.1 (10%): executor.withProportionalJitter(0.1) //10% You may also put hard lower limit on delay time to avoid to short retry times being scheduled: executor.withMinDelay(50) //ms Implementation Details This library was built with Java 8 in mind to take advantage of lambdas and new CompletableFuture abstraction (but Java 7 port with Guava dependency exists). It uses ScheduledExecutorService underneath to run tasks and schedule retries in the future - which allows best thread utilization. But what is really interesting is that the whole library is fully immutable, there is no single mutable field, at all. This might be counter-intuitive at first, take for example this trivial code sample: ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); AsyncRetryExecutor first = new AsyncRetryExecutor(scheduler). retryOn(Exception.class). withExponentialBackoff(500, 2); AsyncRetryExecutor second = first.abortOn(FileNotFoundException.class); AsyncRetryExecutor third = second.withMaxRetries(10); It might seem that all with*() methods or retryOn()/abortOn() mutate existing executor. But that’s not the case, each configuration change creates new instance, leaving the old one untouched. So for example while first executor will retry on FileNotFoundException, the second and third won’t. However they all share the same scheduler. This is the reason why AsyncRetryExecutor does not shut down ScheduledExecutorService (it doesn’t even have any close() method). Since we have no idea how many copies of AsyncRetryExecutor exist pointing to the same scheduler, we don’t even try to manage its lifecycle. However this is typically not a problem (see Spring integration below). You might be wondering, why such an awkward design decision? There are three reasons: when writing a concurrent code immutability can greatly reduce risk of multi-threading bugs. For example RetryContext holds number of retries. But instead of mutating it we simply create new instance (copy) with incremented but final counter. No race condition or visibility can ever occur. if you are given an existing RetryExecutor which is almost exactly what you want but you need one minor tweak, you simply call executor.with...() and get a fresh copy. You don’t have to worry about other places where the same executor was used (see: Spring integration for further examples) functional programming and immutable data structures are sexy these days ;-). N.B.: AsyncRetryExecutor is not marked final, does you can break immutability by subclassing it and adding mutable state. Please don’t do this, subclassing is only permitted to alter behaviour. Dependencies This library requires Java 8 and SLF4J for logging. Java 7 port additionally depends on Guava. Spring Integration If you are just about to use RetryExecutor in Spring - feel free, but the configuration API might not work for you. Spring promotes (or used to promote) the convention of mutable services with plenty of setters. In XML you define bean and invoke setters (via ) on it. This convention assumes the existence of mutating setters. But I found this approach error-prone and counter-intuitive under some circumstances. Let’s say we globally defined org.springframework.transaction.support.TransactionTemplate bean and injected it in multiple places. Great. Now there is this one single request that requires slightly different timeout: @Autowired private TransactionTemplate template; and later in the same class: final int oldTimeout = template.getTimeout(); template.setTimeout(10_000); //do the work template.setTimeout(oldTimeout); This code is wrong on so many levels! First of all if something fails we never restore oldTimeout. OK, finally to the rescue. But also notice how we changed global, shared TransactionTemplate instance. Who knows how many other beans and threads are just about to use it, unaware of changed configuration? And even if you do want to globally change the transaction timeout, fair enough, but it’s still wrong way to do this. private timeout field is not volatile and thus changes made to it may or may not be visible to other threads. What a mess! The same problem appears with many other classes like JmsTemplate. You see where I’m going? Just create one, immutable service class and safely adjust it by creating copies whenever you need it. And using such services is equally simple these days: @Configuration class Beans { @Bean public RetryExecutor retryExecutor() { return new AsyncRetryExecutor(scheduler()). retryOn(SocketException.class). withExponentialBackoff(500, 2); } @Bean(destroyMethod = "shutdownNow") public ScheduledExecutorService scheduler() { return Executors.newSingleThreadScheduledExecutor(); } } Hey! It’s 21st century, we don’t need XML in Spring any more. Bootstrap is simple as well: final ApplicationContext context = new AnnotationConfigApplicationContext(Beans.class); final RetryExecutor executor = context.getBean(RetryExecutor.class); //... context.close(); As you can see integrating modern, immutable services with Spring is just as simple. BTW if you are not prepared for such a big change when designing your own services, at least consider constructor injection. Maturity This library is covered with a strong battery of unit tests. However it wasn’t yet used in any production code and the API is subject to change. Of course you are encouraged to submit bugs, feature requests and pull requests. It was developed with Java 8 in mind but Java 7 backport exists with slightly more verbose API and mandatory Guava dependency (ListenableFuture instead of CompletableFuture from Java 8). Full source code on GitHub.

Fake system clock is a design pattern addressing testability issues of programs heavily relying on system time. If business logic flow depends on current system time, testing various flows becomes cumbersome or even impossible. Examples of such problematic scenarios include: certain business flow runs only (or is ignored) during weekends some logic is triggered only after an hour since some other event when two events occur at the exact same time (typically 1 ms precision), something should happen … Each scenario above poses unique set of challenges. Taken literally our unit tests would have to run only on specific day (1) or sleep for an hour to observe some behaviour. Scenario (3) might even be impossible to test under some circumstances since system clock can tick 1 millisecond at any time, thus making test unreliable. Fake system clock addresses these issues by abstracting system time over simple interface. Essentially you never call new Date(), new GregorianCalendar() or System.currentTimeMillis() but always rely on this: import org.joda.time.{DateTime, Instant} trait Clock { def now(): Instant def dateNow(): DateTime } As you can see I am depending on Joda Time library. Since we are already in the Scala land, one might consider scala-timeor nscala-time wrappers. Moreover the abstract name Clock is not a coincidence. It’s short and descriptive, but more importantly it mimics java.time.Clock class from Java 8 - that happens to address the same problem discussed here at the JDK level! But since Java 8 is still not here, let’s stay with our sweet and small abstraction. The standard implementation that you would normally use simply delegates to system time: import org.joda.time.{Instant, DateTime} object SystemClock extends Clock { def now() = Instant.now() def dateNow() = DateTime.now() } For the purposes of unit testing we will develop other implementations, but first let’s focus on usage scenarios. In a typical Spring/JavaEE applications fake system clock can be turned into a dependency that the container can easily inject. This makes dependence on system time explicit and manageable, especially in tests: @Controller class FooController @Autowired() (fooService: FooService, clock: Clock) { def postFoo(name: String) = fooService store new Foo(name, clock) } Here I am using Spring constructor injection asking the container to provide some Clock implementation. Of course in this case SystemClock is marked as @Service. In unit tests I can pass fake implementation and in integration tests I can place another, @Primary bean in the context, shadowing the SystemClock. This works great, but becomes painful for certain types of objects, namely entity/DTO beans and utility (static) classes. These are typically not managed by Spring so it can’t inject Clock bean to them. This forces us to pass Clock manually from the last “managed” layer: class Foo(fooName: String, clock: Clock) { val name = fooName val time = clock.dateNow() } similarly: object TimeUtil { def firstFridayOfNextMonth(clock: Clock) = //... } It’s not bad from design perspective. Both Foo constructor and firstFridayOfNextMonth() method do rely on system time so let’s make it explicit. On the other hand Clock dependency must be dragged, sometimes through many layers, just so that it can be used in one single method somewhere. Again, this is not bad per se. If your high level method has Clockparameter you know from the beginning that it relies on current time. But still is seems like a lot of boilerplate and overhead for little gain. Luckily Scala can help us here with: implicit parameters Let us refactor our solution a little bit so that Clock is an implicit parameter: @Controller class FooController(fooService: FooService) { def postFoo(name: String)(implicit clock: Clock) = fooService store new Foo(name) } @Service class FooService(fooRepository: FooRepository) { def store(foo: Foo)(implicit clock: Clock) = fooRepository storeInFuture foo } @Repository class FooRepository { def storeInFuture(foo: Foo)(implicit clock: Clock) = { val friday = TimeUtil.firstFridayOfNextMonth() //... } } object TimeUtil { def firstFridayOfNextMonth()(implicit clock: Clock) = //... } Notice how we call fooRepository storeInFuture foo ignoring second clock parameter. However this alone is not enough. We still have to provide some Clock instance as second parameter, otherwise compilation error strikes: could not find implicit value for parameter clock: com.blogspot.nurkiewicz.foo.Clock controller.postFoo("Abc") ^ not enough arguments for method postFoo: (implicit clock: com.blogspot.nurkiewicz.foo.Clock)Unit. Unspecified value parameter clock. controller.postFoo("Abc") ^ The compiler tried to find implicit value for Clock parameter but failed. However we are really close, the simplest solution is to use package object: package com.blogspot.nurkiewicz.foo package object foo { implicit val clock = SystemClock } Where SystemClock was defined earlier. Here is what happens: every time I call a function with implicit clock: Clock parameter inside com.blogspot.nurkiewicz.foo package, the compiler will discover foo.clock implicit variable and pass it transparently. In other words the following code snippets are equivalent but the second one provides explicit Clock, thus ignoring implicits: TimeUtil.firstFridayOfNextMonth() TimeUtil.firstFridayOfNextMonth()(SystemClock) also equivalent (first form is turned into the second by the compiler): fooService.store(foo) fooService.store(foo)(SystemClock) Interestingly in the bytecode level, implicit parameters aren’t any different from normal parameters so if you want to call such method from Java, passing Clock instance is mandatory and explicit. implicit clock parameter seems to work quite well. It hides ubiquitous dependency while still giving possibility to override it. For example in: fooService.store(foo) fooService.store(foo)(SystemClock) Tests The whole point of abstracting system time was to enable unit testing by gaining full control over time flow. Let us begin with a simple fake system clock implementation that always returns the same, specified time: class FakeClock(fixed: DateTime) extends Clock { def now() = fixed.toInstant def dateNow() = fixed } Of course you are free to put any logic here: advancing time by arbitrary value, speeding it up, etc. You get the idea. Now remember, the reason for implicit parameter was to hide Clock from normal production code while still being able to supply alternative implementation. There are two approaches: either pass FakeClock explicitly in tests: val fakeClock = new FakeClock( new DateTime(2013, 7, 15, 0, 0, DateTimeZone.UTC)) controller.postFoo("Abc")(fakeClock) or make it implicit but more specific to the compiler resolution mechanism: implicit val fakeClock = new FakeClock( new DateTime(2013, 7, 15, 0, 0, DateTimeZone.UTC)) controller.postFoo("Abc") The latter approach is easier to maintain as you don’t have to remember about passing fakeClock to method under test all the time. Of course fakeClock can be defined more globally as a field or even inside test package object. No matter which technique of providing fakeClock we choose, it will be used throughout all calls to service, repository and utilities. The moment we given explicit value to this parameter, implicit parameter resolution is ignored. Problems and summary Solution above to testing systems heavily dependant on time is not free from issues on its own. First of all the implicitClock parameter must be propagated throughout all the layers up to the client code. Notice that Clock is only needed in repository/utility layer while we had to drag it up to the controller layer. It’s not a big deal since the compiler will fill it in for us, but sooner or later most of our methods will include this extra parameter. Also Java and frameworks working on top of our code are not aware of Scala implicit resolution happening at compile time. Therefore e.g. our Spring MVC controller will not work as Spring is not aware of SystemClock implicit variable. It can be worked around though with WebArgumentResolver. Fake system clock pattern in general works only when used consistently. If you have even one place when real time is used directly as opposed to Clock abstraction, good luck in finding test failure reason. This applies equally to libraries and SQL queries. Thus if you are designing a library relying on current time, consider providing pluggable Clock abstraction so that client code can supply custom implementation like FakeClock. In SQL, on the other hand, do not rely on functions likeNOW() but always explicitly provide dates from your code (and thus from custom Clock).

In the last post I talked about our need for some improved testing tools, our choice of Spock as something to spike, and how mocking looks in Spock. As that blog got rather long, I saved the next installment for a separate post. Today I want to look at stubbing. Stubbing Mocking is great for checking outputs - in the example in the last post, we're checking that the process of encoding an array calls the right things on the way out, if you like - that the right stuff gets poked onto the bsonWriter. Stubbing is great for faking your inputs (I don't know why this difference never occurred to me before, but Colin's talk at Devoxx UK made this really clear to me). One of the things we need to do in the compatibility layer of the new driver is to wrap all the new style Exceptions that can be thrown by the new architecture layer and turn them into old-style Exceptions, for backwards compatibility purposes. Sometimes testing the exceptional cases is... challenging. So I opted to do this with Spock. class DBCollectionSpecification extends Specification { private final Mongo mongo = Mock() private final ServerSelectingSession session = Mock() private final DB database = new DB(mongo, 'myDatabase', new DocumentCodec()) @Subject private final DBCollection collection = new DBCollection('collectionName', database, new DocumentCodec()) def setup() { mongo.getSession() >> { session } } def 'should throw com.mongodb.MongoException if rename fails'() { setup: session.execute(_) >> { throw new org.mongodb.MongoException('The error from the new Java layer') } when: collection.rename('newCollectionName'); then: thrown(com.mongodb.MongoException) } } So here we can use a real DB class, but with a mock Mongo that will return us a "mock" Session. It's not actually a mock though, it's more of a stub because we want to tell it how to behave when it's called - in this test, we want to force it to throw an org.mongodb.MongoException whenever execute is called. It doesn't matter to us what get passed in to the execute method (that's what the underscore means on line 16), what matters is that when it gets called it throws the correct type of Exception. Like before, the when: section shows the bit we're actually trying to test. In this case, we want to callrename. Then finally the then: section asserts that we received the correct sort of Exception. It's not enormously clear, although I've kept the full namespace in to try and clarify, but the aim is that anyorg.mongodb.MongoException that gets thrown by the new architecture gets turned into the appropriate com.mongodb.MongoException. We're sort of "lucky" because the old code is in the wrong package structure, and in the new architecture we've got a chance to fix this and put stuff into the right place. Once I'd tracked down all the places Exceptions can escape and started writing these sorts of tests to exercise those code paths, not only did I feel more secure that we wouldn't break backwards compatibility by leaking the wrong Exceptions, but we also found our test coverage went up - and more importantly, in the unhappy paths, which are often harder to test. I mentioned in the last post that we already did some simple stubbing to help us test the data driver. Why not just keep using that approach? Well, these stubs end up looking like this: private static class TestAsyncConnectionFactory implements AsyncConnectionFactory { @Override public AsyncConnection create(final ServerAddress serverAddress) { return new AsyncConnection() { @Override public void sendMessage(final List byteBuffers, final SingleResultCallback callback) { throw new UnsupportedOperationException(); } @Override public void receiveMessage(final ResponseSettings responseSettings, final SingleResultCallback callback) { throw new UnsupportedOperationException(); } @Override public void close() { } @Override public boolean isClosed() { throw new UnsupportedOperationException(); } @Override public ServerAddress getServerAddress() { throw new UnsupportedOperationException(); } }; } } Ick. And you end up extending them so you can just override the method you're interested in (particularly in the case of forcing a method to throw an exception). Most irritatingly to me, these stubs live away from the actual tests, so you can't easily see what the expected behaviour is. In the Spock test, the expected stubbed behaviour is defined on line 16, the call that will provoke it is on line 19 and the code that checks the expectation is on line 22. It's all within even the smallest monitor's window. So stubbing in Spock is painless. Next!

Last Saturday, I was a coach at the Legacy Code Retreat Milan 2013 organized by @gabrielelana, along with @filippo and @andreafrancia. Here's a recap of the goals of the event and of the experience. Why? A Code Retreat is a day dedicated to deliberate practice, where programmers reunite and work in pair programming on a single problems for each iteration. Each iteration is composed of a coding (or designing) section of 45' and of a retrospective held by the whole group. The focus of Code Retreats is to improve a particular skill in a setting with no pressure and deadlines; all code is deleted at the end of each iteration to pass the message that it is not the end result of running that is important (getting to the same place where you started from) but the path taken and the training that results. A Legacy Code Retreat is a particular kind of Code Retreat where legacy code is provided in the form of a source code repository containing a version of the Trivial Pursuit game (available for multiple programming languages). In our implementation the goal was different for each iteration: 1st iteration: produce a massive end-to-end automated test for the code, called Golden Master. This is strongly suggested by the existing material on the Legacy Code Retreat: you can only change code that is covered by automated tests. 2nd iteration: make it easy to add a new category of questions. 3rd iteration: add unit tests for the roll() function checking its output and final state. 4th iteration: find all the code smells, and fix 3 of them. 5th iteration: remove all duplication. 6th iteration: make the introduction of different penalty rules a one-line change (an Open/Closed Principle kata). Here is the presentation by Gabriele containing the introduction to the day and the goals. The golden master The first iteration is special, and serve as a setup for the rest of the day; the result of this iteration is not (at least conceptually) thrown away but instead kept and run as an end2end test to find out any regression. The chicken-and-egg problem of legacy code is always the same: it's difficult to test for unclear dependencies and global state, but you can't change it to simplify the job until you have some tests that ensure you're not breaking anything. Both testing and refactoring require each other. In fact, the goal of the first iteration is to produce such an end2end test without modifying the code in the repository: you could say "just add new source files" or "only use safe refactorings" for the languages that allow them. The code itself has no particular global state (such as database tables), unless for the fact that it calls random() functions to generate the result of dice throwing. This means executing the code multiple times always gives different results: "Chet is the current player","They have rolled a 6","Chet's new location is 6",... "Chet is the current player","They have rolled a 4","Chet's new location is 4",... The challenge of producing the golden master is to provide an initialization to the global state of the random number generator so that executions of the game can be repeated. Once you have 100 iterations with different initializations, you can store them and repeat the same process to find out any different in the output of the game. Any difference would signal that refactoring the legacy code has broken it and `git reset` or your favorite undo must be used. As to keep iterations always independent, at the end of the first one we provided a branch golden-master on the repository to all pairs. In this way even who didn't reach the goal (which is normal given the short time frame) or have a rough solution could use a tuned test case for the rest of the day; the presence of more golden masters allows also pairs to change at each iteration, and the components to switch programming languages. How it went? The event was sold out and the 30 people showed up (fortunately an even number so as to make pair programming easier). As coaches, we moved between pairs trying not to interrupt them if they were focused but offering help to the stuck pairs and to the ones wandering off the objective. The final retrospective brought out several goods: good format: each iteration is almost independent. Clearly defined goals. Variety of languages and people. Location and food (Talent Garden in Milan and breakfast offered by XPeppers). And several bads too, to resolve for the next editions: no theoretical introduction on how to work with the legacy code. Difficulties in using Extract Class, with respect to Extract Method and Extract Field which are local changes. Difficulties in introducing unit-level tests.

Everybody who worked with a project which included a StringUtil(s) class with only static methods, raise her hand! Thought so. Are those methods bad? Probably not so much, although I had a word to say about the name, after all if a class is not a utility it isn’t useful (by the definition of Wiktionary) and we hopefully haven’t much of that kind in our projects. But static methods turn bad, when they become more complex than the typical content of a StringUtil class. The problem is your code becomes hard wired to that static method. There is no easy way to replace the reference to the static method with something else, and if you are testing your code using automated tests, this is exactly what you want to do. If you don’t test your code using automated tests, do something about it NOW! Converting a static method to something easily mocked is straight forward once you’ve done it once or twice. Lets start with an example: public class Utility{ public static int doSomething(){ //… } } public class Client{ public void foo(){ //… Utility.doSomething(); //… } } The Client uses a static method in Utility and we want to get rid of that. The first step is to make the doSomethingmethod non-static. It is really as easy as removing the static modifier. Of course now the Client needs and instance ofUtility, so we just create one for now: public class Utility{ public int doSomething(){ //… } } public class Client{ public void foo(){ //… new Utility().doSomething(); //… } } Of course this doesn’t improve the situation much. We still have a static reference to the Utility class, since the constructor is just another static method. But now we can simply inject the dependency from the outside: public class Utility{ public int doSomething(){ //… } } public class Client{ private final Utility utility; public Client(Utility aUtility){ utility = aUtility; } public void foo(){ //… utility.doSomething(); //… } } Now you can replace Utility by a mocked instance for tests, you can use a wrapped instance for logging or make it implement an interface and so one. Basically you are back in OO world. Of course you can use your favorite DI-Framework to inject the dependency (just make sure you do it properly), or if you don’t mind the compile time dependency you can create an alternative constructor in the Client which uses the default implementation.

"A whole leisure day before you, a good novel in hand, and the backlog only just beginning to kindle..." - Backlog Studies, by Charles Dudley Warner A Recap on Backlogs A few weeks ago we took a critical look at Product Backlogs. We considered their purpose, how they are meant to be used, and why the aspirations they represent can so easily fall into a state of "Lost Remembrance". We also saw that a Product Backlog is an ordered list of requirements that are in scope, and if a project is to deliver value, then certain portions of that scope must be delivered in a timely manner. The Product Backlog is an instrument for managing this process. In short it is a queue, and one that is constantly tended and revised by a Product Owner. It is an artifact of diligent curation in which some requirements are determined to be more important than others, and which therefore ought to be delivered first. On the other hand some requirements will be observed to depend upon others, and must therefore be delivered afterwards. Introducing the Sprint Backlog In a very simple agile process - such as an elementary Kanban implementation - there will only be one backlog. Team members will action each item from the backlog in turn. They will be careful to draw only from the top of the queue, in order of priority. More sophisticated methods can include refinements such as “fast track” lanes in which the Quality of Service will be varied. We've already seen how this approach works in the context of managing critical incidents, and also in the context of hybrid agile methods such as Scrumban. Yet when we consider Scrum itself, we see that the Product Backlog is complemented by another of these queues...the Sprint Backlog. The idea is that if the team deliver something of value at regular intervals then the risks of the project can be better managed, and metrics can be generated that show progress towards its completion. Those regular intervals are known as Sprints. The chunk of requirements that the team agrees to work on during Sprint Planning is the Sprint Backlog. All of this is well known to agile developers, and the chances are that most of you reading this will have been working along these lines for years. So now let's challenge some common assumptions that are made about Sprint Backlogs and how a Scrum team is meant to handle them. Have any of these assumptions been made by your team? Assumption: The Sprint Backlog is a subset of the Product Backlog During Sprint Planning, a team will agree with the Product Owner which requirements from the Product Backlog will be worked on and met by the end of the forthcoming iteration. This has lead to the widespread practice of placing corresponding index cards into the Sprint Backlog on the Scrum board. In effect, it's a subset of the Product Backlog. What many teams fail to realize is that although the identification of an appropriate subset of Product Backlog requirements may be fine as a statement of intent, it can hardly be said to represent an actual plan for delivery. Admittedly a suitable plan doesn't have to be documented; it can live entirely in the developers' heads. A Scrum board's Sprint Backlog may indeed only show that subset of Product Backlog requirements which have been chosen for the Sprint. In fact the whole thing may look very like a Kanban board, even to the point that a casual observer might not be able to tell whether Scrum or Kanban rules are in force just by looking at it. The important thing is that a Sprint plan is agreed upon, shared, and understood by the team. Alternatively a task board may be used. Each selected requirement will be planned into tasks, and these will in turn be transcribed onto index cards or sticky notes. The tasks will move across the board in horizontal swim lanes that align each one to its parent requirement. In this model the Sprint Backlog is not represented by a subset of the Product Backlog, but rather by the corresponding tasks that have been planned for delivery. Assumption: A Sprint Backlog consists of tasks If we can see that each User Story has been broken down into tasks, it implies that some attempt has been made at Sprint Planning. It doesn't prove it of course. For all we know, each one of those tasks could have been identified by one person in the back of the pub last night. In other words, the tasks themselves do not amount to a plan. They merely infer by their presence that a team planning session is likely to have occurred, and that a team understanding regarding the delivery of the Sprint Goal has been reached. This means that a Sprint Backlog doesn't have to consist of tasks. It could be that “clean subset” of the Product Backlog we mentioned earlier, and therefore it might consist of User Stories. What matters is whether or not the team have a plan. While tasks imply that such a plan may have been formulated, they are not conclusive evidence of this, and they are certainly not the only way to compose a Sprint Backlog. Assumption: The Sprint Backlog is the Sprint Goal Identifying a meaningful Sprint Goal is usually the hardest part of Sprint Planning. Deciding how many User Stories can be accommodated, and what they should be, is comparatively straightforward. After all the team should know their budget. Time and again, Sprint Planning will boil down to horse-trading with the Product Owner over how many story points can be absorbed. “We've got 13 points left”, is a common refrain in Planning Poker. “We can't do that 20 pointer”. “OK”, says the Product Owner. “I'll bring forward a 5 and an 8 from the next Sprint”. While this satisfies the brutal arithmetic of planning, it does little to help create an increment of value. When the Sprint Backlog consists of disjointed requirements that don't play together as part of a cohesive potential release, the business value you might expect from such a release can hardly be delivered. Product Owners who expect otherwise are doing themselves and the product a disservice, and team members should not be party to such shenanigans. So, can each one of your team members articulate the goal for their current Sprint? Or is the “goal” just to deliver everything that's on the Sprint Backlog? A Sprint Goal is more than the sum of stories to be delivered or the tasks to be performed. It's about releasing business value incrementally and continually. Without that, the Product Owner probably has no idea when the project will reach completion. The common question “When will the project be done” is most often heard when incremental delivery is weak and the corresponding Sprint Goals are shoddy. Assumption: The Product Owner puts the Sprint Backlog in order This assumption is commonly held, but in Scrum terms it's plain wrong. The Development Team wholly own their Sprint Backlog, and it's up to them how they choose to order it. All the Product Owner should care about is whether or not the Sprint Goal is likely to be met by the end of the Sprint. This assumption is commonly held because Scrum is sometimes conflated with Kanban practice. In Kanban, there will normally be just one backlog and a Product Owner might well put it in order, and thereby exercise fine control over what gets delivered and when. Scrum is a different agile method and a different deal. In Scrum, value will be released at the end of the Sprint, not at discrete or arbitrary points within it. Granted, the Development Team should engage with the Product Owner throughout the Sprint, including on such matters as review and signoff, but the schedule for this is up to them. They decide, by creating their Sprint Plan, how the Sprint Backlog will be structured and how the corresponding work will be actioned. Assumption: Developers shouldn't cherry-pick from the Sprint Backlog This is a very good rule, but it is also one that is subject to misunderstanding. The underlying principle is a sound one. Agile teams should be fully cross-trained, and they should action work from a backlog as a team. Kanban team members, for example, should always take the next highest priority item from the backlog, assuming that there is no other work in progress or which is impeded and needs their attention. No team member should ever try and “pre-book” an item on the backlog, regardless of whether they simply want it or because they think they are best qualified to handle it. Each team member should go to where the work is, whatever that work is, and exactly when it needs doing. Scrum fully supports this principle but there is a further consideration that has to be borne in mind...a Scrum Development Team will have a Sprint Plan. When formulating this plan, they will self-organize to meet the Sprint Goal. That means it's quite possible for the team to decide up front, during Sprint Planning and subsequently during each daily Stand-Up, who will do what. It's important to be able to distinguish this behavior from cherry picking. It's also important for a Scrum Master to be able to smell a rat, and to sense when teams genuinely have a good plan or have started to cherry pick or to form undesirable skill silos. Assumption: A team commits to deliver everything in the Sprint Backlog This is a tricky assumption to deal with because until recently it was seen as being quite valid. For a long time, commitment-based planning was pivotal to a Scrum based way of working. Then, in 2011, the Scrum Guide was revised and the Sprint Backlog was positioned as a forecast of what a team could reasonably be expected to deliver. Clearly, a “forecast” is a weaker use of language than “commitment”. The rationale underlying this change is sensible. There are many things that can change during a Sprint, including requirements understanding or the perception of business value. Moreover, estimates are precisely that – estimates. There's something else we have to remember. The Development Team wholly own their Sprint Backlog. Regardless of whether they forecast their deliverables or commit to them, the content of this backlog is up to them and they can revise it at any time. It's the Sprint Goal, and the concomitant potential release of functionality, that is either committed to or forecast for delivery. Assumption: The Sprint Backlog cannot be changed once the Sprint has started This assumption is incorrect, although it is true that the Product Owner can't change the Sprint Backlog unilaterally. Only the Development Team can make such a change, because they wholly own it. If a Product Owner wishes to change something on the Sprint Backlog then that must be negotiated with the team. Now, let's also bear in mind that Scrum does not prescribe how the requirements within a Sprint Backlog are enumerated. User Stories, or the tasks to realize such stories, are the most common form of expression. Since User Stories do not document requirements exhaustively, but are placeholders for a future conversation, it follows that a change in understanding does not necessarily mean a change in the Sprint Backlog itself. Conclusion Sprint Backlogs mean different things to different teams. Some may populate them with tasks, while others may simply transfer over agreed User Stories from the Product Backlog. Either approach is acceptable given that the Development Team wholly own the Sprint Backlog. The important thing is that the team should have a plan for meeting a well defined Sprint Goal that has been agreed with the Product Owner, and they should form their Sprint Backlog in accordance with that plan. The backlog itself should never be mistaken for, or used in lieu of, a coherent goal for delivering a potentially releasable increment of value.

in-line predicates can create a maintenance nightmare. writing in-line lambda expressions and using the stream interfaces to perform common operations on collections can be awesome. assume the following example: list getadultmales (list persons) { return persons.stream().filter(p -> p.getage() > adult && p.getsex() == sexenum.male ).collect(collectors.tolist()); } that’s fun! but things like this also lead to software that is costly to maintain. at least in an enterprise application, where most of your code handles business logic, your development team will grow the tenancy to write the same similar set of predicate rules again and again. that is not what you want on your project. it breaks three important principles for growing maintainable and stable enterprise applications: dry (don’t repeat yourself): writing code more than once is not a good fit for a lazy developer it also makes your software more difficult to maintain because it becomes harder to make your business logic consistent readability : following clean-code best practices, 80% of writing code is reading the code that already exists. having complicated lambda expressions is still a bit hard to read compared to a simple one-line statement. testability : your business logic needs to be well-tested. it is adviced to unit-test your complex predicates. and that is just much easier to do when you separate your business predicate from your operational code. and from a personal point of view… that method still contains too much boilerplate code… imports to the rescue! fortunately, we have a very good suggestion in the world of unit testing on how we could improve on this. imagine the following example: import static somepackage.personpredicate; ... list getadultmales (list persons) { return persons.stream().filter( isadultmale() ).collect(collectors.tolist()); } what we did here was: create a personpredicate class define a “factory” method that creates the lambda predicate for us statically import the factory method into our old class this is how such a predicate class could look like, located next to your person domain entity: public personpredicate { public static predicate isadultmale() { return p -> p.getage() > adult && p.getsex() == sexenum.male; } } wait… why don’t we just create a “ismaleadult” boolean function on the person class itself like we would do in domain driven development? i agreed, that is also an option… but as time goes on and your software project becomes bigger and loaded with functionality and data… you will again break your clean code principles: the class becomes bloated with all kind of function and conditions your class and tests become huge, more difficult to handle and change (*) (*) and yes… even if you do your best to separate your concerns and use composition patterns adding some defaults… working with domain objects, we can imagine that some operations (such as filter) are often executed on domain entities. taking that into account, it would make sense to let our entities implement some interface that offers us some default methods. for example: public interface domainoperations { default list filter(predicate predicate) { return persons.stream().filter( predicate ) .collect(collectors.tolist()); } } when our person entity implements this interface, we can clean-up our code even more: list getadultmales (list persons) { return persons.filter( isadultmale() ); } and there we go… conclusion moving your predicates to a predicate helper class offers some good advantages in the long run: predicate classes are easy to test and change your domain objects remain clean and focussed on representing your domain, not your business logic you optimize the re-usability of your code and, in the end, reduce your maintenance you seperate your business from operational concerns references clean code: a handbook of agile software craftsmanship [robert c. martin] practical unit testing with junit and mockito [tomek kaczanowski] state of the collections [http://cr.openjdk.java.net/~briangoetz/lambda/collections-overview.html] notes the code above is served as an example to illustrate the principles i wanted to discuss. however, i did not proof-run this code yet (it’s still on my todo list). some modifications may be needed for your project.

This is a package of C#, VB.NET Example Project for Spire.BarCode for .NET. Spire.BarCode for .NET is a professional and reliable barcode generation and recognition component. It enables developers to quickly and easily add barcode generation and recognition functionality to their Microsoft .NET applications (ASP.NET, WinForms and .NET) and it supports in C#, VB.NET. Spire.Barcode for.NET is 100% FREE barcode component. First Glance of Spire.BarCode for .NET http://www.e-iceblue.com/images/url/BarCode.png Spire.BarCode for .NET Main Features: Supports rich Barcode types, more than 37 different barcodes. • Code bar Barcode • Code 1 of 1 Barcode • Standard 2 of 5 barcode • Code 3 of 9 barcode • Extended Code 3 of 9 barcode • Code 9 of 3 Barcode • Extended Code 9 of 3 Barcode • Code 128 barcode • EAN-8 barcode • EAN-13 barcode • EAN-128 barcode • EAN-14 barcode • SCC14 barcode • SSCC18 barcode • ITF14 Barcode • ITF-6 Barcode • UPCA barcode • UPCE barcode • Postnet barcode • Planet barcode • MSI barcode • 2D Barcode DataMatrix • QR Code barcode • Pdf417 barcode • Pdf417 Macro barcode • RSS14 barcode • RSS-14 Truncated barcode • RSS Limited Barcode • RSS Expanded Barcode • USPS OneCode barcode • Swiss Post Parcel Barcode • PZN Barcode • OPC(Optical Product Code) Barcode • Deutschen Post Barcode • Deutsche Post Leitcode Barcode • Royal Mail 4-state Customer Code Barcode • Singapore Post Barcode 1.Robust Barcode Recognize and Generation 1D & 2D Barcode. Developers can read most often used Linear, 2D and Postal barcodes, detecting them anywhere, with any orientation. 2.High performance for generating and reading barcode image Developers can create barcode images in any desired output image format like Bitmap, JPG, PNG, EMF, TIFF, GIF and WMF. 3.Superior performance support for reading and writing barcode Developers can easily set barcode image borders, border colors, style, margins and width. You can also rotate barcode images to any angle and produce high quality barcode images. 4.Easy Integration Spire.Barcode for .NET can be easily integrated into any .net applications. There are two main ways to integrate Spire.Barcode in .NET applications, API Mode and Component Mode. • API Mode is just one line of code to create, recognizes barcode. • Component Mode use Visual way to create barcode, then drag Spire.Barcode component to your .NET, Windows or ASP.NET Form. No more code needs. Download Spire.BarCode: Spire.BarCode for .NET is a free barcode library used in .NET applications (in ASP.NET, WinForms and .NET). And you can download Spire.BarCode for .NET and install it on your system. With Spire.BarCode, you can add Enterprise-level barcode formats to your NET applications easily and quickly. Feedback and Support E-iceblue welcomes any kind of questions, bug reports and feedback about this product from our customers. As long as you leave them on Spire.BarCode for .NET Forum or contact us by e-mail, we will offer full support within one business day. Related Links Website:www.e-iceblue.com Product Introduction: http://www.e-iceblue.com/Introduce/barcode-for-net-introduce.html#.UdEuk9hp4Zu Download: http://www.e-iceblue.com/Download/download-barcode-for-net-now.html . Spire.BarCode for .NET is a FREE and professional barcode component specially designed for .NET developers (C#, VB.NET, ASP.NET) to generate, read 1D & 2D barcodes. Developers and programmers can use Spire.BarCode to add Enterprise-Level barcode formats to their .net applications (ASP.NET, WinForms and Web Service) quickly and easily. Spire.BarCode for .NET provides a very easy way to integrate barcode processing. With just one line of code to create, read 1D & 2D barcode, Spire.BarCode supports variable common image formats, such as Bitmap, JPG, PNG, EMF, TIFF, GIF and WMF. Spire.BarCode for .NET is 100% FREE BarCode component, no risk to integrate in your .NET application.

In this blog post, I will show you how to "ETL" all kinds of data to Amazon’s cloud data warehouse Redshift wit Talend’s big data components. Let’s begin with a short introduction to Amazon Redshift (copied from website): "Amazon Redshift is [part of Amazon Web Services (AWS) and] a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud. With a few clicks in the AWS Management Console, customers can launch a Redshift cluster, starting with a few hundred gigabytes and scaling to a petabyte or more, for under $1,000 per terabyte per year. Traditional data warehouses require significant time and resource to administer, especially for large datasets. In addition, the financial cost associated with building, maintaining, and growing self-managed, on-premise data warehouses is very high. Amazon Redshift not only significantly lowers the cost of a data warehouse, but also makes it easy to analyze large amounts of data very quickly.“ Sounds interesting! And indeed, we already see companies using Talend’s Redshift connectors. From Talend perspective it is not much more than just another database. If you have ever used a Talend connector, you can integrate to Redshift within some minutes. In the next sections, I will describe all necessary steps and give some hints regarding configuration issues and performance improvements. Be aware: You need Talend Open Studio for Data Integration (Apache License, open source) or any Talend Enterprise Edition / Platform which contains the Cloud components to see and use Amazon Redshift connectors. The open source edition offers all connectors and functionality to integrate with Amazon Redshift. However, Enterprise versions offer some more features (e.g. versioning), comfort (e.g. wizards) and commercial support. Setup Amazon Redshift Setup of Amazon Redshift is very easy. Just follow Amazon‘s getting started guide: http://docs.aws.amazon.com/redshift/latest/gsg/welcome.html. Like every other AWS guide, it is very easy to understand and use. Be aware, that you just have to do step 1, 2 and 3 of the getting started guide for using it with Talend. Some hints: - Step 1 („before you begin“): Just sign up. Client tools and drivers are not necessary because they are already installed within Talend Studio. - Step 2 („launch a cluster“): Yes, please start your cluster! - Step 3(„authorize access“): If you are not sure what to do here, select Connection Type = CIDR/IP. Find out your IP address (http://whatismyipaddress.com) and enter it with „/32“ at the end. Example: „192.168.1.1/32“ Now you can connect to Amazon Redshift from your Talend Studio on your local computer. Step 4 (connect) and step 5 (create table, data, queries) are not necessary, this will be done from Talend Studio. Of course, you should not forget to delete your cluster (step 7) when you are done. Otherwise, you will pay for every hour, even if you do not access your DWH. Connect to Amazon Redshift from Talend Studio Create a new connection to Amazon Redshift database as you do with every other relational database. The easiest way is to use „DB Connection Wizard“ in metadata. Just enter your connection information and check if it works. You get all information about configuration from Amazon Web Console. The connection string looks something like this: „jdbc:paraccel://talend-demo-cluster.cp8t6c5.eu-west-1.redshift.amazonaws.com:5439/dev“ Next, right click on the created connection and select „retrieve schema“. „public“ is the default schema which you (have to) use. Now, you are ready to use this connection within Talend Jobs to write to Amazon Redshift and read from it. Create Talend Jobs (Write, Read, Delete) Amazon Redshift components work like any other Talend (relational) database components. Look at www.help.talend.com for more information if you have not used them before (or just try them out, they are very self-explanatory). You just have to drag&drop your connection from metadata . Afterwards, you can easily write data (tRedShiftOutput), read data (tRedshiftInput), or do any other queries such as delete or copy (tRedShiftRow). In the following job, I start with deleting all content in the Amazon Redshift table. Then, I read data from a MySQL table and insert it into an Amazon Redshift table. The table is created automatically (as I have configured it this way). After this subjob is finished, I read the data again, and store it to a CSV file (which is also created automatically). Of course, this is no business use case, but it shows how to use different Amazon Redshift components. Query Data from Amazon Redshift You can connect to Amazon Redshift directly from Talend Studio to explore and query data of the DWH. Thus, no other database tool is required. Just right click on your Amazon Redshift connection in metadata and select „edit queries“. Here you can define, execute and save SQL queries. Improve Performance Write performance of Amazon Redshift is relatively low compared to „classical“ relational databases (in your data center) as you have to upload all data into the cloud. Different alternatives exist to improve performance: - Bulk inserts: „Extended insert“ (in advanced settings) improves performance a lot, but still not to hyperspeed… Also, as it is bulk, you can just do inserts! It is not compatible to „rejects“ or „updates“ - AWS S3 and COPY command: S3 is Amazon’s „simple storage service“, a key-value store – also called NoSQL today – for storing very large objects. You can use Amazon Redshift’s COPY command (http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) to transfer data from S3 to Amazon Redshift with good performance. Though, you still have to copy data to S3 before, same „cloud problem“ here. The COPY command can be used with tRedshiftRow, so no problem at all from Talend perspective. To transfer data to S3, you can either use the Talend S3 components from Talendforge, Talend’s open source community (http://www.talendforge.org/exchange), or use camel-s3, an Apache Camel component which is included in Talend ESB. The latter is an option, if you use Talend Data Services which combines Talend DI and Talend ESB in its unified platform. Summary You need not be a cloud or DWH expert, or an expert developer to integrate with Amazon’s cloud data warehouse Redshift. It is very easy with Talend’s integration solutions. Just drag&drop, configure, do some graphical mappings / transformations (if necessary), that’s it. Code is generated. Job runs. You can integrate Amazon Redshift almost as simple as any other relational database. Just be aware of some cloud specific security and performance issues. With Talend, you can easily „ETL“ all data from different sources to Redshift and store it there for under $1,000 per terabyte per year – even with the open source version! Best regards, Kai Wähner (Contact and feedback via @KaiWaehner, www.kai-waehner.de, LinkedIn / Xing) This is content from my blog: http://www.kai-waehner.de/blog/2013/06/26/integration-of-amazon-redshift-cloud-data-warehouse-aws-saas-dwh-with-talend-data-integration-di-big-data-bd-enterprise-service-bus-esb/

Hi, I'm excited to announce the release of our QuartzDesk product. QuartzDesk is an advanced Java Quartz scheduler management and monitoring GUI / tool with many powerful and unique features. To name just a few: Support for Quartz 1.x and 2.x schedulers. Persistent job execution history. Job execution log message capturing. Notifications (email, all popular IM protocols, web-service). Interactive execution statistics and charts. REST API for job / trigger / scheduler monitoring. QuartzAnywhere web-service to manage / monitor Quartz schedulers from applications. and more To keep this announcement short, I kindly refer you to the QuartzDesk Features page for details and screenshots. The product is aimed at Java developers and system administrators. Jan Moravec (Founder) & The QuartzDesk Team

a geofence is a virtual perimeter of interest that can be set up to fire notifications when it is entered or exited, or both. for example, a geofencing app can alert us that our kid has left a previously specified area, or send us a coupon (e.g. the "present this sms an get 20% off" offer type) when we happen to walk or drive in the proximity of a movie theater. now, with the new location apis , google's location algorithm has been rewritten to be more accurate and use significantly less battery life. there is just enough documentation plus sample code and a downloadable sample app ( geofencedetection ) to help us get started creating geofencing apps. prerequisites are: download google play services (via android's sdk manager) and set it up as a library get a google maps v2 api key, and maybe run the sample app. this short quick start guide might help download (again via sdk manager) the support library to cater to older android versions. for practical purposes, let's just start where the sample geofencing app ( geofencedetection ) stops, and introduce a few enhancements to make the app semi-decent and show a sample of possibilities with the new location api. 1. zoom and camera position first, a little taste of google maps api v2. let's choose zoom level and camera angle: import com.google.android.gms.maps.googlemap; import com.google.android.gms.maps.cameraupdatefactory; import com.google.android.gms.maps.model.cameraposition; import com.google.android.gms.maps.model.latlng; //... // inside class, for a given lat/lon cameraposition init = new cameraposition.builder() .target(new latlng(lat, lon)) .zoom( 17.5f ) .bearing( 300f) // orientation .tilt( 50f) // viewing angle .build(); // use googglemap mmap to move camera into position mmap.animatecamera( cameraupdatefactory.newcameraposition(init) ); the code above has a zoom level allowing the viewing of buildings in 3d. google maps v2 uses opengl for embedded systems ( opengl es v2) to render 2d and 3d computer graphics. 2. options menu even if we are not big fans of an options menu, it might be adequate in this case, since we would not want to clutter the map with too much "touch" functionality (we will have plenty of that shortly). we can toggle between "normal" and satellite view: /** * toggle view satellite-normal */ public static void toggleview(){ mmap.setmaptype( mmap.getmaptype() == googlemap.map_type_normal ? googlemap.map_type_satellite : googlemap.map_type_normal); } we can also provide a "flight mode", where we let the camera scroll away. not tremendously useful, but kind of cool nonetheless: import com.google.android.gms.maps.googlemap.cancelablecallback; //... private static cancelablecallback callback = new cancelablecallback() { @override public void onfinish() { scroll(); } @override public void oncancel() {} }; public static void scroll() { // we don't want to scroll too fast since // loading new areas in map takes time mmap.animatecamera( cameraupdatefactory.scrollby(10, -10), callback ); // 10 pix } 3. geocoding/reverse geocoding the sample is here to demonstrate features and makes heavy use of latitude/longitude coordinates. but we need to provide a more user-friendly way to interface with locations on the map, like a street address. we can use geocoding/reverse geocoding to transform a street address to coordinates and vice-versa using android's geocoder . 4. adding geofences ok, now on to geofences. thanks to geocoding, we can request an actual physical address from the user instead of coordinates. we will just change that address to a latitude/longitude pair internally to process user input. notice how we use transparent uis as much as possible to enhance what some might call the user experience. notice also that we provide a spinner so that the user can choose between predefined values. that saves the user some typing and it saves us from validating coordinates values each time. still, if we want to be even more user-friendly, we can give our users the possibility to pre-fill the address field by long-pressing a point on the map. we will then use reverse geocoding to translate the coordinates to a physical address for display (screen below on the right): processing long-presses is pretty straightforward: import com.google.android.gms.maps.googlemap.onmaplongclicklistener; //... public class mainactivity extends fragmentactivity implements onmaplongclicklistener { //... mmap.setonmaplongclicklistener(this); //... @override public void onmaplongclick(latlng point) { // reverse geocode point } } adding /removing geofences is pretty much covered in the sample app (by the geofencerequester and geofenceremover classes). the thing to remember is that the process of adding/removing fences is as follows : a connection to google's location services is requested by our app. once/if the connection is available, the request to add/remove a fence is done using a pendingintent . if a similar request made by our app is still underway, the operation fails. although the method we call (e.g. addgeofences() ) returns at once, we won't know if the request was successful until location services calls back into our app (e.g. onaddgeofencesresultlistener 's onaddgeofencesresult() ) with a success status code. finally, the preceding method will use a broadcast intent to notify other components of our app of success/failure. needless to say, we need to code defensively at almost every step of the way. now, once a geofence is added, we can add a marker (the default or a customized one) and choose between different shapes (circle, polygon, etc.) to delimit the geofence. for instance we can write this code to add the default marker and circle the fence within a specified radius: import com.google.android.gms.maps.model.circle; import com.google.android.gms.maps.model.circleoptions; import com.google.android.gms.maps.model.markeroptions; //... public static void addmarkerforfence(simplegeofence fence){ if(fence == null){ // display en error message and return return; } mmap.addmarker( new markeroptions() .position( new latlng(fence.getlatitude(), fence.getlongitude()) ) .title("fence " + fence.getid()) .snippet("radius: " + fence.getradius()) ).showinfowindow(); //instantiates a new circleoptions object + center/radius circleoptions circleoptions = new circleoptions() .center( new latlng(fence.getlatitude(), fence.getlongitude()) ) .radius( fence.getradius() ) .fillcolor(0x40ff0000) .strokecolor(color.transparent) .strokewidth(2); // get back the mutable circle circle circle = mmap.addcircle(circleoptions); // more operations on the circle... } here are the resulting screens, including the one we get once we "touch to edit" the info window: the sample app has all we need to fire notifications once the circled area above is entered or exited. notice how we set up the marker's info window to allow editing the geofence radius, or removing the geofence altogether. to implement a clickable custom info window, all we need is to create our own infowindowadapter and oninfowindowclicklistener . as for the notifications themselves, this is how they look like in the sample app: we can of course change a notification's appearance and functionality, and... that would be the subject of another article. hopefully, this one gave a glimpse of what is possible with the new location api. have fun with android geofences.