One of the best thing for any developer is to write a code which can be used as many times in an enterprise application. Pipeline templates is one of such capability in Oracle Service Bus in which developer create a prototype of a pipeline called as pipeline template and include all the component which can be use across the enterprise services. One such example can be a Error handler. Here in this post I will try to explain and give an idea of how one can use the pipeline templates in its Oracle Service Bus application. I am going to create template in which I will include these components: Validation Error Handler Logging Alerts Reports As you can see these are the common components which can be used across all the Services in OSB. Let me start with the basic. Pipeline: Pipeline component in OSB provides message processing capability i.e. defines logic for routing, validation, manipulation of messages (transformation) or error handling etc. Primary elements for Pipeline are: Start Node: It is a by default node define automatically when you connect to proxy service. Pipeline Pair: node for request & response pair processing. Branch Node: node to route the message based on condition either on type of operation or values. Route Node: this node set the destination of message otherwise by default echo the request i.e. request is passed to the response. Error Handler: handle errors either on stage or node. Note: Minimum start node and route node required for pipeline. Pipeline Template: A template in pipeline is a prototype of message flow for proxy services. It defines the reusable activities that are common across the business process for example validations on business messages, error handling, transformation of request and response messages etc. How to create pipeline templates in Jdeveloper 1. Right click on the projectà NewàPipeline_Template 2. Template wizard appears, Enter all the information. 3. Click Next to select service type. 4. Finish. 5. Now in the pipeline template add the route node, error handler or any logic which is common across the service. (follow the example) Note: you cannot create pipeline templates in OSB console. Pipeline Template Example Code Structure Two projects i.e. SharedSB and ConsumeTemplate. SharedSB is project in which we store all the common resources that can be used in all the service bus application and In ConsumeTemplate I will create a pipeline on the basis of pipeline template. Create Pipeline Template (CommonComponent.ptx) Create pipeline template CommonComponent.ptx Drag and drop all the components that you think can be re-used in many services. In this case I am using Validity, Error handler, reports, logging, Alerts. After configuring all the activities, you can lock the activities so that the same configuration can be used in all the concrete pipeline i.e. in the pipeline where you are importing the pipleline template. Creating concrete pipeline using template Create proxy service. Create pipleline using the template and use same signature as of your proxy service. After importing template, implement message flow logic if required. Save and deploy both the projects SharedSB and ConsumeTemplate on Oracle Service Bus server. Test your application. References: http://www.codeconfuse.com/2014/10/oracle-osb-12c-pipeline-templates-in.html http://docs.oracle.com/middleware/1213/osb/develop/title.htm#BEGIN For more Oracle Fusion Middleware reference visit- http://www.codeconfuse.com/

[This article was written by Barak Merimovich.] We have discussed OpenStack networking extensively in previous posts. In this post, I’d like to dive into a more advanced OpenStack networking scenario. Many cloud images are not configured to automatically bring up all network cards that are available. They will usually only have a single network card configured. To correctly set up a host in the cloud with multiple network cards, log on to the machine and bring up the additional interfaces. echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 Networks in the cloud A complex network architecture is a mainstay of modern IaaS clouds. Understanding how to configure your cloud-based networks, and hosts, is critical to getting your application working in the cloud. This is especially true with Cloudify, the open source cloud orchestration platform I work on. The cloud, like the world, used to be flat It was not that long a time ago that most IaaS providers only supported flat networks – all of your hosts were in one large network. Separation between services running in the cloud was enforced in software or with firewalls/security-groups. But technically, all of the hosts were connected to the same network and visible to each other. The flat network model is simple, and therefore easy to reason and understand. It was a good choice for the early days of the IaaS cloud and no doubt helped with getting applications into the cloud in the first place. It was one of the things that made EC2 so easy to use for anyone just starting out with the ‘cloud’. This model is in fact still available on Amazon Web Services under the title ‘EC2-Classic’. And for many applications, a flat network is good enough. But as cloud adoption increases, more complex applications are moving into the clouds, and issues like network separation, security, SLA and broadcast domains make more complex networks models a must. Software Defined Networks (SDN) fill that gap. They are now a staple of most major IaaS clouds. AWS has AWS-VPC, OpenStack has the Neutron project and there are many other implementations. Working with SDN requires knowing a bit more about how information moves around between your cloud resources. In this post I am going to discuss how to set up a host in the cloud so it will play nice with complex networks. I’ll be using OpenStack, but the concepts are similar for other cloud infrastructures. Openstack configuration I am going to start with an empty tenant, only the public network is available. First, lets set up out networks and router: neutron router-create demo-router neutron net-create demo-network-1 neutron net-create demo-network-2 neutron subnet-create --name demo-subnet-1 demo-network-1 10.0.0.0/24 neutron subnet-create --name demo-subnet-2 demo-network-2 10.0.1.0/24 neutron router-interface-add demo-router demo-subnet-1 neutron router-interface-add demo-router demo-subnet-2 neutron router-gateway-set demo-router public Note the network IDs: neutron net-list | id | name | subnets | | 2c33efe2-6204-4125-9716-3bc525630016 | demo-network-1 | 928dafa0-83ef-459c-b20d-71d8ea596fa2 10.0.0.0/24 | | aa30627e-c181-4a4b-89bf-5dd7c26c244e | demo-network-2 | 26d573f7-7953-4a54-825b-ed7bbc0661c7 10.0.1.0/24 | | e502de8d-929a-4ee0-bd18-efa297875cf6 | public | d40dab51-a729-452c-9ee6-b9ad08d10808 | We’ll start with a standard Ubuntu cloud image: glance image-create --name "Ubuntu 12.04 Standard" --location "http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img" --disk-format qcow2 --container-format bare Create the keypair and security group: nova keypair-add demo-keypair > demo-keypair.pem chmod 400 demo-keypair.pem nova secgroup-create demo-security-group "Security group for demo" nova secgroup-add-rule demo-security-group tcp 22 22 0.0.0.0/0 Let’s spin up an instance connected to both our networks: nova boot -flavor m1.small --image "Ubuntu 12.04 Standard" --nic net-id=2c33efe2-6204-4125-9716-3bc525630016 --nic net-id=aa30627e-c181-4a4b-89bf-5dd7c26c244e --security-groups demo-security-group --key-name demo-keypair demo-vm And set up floating IPs for the first network: nova list | ID | Name | Status | Task State | Power State | Networks | 2b17588b-8980-4489-9a04-6539a159dc3c | demo-vm | ACTIVE | None | Running | demo-network-1=10.0.0.2; demo-network-2=10.0.1.2 | neutron floatingip-create public neutron floatingip-list | id | fixed_ip_address | floating_ip_address | port_id | | 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 | | 192.168.15.42 | | neutron port-list | id | name | mac_address | fixed_ips | | 1ccfd334-7328-4b22-b93e-24a0888276ab | | fa:16:3e:14:39:39 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.3"} | | a482c4f6-fa74-476e-b1ce-cd8dd0c70815 | | fa:16:3e:18:92:79 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.2"} | | b23d7836-30c5-4bff-b873-15c87ba051f6 | | fa:16:3e:3a:28:40 | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.3"} | | d421b447-2adf-406f-876b-142238683344 | | fa:16:3e:9d:fc:7f | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.2"} | | dcf8696b-cc80-4b48-b09c-61c0f8ab02ac | | fa:16:3e:5b:39:fb | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.1"} | | f6a1666e-495a-4d3f-afa3-754b3cb3cfc0 | | fa:16:3e:8a:1b:fb | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.1"} | neutron floatingip-associate 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 d421b447-2adf-406f-876b-142238683344 Note how we matched the VM’s IP to its port, and associated the floating IP to the port. I wish there was an easier way to do this from the CLI… If everything worked correctly, you should have the following setup: Let’s make sure ssh works correctly: ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] ifconfig eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:230 errors:0 dropped:0 overruns:0 frame:0 TX packets:224 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:46297 (46.2 KB) TX bytes:31130 (31.1 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Huh?! The VM only has one working network interface! Where is my second NIC? Was there a configuration problem with the OpenStack network setup? The answer is here: ssh -i demo-keypair.pem [email protected] ifconfig -a eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:324 errors:0 dropped:0 overruns:0 frame:0 TX packets:332 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:69973 (69.9 KB) TX bytes:47218 (47.2 KB) eth1 Link encap:Ethernet HWaddr fa:16:3e:29:6d:22 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) The second NIC exists, but is not running. The issue is not with the OpenStack network configuration – it’s with the image. The image itself should be configured to work correctly with multiple NICs. All we have to do is bring up the NIC. So we ssh into the instance: ssh -i demo-keypair.pem [email protected] And run the following commands: echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 The second NIC should now be running: ifconfig eth1 eth1 Link encap:Ethernet HWaddr fa:16:3e:18:92:79 inet addr:10.0.1.2 Bcast:10.0.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe18:9279/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:45 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15376 (15.3 KB) TX bytes:3960 (3.9 KB) And there you go – your VM can access both networks. This issue can make life complicated when setting up a complex, or even a not very complex, application. When will this issue hurt you? Well, imagine a scenario where you have a web server and a database server. The web server is connected to both Network1 and Network2, and the database server is only connected to Network2. Network1 is connected to the external world over a router, and Network 2 is completely internal, adding another layer of security to the critical database server. So what happens if the web server only has one network card? If only the NIC for Network1 is up, the web server can’t access the database. If only the NIC for Network2 is up, the web server can’t be reached from the external world. Even worse, if this web server is accessed via a floating IP, this IP will also not work, so you won’t be able to access the web server and fix the issue. Tricky. In conclusion The above commands will bring up your additional network card. You will of-course need to repeat this process for each additional network card, and for each VM. You can use a start-up script (a.k.a. user-data script) or system service to run these commands, but there are better ways. I’ll discuss how to automate the network setup in a follow-up post. This was originally posted at Barak's blog Head in the Clouds, find it here.

For a long time JSF for me was just another web framework I didn’t care too much about. This changed. After being forced to use it for a couple of months now, I consider it a major project risk in almost all cases. Here I present the reasons for this verdict. Bad entanglement of UI and processing Logic The official tutorial claims the following about the benefits of JSF: One of the greatest advantages of Java Server Faces technology is that it offers a clean separation between behavior and presentation for web applications. The opposite is the case. Facelets, the preferred presentation technology of JSF looks at first sight like an ordinary templating technology like the good old JSP or Thyme Leaf. But if you look closer the horror becomes obvious. In the same place where you structure your HTML, you also place the logic what parts of the UI should get updated on an action. A clear violation of the separation of concerns principle in my book. Even better is the immediate attribute which changes the server side life cycle! And if this isn’t enough it does it in different waysdepending on what tag you use it on. You can’t make stuff like this up. It tries to abstract what you can not abstract. Except some weird edge cases clients and server of web application are located on rather different computers, separated by some kind of network. From this follows a simple fact: communication between client and server is slow and unreliable. JSF tries to abstract away the separation of client and server. It processes everything on the backend wildly communicating between client and server in a hard to control way. The result are all kind of failure scenarios just popping into existence because you use JSF. For me the most annoying one is this one: If you open a JSF page, let’s say a simple search page, wait an hour, then hit the submit button you will get an exception because the server side state expired. WAT? Why is there server state of any relevance for a trivial search page? (Yes I know you can change that behavior with the latest versions of JSF, but it is still the way JSF is designed to work) I though everybody learned since EJBs: If you want to abstract over the fact, if two parts of an application run on the same machine or not, you have to assume they don’t. Everything else is just hiding problems until they grow so large that they can eat your project for breakfast. Making stuff complex and complicated that was easy to start with.The architecture of the World Wide Web is a simple one. Simple meaning: It consists of a small set of concepts with limited interaction. This is what made it so widely successful. It also makes it not obvious for beginners how to use it to implement certain features. I’m sure most of us remember the first time they tried to implement something like a shopping cart without having session state. But the solutions for almost all these problems are well known and understood by know. And all you need is a little reading and what you gain is a strong conceptual understanding how to solve this kind of issue. And again, the basics are extremely simple: You send a request to an URL, with some headers and content using a HTTP verb. And you reply with some resource containing links and some headers. And you don’t have state in the server session. Making load balancing and fail over rather simple. Making bookmarkable URLs trivial. Making your site searchable for zero costs. Making your site cachable. Allowing the user to use their back buttons, history and tabs as they wish. Making it trivial to have nice URLs Compare that to the live cycle model of JSF: The page from which a user submitted a request will get synchronized with a model on the server side, then submitted values validated, converted, events generated and processed. As mentioned above the order in which things happen, and if they happen at all are controlled by XML Tags hidden away in a document camouflaged as markup. Apart from hardly anybody properly understanding all this (BalusC seems to be the only one available in the interwebs) it has the following effect on your application: The URLs become ugly. You’ll see the URL of the resource you came from instead of the one you are looking at, thus making bookmarking URLs as useful as a doorknob on your knee. Same for caching, fail over, load balancing and so on. Sure you can fix it with some convention here, and an additional library there. Which of course makes perfect sense when you are in the business of breaking stuff so people have to pay you for fixing it. I personally prefer helping to solve real problems. Hindering testability: I can’t speak for most frameworks but I can compare Spring MVC with JSF. Let me tell you this: If anybody is telling you JSF is nicely testable he probably doesn’t know automatic testing. With JSF you can test your backend beans using unit tests. You can test the whole UI, by deploying the application to a server and hitting it with Selenium. That’s basically it. Just in case you are wondering what else one should be able to test: Load a static version of a page in a browser and testing it with selenium, in order test your Client side UI behavior. Test your generated markup without starting a full blown application server. Test the mapping of attributes/parameter to bean methods. Test your generated markup without bootstrapping a complete application. All this is perfectly possible with Spring MVC and probably with many other sane server side frameworks, but not with JSF … Again: I’m aware there are fixes for many issues, but the simplest fix is> Don’t use JSF.

The last couple of weeks I've been playing around with docker and kubernetes. If you are not familiar with kubernetes let's just say for now that its an open source container cluster management implementation, which I find really really awesome. One of the first things I wanted to try out was running an Apache ZooKeeper ensemble inside kubernetes and I thought that it would be nice to share the experience. For my experiments I used Docker v. 1.3.0 and Openshift V3, which I built from source and includes Kubernetes. ZooKeeper on Docker Managing a ZooKeeper ensemble is definitely not a trivial task. You usually need to configure an odd number of servers and all of the servers need to be aware of each other. This is a PITA on its own, but it gets even more painful when you are working with something as static as docker images. The main difficulty could be expressed as: "How can you create multiple containers out of the same image and have them point to each other?" One approach would be to use docker volumes and provide the configuration externally. This would mean that you have created the configuration for each container, stored it somewhere in the docker host and then pass the configuration to each container as a volume at creation time. I've never tried that myself, I can't tell if its a good or bad practice, I can see some benefits, but I can also see that this is something I am not really excited about. It could look like this: docker run -p 2181:2181 -v /path/to/my/conf:/opt/zookeeper/conf my/zookeeper An other approach would be to pass all the required information as environment variables to the container at creation time and then create a wrapper script which will read the environment variables, modify the configuration files accordingly, launch zookeeper. This is definitely easier to use, but its not that flexible to perform other types of tuning without rebuilding the image itself. Last but not least one could combine the two approaches into one and do something like: Make it possible to provide the base configuration externally using volumes. Use env and scripting to just configure the ensemble. There are plenty of images out there that take one or the other approach. I am more fond of the environment variables approach and since I needed something that would follow some of the kubernetes conventions in terms of naming, I decided to hack an image of my own using the env variables way. Creating a custom image for ZooKeeper I will just focus on the configuration that is required for the ensemble. In order to configure a ZooKeeper ensemble, for each server one has to assign a numeric id and then add in its configuration an entry per zookeeper server, that contains the ip of the server, the peer port of the server and the election port. The server id is added in a file called myid under the dataDir. The rest of the configuration looks like: server.1=server1.example.com:2888:3888 server.2=server2.example.com:2888:3888 server.3=server3.example.com:2888:3888 ... server.current=[bind address]:[peer binding port]:[election biding port]Note that if the server id is X the server.X entry needs to contain the bind ip and ports and not the connection ip and ports. So what we actually need to pass to the container as environment variables are the following: The server id. For each server in the ensemble: The hostname or ip The peer port The election port If these are set, then the script that updates the configuration could look like: if [ ! -z "$SERVER_ID" ]; then echo "$SERVER_ID" > /opt/zookeeper/data/myid #Find the servers exposed in env. for i in `echo {1..15}`;do HOST=`envValue ZK_PEER_${i}_SERVICE_HOST` PEER=`envValue ZK_PEER_${i}_SERVICE_PORT` ELECTION=`envValue ZK_ELECTION_${i}_SERVICE_PORT` if [ "$SERVER_ID" = "$i" ];then echo "server.$i=0.0.0.0:2888:3888" >> conf/zoo.cfg elif [ -z "$HOST" ] || [ -z "$PEER" ] || [ -z "$ELECTION" ] ; then #if a server is not fully defined stop the loop here. break else echo "server.$i=$HOST:$PEER:$ELECTION" >> conf/zoo.cfg fi done fi For simplicity the function that read the keys and values from env are excluded. The complete image and helping scripts to launch zookeeper ensembles of variables size can be found in the fabric8io repository. ZooKeeper on Kubernetes The docker image above, can be used directly with docker, provided that you take care of the environment variables. Now I am going to describe how this image can be used with kubernetes. But first a little rambling... What I really like about using kubernetes with ZooKeeper, is that kubernetes will recreate the container, if it dies or the health check fails. For ZooKeeper this also means that if a container that hosts an ensemble server dies, it will get replaced by a new one. This guarantees that there will be constantly a quorum of ZooKeeper servers. I also like that you don't need to worry about the connection string that the clients will use, if containers come and go. You can use kubernetes services to load balance across all the available servers and you can even expose that outside of kubernetes. Creating a Kubernetes confing for ZooKeeper I'll try to explain how you can create 3 ZooKeeper Server Ensemble in Kubernetes. What we need is 3 docker containers all running ZooKeeper with the right environment variables: { "image": "fabric8/zookeeper", "name": "zookeeper-server-1", "env": [ { "name": "ZK_SERVER_ID", "value": "1" } ], "ports": [ { "name": "zookeeper-client-port", "containerPort": 2181, "protocol": "TCP" }, { "name": "zookeeper-peer-port", "containerPort": 2888, "protocol": "TCP" }, { "name": "zookeeper-election-port", "containerPort": 3888, "protocol": "TCP" } ] } The env needs to specify all the parameters discussed previously. So we need to add along with the ZK_SERVER_ID, the following: ZK_PEER_1_SERVICE_HOST ZK_PEER_1_SERVICE_PORT ZK_ELECTION_1_SERVICE_PORT ZK_PEER_2_SERVICE_HOST ZK_PEER_2_SERVICE_PORT ZK_ELECTION_2_SERVICE_PORT ZK_PEER_3_SERVICE_HOST ZK_PEER_3_SERVICE_PORT ZK_ELECTION_3_SERVICE_PORT An alternative approach could be instead of adding all these manual configuration, to expose peer and election as kubernetes services. I tend to favor the later approach as it can make things simpler when working with multiple hosts. It's also a nice exercise for learning kubernetes. So how do we configure those services? To configure them we need to know: the name of the port the kubernetes pod the provide the service The name of the port is already defined in the previous snippet. So we just need to find out how to select the pod. For this use case, it make sense to have a different pod for each zookeeper server container. So we just need to have a label for each pod, the designates that its a zookeeper server pod and also a label that designates the zookeeper server id. "labels": { "name": "zookeeper-pod", "server": 1 } Something like the above could work. Now we are ready to define the service. I will just show how we can expose the peer port of server with id 1, as a service. The rest can be done in a similar fashion: { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-peer-1", "kind": "Service", "port": 2888, "containerPort": "zookeeper-peer-port", "selector": { "name": "zookeeper-pod", "server": 1 } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } I hope you found it useful. There is definitely room for improvement so feel free to leave comments.

This article explains how errors are handled when using the messaging system with Spring Integration and how to handle route and redirect to specific channel.

This technical tip shows how to read Excel worksheet cells values in multiple threads simultaneously inside android applications. Often you need to read worksheet cells values in multiple threads simultaneously. To do so, set Worksheet.getCells().setMultiThreadReading() to true. If you do not set this property you might get the wrong cell values. Setting it to true, you always get the correct values. To achieve this task first create a workbook and adds a worksheet. Populates the worksheet with some string values then create two threads that simultaneously read values from random cells. If the values read are correct, then nothing happens. But if the values read are incorrect, then a message box shows up in the LogCat window. If you comment this line: testWorkbook.getWorksheets().get(0).getCells().setMultiThreadReading(true); the following message will show up in LogCat window: if (s.equals("R" + row + "C" + col)!=true) { System.out.println("This message box will show up when cells read values are incorrect."); } Otherwise, the program run without showing any message which means all values read from cells are correct. public class ThreadProc implements Runnable { boolean isRunning = true; Workbook testWorkbook; Random r = new Random(); public ThreadProc(Workbook workbook) { this.testWorkbook = workbook; } public int randomNext(int Low, int High) { int R = r.nextInt(High-Low) + Low; return R; } public void kill() { this.isRunning = false; } public void run(){ while(this.isRunning) { int row = randomNext(0, 100); int col = randomNext(0, 10); String s = testWorkbook.getWorksheets().get(0).getCells().get(row, col).getStringValue(); if (s.equals("R" + row + "C" + col)!=true) { System.out.println("This message box will show up when cells read values are incorrect."); } } } } //........MainActivity.java........ //................................. import java.io.File; import java.util.Random; import android.app.Activity; import android.os.Bundle; import android.os.Environment; import android.view.Menu; import com.aspose.cells.CellsHelper; import com.aspose.cells.IWarningCallback; import com.aspose.cells.WarningInfo; import com.aspose.cells.WarningType; import com.aspose.cells.Workbook; public class MainActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); try{ Workbook testWorkbook = new Workbook(); testWorkbook.getWorksheets().clear(); testWorkbook.getWorksheets().add("Sheet1"); for (int row = 0; row < 100; row++) for (int col = 0; col < 10; col++) testWorkbook.getWorksheets().get(0).getCells().get(row, col).setValue("R" + row + "C" + col); //Commenting this line will show a pop-up message testWorkbook.getWorksheets().get(0).getCells().setMultiThreadReading(true); ThreadProc tp = new ThreadProc(testWorkbook); Thread myThread1 = new Thread(tp); myThread1.start(); Thread myThread2 = new Thread(tp); myThread2.start(); Thread.currentThread().sleep(5*1000); tp.kill(); } catch (Exception e) { e.printStackTrace(); } } ......... }

In Parts 1 and 2 we have covered a number of common issues people run into when managing a sharded MongoDB cluster. In this final post of the series we will cover a subtle, but important distinction in terms of balancing a sharded cluster as well as an interesting limitation that can be worked around relatively easily, but is nonetheless surprising when it comes up. 6. Chunk balancing != data balancing != traffic balancing The balancer in a sharded cluster cares about just one thing: Are chunks for a given collection evenly balanced across all shards? If they are not, then it will take steps to rectify that imbalance. This all sounds perfectly logical, and even with extra complexity like tagging involved the logic is pretty straight forward. If we assume that all chunks are equal, then we can rest assured that our data is being evenly balanced across all the shards in our cluster and rest easy at night. Although that is sometimes, perhaps even frequently, the case it is not always true - chunks are not always equal. There can be massive “jumbo” chunks that exceed the maximum chunk size (64MiB), completely empty chunks and everything in between. Let’s use an example from our first pitfall, the monotonically increasing shard key. For our example, we have picked just such a key to shard on (date), and up until this point we have had just one shard and had not sharded the collection. We are about to add a second shard to our cluster and so we enable sharding on the collection and do the necessary admin work to add the new shard into the cluster. Once the collection is enabled for sharding, the first shard contains all the newly minted chunks. Let’s represent them in a simplified table of 10 chunks. This is not representative of a real data set, but it will do for illustrative purposes: Table 1 - Initial Chunk Layout Now we add our second shard. The balancer will kick in and attempt to distribute the chunks evenly. It will do this by moving the lowest range chunks to the new shard until the counts are identical. Once it is finished balancing, our table now looks like this: Table 2 - Balanced Chunk Layout That looks pretty good at the moment, but lets imagine that more recent chunks are more likely to have more activity (updates say) than older chunks. Adding the traffic share estimates for each chunk shows that shard1 is taking far more traffic (72%) than shard2 (28%) despite the chunks seeming balanced overall based on the approximate size. Hence, chunk balancing is not equal to traffic balancing. Using that same example, let’s add another wrinkle - periodic deletion of old data. Every 3 months we run a job to delete any data older than 12 months. Let’s look at the impact of that on our table after we run it for the first time (assuming the first run happens on July 1st 2015). Table 3 - Post-Delete Chunk Layout The distribution of data is now completely skewed toward shard1 - shard2 is in fact empty! However, the balancer is completely unaware of this imbalance - the chunk count has remained the same the entire time, and as far as it is concerned the system is in a steady state. With no data on shard2, our traffic imbalance as seen above will be even worse, and we have essentially negated the benefit of having a second shard for this collection. Possible Mitigation Strategies If data and traffic balance are important, select an appropriate shard key Move chunks manually to address the imbalances - swap “hot” chunks for “cool” chunks, empty chunks for larger chunks 7. Waiting too long to shard a collection (collection too large) This is not very common, but when it falls on your shoulders, it can be quite challenging to solve. There is a maximum data size for a collection when when it is initially split which is a function of the chunk size and data size as noted on the limits page. If your collection contains less than 256GiB of data, then there will be no issue. If the collection size exceeds 256GiB but is less than 400GiB, then MongoDB may be able to do an initial split without any special measures being taken. Otherwise, with larger initial data sizes and the default settings, the initial split will fail. It is worth noting that once split the collection may grow as needed and without any real limitations as long as you can continue to add shards as data size grows. Possible Mitigation Strategies Since the limit is dictated by the chunk size and the data size, and assuming there is not much to be done about the data size, then the remaining variable is the chunk size. This is adjustable (default is 64MiB) and can be raised in order to let a large collection split initially and then reduced once that has been completed. The required chunk size increase will depend on the actual data size. However, this is relatively easy to work out - simply divide your data size by 256GB and then multiply that figure by 64MiB (and round up if it is not a nice even number). As an example, let’s consider a 4TiB collection: 4TiB divided by 256GiB = 16 64MiB x 16 = 1024MiB Hence, set the max chunk size to 1024MiB, then perform the initial sharding of the collection, and then finally reduce the chunk size back to 64MiB using the same procedure. . Thanks for reading through the Sharding Pitfall series! If you want to learn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.

By Adam Comerford, Senior Solutions Engineer In Part I we discussed important considerations when picking a shard key. In this post we will go through some recommendations when running a sharded cluster at scale. Scalability is one of the core benefits of sharding in MongoDB but this can give you a false sense of security; even with that flexibility, you still have to make smart decisions about how and when you deploy resources. In this post, we will cover a couple of common mistakes that people tend to make when it comes to running a sharded cluster. 3. Waiting too long to add a new shard (overloaded) You sharded your database and scaled horizontally for a reason, perhaps it was to add more memory or disk capacity. Whatever the reason, if your application usage grows over time so (generally) does your database utilization. Eventually, your current sharded cluster will pass a certain point, let’s call it 80% utilized (as a nice round estimate), such that it becomes problematic to add another shard. Why? Well, adding a new shard to a cluster is not free, and it is not instantaneous. It consumes resources and (initially) accepts very little traffic. Essentially, at the start of its existence, a newly added shard costs you capacity instead of adding capacity. The length of time it will stay in this state will depend on the balancer and how long it takes for a significant portion of “busy/active” chunks to move onto the new shard. It can often be easier to visualize this process, so let’s make up some hypothetical numbers and set the bar relatively low. Our imaginary existing cluster will be a set of 2 shards, with 2000 chunks (500 considered “active”) and to that we need to add a 3rd shard. This 3rd shard will eventually store one third of the active chunks (and total chunks). The question is, when does this shard stop adding overhead overall and instead become an asset? In reality, this will vary from cluster to cluster and have a lot of dependencies and variables - in other words you need to have good metrics about your cluster, particularly your load bottleneck. Therefore we will once again use our imaginations and go with a relatively low bar: when 5% of active chunks—that is, those chunks seeing most traffic—have migrated to the new shard, you should expect a net gain in performance. In our imaginary system we have evaluated our load levels, the expected impact of migrations and have determine that once that 5% threshold of active chunks has been migrated to the new shard it can be considered a net gain for the overall system. Once all chunks have been balanced, then the migration overhead disappears, but initially this will be an expected trade off. This chart shows how long it would take for new shards to reach net positive contribution in your cluster (the dotted line implies net gain): In this fabricated example, it takes almost 2 hours for the new shard to attain a viable level of active chunks and be considered a net gain for the overall system. Although these numbers are fictional, these numbers are based on setups we have seen in real systems with moderate load. From there it is relatively easy to imagine this set of migrations taking even longer on an overloaded set of shards, and taking far longer for our newly added shard to cross the threshold and become a net gain. As such it is best to be proactive and add capacity before it becomes a necessity. Possible Mitigation Strategies Manual balancing of targeted “hot” chunks (chunk that is being accessed more than others) to move activity to the new shard more quickly Add the shard at low traffic time so that there is less competition for resources Disable balancing on some collections, prioritise balancing busy collections first 4. Under-provisioning Config Servers Provisioning enough resources without being wasteful is always tricky, and all the more so in a complicated distributed system like a MongoDB sharded cluster. Everyone wants to use their hardware, virtual instances, virtual machines, containers and the like in the most efficient way possible, and get the best bang for their buck. Hence it is only natural to take a look at the various pieces of a distributed cluster and look for lower utilized pieces that could be put on less expensive resources. The most common pitfall here with MongoDB are the config servers, which are often neglected when stress testing a cluster. In testing environments and smaller deployments (unless specific measures are taken to stress them) they are relatively lightly loaded and usually identified as candidates for lesser instances/hardware. The problem is that these are critical pieces of infrastructure. They may not be heavily loaded all the time, but when they do see load and struggle to service requests, that can impact all queries (reads, writes, authentication) and add latency to all requests made of the cluster in question. In particular, the first config server in the list supplied to your mongos processes is vital. This is the config server that all mongos processes will default to read from when fetching or refreshing their view of the data distribution in your cluster. Similarly, this is the server that will be hit when attempting to authenticate a user. If it is under-provisioned and cannot service queries, or if it has problems with networking (packet loss, congestion), then the effects will be significant. Possible Mitigation Strategies Ensure the config servers are load tested, slightly over-provisioned (the first config server in particular) If using virtual machines or cloud based instances, investigate increasing available resources Turning off the balancer, disabling chunk splitting will reduce the chances of high read traffic to the config servers (no migrations, no meta data refresh) but this is only a temporary fix unless you have a perfect write distribution and may not eliminate issues completely. 5. Using the count() command on sharded collections This pitfall is very common, and it seems to hit somewhat randomly in terms of how long someone has been running a sharded environment. At some point, a question will arise along the lines of: “How are we tracking/verifying/checking how many documents we have in each collection on each shard, how balanced are they and do they agree with ?” Hopefully no one is actually constructing questions this way in your organization, but you get the basic idea. The most obvious way to do a quick check on this type of thing is to count the documents and see if the numbers make sense and/or agree with counts elsewhere. That thinking naturally leads people to the count command and they proceed to use it to gather figures for their documents and collections. Unfortunately, on a busy, mature sharded cluster, the results will very rarely be what is expected. The reason for this is that the count command as implemented today has several optimizations in place to make it faster to run in general and those speed optimizations essentially bypass a key piece of the sharding functionality needed to return accurate results in this case. This is a known bug and is being tracked in SERVER-3645, but does not stop people from consistently hitting this issue. The nature of the issue means that count will report documents in the results that it should not, for example: Documents that are being deleted as part of a chunk migrations Documents that have been left behind from previous chunk migrations (also known as orphans) Documents currently being copied as part of an in-flight chunk migration A regular query (rather than a count) will have its results filtered by the respective primary and not suffer from the same problem. Hence, if you were to manually count the results from a query client-side you would get an accurate result. This quirk of sharded environments will eventually be fixed, but for now it will inevitably crop up from time to time in all active sharded clusters used by a large team. Possible Mitigation Strategies Do counts on the client side, or use targeted, range based queries (with a primary read preference) to count instead Use cleanUpOrphaned and disable the balancer (make sure it has finished current round) when performing counts across the cluster If you want tolearn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.

We use the Test task in Gradle to run tests. If we use the System.out.println or System.err.println methods in our test we don't see the output when we execute the tests. We can customize the test task to show any output send to standard out or error in the Gradle output. First we show our test class written with Spock, but it could also be a JUnit or TestNG test: // File: src/test/groovy/com/mrhaki/gradle/SampleSpec.groovy package com.mrhaki.gradle import spock.lang.* class SampleSpec extends Specification { def "check that Gradle is Gr8"() { when: def value = 'Gradle is great!' then: // Include a println statement, so // we have output to show. println "Value = [$value]" value == 'Gradle is great!' } } Now we write a simple Gradle build file which can execute our test: // File: build.gradle apply plugin: 'groovy' // Adds test task repositories.jcenter() dependencies { compile 'org.codehaus.groovy:groovy-all:2.3.7' testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' } Let's run the test task from the command line and look at the output: $ gradle test :compileJava UP-TO-DATE :compileGroovy UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :compileTestJava :compileTestGroovy :processTestResources UP-TO-DATE :testClasses :test BUILD SUCCESSFUL Total time: 7.022 secs $ Well at least our test is successful, but we don't see the output of our println method invocation in the test. We customize the test task and add thetestLogging method with a configuration closure. In the closure we set the property showStandardStreams to the value true. Alternatively we can set the events property or use the events method with the values standard_out and standard_err to achieve the same result. In the next build file we use the showStandardStreams property: view sourceprint? 00.// File: build.gradle 01.apply plugin: 'groovy' // Adds test task 02. 03.repositories.jcenter() 04. 05.dependencies { 06.compile 'org.codehaus.groovy:groovy-all:2.3.7' 07.testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' 08.} 09. 10.test { 11.testLogging { 12.// Make sure output from 13.// standard out or error is shown 14.// in Gradle output. 15.showStandardStreams = true 16. 17.// Or we use events method: 18.// events 'standard_out', 'standard_error' 19. 20.// Or set property events: 21.// events = ['standard_out', 'standard_error'] 22. 23.// Instead of string values we can 24.// use enum values: 25.// events org.gradle.api.tasks.testing.logging.TestLogEvent.STANDARD_OUT, 26.// org.gradle.api.tasks.testing.logging.TestLogEvent.STANDARD_ERROR, 27.} 28.} We re-run the test task from the command line and look at the output to see the result from the println method: $ gradle test :compileJava UP-TO-DATE :compileGroovy UP-TO-DATE :processResources UP-TO-DATE :classes UP-TO-DATE :compileTestJava :compileTestGroovy :processTestResources UP-TO-DATE :testClasses :test com.mrhaki.gradle.SampleSpec > check that Gradle is Gr8 STANDARD_OUT Value = [Gradle is great!] BUILD SUCCESSFUL Total time: 8.716 secs $ Written with Gradle 2.1.

in this article i explained how spring built-in transformers works for while transforming object message to map message. sometimes the messages need to be transformed before they can be consumed to achieve a business purpose. for example, a producer uses a plain xml as its payload to produce a message, while a consumer is interested in java object or types like plain text ,name-value pairs, or json model. spring integration provides endpoints such as service activators, channel adapters, message bridges, gateways, transformers, filters, and routers. in this example how transformers endpoint transform object message to map message. references: spring integration spring with jms spring with junit mockrunner sts high level view spring-mockrunner.xml in spring-mockrunner.xml file, i defined mockqueue, mockqueueconnectionfactory for inbound queue, and outbound queue for quick testing purpose. inboundqueue is where you will publish object message from objecttomaptransformertest.java class. outboundqueue where this queue expecting mapmessage type object and this queue is listing mapmessagelistener.java class. for more information mockrunner works please check my previous article mockrunner with spring jms . pom.xml 4.0.0 org.springframework.samples spring-int-jms-basic 0.0.1-snapshot 1.6 utf-8 utf-8 3.2.3.release 1.0.13 1.7.5 4.11 org.springframework spring-context ${spring-framework.version} org.springframework spring-tx ${spring-framework.version} org.springframework.integration spring-integration-core 2.2.4.release org.springframework.integration spring-integration-jmx 2.2.4.release org.springframework.integration spring-integration-jms 2.2.4.release org.slf4j slf4j-api ${slf4j.version} compile ch.qos.logback logback-classic ${logback.version} runtime org.springframework spring-test ${spring-framework.version} test junit junit ${junit.version} test com.mockrunner mockrunner-jms 1.0.3 javax.jms jms 1.1 org.codehaus.jackson jackson-mapper-asl 1.9.3 compile spring-int-jms.xml the endpoint is configured to connect to a jms server, fetch the messages,and publish them onto a local channel i.e inputchannel. where as connection-factory, and destination referred mockqueueconnectionfactory, and mockqueue(inboundqueue) beans from spring-mockrunner.xml file. inputchannel and outputchannel defined as queue channel objecttomaptransformer: object-to-map-transformer element that takes the payload from the input channel original here mockrunner-in-queue object message and emits a name-value paired map object onto the output channel i.e outputchannel and outboundjmsadapter bean fetch this message and publish to queue i.e mockrunner-out-queue. inboundjmsadapter : inbound-channel-adapter bean is responsible for receiving messages from a jms server here it is reading from mock queue name mockrunner-in-queue see objecttomaptransformertest.java class. outboundjmsadapter : outbound-channel-adapter bean is responsible to fetch messages from the channel i.e outputchannel and publish them to jms queue or topic. in this outbounjmsadapter reading message outputchannel as mapmessage and publish to outboundqueue(mockrunner-out-queue). mapmessagelistener.java package com.spijb.listener; import javax.jms.jmsexception; import javax.jms.mapmessage; import javax.jms.session; import org.slf4j.logger; import org.slf4j.loggerfactory; import org.springframework.jms.listener.sessionawaremessagelistener; public class mapmessagelistener implements sessionawaremessagelistener { private static final logger log = loggerfactory.getlogger(mapmessagelistener.class); @override public void onmessage(mapmessage message, session session) throws jmsexception { log.info("message received \r\n"+message); } } it is plain mapmessagelistener class to print received message from queue. department.java package com.spijb.domain; import java.io.serializable; public class department implements serializable{ private static final long serialversionuid = 1l; private final integer deptno; private final string name; private final string location; public department() { deptno=10; name="sales"; location="tx"; } public department(integer dno,string name,string loc) { this.deptno=dno; this.name=name; this.location=loc; } public integer getdeptno() { return deptno; } public string getname() { return name; } public string getlocation() { return location; } @override public string tostring() { return this.deptno+"-> "+this.name+"->"+this.location; } } domain object to send as a message, by default constructor assign deptno 10 , name as sales, location as tx also provide parameter constructor. spring junit class objecttomaptransformertest.java package com.spijb.invoker; import javax.jms.jmsexception; import javax.jms.message; import javax.jms.objectmessage; import javax.jms.session; import org.junit.test; import org.junit.runner.runwith; import org.springframework.beans.factory.annotation.autowired; import org.springframework.jms.core.jmstemplate; import org.springframework.jms.core.messagecreator; import org.springframework.test.context.contextconfiguration; import org.springframework.test.context.junit4.springjunit4classrunner; import com.mockrunner.mock.jms.mockqueue; import com.spijb.domain.department; @runwith(springjunit4classrunner.class) @contextconfiguration({"classpath:spring-mockrunner.xml","classpath:spring-int-jms.xml"}) public class objecttomaptransformertest { @autowired private jmstemplate jmstemplate; @autowired private mockqueue inboundqueue; @test public void shouldsendmessage() throws interruptedexception { final department defaultdepartment = new department(); jmstemplate.send(inboundqueue,new messagecreator() { @override public message createmessage(session session) throws jmsexception { objectmessage objectmessage = session.createobjectmessage(); objectmessage.setobject(defaultdepartment); return objectmessage; } }); thread.sleep(5000); } } spring with junit class where you can send message to inputchannel i.e inboundqueue using mockrunner. output : info: started inboundjmsadapter oct 06, 2014 1:24:25 pm org.springframework.integration.endpoint.abstractendpoint start info: started org.springframework.integration.config.consumerendpointfactorybean#1 13:24:26.882 [org.springframework.jms.listener.defaultmessagelistenercontainer#0-1] info c.spijb.listener.mapmessagelistener - message received com.mockrunner.mock.jms.mockmapmessage: {location=tx, name=sales, deptno=10} oct 06, 2014 1:24:30 pm org.springframework.context.support.abstractapplicationcontext doclose info: closing org.springframework.context.support.genericapplicationcontext@5840979b: startup date [mon oct 06 13:24:25 cdt 2014]; root of context hierarchy oct 06, 2014 1:24:30 pm org.springframework.context.support.defaultlifecycleprocessor$lifecyclegroup stop info: stopping beans in phase 2147483647 in the above highlighted one is output as map.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.

This article shows how to mock your JMS infrastructure using MockRunner and test it using Spring.

There are some instances when you want to store your passwords in files to be used by programs or scripts. But storing your passwords in plain text is not a good idea. Use the SecurePasswordVault to encrypt your passwords before storing and get it decrypted when you want to use it. You can use the SecurePasswordVault described here to store any number of encrypted passwords. Passwords are stored as key value pairs. Key - any name given by the user for the password Value - encrypted password SecurePasswordVault will create a file with the given name in the working directory if it doesn't exist. If a file exists then the information in that file will be read. Passwords are encrypted using the MAC address of the network card. SecurePasswordVault will use the first network card MAC which is not the loop back interface. So the encrypted file can only be decrypted with that particular MAC address. If you want to reset the pass word details, just delete the password file and run the SecurePasswordVault. You can download the sample code from the following GitHub repository https://github.com/jsdjayanga/secure_password com.wso2.devgov; import org.bouncycastle.util.encoders.Base64; import javax.crypto.*; import javax.crypto.spec.SecretKeySpec; import java.io.*; import java.net.NetworkInterface; import java.net.SocketException; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import java.security.Security; import java.util.*; /** * Created by jayanga on 3/31/14. */ public class SecurePasswordVault { private static final int AES_KEY_LEN = 32; private static final int PASSWORD_LEN = 256; private static boolean initialized; private final String secureFile; private final byte[] networkHardwareHaddress; private Map secureDataMap; private List secureDataList; SecretKeySpec secretKey; public SecurePasswordVault(String filename, String[] secureData) throws IOException { Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider()); initialized = false; secureFile = filename; networkHardwareHaddress = SecurePasswordVault.readNetworkHardwareAddress(); secureDataMap = new HashMap(); this.secureDataList = new ArrayList(secureData.length); Collections.addAll(secureDataList, secureData); byte[] key = new byte[AES_KEY_LEN]; Arrays.fill(key, (byte)0); for(int index = 0; index < networkHardwareHaddress.length; index++){ key[index] = networkHardwareHaddress[index]; } secretKey = new SecretKeySpec(key, "AES"); if (!isInitialized()){ readSecureData(secureDataList); persistSecureData(); } readSecureDataFromFile(); } private boolean isInitialized(){ if (initialized == true){ return true; }else{ File file = new File(secureFile); if (file.exists()){ initialized = true; return initialized; } } return false; } private static byte[] readNetworkHardwareAddress() throws SocketException { Enumeration networkInterfaceEnumeration = NetworkInterface.getNetworkInterfaces(); if (networkInterfaceEnumeration != null){ NetworkInterface networkInterface = null; while (networkInterfaceEnumeration.hasMoreElements()){ networkInterface = networkInterfaceEnumeration.nextElement(); if (!networkInterface.isLoopback()){ break; } } if (networkInterface == null){ networkInterface = networkInterfaceEnumeration.nextElement(); } byte[] hwaddr = networkInterface.getHardwareAddress(); return hwaddr; }else{ throw new RuntimeException("Cannot initialize. Failed to generate unique id."); } } private byte[] encrypt(String word) { byte[] password = new byte[PASSWORD_LEN]; Arrays.fill(password, (byte)0); byte[] pw = new byte[0]; try { pw = word.getBytes("UTF-8"); for(int index = 0; index < pw.length; index++){ password[index] = pw[index]; } byte[] cipherText = new byte[password.length]; Cipher cipher = null; try { cipher = Cipher.getInstance("AES/ECB/NoPadding"); try { cipher.init(Cipher.ENCRYPT_MODE, secretKey); int ctLen = 0; try { ctLen = cipher.update(password, 0, password.length, cipherText, 0); ctLen += cipher.doFinal(cipherText, ctLen); return cipherText; } catch (ShortBufferException e) { e.printStackTrace(); } catch (BadPaddingException e) { e.printStackTrace(); } catch (IllegalBlockSizeException e) { e.printStackTrace(); } } catch (InvalidKeyException e) { e.printStackTrace(); } } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (NoSuchPaddingException e) { e.printStackTrace(); } } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return null; } private String decrypt(byte[] cipherText) { byte[] plainText = new byte[PASSWORD_LEN]; Cipher cipher = null; try { cipher = Cipher.getInstance("AES/ECB/NoPadding"); try { cipher.init(Cipher.DECRYPT_MODE, secretKey); int plainTextLen = 0; try { plainTextLen = cipher.update(cipherText, 0, PASSWORD_LEN, plainText, 0); try { plainTextLen += cipher.doFinal(plainText, plainTextLen); String password = new String(plainText); return password.trim(); } catch (IllegalBlockSizeException e) { e.printStackTrace(); } catch (BadPaddingException e) { e.printStackTrace(); } } catch (ShortBufferException e) { e.printStackTrace(); } } catch (InvalidKeyException e) { e.printStackTrace(); } } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (NoSuchPaddingException e) { e.printStackTrace(); } return null; } public void readSecureData(List secureDataList) throws IOException { BufferedReader bufferRead = new BufferedReader(new InputStreamReader(System.in)); for(int index = 0; index < secureDataList.size(); index++){ System.out.println("Please enter the value for :" + secureDataList.get(index)); String value = new String(Base64.encode(encrypt(bufferRead.readLine()))); secureDataMap.put(secureDataList.get(index), value); } } public String getSecureData(String key) { String value = secureDataMap.get(key); if (value != null){ return decrypt(Base64.decode(value.getBytes())); } throw new RuntimeException("Given key is unknown. [key=" + key + "]"); } private void readSecureDataFromFile() throws IOException { BufferedReader br = new BufferedReader(new FileReader(secureFile)); String line; while ((line = br.readLine()) != null){ int dividerPoint = line.indexOf("="); if (dividerPoint > 0){ secureDataMap.put(line.substring(0, dividerPoint), line.substring(dividerPoint + 1)); } } } private void persistSecureData() throws IOException { FileWriter fileWriter = new FileWriter(secureFile); for(String key : secureDataMap.keySet()){ fileWriter.append(key + "=" + secureDataMap.get(key) + "\n"); } fileWriter.close(); } }

I use Charles Proxy extensively for debugging all kinds of applications, and lately I've been using it more with mobile devices. One of the killer features of Charles is its ability to intercept SSL traffic. This is hard - and rightly so, it should be difficult to inspect SSL traffic! Charles handles this by using the server's SSL certificate for the connection from Charles to the remote server, and then using Charles' own SSL certificate for the "last mile" back to your browser or device. This means that the connection will be flagged as insecure; Charles' certificates aren't trusted by your browser or device - but we can easily change that. Get Charles' Certificate On Windows, you can grab the certificate from the Help menu in Charles, but for OS X or Linux you'll need to download it from their website. It's linked from this excellent documentation page about SSL:http://www.charlesproxy.com/documentation/using-charles/ssl-certificates/. Install it On Your Android Device Copy the ``.crt`` file into the root of your SD card (I found this didn't work in a subdirectory, although some documentation suggests it will). Under "Security" -> "Settings", choose "install from device storage" and it should pick up the certificate you put on the SD card. ... that's it :) Check you have enabled the destination for SSL proxying under "Proxy Settings" in Charles, and you're all set! Note, I have one device that refused to see the .crt file on the SD card, but when I emailed it to myself and then saved it from there, was added successfully. Thought I'd mention it! Further Reading Using Charles To Debug PHP SOAP Wireshark Capture on Remote Server View Only Headers with Curl

CodePro Analytix is the premier Java software testing tool for Eclipse developers.

Property based testing is an alternative approach to testing, complementingexample based testing. The latter is what we've been doing all our lives: exercising production code against "examples" - inputs we think are representative. Picking these examples is an art on its own: "ordinary" inputs, edge cases, malformed inputs, etc. But why are we limiting ourselves to just few examples? Why not test hundreds, millions... ALL inputs? There are at least two difficulties with that approach: Scale. A pure function taking just one int input would require 4 billion tests. This means few hundred gigabytes of test source code and several months of execution time. Square it if a function takes two ints. For String it practically goes to infinity. Assume we have these tests, executed on a quantum computer or something. How do you know the expected result for each particular input? You either enter it by hand (good luck) or generate expected output. Bygenerate I mean write a program that produces expected value for every input. But aren't we testing such program already in the first place? Are we suppose to write better, error-free version of code under test just to test it? Also known as ugly mirror antipattern. So you understand testing every single input, although ideal, is just a mental experiment, impossible to implement. That being said property based testing tries to get as close as possible to this testing nirvana. Issue #1 is solved by slamming code under test with hundreds or thousands of random inputs. Not all of them, not even a fraction. But a good, random representation. Issue #2 is surprisingly harder. Property based testing can generate random arguments, but it can't figure out what should be the expected outcome for that random input. Thus we need a different mechanism, giving name to whole philosophy. We have to come up with properties (invariants, behaviours) that code under test exhibits no matter what the input is. This sounds very theoretically, but there are many such properties in various scenarios: Absolute value of any number should never be negative Encoding and decoding any string should yield the same String back for every symmetric encoding Optimized version of some old algorithm should produce the same result as the old one for any input Total money in a bank should remain the same after arbitrary number of intra-bank transactions in any order As you can see there are many properties we can think of that do not mention specific example inputs. This is not exhaustive and strict testing. It's more like sampling and making sure samples are "sane". There are many, many libraries supporting property based testing for virtually every language. In this article we will explore Spock and ScalaCheck later. Spock + custom data generators Spock does not support property based testing out-of-the-box. However with help from data driven testing and 3rd-party data generators we can go quite far. Data tables in Spock can be generalized into so-called data pipes: def 'absolute value of #value should not be negative'() { expect: value.abs() >= 0 where: value << randomInts(100) } private static def List randomInts(int count) { final Random random = new Random() (1..count).collect { random.nextInt() } } Code above will generate 100 random integers and make sure for all of them.abs() is non-negative. You might think this test is quite dumb, but to a great surprise it actually discovers one bug! But first let's kill some boilerplate code. Generating random inputs, especially more complex, is cumbersome and boring. I found two libraries that can help us. spock-genesis: import spock.genesis.Gen def 'absolute value of #value should not be negative'() { expect: value.abs() >= 0 where: value << Gen.int.take(100) } Looks great, but if you want to generate e.g. lists of random integers,net.java.quickcheck has nicer API and is not Groovy-specific: import static net.java.quickcheck.generator.CombinedGeneratorsIterables.someLists import static net.java.quickcheck.generator.PrimitiveGenerators.integers def 'sum of non-negative numbers from #list should not be negative'() { expect: list.findAll{it >= 0}.sum() >= 0 where: list << someLists(integers(), 100) } This test is interesting. It makes sure sum of non-negative numbers is never negative - by generating 100 lists of randoms ints. Sounds reasonable. However multiple tests are failing. First of all due to integer overflow sometimes two positiveints add up to a negative one. Duh! Another type of failure that was discovered is actually frightening. While [1,2,3].sum() is 6, obviously, [].sum() is... null(WAT?) As you can see even silliest and most basic property based tests can be useful in finding unusual corner cases in your data. But wait, I said testing absolute of intdiscovered one bug. Actually it didn't, because of poor (too "random") data generators, not returning known edge values in the first place. We will fix that in the next article.

In this article we wanted to share another configuration property that can provide surprising help when setting up your JBoss BPM Suite. Previously we outlined a basic set of configuration properties to provide you with a few tricks when installing your own JBoss BRMS or JBoss BPM Suite products. As the JBoss BPM Suite is a super set, including full JBoss BRMS functionality, the rest of this article will refer only to JBoss BPM Suite but apply to both products. In this article we will show you how to modify your JBoss EAP container configuration to point the products at a custom deployment repository by adjusting a single configuration property. Maven repository The default setup is that the products will look for your maven setting in the default settings.xml as found set in theM2_HOME variable or in the users home directory at .m2/settings.xml. The following system property can be added to JBoss EAP standalone.xml configuration file to point to any file containing your custom settings. kie.maven.settings.custom Location of the maven configuration file where it can find it's settings. Default: the M2_HOME/conf/settings.xml or users home directory .m2/settings.xml Example usage in JBoss EAP When initially setting up the product for use on JBoss EAP containers, one can adjust configuration with the help of system properties. Below we show how to configure an installation to point to our custom maven deployment repository by using a custom settings file we will call bpmsuite-settings.xml We hope this helps you with configuring your own custom deployment repositories and enables you to tie into existing continuous integration infrastructures that might exist in your organization.

Originally Written by Muhammad Irfan Here on the Percona Support team we often ask customers to retrieve disk stats to monitor disk IO and to measure block devices iops and latency. There are a number of tools available to monitor IO on Linux. iostat is one of the popular tools and Percona Toolkit, which is free, contains the pt-diskstats tool for this purpose. The pt-diskstats tool is similar to iostat but it’s more interactive and contains extended information. pt-diskstats reports current disk activity and shows the statistics for the last second (which by default is 1 second) and will continue until interrupted. The pt-diskstats tool collects samples of /proc/diskstats. In this post, I will share some examples about how to monitor and check to see if the IO subsystem is performing properly or if any disks are a limiting factor – all this by using the pt-diskstats tool. pt-diskstats output consists on number of columns and in order to interpret pt-diskstats output we need to know what each column represents. rd_s tells about number of reads per second while wr_s represents number of writes per second. rd_rt and wr_rt shows average response time in milliseconds for reads & writes respectively, which is similar to iostat tool output await column but pt-diskstats shows individual response time for reads and writes at disk level. Just a note, modern iostat splits read and write latency out, but most distros don’t have the latest iostat in their systat (or equivalent) package. rd_mrg and wr_mrg are other two important columns in pt-diskstats output. *_mrg is telling us how many of the original operations the IO elevator (disk scheduler) was able to merge to reduce IOPS, so *_mrg is telling us a quite important thing by letting us know that the IO scheduler was able to consolidate many or few operations. If rd_mrg/wr_mrg is high% then the IO workload is sequential on the other hand, If rd_mrg/wr_mrg is a low% then IO workload is all random. Binary logs, redo logs (aka ib_logfile*), undo log and doublewrite buffer all need sequential writes. qtime and stime are last two columns in pt-diskstats output where qtime reflects to time spent in disk scheduler queue i.e. average queue time before sending it to physical device and on the other hand stime is average service time which is time accumulated to process the physical device request. Note, that qtime is not discriminated between reads and writes and you can check if response time is higher for qtime than it signal towards disk scheduler. Also note that service time (stime field and svctm field in in pt-diskstats & iostat output respectively) is not reliable on Linux. If you read the iostat manual you will see it is deprecated. Along with that, there are many other parameters for pt-diskstats – you can found full documentation here. Below is an example of pt-disktats in action. I used the –devices-regex option which prints only device information that matches this Perl regex. $ pt-diskstats --devices-regex=sd --interval 5 #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.1 sda 21.6 22.8 0.5 45% 1.2 29.4 275.5 4.0 1.1 0% 40.0 145.1 65% 158 297.1 155.0 2.1 1.1 sdb 15.0 21.0 0.3 33% 0.1 5.2 0.0 0.0 0.0 0% 0.0 0.0 11% 1 15.0 0.5 4.7 1.1 sdc 5.6 10.0 0.1 0% 0.0 5.2 1.9 6.0 0.0 33% 0.0 2.0 3% 0 7.5 0.4 3.6 1.1 sdd 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 5.0 sda 17.0 14.8 0.2 64% 3.1 66.7 404.9 4.6 1.8 14% 140.9 298.5 100% 111 421.9 277.6 1.9 5.0 sdb 14.0 19.9 0.3 48% 0.1 5.5 0.4 174.0 0.1 98% 0.0 0.0 11% 0 14.4 0.9 2.4 5.0 sdc 3.6 27.1 0.1 61% 0.0 3.5 2.8 5.7 0.0 30% 0.0 2.0 3% 0 6.4 0.7 2.4 5.0 sdd 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 These are the stats from 7200 RPM SATA disks. As you can see, the write-response time is very high and most of that is made up of IO queue time. This shows the problem exactly. The problem is that the IO subsystem is not able to handle the write workload because the amount of writes that are being performed are way beyond what it can handle. It means the disks cannot service every request concurrently. The workload would actually depend a lot on where the hot data is stored and as we can see in this particular case the workload only hits a single disk out of the 4 disks. A single 7.2K RPM disk can only do about 100 random writes per second which is not a lot considering heavy workload. It’s not particularly a hardware issue but a hardware capacity issue. The kind of workload that is present and the amount of writes that are performed per second are not something that the IO subsystem is able to handle in an efficient manner. Mostly writes are generated on this server as can be seen by the disk stats. Let me show you a second example. Here you can see read latency. rd_rt is consistently between 10ms-30ms. It depends on how fast the disks are spinning and the number of disks. To deal with it possible solutions would be to optimize queries to avoid table scans, use memcached where possible, use SSD’s as it can provide good I/O performance with high concurrency. You will find this post useful on SSD’s from our CEO, Peter Zaitsev. #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.0 sdb 33.0 29.1 0.9 0% 1.1 34.7 7.0 10.3 0.1 61% 0.0 0.4 99% 1 40.0 2.2 19.5 1.0 sdb1 0.0 0.0 0.0 0% 0.0 0.0 7.0 10.3 0.1 61% 0.0 0.4 1% 0 7.0 0.0 0.4 1.0 sdb2 33.0 29.1 0.9 0% 1.1 34.7 0.0 0.0 0.0 0% 0.0 0.0 99% 1 33.0 3.5 30.2 1.0 sdb 81.9 28.5 2.3 0% 1.1 14.0 0.0 0.0 0.0 0% 0.0 0.0 99% 1 81.9 2.0 12.0 1.0 sdb1 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 1.0 sdb2 81.9 28.5 2.3 0% 1.1 14.0 0.0 0.0 0.0 0% 0.0 0.0 99% 1 81.9 2.0 12.0 1.0 sdb 50.0 25.7 1.3 0% 1.3 25.1 13.0 11.7 0.1 66% 0.0 0.7 99% 1 63.0 3.4 11.3 1.0 sdb1 25.0 21.3 0.5 0% 0.6 25.2 13.0 11.7 0.1 66% 0.0 0.7 46% 1 38.0 3.2 7.3 1.0 sdb2 25.0 30.1 0.7 0% 0.6 25.0 0.0 0.0 0.0 0% 0.0 0.0 56% 0 25.0 3.6 22.2 From the below diskstats output it seems that IO is saturated between both reads and writes. This can be noticed with high value for columns rd_s and wr_s. In this particular case, consider having disks in either RAID 5 (better for read only workload) or RAID 10 array is good option along with battery-backed write cache (BBWC) as single disk can really be bad for performance when you are IO bound. device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime sdb1 362.0 27.4 9.7 0% 2.7 7.5 525.2 20.2 10.3 35% 6.4 8.0 100% 0 887.2 7.0 0.9 sdb1 439.9 26.5 11.4 0% 3.4 7.7 545.7 20.8 11.1 34% 9.8 11.9 100% 0 985.6 9.6 0.8 sdb1 576.6 26.5 14.9 0% 4.5 7.8 400.2 19.9 7.8 34% 6.7 10.9 100% 0 976.8 8.6 0.8 sdb1 410.8 24.2 9.7 0% 2.9 7.1 403.1 18.3 7.2 34% 10.8 17.7 100% 0 813.9 12.5 1.0 sdb1 378.4 24.6 9.1 0% 2.7 7.3 506.1 16.5 8.2 33% 5.7 7.6 100% 0 884.4 6.6 0.9 sdb1 572.8 26.1 14.6 0% 4.8 8.4 422.6 17.2 7.1 30% 1.7 2.8 100% 0 995.4 4.7 0.8 sdb1 429.2 23.0 9.6 0% 3.2 7.4 511.9 14.5 7.2 31% 1.2 1.7 100% 0 941.2 3.6 0.9 The following example reflects write heavy activity but write-response time is very good, under 1ms, which shows disks are healthy and capable of handling high number of IOPS. #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 1.0 dm-0 530.8 16.0 8.3 0% 0.3 0.5 6124.0 5.1 30.7 0% 1.7 0.3 86% 2 6654.8 0.2 0.1 2.0 dm-0 633.1 16.1 10.0 0% 0.3 0.5 6173.0 6.1 36.6 0% 1.7 0.3 88% 1 6806.1 0.2 0.1 3.0 dm-0 731.8 16.0 11.5 0% 0.4 0.5 6064.2 5.8 34.1 0% 1.9 0.3 90% 2 6795.9 0.2 0.1 4.0 dm-0 711.1 16.0 11.1 0% 0.3 0.5 6448.5 5.4 34.3 0% 1.8 0.3 92% 2 7159.6 0.2 0.1 5.0 dm-0 700.1 16.0 10.9 0% 0.4 0.5 5689.4 5.8 32.2 0% 1.9 0.3 88% 0 6389.5 0.2 0.1 6.0 dm-0 774.1 16.0 12.1 0% 0.3 0.4 6409.5 5.5 34.2 0% 1.7 0.3 86% 0 7183.5 0.2 0.1 7.0 dm-0 849.6 16.0 13.3 0% 0.4 0.5 6151.2 5.4 32.3 0% 1.9 0.3 88% 3 7000.8 0.2 0.1 8.0 dm-0 664.2 16.0 10.4 0% 0.3 0.5 6349.2 5.7 35.1 0% 2.0 0.3 90% 2 7013.4 0.2 0.1 9.0 dm-0 951.0 16.0 14.9 0% 0.4 0.4 5807.0 5.3 29.9 0% 1.8 0.3 90% 3 6758.0 0.2 0.1 10.0 dm-0 742.0 16.0 11.6 0% 0.3 0.5 6461.1 5.1 32.2 0% 1.7 0.3 87% 1 7203.2 0.2 0.1 Let me show you a final example. I used –interval and –iterations parameters for pt-diskstats which tells us to wait for a number of seconds before printing the next disk stats and to limit the number of samples respectively. If you notice, you will see in 3rd iteration high latency (rd_rt, wr_rt) mostly for reads. Also, you can notice a high value for queue time (qtime) and service time (stime) where qtime is related to disk IO scheduler settings. For MySQL database servers we usually recommends noop/deadline instead of default cfq. $ pt-diskstats --interval=20 --iterations=3 #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s qtime stime 10.4 hda 11.7 4.0 0.0 0% 0.0 1.1 40.7 11.7 0.5 26% 0.1 2.1 10% 0 52.5 0.4 1.5 10.4 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.4 7.0 0.0 43% 0.0 0.1 0% 0 0.4 0.0 0.1 10.4 hda3 0.0 0.0 0.0 0% 0.0 0.0 0.4 107.0 0.0 96% 0.0 0.2 0% 0 0.4 0.0 0.2 10.4 hda5 0.0 0.0 0.0 0% 0.0 0.0 0.7 20.0 0.0 80% 0.0 0.3 0% 0 0.7 0.1 0.2 10.4 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.1 4.0 0.0 0% 0.0 4.0 0% 0 0.1 0.0 4.0 10.4 hda9 11.7 4.0 0.0 0% 0.0 1.1 39.2 10.7 0.4 3% 0.1 2.7 9% 0 50.9 0.5 1.8 10.4 drbd1 11.7 4.0 0.0 0% 0.0 1.1 39.1 10.7 0.4 0% 0.1 2.8 9% 0 50.8 0.5 1.7 20.0 hda 14.6 4.0 0.1 0% 0.0 1.4 39.5 12.3 0.5 26% 0.3 6.4 18% 0 54.1 2.6 2.7 20.0 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.4 9.1 0.0 56% 0.0 42.0 3% 0 0.4 0.0 42.0 20.0 hda3 0.0 0.0 0.0 0% 0.0 0.0 1.5 22.3 0.0 82% 0.0 1.5 0% 0 1.5 1.2 0.3 20.0 hda5 0.0 0.0 0.0 0% 0.0 0.0 1.1 18.9 0.0 79% 0.1 21.4 11% 0 1.1 0.1 21.3 20.0 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.8 10.4 0.0 62% 0.0 1.5 0% 0 0.8 1.3 0.2 20.0 hda9 14.6 4.0 0.1 0% 0.0 1.4 35.8 11.7 0.4 3% 0.2 4.9 18% 0 50.4 0.5 3.5 20.0 drbd1 14.6 4.0 0.1 0% 0.0 1.4 36.4 11.6 0.4 0% 0.2 5.1 17% 0 51.0 0.5 3.4 20.0 hda 0.9 4.0 0.0 0% 0.2 251.9 28.8 61.8 1.7 92% 4.5 13.1 31% 2 29.6 12.8 0.9 20.0 hda2 0.0 0.0 0.0 0% 0.0 0.0 0.6 8.3 0.0 52% 0.1 98.2 6% 0 0.6 48.9 49.3 20.0 hda3 0.0 0.0 0.0 0% 0.0 0.0 2.0 23.2 0.0 83% 0.0 1.4 0% 0 2.0 1.2 0.3 20.0 hda5 0.0 0.0 0.0 0% 0.0 0.0 4.9 249.4 1.2 98% 4.0 13.2 9% 0 4.9 12.9 0.3 20.0 hda6 0.0 0.0 0.0 0% 0.0 0.0 0.0 0.0 0.0 0% 0.0 0.0 0% 0 0.0 0.0 0.0 20.0 hda9 0.9 4.0 0.0 0% 0.2 251.9 21.3 24.2 0.5 32% 0.4 12.9 31% 2 22.2 10.2 9.7 20.0 drbd1 0.9 4.0 0.0 0% 0.2 251.9 30.6 17.0 0.5 0% 0.7 24.1 30% 5 31.4 21.0 9.5 You can see the busy column in pt-diskstats output which is the same as the util column in iostat – which points to utilization. Actually, pt-diskstats is quite similar to the iostat tool but pt-diskstats is more interactive and has more information. The busy percentage is only telling us for how long the IO subsystem was busy, but is not indicating capacity. So the only time you care about %busy is when it’s 100% and at the same time latency (await in iostat and rd_rt/wr_rt in diskstats output) increases over -say- 5ms. You can estimate capacity of your IO subsystem and then look at the IOPS being consumed (r/s + w/s columns). Also, the system can process more than one request in parallel (in case of RAID) so %busy can go beyond 100% in pt-diskstats output. If you need to check disk throughput, block device IOPS run the following to capture metrics from your IO subsystem and see if utilization matches other worrisome symptoms. I would suggest capturing disk stats during peak load. Output can be grouped by sample or by disk using the –group-by option. You can use the sysbench benchmark tool for this purpose to measure database server performance. You will find this link useful for sysbench tool details. $ pt-diskstats --group-by=all --iterations=7200 > /tmp/pt-diskstats.out; Conclusion: pt-diskstats is one of the finest tools from Percona Toolkit. By using this tool you can easily spot disk bottlenecks, measure the IO subsystem and identify how much IOPS your drive can handle (i.e. disk capacity).

If you use Java for programming, read on to learn about tools like Eclipse IDE, the Java Development Kit, and other must-know tools.

in “ semihosting with kinetis design studio ” i used printf() to exchange text and data between the target board and the host using the debug connection. kinetis design studio (kds) has that semihosting baked into its libraries. what about if using the gnu arm embedded (launchpad) tools and libraries (see “ switching arm gnu tool chain and libraries in kinetis design studio “)? actually it requires two more steps, but is very easy too. semihosting output there are three things to be in place to use semihosting with the gnu arm embedded (launchpad) libraries: option in the gnu linker settings enabling semihosting in the debugger settings initializing the gnu libraries linker option to enable semihosting for the gnu arm embedded ( launchpad ) libraries, i need to add --specs=rdimon.specs to the linker options: linker option to enable semihosting in case i’m using newlib-nano and want to use printf() and/or scanf() with floating point support, i need to pull in some symbols explicitly with the linker options ‘u': -u _scanf_float -u _printf_float debugger settings in the gnu arm eclipse plugins, i need to enable semihosting. segger j-link for segger j-link, i enable the console in the launch configuration: allocated semihosting console for segger additionally i enable semihosting options in the startup options of the debugger: enabled semihosting in the startup options for segger p&e multilink for p&e the following settings are used: semihosting settings for pne settings for openocd the following settings are used for openocd: openocd semihosting settings initializing the gnu libraries if you would now try to use semihosting with running the debugger, you probably will get error messages like this (e.g. from segger j-link): warning: semihosting command sys_flen failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. warning: semihosting command sys_write failed. handle is 0. the reason is that the semihosting needs to be enabled by the application. i need to call initialise_monitor_handles() before i’m using printf() : 1 2 3 4 5 6 7 8 extern void initialise_monitor_handles( void ); /* prototype */ int main( void ) { initialise_monitor_handles(); /* initialize handles */ for (;;) { printf ( "hello world!\r\n" ); } } with this, i can use printf() and scanf() through a debugger connection. semihosting printf output summary while i don’t like printf() for many reasons, sometimes it is useful to exchange data with the host. using semihosting no physical connection is required, as the communication goes through the debugger. it is somewhat intrusive, and adds code and data overhead, but the gnu arm embedded (launchpad) libraries (both newlib and newlib-nano) have semihosting built-in. it is a matter to enable it in the linker and debugger settings, and to initialize the handles in the application. happy semihosting :-)