Learn about the differences between SAML and OAuth plus use cases for each one.

Applications are at the center of the IT universe. As IT shifts its primary goal from connectivity to experience, it will require tighter collaboration between the various infrastructure elements that support application workloads. There are two philosophical approaches to how this orchestration might take place: through a tightly-integrated system, or through a more loose coupling of heterogeneous components. But how should architects make the choice between these approaches? The principles of architecture tend to be most vehemently argued by the vendors competing to sell the underlying solutions. IT vendors generally (and networking in particular) tend to turn these principle discussions into tit-for-tat FUD wars, arguing in absolution that one approach or another is the right way to go. But the ones who put their careers on the line when they select an architectural approach should understand more fully what drives specific architectural selections. The difference between tightly-integrated systems and more loosely federated components is really performance. Whenever two components come together, that boundary is defined by some interface. If you need to extract performance out of the coupled system, you have to make changes on one or both sides of said interface. As a vendor, if you can twiddle the bits on only one side, you can improve the overall system performance up to but not beyond whatever the other side can do. So when performance is the primary objective, you will tend to see solutions where both sides of that interface are owned (or at least controlled) by the same party. The ability to make changes on both sides of the interface is the only way to maximize performance. When the primary objective is not performance, you will see a generalized interface that sits between a decoupled pairing of solution components. Enter SDN. Or network virtualization. Or NFV. Or DevOps. When we talk about performance as an industry, we usually mean capacity and speed. But performance is more than bandwidth and latency. The whole reason any of the SDN technologies is emerging is to satisfy operational issues. Getting applications provisioned, monitored, troubleshot, billed, upgraded, and so on has taken over the top spot on the pain list for many companies. The question we ought to be asking is what are the operational performance requirements. The answer isn't black or white. What does performance even mean in an operational setting? It seems at least plausible that operational performance translates to things like the rate of change (think provisioning changes per second or call setup and teardown rates, for example) or the rate of polling (queries per second, as with monitoring or billing). For some environments, it might be that the scale of configuration management or data querying is quite high. Any company that is doing fine-grained monitoring or rapid state-based network changes, for example, might have very high operational performance requirements. Meanwhile, most normal networks will likely have a much lower performance bar. For the former, the objective has to be to eke out every bit of operational performance from the system. This will demand a more tightly-integrated solution. Both sides of the resource boundary (network and storage, as an example) might need to be within the same system, and the interface between them should appropriately be very specific to the implementation. For the latter, a more generalized interface between infrastructure elements should be more than sufficient. The primary goal is not to maximize performance but rather enable collaboration between components. In these architectures, the generalized interface is the most important thing as it will optimize choice and flexibility between the individual system elements. Both are absolutely valid use cases; there is no judgment in which is the more noble cause. But architects ought to be clear about what it is they are optimizing for. Selecting a generalized interface merely because it is open could be disastrous if it turns out that the performance requirements exceed what that interface provides. Conversely, selecting a tightly-integrated system might be more costly or limiting than is necessary if the real problem is orchestration rather than performance. So where do architects start? Everything starts with requirements. Is the objective to achieve a specific rate of change? Or is the objective merely to make tasks like provisioning and troubleshooting more coordinated across infrastructure silos? Are you planning to do anything exotic in terms of polling data on the system elements? Or are you expecting data to be accessed at a more casual rate? The real point here is that architects should start to express their orchestration requirements in terms of both capability and performance. We do this instinctively when we think about how we move bits back and forth, or how we access storage, or how we allot cycles on a server. But when it comes to management, because our collective capabilities have been so lacking, we have ignored performance. As SDN and other technologies continue to advance, operational performance will take on a more important role. And without knowing what the requirements are, designers will really be flying blind, making tradeoffs that might not even be necessary. [Today's fun fact: In ancient Rome, it was considered a sign of leadership to be born with a crooked nose. If Mike Tyson were born earlier, we'd call him Emperor.]

A quick overview on how to see your heaps in Eclipse.

We'll use a custom CXF interceptor to add these headers.

Apache Shiro can be used in a wide range of applications as part of the Java Security Framework.

there are a lot of design patterns, architectures and design concepts that can be applied to technical aspects of implementing a product. for exapmle, we have mvc architecture that isolates the view, controller and model of the application. we have a factory pattern that defines how to create objects and so on. yet, these all limit themselves to the technical aspects of the product, while there exists no definite pre-defined architecture or patterns to design and build "business functions". the plug and play architecture i want to describe in this article is a "functional architecture", that defines a pattern to design the business functionality to get the most out of it. modular architecture is a design technique where functionality of a program is separated into independent, interchangeable modules, such that each module contains everything necessary to execute one aspect of the desired functionality. typically, in modular architecture, separation is done based on technical aspects. for eg., a module is created for database interaction, another is created for logging and so on. the advantage of a modular architecture is that you can easily replace or add components without affecting the rest of the application. the modules clearly define the interfaces that are used to interact with the module and these are tied at compile time to other modules. what if we could extend this architecture to "business functionality" rather than just limiting it to technical aspects of a product. we already design "business functionality" in a modular manner. but, we do not consciously look at it as modules with clear-cut interfaces for inputs and outputs. defining the plug and play architecture the plug and play architecture extends the techniques of modular architecture to business functions. for eg., a module can be created to encompass all functions related to order, while another can be created to encompass all functions related to quote. i will call these independent modules implementing business functions as flows. an application, in this architecture, is defined as a "collection of flows working together". the same advantage of modular architecture applies here, i.e., you can replace, add or remove flows without affecting the rest of the application. an added advantage to extending modules to business functions, is that the flows can be combined in different paths to get different business functionality. further, extending this definition to be able to add, remove or replace modules at "runtime" rather than tying them together at compile time gives us the flexibility to customize the application at runtime without "in-built flags and if clauses". the beauty of this extension of the architecture is that "it gives us the flexibility to create a set of loosely coupled flows that can be bound together at runtime as opposed to coding one tightly integrated application that is rigid and not malleable". to further see how this architecture works, let us consider a set of loosely coupled flows coded as per the diagram: there are 5 flows coded and deployed. each flow exposes an output and accepts a set of inputs. for eg., the "product listing" flow encapsulates the following business functions create products manually by admin import products by admin provide services to view, search and list products expose an output, "product" data that can be linked to other flows another eg., is the cart flow. this encapsulates the following business functions create a new cart add items to the cart check out the cart expose an output, "cart" data that can be linked to other flows accept an input, "cart item" that can be linked to the output from other flows we can connect the output "product" data from "product listing" flow to the "cart item" input of the cart flow. this is not tightly coupled at compile time and is left as an open gate or an input for the cart flow. at runtime the gate is closed or left open as required by the customer. from the diagram we see that we have 5 gates that can be closed. leading us to create different applications at runtime by closing the correct gates. for eg., we can just enable "product listing" for a customer which provides just the basic features of product listing. gates 4 and 6 can be closed to create an application with the features create, list products add products to a cart send an enquiry for the products added to the cart in another variation instead of gates 4 and 6, gate 7 can be closed. the application now has the features create, list products send an enquiry for single products by plugging the "inputs" of flows with different "outputs" from other flows, varied applications can be created. this is called a plug and play architecture. smart is implemented in this architecture to provide a highly flexible container to build saas products. need for plug and play architecture when a product is exposed as a multi-tenanted saas product, it cannot be created with "one size fits all" tenet. to break this tenet we need a flexible, customizable runtime environment where features can be varied based on a customer requirement without affecting other customers serviced by the same application. the plug and play architecture provides this flexibility to the saas.

When you think of a Doctrine 2 DBAL Type you think of an atomic thing, but how can you work programmatically on this type without defining an event? A DBAL Type doesn't allow access to the Symfony 2 service container, you must use a hack. But before this let me explain the classic way (using events), why you should use this hack and why you shouldn't. The classic way is defined in the Symfony 2 Cookbook: How to Register Event Listeners and Subscribers Doctrine 2 events unlike Symfony 2 events aren't defined by the developer, the developer can only attach listeners on them. Why? Because Doctrine 2 isn't a framework that you can use for everything, persistence is its only job. When should you use this hack? When your stored object isn't a 1:1 representation of the PHP object and its elaboration can be memoizable or really fast. I use this hack for browscaps: with the BrowscapBundle I can convert from an user agent string to a stdClass object (like the get_browser function). Our object is container = $container; } public function prePersist(LifecycleEventArgs $args) { $this->doObjectToString($args); } public function postPersist(LifecycleEventArgs $args) { $this->doStringToObject($args); } public function preUpdate(LifecycleEventArgs $args) { $this->doObjectToString($args); } public function postUpdate(LifecycleEventArgs $args) { $this->doStringToObject($args); } public function postLoad(LifecycleEventArgs $args) { $this->doStringToObject($args); } private function doStringToObject($args) { $entity = $args->getEntity(); if ($entity instanceof Agent && !is_object($entity->getHeader())) { $browscap = $this->container->get('browscap'); $browser = $browscap->getBrowser($entity->getHeader()); $entity->setHeader($browser); } } private function doObjectToString($args) { $entity = $args->getEntity(); if ($entity instanceof Agent && is_object($entity->getHeader())) { $user_agent = $entity->getHeader()->browser_name; $entity->setHeader($user_agent); } } } With this code, everytime you will persist, update or extract a Agent entity from/to related storage system it'll be converted from string to object. The problem is that these callbacks will be invoked everytime and numerous events aren't recommended for your application. But with this hack I can write: services: acme.demo_bundle.event_listener.container_listener: arguments: - "@service_container" class: "Acme\DemoBundle\EventListener\ContainerListener" tags: - { name: doctrine.event_listener, event: getContainer } Doctrine ignores this event but it exists and results attached! container = $container; } public function getContainer() { return $this->container; } } This listener seems useless, but it's the only way for this hack because Doctrine 2 DBAL Type doesn't allow direct access to the service container but allows access to events listeners. getVarcharTypeDeclarationSQL($fieldDeclaration); } public function convertToPHPValue($value, AbstractPlatform $platform) { if (is_null($value)) { return null; } $listeners = $platform->getEventManager()->getListeners('getContainer'); $listener = array_shift($listeners); $container = $listener->getContainer(); return $container->get('browscap')->getBrowser($value); } public function convertToDatabaseValue($value, AbstractPlatform $platform) { if ($value instanceof Browscap) { return $value->getBrowser()->browser_name; } elseif ($value instanceof stdClass) { return $value->browser_name; } return $value; } public function getName() { return 'browscap'; } public function requiresSQLCommentHint(AbstractPlatform $platform) { return true; } } I use this hack to define only the events related to application flow (less events is better). Now that you know when you can use this, you must read why you shouldn't use it. Let me explain the reason with one simple example: imagine that one day PHP will allow external hooks in native classes constructor, how can you work without knowing what you're doing while initializing a new stdClass? The same reason here: everytime you extract a value from the database you want extract it fast (hopefully you'll extract more than one records), but how can you be sure that extraction is fast if every attribute of a single record depends on external libraries and logics? Quoting Ocramius, member of the Doctrine 2 development team: DBAL types are not designed for Dependency Injection. We explicitly avoided using DI for DBAL types because they have to stay simple. We’ve been asked many many times to change this behaviour, but doctrine believes that complex data manipulation should NOT happen within the very core of the persistence layer itself. That should be handled in your service layer.

One piece to Docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it. One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker! Generate The SSL Certificates We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx. # Create the CA Key and Certificate for signing Client Certs openssl genrsa -des3 -out ca.key 4096 openssl rsa -in ca.key -out ca.key # remove password! openssl req -new -x509 -days 365 -key ca.key -out ca.crt # Create the Server Key, CSR, and Certificate openssl genrsa -des3 -out server.key 1024 openssl rsa -in server.key -out server.key # remove password! openssl req -new -key server.key -out server.csr # We're self signing our own server cert here. This is a no-no in production. openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt # Create the Client Key and CSR openssl genrsa -des3 -out client.key 1024 openssl rsa -in client.key -out client.key # no password! openssl req -new -key client.key -out client.csr # Sign the client certificate with our CA cert. Unlike signing our own server cert, this is what we want to do. openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security. We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs. Setup Nginx The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default. upstream docker { server unix:/var/run/docker.sock fail_timeout=0; } server { listen 4242; server localhost; ssl on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; ssl_client_certificate /etc/nginx/certs/ca.crt; ssl_verify_client on; access_log on; error_log /dev/null; location / { proxy_pass http://docker; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 120; proxy_read_timeout 120; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else! Hack It Up! With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one. # Is normal http traffic denied? curl -v http://localhost:4242/info # How about https, sans client cert and key? curl -v -s -k https://localhost:4242/info # And the final good request! curl -v -s -k --key client.key --cert client.crt https://localhost:4242/info For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot! But wait there’s more… let’s build a container that can call the service to launch other containers! For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting). First, the node.js application that will connect and display information: https = require 'https' fs = require 'fs' options = host: 172.42.1.62 port: 4242 method: 'GET' path: '/containers/json' key: fs.readFileSync('ssl/client.key') cert: fs.readFileSync('ssl/client.crt') headers: { 'Accept': 'application/json'} # not required, but being semantic here! req = https.request options, (res) -> console.log res req.end() And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it! FROM shykes/nodejs MAINTAINER James R. Carr ADD ssl/client* /srv/app/ssl ADD package.json /srv/app/package.json ADD app.coffee /srv/app/app.coffee RUN cd /srv/app && npm install . CMD cd /srv/app && npm start That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!

Since git hooks can be any executable script with an appropriate #! line, Python is more than suitable for writing your git hooks. Simply stated, git hooks are scripts which are called at different points of time in the life cycle of working with your git repository. Let’s start by creating a new git repository: ~/work> git init git-hooks-exp Initialized empty Git repository in /home/gene/work/git-hooks-exp/.git/ ~/work> cd git-hooks-exp/ ~/work/git-hooks-exp (master)> tree -al .git/ .git/ ├── branches ├── config ├── description ├── HEAD ├── hooks │ ├── applypatch-msg.sample │ ├── commit-msg.sample │ ├── post-update.sample │ ├── pre-applypatch.sample │ ├── pre-commit.sample │ ├── prepare-commit-msg.sample │ ├── pre-rebase.sample │ └── update.sample ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags 9 directories, 12 files Inside the .git are a number of directories and files, one of them being hooks/ which is where the hooks live. By default, you will have a number of hooks with the file names ending in .sample. They may be useful as starting points for your own scripts. However, since they all have an extension .sample, none of the hooks are actually activated. For a hook to be activated, it must have the right file name and it should be executable. Let’s see how we can write a hook using Python. We will write a post-commit hook. This hook is called immediately after you have made a commit. We are going to do something fairly useless, but quite interesting in this hook. We will take the commit SHA1 of this commit, and print how it may look like in a more human form. I do the latter using the humanhash module. You will need to have it installed. Here is how the hook looks like: #!/usr/bin/python import subprocess import humanhash # get the last commit SHA and print it after humanizing it # https://github.com/zacharyvoase/humanhash print humanhash.humanize( subprocess.check_output( ['git','rev-parse','HEAD'])) I use the subprocess.check_output() function to execute the command git rev-parse HEAD so that I can get the commit SHA1 and then call the humanhash.humanize() function with it. Save the hook as a file, post-commit in your hooks/ directory and make it executable using chmod +x .git/hooks/post-commit. Let’s see the hook in action: ~/work/git-hooks-exp (master)> touch file ~/work/git-hooks-exp (master)> git add file ~/work/git-hooks-exp (master)> git commit -m "Added a file" carbon-network-connecticut-equal [master (root-commit) 2d7880b] Added a file 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 file The commit SHA1 for the commit turned out to be 2d7880be746a1c1e75844fc1aa161e2b8d955427. Let’s check it with the humanize function and check if we get the same message as above: >>> humanhash.humanize('2d7880be746a1c1e75844fc1aa161e2b8d955427') 'carbon-network-connecticut-equal' And you can see the same message above as well. For some of the hooks, you will see that they are called with some parameters. In Python you can access them using the sys.argv attribute from the sys module, with the first member being the name of the hook of course and the others will be the parameters that the hook is called with.

This blog post demonstrates how easy it is to use Apache Camel and its new json-path component along with the camel-sqs component to produce and consume messages on Amazon SQS. Amazon Web Services SQS is a message queuing “software as a service” (SaaS) in the cloud. To be able to use it, you need to sign up for AWS. It’s primary access mechanism is XML over HTTP through various AWS SDK clients provided by Amazon. Please check out the SQS documentation for more. And as “luck” would have it, one of the users in the Apache Camel community created a component to be able to integrate with SQS. This makes it trivial to add a producer or consumer to an SQS queue and plugs in nicely with the Camel DSL. SQS, however, is not a “one-size fits all” queueing service; you must be aware of your use case and make sure it fits (current requirements as well as somewhat into the future…). There are limitations that, if not studied and accounted for ahead of time, could come back to sink your project. An example of a viable alternative, and one that more closely fits the profile of a high performance and full featured message queue is Apache ActiveMQ. For example, one limitation to keep in mind is that unlike traditional JMS consumers, you cannot create a subscription to a queue that filters messages based on some predicate (at least not using the AWS-SQS API — you’d have to build that into your solution). Some other things to keep in mind when using SQS: The queue does not preserve FIFO messaging That is, message order is not preserved. They can arrive out of order from when they were sent. Apache Camel can help with its resequencer pattern. Bilgin Ibryam, now a colleague of mine at Red Hat, has written a great blog post about how to restore message order using the resequencer pattern. Message size is limited to 256K This is probably sufficient, but if your message sizes are variable, or contain more data that 256K, you will have to chunk them and send in smaller chunks. No selector or selective consumption If you’re familiar with JMS, you know that you can specify consumers to use a “selector” or a predicate expression that is evaluated on the broker side to determine whether or not a specific message should be dispatched to a specific consumer. For example, Durability constraints Some use cases call for the message broker to store messages until consumers return. SQS allows a limit of up to 14 days. This is most likely sufficient, but something to keep in mind. Binary payloads not allowed SQS only allows text-based messages, e.g., XML, JSON, fixed format text, etc. Binary such as Avro, Protocol Buffers, or Thrift are not allowed. For some of these limitations, you can work around them by building out the functionality yourself. I would always recommend taking a look at how an integration library like Apache Camel can help — which has out-of-the-box support for doing some of these things. Doing JMS-style selectors So the basic problem is we want to subscribe to a SQS queue, but we want to filter which messages we process. For those messages that we do not process, those should be left in the queue. To do this, we will make use of Apache Camel’s Filter EIP as well as the visibility timeouts available on the SQS queue. By default, SQS will dispatch all messages in its queue when it’s queried. We cannot change this, and thus not avoid the message being dispatched to us — we’ll have to do the filtering on our side (this is different than how a full-featured broker like ActiveMQ does it, i.e., filtering is done on the broker side so the consumer doesn’t even see the message it does not want to see). Once SQS dispatches a message, it does not remove it from the queue unless the consumer has acknowledged that it has it and is finished with it. The consumer does this by sending a DeleteMessage command. Until the DeleteMessage command is sent, the message is always in the queue, however visibility comes in to play here. When a message is dispatched to a consumer, there is a period of time which it will not be visible to other consumers. So if you browsed the queue, you would not see it (it should appear in the stats as “in-flight”). However, there is a configurable period of time you can specify for how long this “visibility timeout” should be active. So if you set the visibility to a lower time period (default is 30 seconds), you can more quickly get messages re-dispatched to consumers that would be able to handle the message. Take a look at the following Camel route which does just that: @Override public void configure() throws Exception { // every two seconds, send a message to the "demo" queue in SQS from("timer:kickoff?period=5000") .setBody().method(this, "generateJsonString") .to("aws-sqs://demo?amazonSQSClient=#sqsClient&defaultVisibilityTimeout=2"); } In the above Camel Route, we create a new message every 5 seconds and send it to an SQS queue named demo — note we set the defaultVisibilityTimeout to 2 seconds. This means that after a message gets dispatched to a consumer, SQS will wait about 2 seconds before considering it eligible to be dispatched to another consumer if it has not been deleted. On the consumer side, we take advantage of a couple Apache Camel conveniences Using JSON Path + Filter EIP Camel has an excellent new component named JSON-Path. Claus Ibsen tweeted about it when he hacked it up. This allows you to do Content-Based Routing on a JSON payload very easily by using XPath-style expressions to pick out and evaluate attributes in a JSON encoded object. So in the following example, we can test an attribute named ‘type’ to be equal to ‘LOGIN’ and use Camel’s Filter EIP to allow only those messages that match to go through and continue processing: public class ConsumerRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { from("aws-sqs://demo?amazonSQSClient=#sqsClient&deleteIfFiltered=false") .setHeader("identity").jsonpath("$['type']") .filter(simple("${header.identity} == 'login'")) .log("We have a message! ${body}") .to("file:target/output?fileName=login-message-${date:now:MMDDyy-HHmmss}.json"); } } To complete the functionality, we have to pay attention to a new configuration option added for the Camel-SQS component: deleteIfFiltered — Whether or not to send the DeleteMessage to the SQS queue if an exchange fails to get through a filter. If ‘false’ and exchange does not make it through a Camel filter upstream in the route, then don’t send DeleteMessage. By default, Camel will send the “DeleteMessage” command to SQS after a route has completed successfully (without an exception). However, in this case, we are specifying to not send the DeleteMessage command if the message had been previously filtered by Camel. This example demonstrates how easy it is to use Apache Camel and its new json-path component along with the camel-sqs component to produce and consume messages on Amazon SQS. Please take a look at the source code on my github repo to play with the live code and try it out yourself.

The examples in this post were updated in September to work with the current version of the Windows Azure Storage REST API. In the Windows Azure MSDN Azure Forum there are occasional questions about the Windows Azure Storage Services REST API. I have occasionally responded to these with some code examples showing how to use the API. I thought it would be useful to provide some examples of using the REST API for tables, blobs and queues – if only so I don’t have to dredge up examples when people ask how to use it. This post is not intended to provide a complete description of the REST API. The REST API is comprehensively documented (other than the lack of working examples). Since the REST API is the definitive way to address Windows Azure Storage Services I think people using the higher level Storage Client API should have a passing understanding of the REST API to the level of being able to understand the documentation. Understanding the REST API can provide a deeper understanding of why the Storage Client API behaves the way it does. Fiddler The Fiddler Web Debugging Proxy is an essential tool when developing using the REST (or Storage Client) API since it captures precisely what is sent over the wire to the Windows Azure Storage Services. Authorization Nearly every request to the Windows Azure Storage Services must be authenticated. The exception is access to blobs with public read access. The supported authentication schemes for blobs, queues and tables and these are described here. The requests must be accompanied by an Authorization header constructed by making a hash-based message authentication code using the SHA-256 hash. The following is an example of performing the SHA-256 hash for the Authorization header: public static String CreateAuthorizationHeader(String canonicalizedString) { String signature = String.Empty; using (HMACSHA256 hmacSha256 = new HMACSHA256( Convert.FromBase64String(storageAccountKey) )) { Byte[] dataToHmac = System.Text.Encoding.UTF8.GetBytes(canonicalizedString); signature = Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac)); } String authorizationHeader = String.Format( CultureInfo.InvariantCulture, "{0} {1}:{2}", AzureStorageConstants.SharedKeyAuthorizationScheme, AzureStorageConstants.Account, signature ); return authorizationHeader; } This method is used in all the examples in this post. AzureStorageConstants is a helper class containing various constants. Key is a secret key for Windows Azure Storage Services account specified by Account. In the examples given here, SharedKeyAuthorizationScheme is SharedKey. The trickiest part in using the REST API successfully is getting the correct string to sign. Fortunately, in the event of an authentication failure the Blob Service and Queue Service responds with the authorization string they used and this can be compared with the authorization string used in generating the Authorization header. This has greatly simplified the us of the REST API. Table Service API The Table Service API supports the following table-level operations: Create Table Delete Table Query Tables The Table Service API supports the following entity-level operations: Delete Entity Insert Entity Merge Entity Update Entity Query Entities These operations are implemented using the appropriate HTTP VERB: DELETE – delete GET – query MERGE – merge POST – insert PUT – update This section provides examples of the Insert Entity and Query Entities operations. Insert Entity The InsertEntity() method listed in this section inserts an entity with two String properties, Artist and Title, into a table. The entity is submitted as an ATOM entry in the body of a request POSTed to the Table Service. In this example, the ATOM entry is generated by the GetRequestContentInsertXml() method. The date must be in RFC 1123 format in the x-ms-date header supplied to the canonicalized resource used to create the Authorization string. Note that the storage service version is set to “2012-02-12″ which requires the DataServiceVersion and MaxDataServiceVersion to be set appropriately. public void InsertEntity(String tableName, String artist, String title) { String requestMethod = "POST"; String urlPath = tableName; String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String contentMD5 = String.Empty; String contentType = "application/atom+xml"; String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n{1}\n{2}\n{3}\n{4}", requestMethod, contentMD5, contentType, dateInRfc1123Format, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); UTF8Encoding utf8Encoding = new UTF8Encoding(); Byte[] content = utf8Encoding.GetBytes(GetRequestContentInsertXml(artist, title)); Uri uri = new Uri(AzureStorageConstants.TableEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Accept = "application/atom+xml,application/xml"; request.ContentLength = content.Length; request.ContentType = contentType; request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.Headers.Add("Accept-Charset", "UTF-8"); request.Headers.Add("DataServiceVersion", "2.0;NetFx"); request.Headers.Add("MaxDataServiceVersion", "2.0;NetFx"); using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(content, 0, content.Length); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (StreamReader reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } } private String GetRequestContentInsertXml(String artist, String title) { String defaultNameSpace = "http://www.w3.org/2005/Atom"; String dataservicesNameSpace = "http://schemas.microsoft.com/ado/2007/08/dataservices"; String metadataNameSpace = "http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"; XmlWriterSettings xmlWriterSettings = new XmlWriterSettings(); xmlWriterSettings.OmitXmlDeclaration = false; xmlWriterSettings.Encoding = Encoding.UTF8; StringBuilder entry = new StringBuilder(); using (XmlWriter xmlWriter = XmlWriter.Create(entry)) { xmlWriter.WriteProcessingInstruction("xml", "version=\"1.0\" encoding=\"UTF-8\""); xmlWriter.WriteWhitespace("\n"); xmlWriter.WriteStartElement("entry", defaultNameSpace); xmlWriter.WriteAttributeString("xmlns", "d", null, dataservicesNameSpace); xmlWriter.WriteAttributeString("xmlns", "m", null, metadataNameSpace); xmlWriter.WriteElementString("title", null); xmlWriter.WriteElementString("updated", String.Format("{0:o}", DateTime.UtcNow)); xmlWriter.WriteStartElement("author"); xmlWriter.WriteElementString("name", null); xmlWriter.WriteEndElement(); xmlWriter.WriteElementString("id", null); xmlWriter.WriteStartElement("content"); xmlWriter.WriteAttributeString("type", "application/xml"); xmlWriter.WriteStartElement("properties", metadataNameSpace); xmlWriter.WriteElementString("PartitionKey", dataservicesNameSpace, artist); xmlWriter.WriteElementString("RowKey", dataservicesNameSpace, title); xmlWriter.WriteElementString("Artist", dataservicesNameSpace, artist); xmlWriter.WriteElementString("Title", dataservicesNameSpace, title + "\n" + title); xmlWriter.WriteEndElement(); xmlWriter.WriteEndElement(); xmlWriter.WriteEndElement(); xmlWriter.Close(); } String requestContent = entry.ToString(); return requestContent; } This generates the following request (as captured by Fiddler): POST https://STORAGE_ACCOUNT.table.core.windows.net/authors HTTP/1.1 Accept: application/atom+xml,application/xml Content-Type: application/atom+xml x-ms-date: Sun, 08 Sep 2013 06:31:12 GMT x-ms-version: 2012-02-12 Authorization: SharedKey STORAGE_ACCOUNT:w7Uu4wHZx4fFwa2bsxd/TJVZZ1AqMPwxvW+pYtoWHd0= Accept-Charset: UTF-8 DataServiceVersion: 2.0;NetFx MaxDataServiceVersion: 2.0;NetFx Host: STORAGE_ACCOUNT.table.core.windows.net Content-Length: 514 Expect: 100-continue Connection: Keep-Alive The body of the request is: 2013-09-08T07:19:07Z Beckett Molloy 2013-09-08T07:19:07.2189243Z Beckett Molloy Molloy Note that I should have URLEncoded the PartitionKey and RowKey but did not do so for simplicity. There are, in fact, some issues with the URL encoding of spaces and other symbols. Get Entity The GetEntity() method described in this section retrieves the single entity inserted in the previous section. The particular entity to be retrieved is identified directly in the URL. public void GetEntity(String tableName, String partitionKey, String rowKey) { String requestMethod = "GET"; String urlPath = String.Format("{0}(PartitionKey='{1}',RowKey='{2}')", tableName, partitionKey, rowKey); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n{2}", requestMethod, dateInRfc1123Format, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.TableEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.Headers.Add("Accept-Charset", "UTF-8"); request.Accept = "application/atom+xml,application/xml"; request.Headers.Add("DataServiceVersion", "2.0;NetFx"); request.Headers.Add("MaxDataServiceVersion", "2.0;NetFx"); using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (StreamReader reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } } This generates the following request (as captured by Fiddler): GET https://STORAGE_ACCOUNT.table.core.windows.net/authors(PartitionKey='Beckett',RowKey='Molloy') HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:31:14 GMT x-ms-version: 2012-02-12 Authorization: SharedKey STORAGE_ACCOUNT:1hWbr4aNq4JWCpNJY3rsLH1SkIyeFTJflbqyKMPQ1Gk= Accept-Charset: UTF-8 Accept: application/atom+xml,application/xml DataServiceVersion: 2.0;NetFx MaxDataServiceVersion: 2.0;NetFx Host: STORAGE_ACCOUNT.table.core.windows.net The Table Service generates the following response: HTTP/1.1 200 OK Cache-Control: no-cache Content-Type: application/atom+xml;charset=utf-8 ETag: W/"datetime'2013-09-08T06%3A31%3A14.1579056Z'" Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: f4bd4c77-6fb6-42a8-8dff-81ea8d28fa2e x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:31:15 GMT Content-Length: 1108 The returned entities, in this case a single entity, are returned in ATOM entry format in the response body: https://STORAGE_ACCOUNT.table.core.windows.net/authors(PartitionKey='Beckett',RowKey='Molloy') 2013-09-08T06:31:15Z Beckett Molloy 2013-09-08T06:31:14.1579056Z Beckett Molloy Molloy Blob Service API The Blob Service API supports the following account-level operation: List Containers The Blob Service API supports the following container-level operation: Create Container Delete Container Get Container ACL Get Container Properties Get Container Metadata List Blobs Set Container ACL Set Container Metadata The Blob Service API supports the following blob-level operation: Copy Blob Delete Blob Get Blob Get Blob Metadata Get Blob Properties Lease Blob Put Blob Set Blob Metadata Set Blob Properties Snapshot Blob The Blob Service API supports the following operations on block blobs: Get Block List Put Block Put Block List The Blob Service API supports the following operations on page blobs: Get Page Regions Put Page This section provides examples of the Put Blob and Lease Blob operations. Put Blob The Blob Service and Queue Service use a different form of shared-key authentication from the Table Service so care should be taken in creating the string to be signed for authorization. The blob type, BlockBlob or PageBlob, must be specified as a request header and consequently appears in the authorization string. public void PutBlob(String containerName, String blobName) { String requestMethod = "PUT"; String urlPath = String.Format("{0}/{1}", containerName, blobName); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String content = "Andrew Carnegie was born in Dunfermline"; UTF8Encoding utf8Encoding = new UTF8Encoding(); Byte[] blobContent = utf8Encoding.GetBytes(content); Int32 blobLength = blobContent.Length; const String blobType = "BlockBlob"; String canonicalizedHeaders = String.Format( "x-ms-blob-type:{0}\nx-ms-date:{1}\nx-ms-version:{2}", blobType, dateInRfc1123Format, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n\n\n\n\n\n\n\n\n{2}\n{3}", requestMethod, blobLength, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.BlobEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-blob-type", blobType); request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.ContentLength = blobLength; using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(blobContent, 0, blobLength); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { String ETag = response.Headers["ETag"]; } } This generates the following request: PUT https://STORAGE_ACCOUNT.blob.core.windows.net/fife/dunfermline HTTP/1.1 x-ms-blob-type: BlockBlob x-ms-date: Sun, 08 Sep 2013 06:28:29 GMT x-ms-version: 2012-02-12 Authorization: SharedKey STORAGE_ACCOUNT:ntvh/lamVmikvwHhy6vRVBIh87kibkPlEOiHyLDia6g= Host: STORAGE_ACCOUNT.blob.core.windows.net Content-Length: 39 Expect: 100-continue Connection: Keep-Alive The body of the request is: Andrew Carnegie was born in Dunfermline The Blob Service generates the following response: HTTP/1.1 201 Created Transfer-Encoding: chunked Content-MD5: RYJnWGXLyt94l5jG82LjBw== Last-Modified: Sun, 08 Sep 2013 06:28:31 GMT ETag: "0x8D07A73C5704A86" Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: b74ef0a2-294d-4581-b8f1-6cda724bbdbf x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:28:30 GMT Lease Blob The Blob Service allows a user to lease a blob for a minute at a time and so acquire a write lock on it. The use case for this is the locking of a page blob used to store the VHD backing an writeable Azure Drive. The LeaseBlob() example in this section demonstrates a subtle issue with the creation of authorization strings. The URL has a query string, comp=lease. Rather than using this directly in creating the authorization string it must be converted into comp:lease with a colon replacing the equal symbol – see modifiedURL in the example. Furthermore, the Lease Blob operation requires the use of an x-ms-lease-action to indicate whether the lease is being acquired, renewed, released or broken. public void LeaseBlob(String containerName, String blobName) { String requestMethod = "PUT"; String urlPath = String.Format("{0}/{1}?comp=lease", containerName, blobName); String modifiedUrlPath = String.Format("{0}/{1}\ncomp:lease", containerName, blobName); const Int32 contentLength = 0; String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String leaseAction = "acquire"; String leaseDuration = "60"; String canonicalizedHeaders = String.Format( "x-ms-date:{0}\nx-ms-lease-action:{1}\nx-ms-lease-duration:{2}\nx-ms-version:{3}", dateInRfc1123Format, leaseAction, leaseDuration, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, modifiedUrlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n\n\n\n\n\n\n\n\n{2}\n{3}", requestMethod, contentLength, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.BlobEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-lease-action", leaseAction); request.Headers.Add("x-ms-lease-duration", leaseDuration); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.ContentLength = contentLength; using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { String leaseId = response.Headers["x-ms-lease-id"]; } } This generates the following request: PUT https://STORAGE_ACCOUNT.blob.core.windows.net/fife/dunfermline?comp=lease HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:28:31 GMT x-ms-lease-action: acquire x-ms-lease-duration: 60 x-ms-version: 2012-02-12 Authorization: SharedKey rebus:+SQ5+RFZg3hUaws5XCRHxsDgXb1ycdRIz5EKyHJWP7s= Host: rebus.blob.core.windows.net Content-Length: 0 The Blob Service generates the following response: HTTP/1.1 201 Created Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: 4b6ff77f-f885-4f74-803a-c92920d225c3 x-ms-version: 2012-02-12 x-ms-lease-id: b1320c2c-65ad-41d6-a7bd-85a4242c0ac5 Date: Sun, 08 Sep 2013 06:28:31 GMT Content-Length: 0 Queue Service API The Queue Service API supports the following queue-level operation: List Queues The Queue Service API supports the following queue-level operation: Create Queue Delete Queue Get Queue Metadata Set Queue Metadata The Queue Service API supports the following message-level operations: Clear Messages Delete Message Get Messages Peek Messages Put Message This section provides examples of the Put Message and Get Message operations. Put Message The most obvious curiosity about Put Message is that it uses the HTTP verb POST rather than PUT. The issue is presumably the interaction of the English language and the HTTP standard which states that PUT should be idempotent and that the Put Message operation is clearly not since each invocation merely adds another message to the queue. Regardless, it did catch me out when I failed to read the documentation well enough – so take that as a warning. The content of a message posted to the queue must be formatted in a specified XML schema and must then be UTF8 encoded. public void PutMessage(String queueName, String message) { String requestMethod = "POST"; String urlPath = String.Format("{0}/messages", queueName); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String messageText = String.Format( "{0}", message); UTF8Encoding utf8Encoding = new UTF8Encoding(); Byte[] messageContent = utf8Encoding.GetBytes(messageText); Int32 messageLength = messageContent.Length; String canonicalizedHeaders = String.Format( "x-ms-date:{0}\nx-ms-version:{1}", dateInRfc1123Format, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n\n\n\n\n\n\n\n\n{2}\n{3}", requestMethod, messageLength, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.QueueEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.ContentLength = messageLength; using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(messageContent, 0, messageLength); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { String requestId = response.Headers["x-ms-request-id"]; } } This generates the following request: POST https://rebus.queue.core.windows.net/revolution/messages HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:34:08 GMT x-ms-version: 2012-02-12 Authorization: SharedKey rebus:nyASTVWifnxHKnj2wXwuzzzXz5CxUBZj58SToV5QFK8= Host: rebus.queue.core.windows.net Content-Length: 76 Expect: 100-continue Connection: Keep-Alive The body of the request is: Saturday in the cafe The Queue Service generates the following response: HTTP/1.1 201 Created Server: Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: 14c6e73b-15d9-480c-b251-c4c01b48e529 x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:34:09 GMT Content-Length: 0 Get Messages The Get Messages operation described in this section retrieves a single message with the default message visibility timeout of 30 seconds. public void GetMessage(String queueName) { string requestMethod = "GET"; String urlPath = String.Format("{0}/messages", queueName); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String canonicalizedHeaders = String.Format( "x-ms-date:{0}\nx-ms-version:{1}", dateInRfc1123Format, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n\n\n\n\n\n\n\n\n\n{1}\n{2}", requestMethod, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.QueueEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.Accept = "application/atom+xml,application/xml"; using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (StreamReader reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } } This generates the following request: GET https://rebus.queue.core.windows.net/revolution/messages HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:34:11 GMT x-ms-version: 2012-02-12 Authorization: SharedKey rebus:K67XooYhokw0i0AlCzYQ4GeLLrJih1r1vSqiO9DBo0c= Accept: application/atom+xml,application/xml Host: rebus.queue.core.windows.net The Queue Service generates the following response: HTTP/1.1 200 OK Content-Type: application/xml Server: Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: efb21a86-7d66-47fd-b13d-7aa74fce0568 x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:34:12 GMT Content-Length: 484 The message is returned in the response body as follows: 05fd902f-6031-4ef4-8298-ef3844ec3bc6Sun, 08 Sep 2013 06:34:11 GMTSun, 15 Sep 2013 06:34:11 GMT1AgAAAAMAAAAAAAAAAL+zgF2szgE=Sun, 08 Sep 2013 06:34:43 GMTSaturday in the cafe I noticed that some newline specifiers in strings (\n) were lost when the blog was auto-ported from Windows Live Spaces to WordPress. I have put them back in but it is possible I missed some. Consequently, in the event of a problem you should check the newlines in canonicalizedHeaders and stringToSign.

Previous posts such as this one have shown using embedded Jetty to REST-enable a standalone Java program. Those posts were lacking an important feature for real applications: packaging into a JAR so the application will run outside of Eclipse and won’t be dependent on Maven and jetty:run. To make this happen, we will use Maven to build an executable JAR that also includes all of the Jetty and Spring dependencies we need. The goal of this work is to get to the point where we can run the example application by: Cloning the Git repository. Running mvn package. Running java -jar target/webmvc-standalone.jar When I started adding the necessary bits to the pom.xml file of my sample application, I expected a relatively straightforward solution. I ended up with a relatively straightforward solution that was completely different from what I expected. So I think it’s worth a detailed discussion of how this solution works and what Maven is doing for us. Our desire to make an executable JAR is complicated by the fact that we want our Maven project to build a WAR as a default package, so that we can use this code in a Java web container if desired. Additionally, we introduce some complexity by making a single JAR with all dependencies, because that causes files in the Spring JARs to collide. I’ll show what I did to address each of these. Build both JAR and WAR The basic idea here is that we want Maven to make both a JAR file and a WAR file during the “package” phase. Our pom.xml file specifies war as the packaging for this project, so the WAR file will be created as expected. We need to add the JAR file without disturbing this. I found a great post here that got me started. The basic idea is to add the following to pom.xml under build/plugins: org.apache.maven.plugins maven-jar-plugin 2.4 package-jar package jar This is the behavior we would get for “free” if we used jar packaging inpom.xml. The execution section ties it to the package phase so that it runs during the default build process. The jar goal tells the plugin what to make. This gets us a basic JAR with the classes in the normal place for a JAR (rather than in WEB-INF/classes as they must be in the WAR file). At the same time, we need to deal with the fact that the Maven resources plugin considers only src/main/resources to be a resources directory, while in our case we have files in src/main/webapp that also need to be included. We want to copy these resources to the target directory so the JAR plugin will pick them up. (This is an important distinction; the typical Maven question, “how do I include extra resources in my JAR?” should really be “how do I get extra resources into target so the JAR plugin will pick them up?”) We add this to the build section of pom.xml: src/main/resources src/main/webapp This causes our new webmvc.jar file to include the HTML, JavaScript, etc. required for our embedded Jetty webapp. JAR with dependencies Next, we make an additional JAR that has the correct Main-Class entry in theMANIFEST.MF file and includes the necessary dependencies so we only have to ship one file. This is done using the Maven assembly plugin. The assembly plugin does repackaging only; that’s why we had to add a JAR artifact above. Without that JAR artifact to work from, the assembly plugin repackages the WAR, and we end up with classes in WEB-INF/classes. This causes Java to complain that it can’t find our main class when we try to run the JAR. The assembly plugin comes with a jar-with-dependencies configuration that can be used simply by adding it as a descriptorRef to the relevant section of pom.xml, as shown in this StackOverflow question. However, this configuration doesn’t work in our particular case, as the Spring dependencies we need have files with overlapping names. As a result, we need to make our own assembly configuration. Fortunately, this is pretty simple. We first add this to the build/plugins section of pom.xml: org.apache.maven.plugins maven-assembly-plugin 2.4 src/assemble/distribution.xml org.anvard.webmvc.server.EmbeddedServer package single As before, we use the executions section to make sure this is run automaticaly during package. We also specify the main class for our application. Finally, we point the plugin to our assembly configuration file, which lives in src/assemble. I present the assembly configuration below, but first we need to talk about the issue with the Spring JARs that made this custom assembly necessary. Spring schemas and handlers With this sample application, we use Spring WebMVC to provide a REST API for ordinary Java classes, as discussed in this post. The Spring code we use is spread across a few different JARs. Recent versions of Spring added a “custom XML namespace” feature that allows the contents of a Spring XML configuration file to be very extensible. Spring WebMVC, and other Spring libraries, use this feature to provide custom XML tags. In order to parse the XML file with these custom tags, Spring needs to be able to match these custom namespaces to handlers. To do this, Spring expects to find files called spring.handlers andspring.schemas in the META-INF directory of any JAR providing a Spring custom namespace. Several of the Spring JARs used by this application include thosespring.handlers and spring.schemas files. Of course, each JAR only includes its own handlers and schemas. When the Maven assembly plugin uses the jar-with-dependencies configuration, only one copy of those files “wins” and makes it into the executable JAR. We really just need a single spring.handlers and spring.schemas that are the concatentation of the respective files. There is probably some Maven magic to accomplish this, but I elected to do it manually as my Bash-fu is much greater than my Maven-fu. I added two files to the src/assemble directory that have the combined contents of the various files in the Spring JARs. Maven assembly configuration The assembly file looks like this: standalone jar true META-INF/spring.handlers META-INF/spring.schemas src/assemble/spring.handlers /META-INF false src/assemble/spring.schemas /META-INF false The id will be used to name this assembly. The baseDirectory tells the assembly plugin that the pieces it assembles should go at the root of the new JAR. (Otherwise they would go into a directory using the project name, in this case “webapp”.) The next two sections are important. We want to exclude thespring.handlers and spring.schemas from the Spring JARs (a.k.a. the dependency set). Instead, we want to explicitly include them from oursrc/assemble directory, and put them into the right place. We also want the assembly plugin to unpack the dependency set JARs so we wind up with Java class files in our new JAR, rather than just JAR-files-inside-JAR-file, which would not run correctly. Notice that there is no directive telling Spring to include all dependencies from the dependency set, including transitive dependencies. This is the default so we don’t need to specify it. It’s also the default to include the unpacked files from our own artifact (webmvc.jar) into the new JAR. Conclusion A real-world application would probably pick either WAR packaging or executable JAR packaging, and be simpler. Additionally, it would be possible to use multiple Maven modules to build a JAR and embed it in the WAR. But it’s interesting to see how to implement a more complex solution that builds everything we need from a single project.

With these changes, we can access the REST API equally well fromhttp://:9999 and https://:9998.

qunit , used by projects like jquery and jquery mobile , is a rather popular javascript testing framework. for tests written using qunit, how do we measure its code coverage ? a possible solution which is quite easy to setup is to leverage the deadly combination of karma and istanbul . just like our previous adventure with jasmine code coverage , let's take a look at some simple code we need to test. this function my.sqrt is a reimplementation of math.sqrt which may throw an exception if the input is invalid. var my = { sqrt: function(x) { if (x < 0) throw new error("sqrt can't work on negative number"); return math.exp(math.log(x)/2); } }; a very simple qunit-based test for the above code is as follows. test("sqrt", function() { deepequal(my.sqrt(4), 2, "square root of 4 is 2"); }); manually running the test is easy as opening the test runner in a web browser: for a smoothed development workflow, an automated way to run the tests will be much preferred. this is where karma becomes very useful. karma also has the ability to launch a predetermined collection of browsers, or even to use phantomjs for a pure headless execution (suitable for smoke testing and/or continuous delivery). before we can use karma, installation is necessary: npm install karma karma-qunit karma-coverage karma requires a configuration file. for this purpose, the config file is very simple. as an illustration, the execution is done by phantomjs but it is easy to include other browsers as well. module.exports = function(config) { config.set({ basepath: '', frameworks: ['qunit'], files: [ '*.js', 'test/spec/*.js' ], browsers: ['phantomjs'], singlerun: true, reporters: ['progress', 'coverage'], preprocessors: { '*.js': ['coverage'] } }); }; now you can start karma with the above configuration, it would say that the test passes just fine. should you encounter some problems, you can look at an example repository i have setup github.com/ariya/coverage-qunit-istanbul-karma , it may be useful as a starting point or a reference for your own project. as a convenience, the test in that repository can be executed via npm test . what is more interesting here is that karma runs its coverage processor, as indicated by preprocessors in the above configuration. karma will run istanbul , a full-featured instrumenter and coverage tracker. essentially, istanbul grabs the original javascript source and injects extra instrumentation code so that it can gather the execution metrics once the process finishes (read also my previous blog post on javascript code coverage with istanbul ). in this karma and istanbul combo, the generated coverage report is available in the under the subdirectory coverage . the above report indicates that the single test for my.sqrt is still missing the test for an invalid input, thanks to branch coverage feature of istanbul. the i indicator next to the conditional statement tells us that the if branch was never taken. of course, once the issue is known, adding another test which will cover that branch is easy (left as an exercise for the reader). now that code coverage is tracker, perhaps you are ready for the next level? it is about setting the hard threshold so that future coverage regression will never happen. protect yourself and your team from carelessness, overconfidence, or honest mistakes!

for modern web application development, having dozens of unit tests is not enough anymore. the actual code coverage of those tests would reveal if the application is thoroughly stressed or not. for tests written using the famous jasmine test library, an easy way to have the coverage report is via istanbul and karma . for this example, let’s assume that we have a simple library sqrt.js which contains an alternative implementation of math.sqrt . note also how it will throw an exception instead of returning nan for an invalid input. var my = { sqrt: function(x) { if (x < 0) throw new error("sqrt can't work on negative number"); return math.exp(math.log(x)/2); } }; using jasmine placed under test/lib/jasmine-1.3.1 , we can craft a test runner that includes the following spec: describe("sqrt", function() { it("should compute the square root of 4 as 2", function() { expect(my.sqrt(4)).toequal(2); }); }); opening the spec runner in a web browser will give the expected outcome: so far so good. now let's see how the code coverage of our test setup can be measured. the first order of business is to install karma . if you are not familiar with karma, it is basically a test runner which can launch and connect to a specific set of web browsers, run your tests, and then gather the report. using node.js, what we need to do is: npm install karma karma-coverage before launching karma, we need to specify its configuration . it could be as simple as the following my.conf.js (most entries are self-explained). note that the tests are executed using phantomjs for simplicity, it is however quite trivial to add other web browsers such as chrome and firefox. module.exports = function(config) { config.set({ basepath: '', frameworks: ['jasmine'], files: [ '*.js', 'test/spec/*.js' ], browsers: ['phantomjs'], singlerun: true, reporters: ['progress', 'coverage'], preprocessors: { '*.js': ['coverage'] } }); }; running the tests, as well as performing code coverage at the same time, can be triggered via: node_modules/.bin/karma start my.conf.js which will dump the output like: info [karma]: karma v0.10.2 server started at http://localhost:9876/ info [launcher]: starting browser phantomjs info [phantomjs 1.9.2 (linux)]: connected on socket n9ndnhj0np92ntspgx-x phantomjs 1.9.2 (linux): executed 1 of 1 success (0.029 secs / 0.003 secs) as expected (from the previous manual invocation of the spec runner), the test passed just fine. however, the most particular interesting piece here is the code coverage report, it is stored (in the default location) under the subdirectory coverage . open the report in your favorite browser and there you'll find the coverage analysis report. behind the scene, karma is using istanbul , a comprehensive javascript code coverage tool (read also my previous blog post on javascript code coverage with istanbul ). istanbul parses the source file, in this example sqrt.js , using esprima and then adds some extra instrumentation which will be used to gather the execution statistics. the above report that you see is one of the possible outputs, istanbul can also generate lcov report which is suitable for many continuous integration systems (jenkins, teamcity, etc). an extensive analysis of the coverage data should also prevent any future coverage regression, check out my other post hard thresholds on javascript code coverage . one important thing about code coverage is branch coverage . if you pay attention carefully, our test above is still not exercising the situation where the input to my.sqrt is negative. there is a big "i" marking in the third-line of the code, this is istanbul telling us that the if branch is not taken at all (for the else branch, it will be an "e" marker). once this missing branch is noticed, improving the situation is as easy as adding one more test to the spec: it("should throw an exception if given a negative number", function() { expect(function(){ my.sqrt(-1); }). tothrow(new error("sqrt can't work on negative number")); }); once the test is executed again, the code coverage report looks way better and everyone is happy. if you have some difficulties following the above step-by-step instructions, take a look at a git repository i have prepared: github.com/ariya/coverage-jasmine-istanbul-karma . feel free to play with it and customize it to suit your workflow!

I’m going to start by discussing the Spring WebMVC configuration and move on from there in future posts.

One of the problems with memory-mapped files is that you can’t actually map beyond the end of the file. So you can’t use that to extend your file. I had a thought about and set out to check out what happens when I create a sparse file, a file that only take space when you write to it, and at the same time, map it. As it turns out, this actually works pretty well in practice. You can do so without any issues. Here is how it works: using (var f = File.Create(path)) { int bytesReturned = 0; var nativeOverlapped = new NativeOverlapped(); if (!NativeMethod.DeviceIoControl(f.SafeFileHandle, EIoControlCode.FsctlSetSparse, IntPtr.Zero, 0, IntPtr.Zero, 0, ref bytesReturned, ref nativeOverlapped)) { throw new Win32Exception(); } f.SetLength(1024*1024*1024*64L); } This creates a sparse file that is 64 GB in size. Then we can map it normally: using (var mmf = MemoryMappedFile.CreateFromFile(path)) using (var memoryMappedViewAccessor = mmf.CreateViewAccessor(0, 1024*1024*1024*64L)) { for (long i = 0; i < memoryMappedViewAccessor.Capacity; i += buffer.Length) { memoryMappedViewAccessor.WriteArray(i, buffer, 0, buffer.Length); } } And then we can do stuff to it. And that includes writing to yet-unallocated parts of the file. This also means that you don’t have to worry about writing past the end of the file, the OS will take care of all of that for you. Happy happy, joy joy, etc. There is one problem with this method, however. It means that you have a 64 GB file, but you don’t have that much allocated. What that means in turn is that you might not have that much space available for the file. Which brings up an interesting question, what happens when you are trying to commit a new page, and the disk is out of space? Using file I/O you would get an I/O error with the right code. But when using memory mapped files, the error would actually turn up during access, which can happen pretty much anywhere. It also means that it is a Standard Exception Handling error in Windows, which requires special treatment. To test this out, I wrote the following so it would write to a disk that had only about 50 GB free. I wanted to know what would happen when it ran out of space. That is actually something that happens, and we need to be able to address this issue robustly. The kicker is that this might actually happen at any time, so that would really result is some… interesting behavior with regards to robustness. In other words, I don’t think that this is a viable option, it is a really cool trick, but I don’t think it is a very well thought out option. By the way, the result of my experiment was that we had an effectively a frozen process. No errors, nothing, just a hung. Also, I am pretty sure that WriteArray() is really slow, but I’ll check this out at another pointer in time.

ElasticSearch provides Java API, thus it executes all operations asynchronously by using client object.

My last couple of blogs have talked about long polling and Spring's DeferredResult technique and to demonstrate these concepts I've shoehorned the code from my Producer Consumer project into a web application. Although the code demonstrates the points made by the blogs it does contain a large number of holes in its logic. Apart from the fact that in a real application you wouldn't use a simple LinkedBlockingQueue, but would choose JMS or some other industrial-strength messaging service, and the fact that only one user can get a hold of the match updates, there's also the problem that it spawns badly behaved threads that don't close down when the JVM terminates. You may wonder why this should be a problem… well to you, as a developer, it isn't really a problem at all, it's only a little bit of sloppy programming. But, to one of your operations guys it can make life unnecessarily difficult. The reason for this is that if you have too many badly behaved threads then typing Tomcat's shutdown.sh command will have very little effect and you have to savagely kill your web server by typing something like: ps -ef | grep java to get the pid and then kill -9 <> …and when you have a field of Tomcat web servers to restart all this extra kerfuffle that becomes a severe pain. When you type shutdown.sh you want Tomcat to stop. In my last couple of blogs the badly behaved threads I created had the following run() methods with the first of these, shown below, being really badly behaved: @Override public void run() { while (true) { try { DeferredResult result = resultQueue.take(); Message message = queue.take(); result.setResult(message); } catch (InterruptedException e) { throw new UpdateException("Cannot get latest update. " + e.getMessage(), e); } } } In this code I've used an infinite while(true), which means that the thread will just keep running and never terminate. @Override public void run() { sleep(5); // Sleep to allow the reset of the app to load logger.info("The match has now started..."); long now = System.currentTimeMillis(); List matchUpdates = match.getUpdates(); for (Message message : matchUpdates) { delayUntilNextUpdate(now, message.getTime()); logger.info("Add message to queue: {}", message.getMessageText()); queue.add(message); } start = true; // Game over, can restart logger.warn("GAME OVER"); } The second example, above, is also pretty badly behaved. It'll keep taking messages from MatchUpdates list and adding them to the message queue at the appropriate moment. Their only saving grace is that they may throw an InterruptedException, which if handled correctly would cause thread termination; however, this cannot be guaranteed. There's a quick fix for this, really… all you need to do is to ensure that any threads you create are daemon threads. The definition of a daemon thread is a thread that doesn't prevent the JVM from exiting when the program finishes but, the thread is still running. The usual example of a daemon thread is the JVM's garbage collection thread. To turn your threads into daemon threads you simply call: thread.setDaemon(true); ...and when you type shutdown.sh then, WHAM, all your threads will disappear. There is, however, a problem with this. What if one of your daemon's threads was doing something important and chopping it down in its prime, resulting in the loss of some pretty important data? What you need to do is to ensure that all your threads shut down gracefully, completing any work they may be currently undertaking. The rest of this blog demonstrates a fix for these errant threads, gracefully coordinating their shutdown by using a ShutdownHook. According to the documentation, a "shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently." So, after reading the last sentence you may have guessed that what you need to do is to create a thread that has the responsibility of shutting down all your other threads and is passed to the JVM as a shutdown hook. All of this can be generically implemented in a couple of small classes and by performing some jiggery-pokery on your existing thread run() methods. The two classes to create are a ShutdownService and a Hook. The Hook class, which I'll demonstrate first, is used to link the ShutdownService to your threads. The code for Hook is as follows: public class Hook { private static final Logger logger = LoggerFactory.getLogger(Hook.class); private boolean keepRunning = true; private final Thread thread; Hook(Thread thread) { this.thread = thread; } /** * @return True if the daemon thread is to keep running */ public boolean keepRunning() { return keepRunning; } /** * Tell the client daemon thread to shutdown and wait for it to close gracefully. */ public void shutdown() { keepRunning = false; thread.interrupt(); try { thread.join(); } catch (InterruptedException e) { logger.error("Error shutting down thread with hook", e); } } } The Hook contains two instance variables: keepRunning and thread. thread is a reference to the thread that this instance of Hook is responsible for shutting down, while keepRunning tells the thread to… keep running. Hook has two public methods: keepRunning() and shutdown(). keepRunning() is called by the thread to figure out whether it should keep running, and shutdown() is called by the ShutdownService's shutdown hook thread to get your thread to shut down. This is the most interesting of the two methods. Firstly it sets the keepRunning variable to false. It then calls thread.interrupt() to interrupt the thread forcing it to throw an InterruptedException. Lastly, it calls thread.join() and waits for the thread instance to shutdown. Note that this technique relies on all your threads cooperating. If there's one badly behaved thread in the mix, then the whole thing could hang. To get around this problem add a timeout to thread.join(…). @Service public class ShutdownService { private static final Logger logger = LoggerFactory.getLogger(ShutdownService.class); private final List hooks; public ShutdownService() { logger.debug("Creating shutdown service"); hooks = new ArrayList(); createShutdownHook(); } /** * Protected for testing */ @VisibleForTesting protected void createShutdownHook() { ShutdownDaemonHook shutdownHook = new ShutdownDaemonHook(); Runtime.getRuntime().addShutdownHook(shutdownHook); } protected class ShutdownDaemonHook extends Thread { /** * Loop and shutdown all the daemon threads using the hooks * * @see java.lang.Thread#run() */ @Override public void run() { logger.info("Running shutdown sync"); for (Hook hook : hooks) { hook.shutdown(); } } } /** * Create a new instance of the hook class */ public Hook createHook(Thread thread) { thread.setDaemon(true); Hook retVal = new Hook(thread); hooks.add(retVal); return retVal; } @VisibleForTesting List getHooks() { return hooks; } } The ShutdownService is a Spring service that contains a list of Hook classes, and therefore by inference threads, that it is responsible for shutting down. It also contains an inner class ShutdownDaemonHook, which extends Thread. An instance of ShutdownDaemonHook is created during the construction of ShutdownService, which is then passed to the JVM as a shutdown hook by calling: Runtime.getRuntime().addShutdownHook(shutdownHook); The ShutdownService has one public method: createHook(). The first thing that this class does is to ensure that any thread passed to it is converted into a daemon thread. It then creates a new Hook instance, passing in the thread as the argument, before finally both storing the result in a list and returning it to the caller. The only thing left to do now is to integrate the ShutdownService into DeferredResultService and MatchReporter, the two classes that contain the badly behaved threads. @Service("DeferredService") public class DeferredResultService implements Runnable { private static final Logger logger = LoggerFactory.getLogger(DeferredResultService.class); private final BlockingQueue> resultQueue = new LinkedBlockingQueue<>(); private Thread thread; private volatile boolean start = true; @Autowired private ShutdownService shutdownService; private Hook hook; @Autowired @Qualifier("theQueue") private LinkedBlockingQueue queue; @Autowired @Qualifier("BillSkyes") private MatchReporter matchReporter; public void subscribe() { logger.info("Starting server"); matchReporter.start(); startThread(); } private void startThread() { if (start) { synchronized (this) { if (start) { start = false; thread = new Thread(this, "Studio Teletype"); hook = shutdownService.createHook(thread); thread.start(); } } } } @Override public void run() { logger.info("DeferredResultService - Thread running"); while (hook.keepRunning()) { try { DeferredResult result = resultQueue.take(); Message message = queue.take(); result.setResult(message); } catch (InterruptedException e) { System.out.println("Interrupted when waiting for latest update. " + e.getMessage()); } } System.out.println("DeferredResultService - Thread ending"); } public void getUpdate(DeferredResult result) { resultQueue.add(result); } } The first change to this class was to autowire in the Shutdown service instance. The next thing to do is to use the ShutdownService to create an instance of Hook after the creation of the thread but before thread.start() is called: thread = new Thread(this, "Studio Teletype"); hook = shutdownService.createHook(thread); thread.start(); The final change is to replace while(true) with: while (hook.keepRunning()) { … telling the thread when to quit the while loop and shutdown. You may have also noticed that there are a few System.out.println() calls thrown into the above code. There is a reason for this and it's because of the undetermined order in which the shutdown hook threads are executed. Remember that not only are your classes trying to shutdown gracefully, but other sub-systems and shutting down too. This means that my original code, which called logger.info(…) failed throwing the following exception: Exception in thread "Studio Teletype" java.lang.NoClassDefFoundError: org/apache/log4j/spi/ThrowableInformation at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:159) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:382) at com.captaindebug.longpoll.service.DeferredResultService.run(DeferredResultService.java:75) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.spi.ThrowableInformation at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559) ... 6 more This is because the logger has already been unloaded when I try to call it; hence the failure.Again, as the documentation states: "Shutdown hooks run at a delicate time in the life cycle of a virtual machine and should therefore be coded defensively. They should, in particular, be written to be thread-safe and to avoid deadlocks insofar as possible. They should also not rely blindly upon services that may have registered their own shutdown hooks and therefore may themselves in the process of shutting down. Attempts to use other thread-based services such as the AWT event-dispatch thread, for example, may lead to deadlocks." The MatchReport class has some very similar modifications. The major difference is that the hook.keepRunning() code is inside the run() method's for loop. public class MatchReporter implements Runnable { private static final Logger logger = LoggerFactory.getLogger(MatchReporter.class); private final Match match; private final Queue queue; private volatile boolean start = true; @Autowired private ShutdownService shutdownService; private Hook hook; public MatchReporter(Match theBigMatch, Queue queue) { this.match = theBigMatch; this.queue = queue; } /** * Called by Spring after loading the context. Will "kick off" the match... */ public void start() { if (start) { synchronized (this) { if (start) { start = false; logger.info("Starting the Match Reporter..."); String name = match.getName(); Thread thread = new Thread(this, name); hook = shutdownService.createHook(thread); thread.start(); } } } else { logger.warn("Game already in progress"); } } /** * The main run loop */ @Override public void run() { sleep(5); // Sleep to allow the reset of the app to load logger.info("The match has now started..."); long now = System.currentTimeMillis(); List matchUpdates = match.getUpdates(); for (Message message : matchUpdates) { delayUntilNextUpdate(now, message.getTime()); if (!hook.keepRunning()) { break; } logger.info("Add message to queue: {}", message.getMessageText()); queue.add(message); } start = true; // Game over, can restart logger.warn("GAME OVER"); } private void sleep(int deplay) { try { TimeUnit.SECONDS.sleep(10); } catch (InterruptedException e) { logger.info("Sleep interrupted..."); } } private void delayUntilNextUpdate(long now, long messageTime) { while (System.currentTimeMillis() < now + messageTime) { try { Thread.sleep(100); } catch (InterruptedException e) { logger.info("MatchReporter Thread interrupted..."); } } } } The ultimate test of this code is to issue a Tomcat shutdown.sh command half way through the match update sequence. As the JVM terminates it'll call the shutdown hook from the ShutdownDaemonHook class. As this class's run() method executes it loops throughout the list of Hook instances telling them to close down their respective threads. If you tail -f your server's log file (in my case catalina.out, but your Tomcat maybe configured differently to mine), you'll see the trail of entries shutting your server shutdown gracefully.

Intro If you want to manage your SQL Databases in Azure using tools that you’re a little more familiar and comfortable with – for example – SQL Management Studio, how do you go about connecting? You could read the help article from Microsoft, or you can follow my intuitive screen-based instructions, below: Assumptions 1. I’m assuming you have a version of SQL Management Studio already installed. I believe you’ll need at least SQL Server 2008 R2’s version or newer 2. I’m further assuming you’ve already created a SQL Database in Azure Steps to Connect SSMS to SQL Azure 1. Authenticate to the Azure Portal 2. Click on SQL Databases 3. Click on Servers 4. Click on the name of the Server you wish to connect to… 5. Click on Configure… If not already in place, click on ‘Add to the allowed IP addresses’ to add your current IP address (or specify an address you wish to connect from) and click ‘Save’ 6. Open SQL Management Studio and connect to Database services (usually comes up by default) Enter the fully qualified server name (.database.windows.net) Change to SQL Server Authentication Enter the login preferred (if a new database, the username you specified when yuo created the DB server) Enter the correct password 7. Hit the Connect button Troubleshooting Ensure you have the appropriate ports open outbound from your local network or connection (typically port 1433) Ensure you have allowed the correct public IP address you’re trying to connect from via the Azure Portal (steps 1-5 above) Ensure you are using the correct server name and user name For SSMS, this is the server name (in step 4) followed by .database.windows.net Ensure you are using SQL Server Authentication For SSMS the username format is If you forgot the password of your username, you can reset the password in the Azure Portal, in step 4, click on Dashboard: Lastly… You can click on the Database (in step 2) to see your connection options: