Apache Shiro can be used in a wide range of applications as part of the Java Security Framework.

there are a lot of design patterns, architectures and design concepts that can be applied to technical aspects of implementing a product. for exapmle, we have mvc architecture that isolates the view, controller and model of the application. we have a factory pattern that defines how to create objects and so on. yet, these all limit themselves to the technical aspects of the product, while there exists no definite pre-defined architecture or patterns to design and build "business functions". the plug and play architecture i want to describe in this article is a "functional architecture", that defines a pattern to design the business functionality to get the most out of it. modular architecture is a design technique where functionality of a program is separated into independent, interchangeable modules, such that each module contains everything necessary to execute one aspect of the desired functionality. typically, in modular architecture, separation is done based on technical aspects. for eg., a module is created for database interaction, another is created for logging and so on. the advantage of a modular architecture is that you can easily replace or add components without affecting the rest of the application. the modules clearly define the interfaces that are used to interact with the module and these are tied at compile time to other modules. what if we could extend this architecture to "business functionality" rather than just limiting it to technical aspects of a product. we already design "business functionality" in a modular manner. but, we do not consciously look at it as modules with clear-cut interfaces for inputs and outputs. defining the plug and play architecture the plug and play architecture extends the techniques of modular architecture to business functions. for eg., a module can be created to encompass all functions related to order, while another can be created to encompass all functions related to quote. i will call these independent modules implementing business functions as flows. an application, in this architecture, is defined as a "collection of flows working together". the same advantage of modular architecture applies here, i.e., you can replace, add or remove flows without affecting the rest of the application. an added advantage to extending modules to business functions, is that the flows can be combined in different paths to get different business functionality. further, extending this definition to be able to add, remove or replace modules at "runtime" rather than tying them together at compile time gives us the flexibility to customize the application at runtime without "in-built flags and if clauses". the beauty of this extension of the architecture is that "it gives us the flexibility to create a set of loosely coupled flows that can be bound together at runtime as opposed to coding one tightly integrated application that is rigid and not malleable". to further see how this architecture works, let us consider a set of loosely coupled flows coded as per the diagram: there are 5 flows coded and deployed. each flow exposes an output and accepts a set of inputs. for eg., the "product listing" flow encapsulates the following business functions create products manually by admin import products by admin provide services to view, search and list products expose an output, "product" data that can be linked to other flows another eg., is the cart flow. this encapsulates the following business functions create a new cart add items to the cart check out the cart expose an output, "cart" data that can be linked to other flows accept an input, "cart item" that can be linked to the output from other flows we can connect the output "product" data from "product listing" flow to the "cart item" input of the cart flow. this is not tightly coupled at compile time and is left as an open gate or an input for the cart flow. at runtime the gate is closed or left open as required by the customer. from the diagram we see that we have 5 gates that can be closed. leading us to create different applications at runtime by closing the correct gates. for eg., we can just enable "product listing" for a customer which provides just the basic features of product listing. gates 4 and 6 can be closed to create an application with the features create, list products add products to a cart send an enquiry for the products added to the cart in another variation instead of gates 4 and 6, gate 7 can be closed. the application now has the features create, list products send an enquiry for single products by plugging the "inputs" of flows with different "outputs" from other flows, varied applications can be created. this is called a plug and play architecture. smart is implemented in this architecture to provide a highly flexible container to build saas products. need for plug and play architecture when a product is exposed as a multi-tenanted saas product, it cannot be created with "one size fits all" tenet. to break this tenet we need a flexible, customizable runtime environment where features can be varied based on a customer requirement without affecting other customers serviced by the same application. the plug and play architecture provides this flexibility to the saas.

One piece to Docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it. One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker! Generate The SSL Certificates We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx. # Create the CA Key and Certificate for signing Client Certs openssl genrsa -des3 -out ca.key 4096 openssl rsa -in ca.key -out ca.key # remove password! openssl req -new -x509 -days 365 -key ca.key -out ca.crt # Create the Server Key, CSR, and Certificate openssl genrsa -des3 -out server.key 1024 openssl rsa -in server.key -out server.key # remove password! openssl req -new -key server.key -out server.csr # We're self signing our own server cert here. This is a no-no in production. openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt # Create the Client Key and CSR openssl genrsa -des3 -out client.key 1024 openssl rsa -in client.key -out client.key # no password! openssl req -new -key client.key -out client.csr # Sign the client certificate with our CA cert. Unlike signing our own server cert, this is what we want to do. openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security. We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs. Setup Nginx The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default. upstream docker { server unix:/var/run/docker.sock fail_timeout=0; } server { listen 4242; server localhost; ssl on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; ssl_client_certificate /etc/nginx/certs/ca.crt; ssl_verify_client on; access_log on; error_log /dev/null; location / { proxy_pass http://docker; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 120; proxy_read_timeout 120; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else! Hack It Up! With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one. # Is normal http traffic denied? curl -v http://localhost:4242/info # How about https, sans client cert and key? curl -v -s -k https://localhost:4242/info # And the final good request! curl -v -s -k --key client.key --cert client.crt https://localhost:4242/info For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot! But wait there’s more… let’s build a container that can call the service to launch other containers! For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting). First, the node.js application that will connect and display information: https = require 'https' fs = require 'fs' options = host: 172.42.1.62 port: 4242 method: 'GET' path: '/containers/json' key: fs.readFileSync('ssl/client.key') cert: fs.readFileSync('ssl/client.crt') headers: { 'Accept': 'application/json'} # not required, but being semantic here! req = https.request options, (res) -> console.log res req.end() And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it! FROM shykes/nodejs MAINTAINER James R. Carr ADD ssl/client* /srv/app/ssl ADD package.json /srv/app/package.json ADD app.coffee /srv/app/app.coffee RUN cd /srv/app && npm install . CMD cd /srv/app && npm start That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!

Since git hooks can be any executable script with an appropriate #! line, Python is more than suitable for writing your git hooks. Simply stated, git hooks are scripts which are called at different points of time in the life cycle of working with your git repository. Let’s start by creating a new git repository: ~/work> git init git-hooks-exp Initialized empty Git repository in /home/gene/work/git-hooks-exp/.git/ ~/work> cd git-hooks-exp/ ~/work/git-hooks-exp (master)> tree -al .git/ .git/ ├── branches ├── config ├── description ├── HEAD ├── hooks │ ├── applypatch-msg.sample │ ├── commit-msg.sample │ ├── post-update.sample │ ├── pre-applypatch.sample │ ├── pre-commit.sample │ ├── prepare-commit-msg.sample │ ├── pre-rebase.sample │ └── update.sample ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags 9 directories, 12 files Inside the .git are a number of directories and files, one of them being hooks/ which is where the hooks live. By default, you will have a number of hooks with the file names ending in .sample. They may be useful as starting points for your own scripts. However, since they all have an extension .sample, none of the hooks are actually activated. For a hook to be activated, it must have the right file name and it should be executable. Let’s see how we can write a hook using Python. We will write a post-commit hook. This hook is called immediately after you have made a commit. We are going to do something fairly useless, but quite interesting in this hook. We will take the commit SHA1 of this commit, and print how it may look like in a more human form. I do the latter using the humanhash module. You will need to have it installed. Here is how the hook looks like: #!/usr/bin/python import subprocess import humanhash # get the last commit SHA and print it after humanizing it # https://github.com/zacharyvoase/humanhash print humanhash.humanize( subprocess.check_output( ['git','rev-parse','HEAD'])) I use the subprocess.check_output() function to execute the command git rev-parse HEAD so that I can get the commit SHA1 and then call the humanhash.humanize() function with it. Save the hook as a file, post-commit in your hooks/ directory and make it executable using chmod +x .git/hooks/post-commit. Let’s see the hook in action: ~/work/git-hooks-exp (master)> touch file ~/work/git-hooks-exp (master)> git add file ~/work/git-hooks-exp (master)> git commit -m "Added a file" carbon-network-connecticut-equal [master (root-commit) 2d7880b] Added a file 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 file The commit SHA1 for the commit turned out to be 2d7880be746a1c1e75844fc1aa161e2b8d955427. Let’s check it with the humanize function and check if we get the same message as above: >>> humanhash.humanize('2d7880be746a1c1e75844fc1aa161e2b8d955427') 'carbon-network-connecticut-equal' And you can see the same message above as well. For some of the hooks, you will see that they are called with some parameters. In Python you can access them using the sys.argv attribute from the sys module, with the first member being the name of the hook of course and the others will be the parameters that the hook is called with.

This blog post demonstrates how easy it is to use Apache Camel and its new json-path component along with the camel-sqs component to produce and consume messages on Amazon SQS. Amazon Web Services SQS is a message queuing “software as a service” (SaaS) in the cloud. To be able to use it, you need to sign up for AWS. It’s primary access mechanism is XML over HTTP through various AWS SDK clients provided by Amazon. Please check out the SQS documentation for more. And as “luck” would have it, one of the users in the Apache Camel community created a component to be able to integrate with SQS. This makes it trivial to add a producer or consumer to an SQS queue and plugs in nicely with the Camel DSL. SQS, however, is not a “one-size fits all” queueing service; you must be aware of your use case and make sure it fits (current requirements as well as somewhat into the future…). There are limitations that, if not studied and accounted for ahead of time, could come back to sink your project. An example of a viable alternative, and one that more closely fits the profile of a high performance and full featured message queue is Apache ActiveMQ. For example, one limitation to keep in mind is that unlike traditional JMS consumers, you cannot create a subscription to a queue that filters messages based on some predicate (at least not using the AWS-SQS API — you’d have to build that into your solution). Some other things to keep in mind when using SQS: The queue does not preserve FIFO messaging That is, message order is not preserved. They can arrive out of order from when they were sent. Apache Camel can help with its resequencer pattern. Bilgin Ibryam, now a colleague of mine at Red Hat, has written a great blog post about how to restore message order using the resequencer pattern. Message size is limited to 256K This is probably sufficient, but if your message sizes are variable, or contain more data that 256K, you will have to chunk them and send in smaller chunks. No selector or selective consumption If you’re familiar with JMS, you know that you can specify consumers to use a “selector” or a predicate expression that is evaluated on the broker side to determine whether or not a specific message should be dispatched to a specific consumer. For example, Durability constraints Some use cases call for the message broker to store messages until consumers return. SQS allows a limit of up to 14 days. This is most likely sufficient, but something to keep in mind. Binary payloads not allowed SQS only allows text-based messages, e.g., XML, JSON, fixed format text, etc. Binary such as Avro, Protocol Buffers, or Thrift are not allowed. For some of these limitations, you can work around them by building out the functionality yourself. I would always recommend taking a look at how an integration library like Apache Camel can help — which has out-of-the-box support for doing some of these things. Doing JMS-style selectors So the basic problem is we want to subscribe to a SQS queue, but we want to filter which messages we process. For those messages that we do not process, those should be left in the queue. To do this, we will make use of Apache Camel’s Filter EIP as well as the visibility timeouts available on the SQS queue. By default, SQS will dispatch all messages in its queue when it’s queried. We cannot change this, and thus not avoid the message being dispatched to us — we’ll have to do the filtering on our side (this is different than how a full-featured broker like ActiveMQ does it, i.e., filtering is done on the broker side so the consumer doesn’t even see the message it does not want to see). Once SQS dispatches a message, it does not remove it from the queue unless the consumer has acknowledged that it has it and is finished with it. The consumer does this by sending a DeleteMessage command. Until the DeleteMessage command is sent, the message is always in the queue, however visibility comes in to play here. When a message is dispatched to a consumer, there is a period of time which it will not be visible to other consumers. So if you browsed the queue, you would not see it (it should appear in the stats as “in-flight”). However, there is a configurable period of time you can specify for how long this “visibility timeout” should be active. So if you set the visibility to a lower time period (default is 30 seconds), you can more quickly get messages re-dispatched to consumers that would be able to handle the message. Take a look at the following Camel route which does just that: @Override public void configure() throws Exception { // every two seconds, send a message to the "demo" queue in SQS from("timer:kickoff?period=5000") .setBody().method(this, "generateJsonString") .to("aws-sqs://demo?amazonSQSClient=#sqsClient&defaultVisibilityTimeout=2"); } In the above Camel Route, we create a new message every 5 seconds and send it to an SQS queue named demo — note we set the defaultVisibilityTimeout to 2 seconds. This means that after a message gets dispatched to a consumer, SQS will wait about 2 seconds before considering it eligible to be dispatched to another consumer if it has not been deleted. On the consumer side, we take advantage of a couple Apache Camel conveniences Using JSON Path + Filter EIP Camel has an excellent new component named JSON-Path. Claus Ibsen tweeted about it when he hacked it up. This allows you to do Content-Based Routing on a JSON payload very easily by using XPath-style expressions to pick out and evaluate attributes in a JSON encoded object. So in the following example, we can test an attribute named ‘type’ to be equal to ‘LOGIN’ and use Camel’s Filter EIP to allow only those messages that match to go through and continue processing: public class ConsumerRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { from("aws-sqs://demo?amazonSQSClient=#sqsClient&deleteIfFiltered=false") .setHeader("identity").jsonpath("$['type']") .filter(simple("${header.identity} == 'login'")) .log("We have a message! ${body}") .to("file:target/output?fileName=login-message-${date:now:MMDDyy-HHmmss}.json"); } } To complete the functionality, we have to pay attention to a new configuration option added for the Camel-SQS component: deleteIfFiltered — Whether or not to send the DeleteMessage to the SQS queue if an exchange fails to get through a filter. If ‘false’ and exchange does not make it through a Camel filter upstream in the route, then don’t send DeleteMessage. By default, Camel will send the “DeleteMessage” command to SQS after a route has completed successfully (without an exception). However, in this case, we are specifying to not send the DeleteMessage command if the message had been previously filtered by Camel. This example demonstrates how easy it is to use Apache Camel and its new json-path component along with the camel-sqs component to produce and consume messages on Amazon SQS. Please take a look at the source code on my github repo to play with the live code and try it out yourself.

The examples in this post were updated in September to work with the current version of the Windows Azure Storage REST API. In the Windows Azure MSDN Azure Forum there are occasional questions about the Windows Azure Storage Services REST API. I have occasionally responded to these with some code examples showing how to use the API. I thought it would be useful to provide some examples of using the REST API for tables, blobs and queues – if only so I don’t have to dredge up examples when people ask how to use it. This post is not intended to provide a complete description of the REST API. The REST API is comprehensively documented (other than the lack of working examples). Since the REST API is the definitive way to address Windows Azure Storage Services I think people using the higher level Storage Client API should have a passing understanding of the REST API to the level of being able to understand the documentation. Understanding the REST API can provide a deeper understanding of why the Storage Client API behaves the way it does. Fiddler The Fiddler Web Debugging Proxy is an essential tool when developing using the REST (or Storage Client) API since it captures precisely what is sent over the wire to the Windows Azure Storage Services. Authorization Nearly every request to the Windows Azure Storage Services must be authenticated. The exception is access to blobs with public read access. The supported authentication schemes for blobs, queues and tables and these are described here. The requests must be accompanied by an Authorization header constructed by making a hash-based message authentication code using the SHA-256 hash. The following is an example of performing the SHA-256 hash for the Authorization header: public static String CreateAuthorizationHeader(String canonicalizedString) { String signature = String.Empty; using (HMACSHA256 hmacSha256 = new HMACSHA256( Convert.FromBase64String(storageAccountKey) )) { Byte[] dataToHmac = System.Text.Encoding.UTF8.GetBytes(canonicalizedString); signature = Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac)); } String authorizationHeader = String.Format( CultureInfo.InvariantCulture, "{0} {1}:{2}", AzureStorageConstants.SharedKeyAuthorizationScheme, AzureStorageConstants.Account, signature ); return authorizationHeader; } This method is used in all the examples in this post. AzureStorageConstants is a helper class containing various constants. Key is a secret key for Windows Azure Storage Services account specified by Account. In the examples given here, SharedKeyAuthorizationScheme is SharedKey. The trickiest part in using the REST API successfully is getting the correct string to sign. Fortunately, in the event of an authentication failure the Blob Service and Queue Service responds with the authorization string they used and this can be compared with the authorization string used in generating the Authorization header. This has greatly simplified the us of the REST API. Table Service API The Table Service API supports the following table-level operations: Create Table Delete Table Query Tables The Table Service API supports the following entity-level operations: Delete Entity Insert Entity Merge Entity Update Entity Query Entities These operations are implemented using the appropriate HTTP VERB: DELETE – delete GET – query MERGE – merge POST – insert PUT – update This section provides examples of the Insert Entity and Query Entities operations. Insert Entity The InsertEntity() method listed in this section inserts an entity with two String properties, Artist and Title, into a table. The entity is submitted as an ATOM entry in the body of a request POSTed to the Table Service. In this example, the ATOM entry is generated by the GetRequestContentInsertXml() method. The date must be in RFC 1123 format in the x-ms-date header supplied to the canonicalized resource used to create the Authorization string. Note that the storage service version is set to “2012-02-12″ which requires the DataServiceVersion and MaxDataServiceVersion to be set appropriately. public void InsertEntity(String tableName, String artist, String title) { String requestMethod = "POST"; String urlPath = tableName; String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String contentMD5 = String.Empty; String contentType = "application/atom+xml"; String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n{1}\n{2}\n{3}\n{4}", requestMethod, contentMD5, contentType, dateInRfc1123Format, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); UTF8Encoding utf8Encoding = new UTF8Encoding(); Byte[] content = utf8Encoding.GetBytes(GetRequestContentInsertXml(artist, title)); Uri uri = new Uri(AzureStorageConstants.TableEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Accept = "application/atom+xml,application/xml"; request.ContentLength = content.Length; request.ContentType = contentType; request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.Headers.Add("Accept-Charset", "UTF-8"); request.Headers.Add("DataServiceVersion", "2.0;NetFx"); request.Headers.Add("MaxDataServiceVersion", "2.0;NetFx"); using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(content, 0, content.Length); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (StreamReader reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } } private String GetRequestContentInsertXml(String artist, String title) { String defaultNameSpace = "http://www.w3.org/2005/Atom"; String dataservicesNameSpace = "http://schemas.microsoft.com/ado/2007/08/dataservices"; String metadataNameSpace = "http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"; XmlWriterSettings xmlWriterSettings = new XmlWriterSettings(); xmlWriterSettings.OmitXmlDeclaration = false; xmlWriterSettings.Encoding = Encoding.UTF8; StringBuilder entry = new StringBuilder(); using (XmlWriter xmlWriter = XmlWriter.Create(entry)) { xmlWriter.WriteProcessingInstruction("xml", "version=\"1.0\" encoding=\"UTF-8\""); xmlWriter.WriteWhitespace("\n"); xmlWriter.WriteStartElement("entry", defaultNameSpace); xmlWriter.WriteAttributeString("xmlns", "d", null, dataservicesNameSpace); xmlWriter.WriteAttributeString("xmlns", "m", null, metadataNameSpace); xmlWriter.WriteElementString("title", null); xmlWriter.WriteElementString("updated", String.Format("{0:o}", DateTime.UtcNow)); xmlWriter.WriteStartElement("author"); xmlWriter.WriteElementString("name", null); xmlWriter.WriteEndElement(); xmlWriter.WriteElementString("id", null); xmlWriter.WriteStartElement("content"); xmlWriter.WriteAttributeString("type", "application/xml"); xmlWriter.WriteStartElement("properties", metadataNameSpace); xmlWriter.WriteElementString("PartitionKey", dataservicesNameSpace, artist); xmlWriter.WriteElementString("RowKey", dataservicesNameSpace, title); xmlWriter.WriteElementString("Artist", dataservicesNameSpace, artist); xmlWriter.WriteElementString("Title", dataservicesNameSpace, title + "\n" + title); xmlWriter.WriteEndElement(); xmlWriter.WriteEndElement(); xmlWriter.WriteEndElement(); xmlWriter.Close(); } String requestContent = entry.ToString(); return requestContent; } This generates the following request (as captured by Fiddler): POST https://STORAGE_ACCOUNT.table.core.windows.net/authors HTTP/1.1 Accept: application/atom+xml,application/xml Content-Type: application/atom+xml x-ms-date: Sun, 08 Sep 2013 06:31:12 GMT x-ms-version: 2012-02-12 Authorization: SharedKey STORAGE_ACCOUNT:w7Uu4wHZx4fFwa2bsxd/TJVZZ1AqMPwxvW+pYtoWHd0= Accept-Charset: UTF-8 DataServiceVersion: 2.0;NetFx MaxDataServiceVersion: 2.0;NetFx Host: STORAGE_ACCOUNT.table.core.windows.net Content-Length: 514 Expect: 100-continue Connection: Keep-Alive The body of the request is: 2013-09-08T07:19:07Z Beckett Molloy 2013-09-08T07:19:07.2189243Z Beckett Molloy Molloy Note that I should have URLEncoded the PartitionKey and RowKey but did not do so for simplicity. There are, in fact, some issues with the URL encoding of spaces and other symbols. Get Entity The GetEntity() method described in this section retrieves the single entity inserted in the previous section. The particular entity to be retrieved is identified directly in the URL. public void GetEntity(String tableName, String partitionKey, String rowKey) { String requestMethod = "GET"; String urlPath = String.Format("{0}(PartitionKey='{1}',RowKey='{2}')", tableName, partitionKey, rowKey); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n{2}", requestMethod, dateInRfc1123Format, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.TableEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.Headers.Add("Accept-Charset", "UTF-8"); request.Accept = "application/atom+xml,application/xml"; request.Headers.Add("DataServiceVersion", "2.0;NetFx"); request.Headers.Add("MaxDataServiceVersion", "2.0;NetFx"); using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (StreamReader reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } } This generates the following request (as captured by Fiddler): GET https://STORAGE_ACCOUNT.table.core.windows.net/authors(PartitionKey='Beckett',RowKey='Molloy') HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:31:14 GMT x-ms-version: 2012-02-12 Authorization: SharedKey STORAGE_ACCOUNT:1hWbr4aNq4JWCpNJY3rsLH1SkIyeFTJflbqyKMPQ1Gk= Accept-Charset: UTF-8 Accept: application/atom+xml,application/xml DataServiceVersion: 2.0;NetFx MaxDataServiceVersion: 2.0;NetFx Host: STORAGE_ACCOUNT.table.core.windows.net The Table Service generates the following response: HTTP/1.1 200 OK Cache-Control: no-cache Content-Type: application/atom+xml;charset=utf-8 ETag: W/"datetime'2013-09-08T06%3A31%3A14.1579056Z'" Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: f4bd4c77-6fb6-42a8-8dff-81ea8d28fa2e x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:31:15 GMT Content-Length: 1108 The returned entities, in this case a single entity, are returned in ATOM entry format in the response body: https://STORAGE_ACCOUNT.table.core.windows.net/authors(PartitionKey='Beckett',RowKey='Molloy') 2013-09-08T06:31:15Z Beckett Molloy 2013-09-08T06:31:14.1579056Z Beckett Molloy Molloy Blob Service API The Blob Service API supports the following account-level operation: List Containers The Blob Service API supports the following container-level operation: Create Container Delete Container Get Container ACL Get Container Properties Get Container Metadata List Blobs Set Container ACL Set Container Metadata The Blob Service API supports the following blob-level operation: Copy Blob Delete Blob Get Blob Get Blob Metadata Get Blob Properties Lease Blob Put Blob Set Blob Metadata Set Blob Properties Snapshot Blob The Blob Service API supports the following operations on block blobs: Get Block List Put Block Put Block List The Blob Service API supports the following operations on page blobs: Get Page Regions Put Page This section provides examples of the Put Blob and Lease Blob operations. Put Blob The Blob Service and Queue Service use a different form of shared-key authentication from the Table Service so care should be taken in creating the string to be signed for authorization. The blob type, BlockBlob or PageBlob, must be specified as a request header and consequently appears in the authorization string. public void PutBlob(String containerName, String blobName) { String requestMethod = "PUT"; String urlPath = String.Format("{0}/{1}", containerName, blobName); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String content = "Andrew Carnegie was born in Dunfermline"; UTF8Encoding utf8Encoding = new UTF8Encoding(); Byte[] blobContent = utf8Encoding.GetBytes(content); Int32 blobLength = blobContent.Length; const String blobType = "BlockBlob"; String canonicalizedHeaders = String.Format( "x-ms-blob-type:{0}\nx-ms-date:{1}\nx-ms-version:{2}", blobType, dateInRfc1123Format, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n\n\n\n\n\n\n\n\n{2}\n{3}", requestMethod, blobLength, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.BlobEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-blob-type", blobType); request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.ContentLength = blobLength; using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(blobContent, 0, blobLength); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { String ETag = response.Headers["ETag"]; } } This generates the following request: PUT https://STORAGE_ACCOUNT.blob.core.windows.net/fife/dunfermline HTTP/1.1 x-ms-blob-type: BlockBlob x-ms-date: Sun, 08 Sep 2013 06:28:29 GMT x-ms-version: 2012-02-12 Authorization: SharedKey STORAGE_ACCOUNT:ntvh/lamVmikvwHhy6vRVBIh87kibkPlEOiHyLDia6g= Host: STORAGE_ACCOUNT.blob.core.windows.net Content-Length: 39 Expect: 100-continue Connection: Keep-Alive The body of the request is: Andrew Carnegie was born in Dunfermline The Blob Service generates the following response: HTTP/1.1 201 Created Transfer-Encoding: chunked Content-MD5: RYJnWGXLyt94l5jG82LjBw== Last-Modified: Sun, 08 Sep 2013 06:28:31 GMT ETag: "0x8D07A73C5704A86" Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: b74ef0a2-294d-4581-b8f1-6cda724bbdbf x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:28:30 GMT Lease Blob The Blob Service allows a user to lease a blob for a minute at a time and so acquire a write lock on it. The use case for this is the locking of a page blob used to store the VHD backing an writeable Azure Drive. The LeaseBlob() example in this section demonstrates a subtle issue with the creation of authorization strings. The URL has a query string, comp=lease. Rather than using this directly in creating the authorization string it must be converted into comp:lease with a colon replacing the equal symbol – see modifiedURL in the example. Furthermore, the Lease Blob operation requires the use of an x-ms-lease-action to indicate whether the lease is being acquired, renewed, released or broken. public void LeaseBlob(String containerName, String blobName) { String requestMethod = "PUT"; String urlPath = String.Format("{0}/{1}?comp=lease", containerName, blobName); String modifiedUrlPath = String.Format("{0}/{1}\ncomp:lease", containerName, blobName); const Int32 contentLength = 0; String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String leaseAction = "acquire"; String leaseDuration = "60"; String canonicalizedHeaders = String.Format( "x-ms-date:{0}\nx-ms-lease-action:{1}\nx-ms-lease-duration:{2}\nx-ms-version:{3}", dateInRfc1123Format, leaseAction, leaseDuration, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, modifiedUrlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n\n\n\n\n\n\n\n\n{2}\n{3}", requestMethod, contentLength, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.BlobEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-lease-action", leaseAction); request.Headers.Add("x-ms-lease-duration", leaseDuration); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.ContentLength = contentLength; using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { String leaseId = response.Headers["x-ms-lease-id"]; } } This generates the following request: PUT https://STORAGE_ACCOUNT.blob.core.windows.net/fife/dunfermline?comp=lease HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:28:31 GMT x-ms-lease-action: acquire x-ms-lease-duration: 60 x-ms-version: 2012-02-12 Authorization: SharedKey rebus:+SQ5+RFZg3hUaws5XCRHxsDgXb1ycdRIz5EKyHJWP7s= Host: rebus.blob.core.windows.net Content-Length: 0 The Blob Service generates the following response: HTTP/1.1 201 Created Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: 4b6ff77f-f885-4f74-803a-c92920d225c3 x-ms-version: 2012-02-12 x-ms-lease-id: b1320c2c-65ad-41d6-a7bd-85a4242c0ac5 Date: Sun, 08 Sep 2013 06:28:31 GMT Content-Length: 0 Queue Service API The Queue Service API supports the following queue-level operation: List Queues The Queue Service API supports the following queue-level operation: Create Queue Delete Queue Get Queue Metadata Set Queue Metadata The Queue Service API supports the following message-level operations: Clear Messages Delete Message Get Messages Peek Messages Put Message This section provides examples of the Put Message and Get Message operations. Put Message The most obvious curiosity about Put Message is that it uses the HTTP verb POST rather than PUT. The issue is presumably the interaction of the English language and the HTTP standard which states that PUT should be idempotent and that the Put Message operation is clearly not since each invocation merely adds another message to the queue. Regardless, it did catch me out when I failed to read the documentation well enough – so take that as a warning. The content of a message posted to the queue must be formatted in a specified XML schema and must then be UTF8 encoded. public void PutMessage(String queueName, String message) { String requestMethod = "POST"; String urlPath = String.Format("{0}/messages", queueName); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String messageText = String.Format( "{0}", message); UTF8Encoding utf8Encoding = new UTF8Encoding(); Byte[] messageContent = utf8Encoding.GetBytes(messageText); Int32 messageLength = messageContent.Length; String canonicalizedHeaders = String.Format( "x-ms-date:{0}\nx-ms-version:{1}", dateInRfc1123Format, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n{1}\n\n\n\n\n\n\n\n\n{2}\n{3}", requestMethod, messageLength, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.QueueEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.ContentLength = messageLength; using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(messageContent, 0, messageLength); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { String requestId = response.Headers["x-ms-request-id"]; } } This generates the following request: POST https://rebus.queue.core.windows.net/revolution/messages HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:34:08 GMT x-ms-version: 2012-02-12 Authorization: SharedKey rebus:nyASTVWifnxHKnj2wXwuzzzXz5CxUBZj58SToV5QFK8= Host: rebus.queue.core.windows.net Content-Length: 76 Expect: 100-continue Connection: Keep-Alive The body of the request is: Saturday in the cafe The Queue Service generates the following response: HTTP/1.1 201 Created Server: Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: 14c6e73b-15d9-480c-b251-c4c01b48e529 x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:34:09 GMT Content-Length: 0 Get Messages The Get Messages operation described in this section retrieves a single message with the default message visibility timeout of 30 seconds. public void GetMessage(String queueName) { string requestMethod = "GET"; String urlPath = String.Format("{0}/messages", queueName); String storageServiceVersion = "2012-02-12"; String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture); String canonicalizedHeaders = String.Format( "x-ms-date:{0}\nx-ms-version:{1}", dateInRfc1123Format, storageServiceVersion); String canonicalizedResource = String.Format("/{0}/{1}", AzureStorageConstants.Account, urlPath); String stringToSign = String.Format( "{0}\n\n\n\n\n\n\n\n\n\n\n\n{1}\n{2}", requestMethod, canonicalizedHeaders, canonicalizedResource); String authorizationHeader = Utility.CreateAuthorizationHeader(stringToSign); Uri uri = new Uri(AzureStorageConstants.QueueEndPoint + urlPath); HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = requestMethod; request.Headers.Add("x-ms-date", dateInRfc1123Format); request.Headers.Add("x-ms-version", storageServiceVersion); request.Headers.Add("Authorization", authorizationHeader); request.Accept = "application/atom+xml,application/xml"; using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (StreamReader reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } } This generates the following request: GET https://rebus.queue.core.windows.net/revolution/messages HTTP/1.1 x-ms-date: Sun, 08 Sep 2013 06:34:11 GMT x-ms-version: 2012-02-12 Authorization: SharedKey rebus:K67XooYhokw0i0AlCzYQ4GeLLrJih1r1vSqiO9DBo0c= Accept: application/atom+xml,application/xml Host: rebus.queue.core.windows.net The Queue Service generates the following response: HTTP/1.1 200 OK Content-Type: application/xml Server: Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: efb21a86-7d66-47fd-b13d-7aa74fce0568 x-ms-version: 2012-02-12 Date: Sun, 08 Sep 2013 06:34:12 GMT Content-Length: 484 The message is returned in the response body as follows: 05fd902f-6031-4ef4-8298-ef3844ec3bc6Sun, 08 Sep 2013 06:34:11 GMTSun, 15 Sep 2013 06:34:11 GMT1AgAAAAMAAAAAAAAAAL+zgF2szgE=Sun, 08 Sep 2013 06:34:43 GMTSaturday in the cafe I noticed that some newline specifiers in strings (\n) were lost when the blog was auto-ported from Windows Live Spaces to WordPress. I have put them back in but it is possible I missed some. Consequently, in the event of a problem you should check the newlines in canonicalizedHeaders and stringToSign.

With these changes, we can access the REST API equally well fromhttp://:9999 and https://:9998.

I’m going to start by discussing the Spring WebMVC configuration and move on from there in future posts.

ElasticSearch provides Java API, thus it executes all operations asynchronously by using client object.

My last couple of blogs have talked about long polling and Spring's DeferredResult technique and to demonstrate these concepts I've shoehorned the code from my Producer Consumer project into a web application. Although the code demonstrates the points made by the blogs it does contain a large number of holes in its logic. Apart from the fact that in a real application you wouldn't use a simple LinkedBlockingQueue, but would choose JMS or some other industrial-strength messaging service, and the fact that only one user can get a hold of the match updates, there's also the problem that it spawns badly behaved threads that don't close down when the JVM terminates. You may wonder why this should be a problem… well to you, as a developer, it isn't really a problem at all, it's only a little bit of sloppy programming. But, to one of your operations guys it can make life unnecessarily difficult. The reason for this is that if you have too many badly behaved threads then typing Tomcat's shutdown.sh command will have very little effect and you have to savagely kill your web server by typing something like: ps -ef | grep java to get the pid and then kill -9 <> …and when you have a field of Tomcat web servers to restart all this extra kerfuffle that becomes a severe pain. When you type shutdown.sh you want Tomcat to stop. In my last couple of blogs the badly behaved threads I created had the following run() methods with the first of these, shown below, being really badly behaved: @Override public void run() { while (true) { try { DeferredResult result = resultQueue.take(); Message message = queue.take(); result.setResult(message); } catch (InterruptedException e) { throw new UpdateException("Cannot get latest update. " + e.getMessage(), e); } } } In this code I've used an infinite while(true), which means that the thread will just keep running and never terminate. @Override public void run() { sleep(5); // Sleep to allow the reset of the app to load logger.info("The match has now started..."); long now = System.currentTimeMillis(); List matchUpdates = match.getUpdates(); for (Message message : matchUpdates) { delayUntilNextUpdate(now, message.getTime()); logger.info("Add message to queue: {}", message.getMessageText()); queue.add(message); } start = true; // Game over, can restart logger.warn("GAME OVER"); } The second example, above, is also pretty badly behaved. It'll keep taking messages from MatchUpdates list and adding them to the message queue at the appropriate moment. Their only saving grace is that they may throw an InterruptedException, which if handled correctly would cause thread termination; however, this cannot be guaranteed. There's a quick fix for this, really… all you need to do is to ensure that any threads you create are daemon threads. The definition of a daemon thread is a thread that doesn't prevent the JVM from exiting when the program finishes but, the thread is still running. The usual example of a daemon thread is the JVM's garbage collection thread. To turn your threads into daemon threads you simply call: thread.setDaemon(true); ...and when you type shutdown.sh then, WHAM, all your threads will disappear. There is, however, a problem with this. What if one of your daemon's threads was doing something important and chopping it down in its prime, resulting in the loss of some pretty important data? What you need to do is to ensure that all your threads shut down gracefully, completing any work they may be currently undertaking. The rest of this blog demonstrates a fix for these errant threads, gracefully coordinating their shutdown by using a ShutdownHook. According to the documentation, a "shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently." So, after reading the last sentence you may have guessed that what you need to do is to create a thread that has the responsibility of shutting down all your other threads and is passed to the JVM as a shutdown hook. All of this can be generically implemented in a couple of small classes and by performing some jiggery-pokery on your existing thread run() methods. The two classes to create are a ShutdownService and a Hook. The Hook class, which I'll demonstrate first, is used to link the ShutdownService to your threads. The code for Hook is as follows: public class Hook { private static final Logger logger = LoggerFactory.getLogger(Hook.class); private boolean keepRunning = true; private final Thread thread; Hook(Thread thread) { this.thread = thread; } /** * @return True if the daemon thread is to keep running */ public boolean keepRunning() { return keepRunning; } /** * Tell the client daemon thread to shutdown and wait for it to close gracefully. */ public void shutdown() { keepRunning = false; thread.interrupt(); try { thread.join(); } catch (InterruptedException e) { logger.error("Error shutting down thread with hook", e); } } } The Hook contains two instance variables: keepRunning and thread. thread is a reference to the thread that this instance of Hook is responsible for shutting down, while keepRunning tells the thread to… keep running. Hook has two public methods: keepRunning() and shutdown(). keepRunning() is called by the thread to figure out whether it should keep running, and shutdown() is called by the ShutdownService's shutdown hook thread to get your thread to shut down. This is the most interesting of the two methods. Firstly it sets the keepRunning variable to false. It then calls thread.interrupt() to interrupt the thread forcing it to throw an InterruptedException. Lastly, it calls thread.join() and waits for the thread instance to shutdown. Note that this technique relies on all your threads cooperating. If there's one badly behaved thread in the mix, then the whole thing could hang. To get around this problem add a timeout to thread.join(…). @Service public class ShutdownService { private static final Logger logger = LoggerFactory.getLogger(ShutdownService.class); private final List hooks; public ShutdownService() { logger.debug("Creating shutdown service"); hooks = new ArrayList(); createShutdownHook(); } /** * Protected for testing */ @VisibleForTesting protected void createShutdownHook() { ShutdownDaemonHook shutdownHook = new ShutdownDaemonHook(); Runtime.getRuntime().addShutdownHook(shutdownHook); } protected class ShutdownDaemonHook extends Thread { /** * Loop and shutdown all the daemon threads using the hooks * * @see java.lang.Thread#run() */ @Override public void run() { logger.info("Running shutdown sync"); for (Hook hook : hooks) { hook.shutdown(); } } } /** * Create a new instance of the hook class */ public Hook createHook(Thread thread) { thread.setDaemon(true); Hook retVal = new Hook(thread); hooks.add(retVal); return retVal; } @VisibleForTesting List getHooks() { return hooks; } } The ShutdownService is a Spring service that contains a list of Hook classes, and therefore by inference threads, that it is responsible for shutting down. It also contains an inner class ShutdownDaemonHook, which extends Thread. An instance of ShutdownDaemonHook is created during the construction of ShutdownService, which is then passed to the JVM as a shutdown hook by calling: Runtime.getRuntime().addShutdownHook(shutdownHook); The ShutdownService has one public method: createHook(). The first thing that this class does is to ensure that any thread passed to it is converted into a daemon thread. It then creates a new Hook instance, passing in the thread as the argument, before finally both storing the result in a list and returning it to the caller. The only thing left to do now is to integrate the ShutdownService into DeferredResultService and MatchReporter, the two classes that contain the badly behaved threads. @Service("DeferredService") public class DeferredResultService implements Runnable { private static final Logger logger = LoggerFactory.getLogger(DeferredResultService.class); private final BlockingQueue> resultQueue = new LinkedBlockingQueue<>(); private Thread thread; private volatile boolean start = true; @Autowired private ShutdownService shutdownService; private Hook hook; @Autowired @Qualifier("theQueue") private LinkedBlockingQueue queue; @Autowired @Qualifier("BillSkyes") private MatchReporter matchReporter; public void subscribe() { logger.info("Starting server"); matchReporter.start(); startThread(); } private void startThread() { if (start) { synchronized (this) { if (start) { start = false; thread = new Thread(this, "Studio Teletype"); hook = shutdownService.createHook(thread); thread.start(); } } } } @Override public void run() { logger.info("DeferredResultService - Thread running"); while (hook.keepRunning()) { try { DeferredResult result = resultQueue.take(); Message message = queue.take(); result.setResult(message); } catch (InterruptedException e) { System.out.println("Interrupted when waiting for latest update. " + e.getMessage()); } } System.out.println("DeferredResultService - Thread ending"); } public void getUpdate(DeferredResult result) { resultQueue.add(result); } } The first change to this class was to autowire in the Shutdown service instance. The next thing to do is to use the ShutdownService to create an instance of Hook after the creation of the thread but before thread.start() is called: thread = new Thread(this, "Studio Teletype"); hook = shutdownService.createHook(thread); thread.start(); The final change is to replace while(true) with: while (hook.keepRunning()) { … telling the thread when to quit the while loop and shutdown. You may have also noticed that there are a few System.out.println() calls thrown into the above code. There is a reason for this and it's because of the undetermined order in which the shutdown hook threads are executed. Remember that not only are your classes trying to shutdown gracefully, but other sub-systems and shutting down too. This means that my original code, which called logger.info(…) failed throwing the following exception: Exception in thread "Studio Teletype" java.lang.NoClassDefFoundError: org/apache/log4j/spi/ThrowableInformation at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:159) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:382) at com.captaindebug.longpoll.service.DeferredResultService.run(DeferredResultService.java:75) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.spi.ThrowableInformation at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559) ... 6 more This is because the logger has already been unloaded when I try to call it; hence the failure.Again, as the documentation states: "Shutdown hooks run at a delicate time in the life cycle of a virtual machine and should therefore be coded defensively. They should, in particular, be written to be thread-safe and to avoid deadlocks insofar as possible. They should also not rely blindly upon services that may have registered their own shutdown hooks and therefore may themselves in the process of shutting down. Attempts to use other thread-based services such as the AWT event-dispatch thread, for example, may lead to deadlocks." The MatchReport class has some very similar modifications. The major difference is that the hook.keepRunning() code is inside the run() method's for loop. public class MatchReporter implements Runnable { private static final Logger logger = LoggerFactory.getLogger(MatchReporter.class); private final Match match; private final Queue queue; private volatile boolean start = true; @Autowired private ShutdownService shutdownService; private Hook hook; public MatchReporter(Match theBigMatch, Queue queue) { this.match = theBigMatch; this.queue = queue; } /** * Called by Spring after loading the context. Will "kick off" the match... */ public void start() { if (start) { synchronized (this) { if (start) { start = false; logger.info("Starting the Match Reporter..."); String name = match.getName(); Thread thread = new Thread(this, name); hook = shutdownService.createHook(thread); thread.start(); } } } else { logger.warn("Game already in progress"); } } /** * The main run loop */ @Override public void run() { sleep(5); // Sleep to allow the reset of the app to load logger.info("The match has now started..."); long now = System.currentTimeMillis(); List matchUpdates = match.getUpdates(); for (Message message : matchUpdates) { delayUntilNextUpdate(now, message.getTime()); if (!hook.keepRunning()) { break; } logger.info("Add message to queue: {}", message.getMessageText()); queue.add(message); } start = true; // Game over, can restart logger.warn("GAME OVER"); } private void sleep(int deplay) { try { TimeUnit.SECONDS.sleep(10); } catch (InterruptedException e) { logger.info("Sleep interrupted..."); } } private void delayUntilNextUpdate(long now, long messageTime) { while (System.currentTimeMillis() < now + messageTime) { try { Thread.sleep(100); } catch (InterruptedException e) { logger.info("MatchReporter Thread interrupted..."); } } } } The ultimate test of this code is to issue a Tomcat shutdown.sh command half way through the match update sequence. As the JVM terminates it'll call the shutdown hook from the ShutdownDaemonHook class. As this class's run() method executes it loops throughout the list of Hook instances telling them to close down their respective threads. If you tail -f your server's log file (in my case catalina.out, but your Tomcat maybe configured differently to mine), you'll see the trail of entries shutting your server shutdown gracefully.

In Java 5.0, a new addition called Reentrant Lock was made to enhance intrinsic locking capabilities. Prior to this, "synchronized" and "volatile" were the means for achieving concurrency. public synchronized void doAtomicTransfer(){ //enter synchronized block , acquire lock over this object. operation1() operation2(); } // exiting synchronized block, release lock over this object. Synchronized uses intrinsic locks or monitors. Every object in Java has an intrinsic lock associated with it. Whenever a thread tries to access a synchronized block or method, it acquires the intrinsic lock or the monitor on that object. In case of static methods, the thread acquires the lock over the class object. An intrinsic locking mechanism is a clean approach in terms of writing code, and is pretty good for most of the use-cases. So why do we need the additional feature of explicit locks? Let's discuss. An intrinsic locking mechanism can have some functional limitations, such as: 1.) It is not possible to interrupt a thread waiting to acquire a lock (lock Interruptibly). 2.) It is not possible to attempt to acquire a lock without being willing to wait for it forever (try lock). 3.) Cannot implement non-block-structured locking disciplines, as intrinsic locks must be released in the same block in which they are acquired. Aside from that, ReentrantLock supports lock polling, and interruptible lock waits that support time-out. ReentrantLock also has support for configurable fairness policy, allowing more flexible thread scheduling. Source: Stack Overflow Lets see a few of the methods implemented by ReentrantLock class (which implements Lock): void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit) throws InterruptedException; ..... Lets try and understand the use of these and see what benefits we can get. 1.) Polled and Timed Lock Acquisition Let's see some example code: public void transferMoneyWithSync(Account fromAccount, Account toAccount, float amount) throws InsufficientAmountException { synchronized (fromAccount) { // acquired lock on fromAccount Object synchronized (toAccount) { // acquired lock on toAccount Object if (amount > fromAccount.getCurrentAmount()) { throw new InsufficientAmountException( "Insufficient Balance"); } else { fromAccount.debit(amount); toAccount.credit(amount); } } } } In the transferMoney() method above, there is a possibility of deadlock when two threads A and B are trying to transfer money at almost the same time. A: transferMoney(acc1, acc2, 20); B: transferMoney(acc2, acc1 ,25); It is possible that thread A has acquired a lock on the acc1 object and is waiting to acquire a lock on the acc2 object. Meanwhile, thread B has acquired a lock on the acc2 object and is waiting for a lock on acc1. This will lead to deadlock, and the system would have to be restarted! There is, however, a way to avoid this, which is called "lock ordering." Personally, I find this a bit complex. A cleaner approach is implemented by ReentrantLock with the use of tryLock() method. This approach is called the "timed and polled lock-acquisition." It lets you regain control if you cannot acquire all the required locks, release the ones you have acquired and retry. So, using tryLock, we will attempt to acquire both locks. If we cannot attain both, we will release if one of these has been acquired, then retry. public boolean transferMoneyWithTryLock(Account fromAccount, Account toAccount, float amount) throws InsufficientAmountException, InterruptedException { // we are defining a stopTime long stopTime = System.nanoTime() + 5000; while (true) { if (fromAccount.lock.tryLock()) { try { if (toAccount.lock.tryLock()) { try { if (amount > fromAccount.getCurrentAmount()) { throw new InsufficientAmountException( "Insufficient Balance"); } else { fromAccount.debit(amount); toAccount.credit(amount); } } finally { toAccount.lock.unlock(); } } } finally { fromAccount.lock.unlock(); } } if(System.nanoTime() < stopTime) return false; Thread.sleep(100); }//while } Here we implemented a timed lock, so if the locks cannot be acquired within the specified time, the transferMoney method will return a failure notice and exit gracefully. We can also maintain time budget activities using this concept. 2.) Interruptible Lock Acquisition Interruptible lock acquisition allows locking to be used within cancellable activities. The lockInterruptibly method allows us to try and acquire a lock while being available for interruption. So, basically it allows the thread to immediately react to the interrupt signal sent to it from another thread. This can be helpful when we want to send a KILL signal to all the waiting locks. Let's see one example: Suppose we have a shared line to send messages. We would want to design it in such a way that if another thread comes and interrupts the current thread, the lock should release and perform the exit or shut down operations to cancel the current task. public boolean sendOnSharedLine(String message) throws InterruptedException{ lock.lockInterruptibly(); try{ return cancellableSendOnSharedLine(message); } finally { lock.unlock(); } } private boolean cancellableSendOnSharedLine(String message){ ....... The timed tryLock is also responsive to interruption. 3.) Non-block Structured Locking: In intrinsic locks, acquire-release pairs are block-structured. In other words, a lock is always released in the same basic block in which it was acquired, regardless of how control exits the block. Extrinsic locks allow the facility to have more explicit control. Some concepts, like Lock Strapping, can be achieved more easily using extrinsic locks. Some use cases are seen in hash-bashed collections and linked lists. 4.) Fairness: The ReentrantLock constructor offers a choice of two fairness options: create a non-fair lock or a fair lock. With fair locking, threads can acquire locks only in the order in which they were requested, whereas an unfair lock allows a lock to acquire it out of its turn. This is called barging (breaking the queue and acquiring the lock when it became available). Fair locking has a significant performance cost because of the overhead of suspending and resuming threads. There could be cases where there is a significant delay between when a suspended thread is resumed and when it actually runs. Let's see a situation: A -> holds a lock. B -> has requested and is in a suspended state waiting for A to release the lock. C -> requests the lock at the same time that A releases the lock, and has not yet gone to a suspended state. As C has not yet gone to a suspended state, there is a chance that it can acquire the lock released by A, use it, and release it before B even finishes waking up. So, in this context, unfair lock has a significant performance advantage. Intrinsic locks and extrinsic locks have the same mechanism inside for locking, so the performance improvement is purely subjective. It depends on the use cases we discussed above. Extrinsic locks give a more explicit control mechanism for better handling of deadlocks, starvation, and so on. In future articles, I will cover more use cases to exhibit extrinsic locks.

Learn about deploying and debugging Visual Studio packages.

About a year ago, we published in our blog a series of articles on the development of Visual Studio plugins in C#. We have recently revised those materials and added new sections and now invite you to have a look at the updated version of the manual as a series of articles here on DZone. The other articles in the series can be found here: Part 1 - How to Create, Debug and Deploy Visual Studio Extension Packages Part 2 - Using the Visual Studio Automation Object Model This article (the second in the series) contains an overview of the Visual Studio Automation Object Model. In it, we examine the model's overall structure and the means of obtaining access to its interfaces through DTE/DTE2 top-level objects. Several examples of utilizing elements of the model are provided. Also discussed are the issues of using the model's interfaces within multithreaded applications; an example of implementing such a mechanism for multithreaded interaction with COM interfaces in managed code is provided as well. Introduction The Visual Studio development environment is built upon the principles of automation and extensibility, providing developers with the ability to integrate almost any custom element into the IDE and allowing for an easy interaction with its default and user-created components. As the means of implementing these tasks, Visual Studio users are provided with several cross-complementing toolsets; the most basic and versatile among these is the Visual Studio Automation Object Model. The Automation Object Model is represented by a series of libraries containing a vast and well-structured API set that covers all aspects of IDE automation and the majority of its extensibility capabilities. Although, in comparison to other IDE extensibility tools, this model does not provide access to some portions of Visual Studio (this applies mostly to the extension of some IDE's features), it is nonetheless the most flexible and versatile among them. The majority of the model's interfaces are accessible from within every type of IDE extension module, which allows interacting with the environment even from an external independent process. Moreover, the model itself could be extended along with the extension of Visual Studio IDE, providing other third-party developers with an access to user-created custom components. Automation Object Model Structure The Visual Studio automation model is composed of several interconnected functional object groups covering all aspects of the development environment; it also provides capabilities for controlling and extending these groups. Accessing any of them is possible through the top-level global DTE interface (Development Tools Environment). Figure 1 shows the overall structure of the automation model and how it is divided among functionality groups. Figure 1 — Visual Studio Automation Object Model A user could extend the model in one of the following groups: Project models (implementing new project types, support for new languages); Document models (implementing new document types and document editors) Code editor level models (support for specific language constructs) Project build-level models An automation model could be extended from plug-ins of VSPackage type only. All of the automation model's interfaces could be conventionally subdivided into two large groups. The first group consists of the interfaces of the EnvDTE and Visual Studio Interop namespaces. These interfaces allow interactions with basic common components of the IDE itself, such as tool windows, editors, event handling services and so on. The second group consists of the interfaces of the specific project model. The figure above specifies this interface group as late-bound properties, i.e. these interfaces are implemented in a separate dynamically loaded library. Each standard (i.e. the one that is included in a regular Visual Studio distribution) project model, such as Visual C++ or Visual Basic, provides a separate implementation for these interfaces. Third-party developers are able to extend the automation model by adding their own custom project models and by providing an implementation of these automation interfaces. Also worth noting is that the interfaces of the 1st group, which was specified above, are universal, meaning that they could be utilized for interaction with any of the project models or Visual Studio editions, including the integrated\isolated Visual Studio shells. In this article, we will examine this group in more detail. But still, despite the model's versatility, not every group belonging to the model could be equally utilized from all the types of IDE extensions. For instance, some of the model's capabilities are inaccessible to external processes; these capabilities are tied to specific extension types, such as add-in or VSPackage. Therefore, when selecting the type for the extension to be developed, it is important to consider the functionality that this extension will require. The Microsoft.VisualStudio.Shell.Interop namespace also provides a group of COM interfaces, which can be used to extend and automate Visual Studio application from managed code. Managed Package Framework (MPF) classes, which we utilized earlier for creating a VSPackage plugin, are actually themselves based on these interfaces. Although theses interfaces are not a part of EnvDTE automation model described above, nevertheless they greatly enhance this model by providing additional functionality for VSPackage extensions, which is otherwise unavailable for extensions of other types. Obtaining References to DTE/DTE2 Objects In order to create a Visual Studio automation application, it is necessary to obtain access to the automation objects themselves in the first place. To accomplish this, first of all it is necessary to hook up the correct versions of libraries containing the required managed API wrappers in the EnvDTE namespace. Secondly, the reference to the automation model top-level object, that is the DTE2 interface, should be obtained. In the course of Visual Studio’s evolution, several of its automation objects had been modified or received some additional functionality. So, to maintain a backward compatibility with existing extension packages, new EnvDTE80, EnvDTE90, EnvDTE100 etc. namespaces were created instead of updating the interfaces from the original EnvDTE namespace. The majority of such updated interfaces from these new namespaces do maintain the same names as in the original ones, but with the addition of an ordinal number at the end of the name, for example Solution and Solution2. It is advised that these updated interfaces should be utilized when creating a new project, as they do contain the most recent functionality. It's worth noting that properties and methods of DTE2 interface usually return object references with types corresponding to the original DTE. For example, accessing dte2.Solution will return Solution and not the Solution2 as it would seem. Although these new EnvDTE80, EnvDTE90, EnvDTE100 namespaces do contain some of the updated functionality as mentioned above, still it is the EnvDTE interface that contains the majority of automation objects. Therefore, in order to possess access to all of the existing interfaces, it is necessary to link all versions of the managed COM wrapper libraries to the project, as well as to obtain the references to DTE and also to DTE2. The way of obtaining a top-level EnvDTE object reference is dependent upon the type of IDE extension being developed. Let's examine three such extension types: add-in, VSPackage and an MSVS-independent external process. Add-in Extension In the case of an add-in extension, access to the DTE interface can be obtained inside the OnConnection method, which should be implemented for the IDTExtensibility interface that provides access to the extension-environment interaction events. The OnConnection method is called at the moment when the module is loaded by the IDE. It can happen either when the environment is being loaded itself or after the extension was called for the first time in the IDE session. The example of obtaining the reference follows: public void OnConnection(object application, ext_ConnectMode connectMode, object addInInst, ref Array custom) { _dte2 = (DTE2)application; ... } An add-in module can be initialized either at the moment of IDE start-up, or when it is called for the first time in the current IDE session. So, the connectMode can be used to correctly determine the moment of initialization inside the OnConnection method. switch(connectMode) { case ext_ConnectMode.ext_cm_UISetup: ... break; case ext_ConnectMode.ext_cm_Startup: ... break; case ext_ConnectMode.ext_cm_AfterStartup: ... break; case ext_ConnectMode.ext_cm_CommandLine: ... break; } As in the example above, add-in could be loaded either simultaneously with the IDE itself (if the startup option in the add-in manager is checked), when it is called the first time or when it is called through the command line. The ext_ConnectMode.ext_cm_UISetup option is invoked only for a single time in the plug-in's overall lifetime, which is during its first initialization. This case should be used for initializing user UI elements that are to be integrated into the environment (more on this later on). If an add-in is being loaded during Visual Studio start-up (ext_ConnectMode.ext_cm_Startup), then at the moment OnConnect method receives control for the first time, it is possible that the IDE still is not fully initialized itself. In such a case, it is advised to postpone the acquisition of the DTE reference until the environment is fully loaded. The OnStartupComplete handler provided by the IDTExtensibility can be used for this. public void OnStartupComplete(ref Array custom) { ... } VSPackage Extension For VSPackage type of extension, the DTE could be obtained through the global Visual Studio service with the help of GetService method of a Package subclass: DTE dte = MyPackage.GetService(typeof(DTE)) as DTE; Please note that the GetService method could potentially return null in case Visual Studio is not fully loaded or initialized at the moment of such access, i.e. it is in the so called "zombie" state. To correctly handle this situation, it is advised that the acquisition of DTE reference should be postponed until this interface is inquired. But in case the DTE reference is required inside the Initialize method itself, the IVsShellPropertyEvents interface can be utilized (also by deriving our Package subclass from it) and then the reference could be safely obtained inside the OnShellPropertyChange handler. DTE dte; uint cookie; protected override void Initialize() { base.Initialize(); IVsShell shellService = GetService(typeof(SVsShell)) as IVsShell; if (shellService != null) ErrorHandler.ThrowOnFailure( shellService.AdviseShellPropertyChanges(this,out cookie)); ... } public int OnShellPropertyChange(int propid, object var) { // when zombie state changes to false, finish package initialization if ((int)__VSSPROPID.VSSPROPID_Zombie == propid) { if ((bool)var == false) { this.dte = GetService(typeof(SDTE)) as DTE; IVsShell shellService = GetService(typeof(SVsShell)) as IVsShell; if (shellService != null) ErrorHandler.ThrowOnFailure( shellService.UnadviseShellPropertyChanges(this.cookie) ); this.cookie = 0; } } return VSConstants.S_OK; } It should be noted that the process of VSPackage module initialization at IDE startup could vary for different Visual Studio versions. For instance, in case of VS2005 and VS2008, an attempt at accessing DTE during IDE startup will almost always result in null being returned, owning to the relative fast loading times of these versions. But, one does not simply obtain access into DTE. In Visual Studio 2010 case, it mistakenly appears that one could simply obtain an access to the DTE from inside the Initialize() method. In fact, this impression is a false one, as such a method of DTE acquisition could potentially cause the occasional appearance of "floating" errors which are hard to identify and debug, and even the DTE itself may be still uninitialized when the reference is acquired. Because of these disparities, the aforementioned acquisition method for handling IDE loading states should not be ignored on any version of Visual Studio. Independent External Process The DTE interface is a top-level abstraction for Visual Studio environment in the automation model. In order to acquire a reference to this interface from an external application, its ProgID COM identifier could be utilized; for instance, it will be "VisualStudio.DTE.10.0" for Visual Studio 2010. Consider this example of initializing a new IDE instance and when obtaining a reference to the DTE interface. // Get the ProgID for DTE 8.0. System.Type t = System.Type.GetTypeFromProgID( "VisualStudio.DTE.10.0", true); // Create a new instance of the IDE. object obj = System.Activator.CreateInstance(t, true); // Cast the instance to DTE2 and assign to variable dte. EnvDTE80.DTE2 dte = (EnvDTE80.DTE2)obj; // Show IDE Main Window dte.MainWindow.Activate(); In the example above, we've actually created a new DTE object, starting deven.exe process by the CreateInstance method. But at the same time, the GUI window of the environment will be displayed only after the Activate method is called. Next, let's review a simple example of obtaining the DTE reference from an already running Visual Studio Instance: EnvDTE80.DTE2 dte2; dte2 = (EnvDTE80.DTE2) System.Runtime.InteropServices.Marshal.GetActiveObject( "VisualStudio.DTE.10.0"); However, in case several instances of the Visual Studio are executing at the moment of our inquiry, the GetActiveObject method will return a reference to the IDE instance that was started the earliest. Let's examine a possible way of obtaining the reference to DTE from a running Visual Studio instance by the PID of its process. using EnvDTE80; using System.Diagnostics; using System.Runtime.InteropServices; using System.Runtime.InteropServices.ComTypes; [DllImport("ole32.dll")] private static extern void CreateBindCtx(int reserved, out IBindCtx ppbc); [DllImport("ole32.dll")] private static extern void GetRunningObjectTable(int reserved, out IRunningObjectTable prot); public static DTE2 GetByID(int ID) { //rot entry for visual studio running under current process. string rotEntry = String.Format("!VisualStudio.DTE.10.0:{0}", ID); IRunningObjectTable rot; GetRunningObjectTable(0, out rot); IEnumMoniker enumMoniker; rot.EnumRunning(out enumMoniker); enumMoniker.Reset(); IntPtr fetched = IntPtr.Zero; IMoniker[] moniker = new IMoniker[1]; while (enumMoniker.Next(1, moniker, fetched) == 0) { IBindCtx bindCtx; CreateBindCtx(0, out bindCtx); string displayName; moniker[0].GetDisplayName(bindCtx, null, out displayName); if (displayName == rotEntry) { object comObject; rot.GetObject(moniker[0], out comObject); return (EnvDTE80.DTE2)comObject; } } return null; } Here we've acquired the DTE interface by identifying the required instance of the IDE in the table of running COM objects (ROT, Running Object Table) by its process identifier. Now we can access the DTE for every of the executing instances of Visual Studio, for example: Process Devenv; ... //Get DTE by Process ID EnvDTE80.DTE2 dte2 = GetByID(Devenv.Id); Additionally, to acquire any project-specific interface (including custom model extensions), for example the CSharpProjects model, through a valid DTE interface, the GetObject method should be utilized: Projects projects = (Projects)dte.GetObject("CSharpProjects"); The GetObject method will return a Projects collection of regular Project objects, and each one of them will contain a reference to our project-specific properties, among other regular ones. Visual Studio Text Editor Documents An automation model represents Visual Studio text documents through the TextDocument interface. For example, C/C++ source code files are opened by the environment as text documents. The TextDocument is based upon the common automation model document interface (the Document interface), which represents files of any type opened in Visual Studio editor or designer. A reference to the text document object can be obtained through the 'Object' field of the Document object. Let's acquire a text document for the currently active (i.e. the one possessing focus) document from IDE's text editor: EnvDTE.TextDocument objTextDoc = (TextDocument)PVSStudio.DTE.ActiveDocument.Object("TextDocument"); Modifying Documents The TextSelection document allows controlling the text selection or to modify it. The methods of this interface represent the functionality of Visual Studio text editor, i.e. they allow the interaction with the text as it is presented directly by the UI. EnvDTE.TextDocument Doc = (TextDocument)PVSStudio.DTE.ActiveDocument.Object(string.Empty); Doc.Selection.SelectLine(); TextSelection Sel = Doc.Selection; int CurLine = Sel.TopPoint.Line; String Text = Sel.Text; Sel.Insert("test\r\n"); In this example, we selected a text line under the cursor, read the selected text and replaced it with a 'test' string. The TextDocument interface also allows text modification through the EditPoint interface. This interface is somewhat similar to the TextSelection, but instead of operating with the text through the editor UI, it directly manipulates text buffer data. The difference between them is that the text buffer is not influenced by such editor-specific notions as WordWrap and Virtual Spaces. It should be noted that both of these editing methods are not able to modify read-only text blocks. Let's examine the example of modifying text with EditPoint by placing additional lines at the end of current line with a cursor: objEditPt = objTextDoc.StartPoint.CreateEditPoint(); int lineNumber = objTextDoc.Selection.CurrentLine; objEditPt.LineDown(lineNumber - 1); EditPoint objEditPt2 = objTextDoc.StartPoint.CreateEditPoint(); objEditPt2.LineDown(lineNumber - 1); objEditPt2.CharRight(objEditPt2.LineLength); String line = objEditPt.GetText(objEditPt.LineLength); String newLine = line + "test"; objEditPt.ReplaceText(objEditPt2, newLine, (int)vsEPReplaceTextOptions.vsEPReplaceTextKeepMarkers); Navigating the Documents VSPackage modules are able to obtain access to a series of global services that could be used for opening and handling environment documents. These services could be acquired by the Package.GetGlobalService() method from Managed Package Framework. It should be noted that the services described here are not part of the EnvDTE model and are accessible only from a Package-type extension, and therefore they could not be utilized in other types of Visual Studio extensions. Nonetheless, they can be quite useful for handling IDE documents when they are utilized in addition to the Documents interface described earlier. Next, we'll examine these services in more detail. The IVsUIShellOpenDocument interface controls the state of documents opened in the environment. Following is the example that uses this interface to open a document through path to a file, which this document will represent: String path = "C:\Test\test.cpp"; IVsUIShellOpenDocument openDoc = Package.GetGlobalService(typeof(IVsUIShellOpenDocument)) as IVsUIShellOpenDocument; IVsWindowFrame frame; Microsoft.VisualStudio.OLE.Interop.IServiceProvider sp; IVsUIHierarchy hier; uint itemid; Guid logicalView = VSConstants.LOGVIEWID_Code; if (ErrorHandler.Failed( openDoc.OpenDocumentViaProject(path, ref logicalView, out sp, out hier, out itemid, out frame)) || frame == null) { return; } object docData; frame.GetProperty((int)__VSFPROPID.VSFPROPID_DocData, out docData); The file will be opened in a new editor or will receive focus in case it already has been opened earlier. Next, let's read a VsTextBuffer text buffer from this document we opened: // Get the VsTextBuffer VsTextBuffer buffer = docData as VsTextBuffer; if (buffer == null) { IVsTextBufferProvider bufferProvider = docData as IVsTextBufferProvider; if (bufferProvider != null) { IVsTextLines lines; ErrorHandler.ThrowOnFailure(bufferProvider.GetTextBuffer( out lines)); buffer = lines as VsTextBuffer; Debug.Assert(buffer != null, "IVsTextLines does not implement IVsTextBuffer"); if (buffer == null) { return; } } } The IVsTextManager interface controls all of the active text buffers in the environment. For example, we can navigate a text document using the NavigateToLineAndColumn method of this manager on a buffer we've acquired earlier: IVsTextManager mgr = Package.GetGlobalService(typeof(VsTextManagerClass)) as IVsTextManager; mgr.NavigateToLineAndColumn(buffer, ref logicalView, line, column, line, column); Subscribing and Handling Events Automation objects events are represented by the DTE.Events property. This element references all of the common IDE events (such as CommandEvents, SolutionEvents), as well as the events of separate environment components (project types, editors, tools etc.), also including the ones designed by third-party developers. To acquire a reference for this automation object, the GetObject method could be utilized. When subscribing to the DTE events, one should remember that this interface could be still unavailable at the moment of extension being initialized. So it is always important to consider the sequence of your extension initialization process if the access to DTE.Events is required in the Initialize() method of your extension package. The correct handling of initialization sequence will vary for different extension types, as it was described earlier. Let's acquire a reference for an events object of Visual C++ project model defined by the VCProjectEngineEvents interface and assign a handler for the removal of an element from the Solution Explorer tree: VCProjectEngineEvents m_ProjectItemsEvents = PVSStudio.DTE.Events.GetObject("VCProjectEngineEventsObject") as VCProjectEngineEvents; m_ProjectItemsEvents.ItemRemoved += new _dispVCProjectEngineEvents_ItemRemovedEventHandler( m_ProjectItemsEvents_ItemRemoved); MDI Windows Events The Events.WindowEvents property could be utilized to handle regular events of an environment MDI window. This interface permits the assignment of a separate handler for a single window (defined through the EnvDTE.Window interface) or the assignment of a common handler for all of the environment's windows. The following example contains the assignment of a handler for the event of switching between IDE windows: WindowEvents WE = PVSStudio.DTE.Events.WindowEvents; WE.WindowActivated += new _dispWindowEvents_WindowActivatedEventHandler( Package.WE_WindowActivated); The next example is the assignment of a handler for window switching to the currently active MDI window through the WindowEvents indexer: WindowEvents WE = m_dte.Events.WindowEvents[MyPackage.DTE.ActiveWindow]; WE.WindowActivated += new _dispWindowEvents_WindowActivatedEventHandler( MyPackage.WE_WindowActivated); IDE Commands Events The actual handling of the environment's commands and their extension through the automation model is covered in a separate article of this series. In this section, we will examine the handling of the events related to these commands (and not of the execution of the commands themselves). Assigning the handlers to these events is possible through the Events.CommandEvents interface. The CommandEvents property, as in the case of MDI windows events, also permits the assignment of a handler either for all of the commands or for a single one through the indexer. Let's examine the assignment of a handler for the event of a command execution being complete (i.e. when the command finishes its execution): CommandEvents CEvents = DTE.Events.CommandEvents; CEvents.AfterExecute += new _dispCommandEvents_AfterExecuteEventHandler(C_AfterExecute); But in order to assign such a handler for an individual command, it is necessary to identify this command in the first place. Each command of the environment is identified by a pair of GUID:ID, and in case of user-created commands, these values are specified directly by the developer during their integration, for example through the VSCT table. Visual Studio possesses a special debug mode that allows identifying any of the environment's commands. To activate this mode, it is required that the following key is to be added to the system registry (an example for Visual Studio 2010): [HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\10.0\General] "EnableVSIPLogging"=dword:00000001 Now, after restarting the IDE, hovering your mouse over the menu or toolbar elements with CTRL+SHIFT being simultaneously pressed (though sometime it will not work until you left-click it) will display a dialog window containing all of the command's internal identifiers. We are interested in the values of Guid and CmdID. Let's examine the handling of events for the File.NewFile command: CommandEvents CEvents = DTE.Events.CommandEvents[ "{5EFC7975-14BC-11CF-9B2B-00AA00573819}", 221]; CEvents.AfterExecute += new _dispCommandEvents_AfterExecuteEventHandler(C_AfterExecute); The handler obtained in this way will receive control only after the command execution is finished. void C_AfterExecute(string Guid, int ID, object CustomIn, object CustomOut) { ... } This handler should not be confused with an immediate handler for the execution of the command itself which could be assigned during this command's initialization (from an extension package and in case the command is user-created). Handling the IDE commands is described in a separate article that is entirely devoted to IDE commands. In concluding this section, it should be mentioned that in the process of developing our own VSPackage extension, we've encountered the necessity to store the references to interface objects containing our handler delegates (such as CommandEvents, WindowEvents etc.) on the top-level fields of our main Package subclass. The reason for this is that in the case of the handler being assigned through a function-level local variable, it is lost immediately after leaving the method. Such behavior could probably be attributed to the .NET garbage collector, although we've obtained these references from the DTE interface that definitely exists during the entire lifetime of our extension package. Handling Project and Solution Events for VSPackage Extensions Let's examine some of the interfaces from the Microsoft.VisualStudio.Shell.Interop namespace -- the ones that permit us to handle the events related to Visual Studio projects and solutions to be more precise. Although these interfaces are not a part of the EnvDTE automation model, they could be implemented by the main class of the VSPackage extension (that is the class that was inherited from the Package base class of Managed Package Framework). That is why, if you are developing the extension of this type, these interfaces conveniently supplement the basic set of interfaces provided by the DTE object. By the way, this is another argument for creating a full-fledged VSPackage plugin using MPF. The IVsSolutionEvents could be implemented by the class inherited from Package, and it is available starting from Visual Studio version 2005, and the isolated\integrated shells based applications. This interface permits you to track the loading, unloading, opening and closing of projects or even the whole solutions in the development environment by implementing such of its methods as OnAfterCloseSolution, OnBeforeCloseProject and OnQueryCloseSolution. For example: public int OnAfterLoadProject(IVsHierarchy pStubHierarchy, IVsHierarchy pRealHierarchy) { //your custom handler code return VSConstants.S_OK; } As you can see, this method takes the IVsHierarchy object as an input parameter that represents the loading project. Managing such objects will be examined in another article devoted to the interaction with the Visual Studio project model. The IVsSolutionLoadEvents interface, in a similar fashion to the interface described above, should be implemented by the Package subclass and is available to versions of Visual Studio starting from 2010 and above. This interface allows you to handle such interesting aspects as batch loading of project groups and background solution loadings (the OnBeforeLoadProjectBatch and OnBeforeBackgroundSolutionLoadBegins methods), and also to intercept the end of this background loading operation as well (the OnAfterBackgroundSolutionLoadComplete method). Such event handlers should come in handy in case your plug-in needs to execute some code immediately after its initialization, and, at the same time, the plug-in depends on projects\solutions that are loaded inside the IDE. In this case, executing such a code without waiting for the solution loading to be finished could lead to either incorrect (incomplete) results because of the incompletely formed projects tree, or even to runtime exceptions. While developing the PVS-Studio IDE plug-in, we've encountered another interesting aspect of VSPackage plug-in initialization. When one Package plug-in enters a waiting state (for instance, by displaying a dialog window to the user), further initialization of VSPackage extensions is suspended until the blocking plug-in returns. So, when handling loading and initialization inside the environment, one should always remember this possible scenario as well. And finally, I want to return one final time to the fact that for the interface methods described above to operate correctly, you should inherit your main class from theses interfaces: class MyPackage: Package, IVsSolutionLoadEvents, IVsSolutionEvents { //Implementation of Package, IVsSolutionLoadEvents, IVsSolutionEvents ... } Supporting Visual Studio Color Schemes If the extension you are developing will be integrated into the interface of the development environment, for instance, by creating custom tool windows or document MDI windows (and the most convenient way for such an integration is a VSPackage extesnion), it is advisable that the coloring of your custom UI components should match the common color scheme used by Visual Studio itself. The importance of this task was elevated with the release of Visual Studio 2012, containing two hugely opposite color themes (Dark and Light), which the user could switch "on the fly" from the IDE options window. The GetVSSysColorEx method from Visual Studio Interop interface IVsUIShell2 could be utilized to obtain an environment's color settings. This interface is available to VSPackage plugins only: IVsUIShell2 vsshell = this.GetService(typeof(SVsUIShell)) as IVsUIShell2; By passing the __VSSYSCOLOREX and __VSSYSCOLOREX3 enums to the GetVSSysColorEx method, you can get the currently selected color for any of the Visual Studio UI elements. For example, let's obtain one of the colors from the context menu's background gradient: uint Win32Color; vsshell.GetVSSysColorEx((int)__VSSYSCOLOREX3.VSCOLOR_COMMANDBAR_MENU_BACKGROUND_GRADIENTBEGIN, out Win32Color); Color BackgroundGradient1 = ColorTranslator.FromWin32((int)Win32Color); Now we can use this Color object to "paint" our custom context menus. To determine the point in the time at which the color theme of your components should be reapplied, you can, for example, utilize events of the environment command responsible for opening of IDE's settings window (Tools -> Options). How to subscribe your handlers to such an event was described earlier in this article. But if you are, for some reason, unable to utilize the IVsUIShell2 object (for instance, in case you are developing a non-VSPackage extension), but at the same time you still need to support Visual Studio color themes, then it is possible to obtain color values for your environment's various UI components directly from the system registry. We will not cover this approach in the article, but here you can download a free and open-source tool designed for Visual Studio color theme editing. The tool is written in C# and it contains all the code required for reading and modifying Visual Studio 2012 color themes from the managed code. Interacting with COM Interfaces from Within a Multithreaded Application Initially the PVS-Studio extension package had not contained any specific thread-safety mechanisms for its interaction with Visual Studio APIs. At the same time, we had been attempting to confine the interactions with these APIs within a single background thread that was created and owned by our plug-in. And such an approach functioned flawlessly for quite a long period. However, several bug reports from our users, each one containing a similar ComExeption error, prompted us to examine this issue in more detail and to implement a threading safety mechanism for our COM Interop. Although the Visual Studio automation model is not a thread-safe one, it still provides a way for interacting with multi-threaded applications. The Visual Studio application is a COM (Component Object Mode) server. For the task of handling calls from COM clients (in our case, this will be our extension package) to thread-unsafe servers, COM provides a mechanism known as an STA (single-threaded apartment) model. In the terms of COM, an apartment represents a logical container inside a process in which objects and threads share the same thread access rules. STA can hold only a single thread, but an unlimited number of objects, inside such container. Calls from other threads to such thread-unsafe objects inside STA are converted into messages and posted to a message queue. Messages are retrieved from the message queue and converted back into method calls one at a time by the thread running in the STA, so it becomes possible for only a single thread to access these unsafe objects on the server. Utilizing Apartment Mechanism Inside Managed Code The .NET Framework does not utilize COM Apartment mechanics directly. Therefore, when a managed application calls a COM object in the COM interoperation scenarios, CLR (Common Language Runtime) creates and initializes apartment container. A managed thread is able to create and enter either an MTA (multi-threaded apartment, a container that, contrary to an STA, can host several threads at the same time), or an STA, though a thread will be started as an MTA by default. The type of the apartment could be specified before thread is launched: Thread t = new Thread(ThreadProc); t.SetApartmentState(ApartmentState.STA); ... t.Start(); As an apartment type could not be changed once a thread had been started, the STAThread attribute should be used to specify the main thread of a managed application as an STA: [STAThread] static void Main(string[] args) {...} Implementing Message Filters for COM Interoperation Errors in a Managed Environment STA serializes all calls to the COM server, so one of the calling clients could potentially be blocked or even rejected when the server is busy, processing different calls or another thread is already inside the apartment container. In case the COM server rejects its client, .NET COM interop will generate a System.Runtime.InteropServices.COMException ("The message filter indicated that the application is busy"). When working on a Visual Studio module (add-in, vspackage) or a macro, the execution control usually passes into the module from the environment's main STA UI thread (such as in the case of handling events or environment state changes, etc.). Calling automation COM interfaces from this main IDE thread is safe. But if other background threads are planned to be utilized and EnvDTE COM interfaces are to be called from these background threads (as in the case of long calculations that could potentially hang the IDE's interface, if these are performed on the main UI thread), then it is advised to implement a mechanism for handling calls rejected by a server. While working on the PVS-Studio plug-in, we've often encountered these kinds of COM exceptions in situations when other third-party extensions were active inside the IDE simultaneously with the PVS-Studio plug-in. Heavy user interaction with the UI also was the usual cause for such issues. It is quite logical that these situations often resulted in simultaneous parallel calls to COM objects inside STA and consequently to the rejection of some of them. To selectively handle incoming and outgoing calls, COM provides the IMessageFilter interface. If the server implements it, all of the calls are passed to the HandleIncomingCall method, and the client is informed on the rejected calls through the RetryRejectedCall method. This in turn allows the rejected calls to be repeated, or at least to correctly present this rejection to a user (for example, by displaying a dialog with a 'server is busy' message). Following is the example of implementing the rejected call handling for a managed application: [ComImport()] [Guid("00000016-0000-0000-C000-000000000046")] [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] public interface IMessageFilter { [PreserveSig] int HandleInComingCall( int dwCallType, IntPtr hTaskCaller, int dwTickCount, IntPtr lpInterfaceInfo); [PreserveSig] int RetryRejectedCall( IntPtr hTaskCallee, int dwTickCount, int dwRejectType); [PreserveSig] int MessagePending( IntPtr hTaskCallee, int dwTickCount, int dwPendingType); } class MessageFilter : MarshalByRefObject, IDisposable, IMessageFilter { [DllImport("ole32.dll")] [PreserveSig] private static extern int CoRegisterMessageFilter( IMessageFilter lpMessageFilter, out IMessageFilter lplpMessageFilter); private IMessageFilter oldFilter; private const int SERVERCALL_ISHANDLED = 0; private const int PENDINGMSG_WAITNOPROCESS = 2; private const int SERVERCALL_RETRYLATER = 2; public MessageFilter() { //Starting IMessageFilter for COM objects int hr = MessageFilter.CoRegisterMessageFilter( (IMessageFilter)this, out this.oldFilter); System.Diagnostics.Debug.Assert(hr >= 0, "Registering COM IMessageFilter failed!"); } public void Dispose() { //disabling IMessageFilter IMessageFilter dummy; int hr = MessageFilter.CoRegisterMessageFilter(this.oldFilter, out dummy); System.Diagnostics.Debug.Assert(hr >= 0, "De-Registering COM IMessageFilter failed!") System.GC.SuppressFinalize(this); } int IMessageFilter.HandleInComingCall(int dwCallType, IntPtr threadIdCaller, int dwTickCount, IntPtr lpInterfaceInfo) { // Return the ole default (don't let the call through). return MessageFilter.SERVERCALL_ISHANDLED; } int IMessageFilter.RetryRejectedCall(IntPtr threadIDCallee, int dwTickCount, int dwRejectType) { if (dwRejectType == MessageFilter.SERVERCALL_RETRYLATER) { // Retry the thread call immediately if return >=0 & // <100. return 150; //waiting 150 mseconds until retry } // Too busy; cancel call. SERVERCALL_REJECTED return -1; //Call was rejected by callee. //(Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED)) } int IMessageFilter.MessagePending( IntPtr threadIDCallee, int dwTickCount, int dwPendingType) { // Perform default processing. return MessageFilter.PENDINGMSG_WAITNOPROCESS; } } Now we can utilize our MessageFilter while calling COM interfaces from a background thread: using (new MessageFilter()) { //COM-interface dependent code ... } References MSDN. Referencing Automation Assemblies and the DTE2 Object. MSDN. Functional Automation Groups. MZ-Tools. HOWTO: Use correctly the OnConnection method of a Visual Studio add-in. The Code Project. Understanding The COM Single-Threaded Apartment. MZ-Tools. HOWTO: Add an event handler from a Visual Studio add-in. Dr. eX's Blog. Using EnableVSIPLogging to identify menus and commands with VS 2005 + SP1.

The seda component in Apache Camel is very similar to the direct component that I’ve presented in previous blog, but in a asynchronous manner.

EasyNetQ is my little, easy to use, client API for RabbitMQ. It’s been doing really well recently. As I write this, it has 24,653 downloads on NuGet, making it by far the most popular high-level RabbitMQ API. The goal of EasyNetQ is to make working with RabbitMQ as easy as possible. I wanted junior developers to be able to use basic messaging patterns out-of-the-box with just a few lines of code and have EasyNetQ do all the heavy lifting: exchange-binding-queue configuration, error management, connection management, serialization, thread handling; all the things that make working against the low level AMQP C# API, provided by RabbitMQ, such a steep learning curve. To meet this goal, EasyNetQ has to be a very opinionated library. It has a set way of configuring exchanges, bindings and queues based on the .NET type of your messages. However, right from the first release, many users said that they liked the connection management, thread handling, and error management, but wanted to be able to set up their own broker topology. To support this, we introduced the advanced API, an idea stolen shamelessly from Ayende’s RavenDB client. You access the advanced bus (IAdvancedBus) via the Advanced property on IBus: var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced; Sometimes something can seem like a good idea at the time, and then later you think, “WTF! Why on earth did I do that?” It happens to me all the time. I thought it would be cool if I created the exchange-binding-queue topology and then passed it to the publish and subscribe methods, which would then internally declare the exchanges and queues and do the binding. I implemented a tasty little visitor pattern in my ITopologyVisitor. I optimized for my own programming pleasure, rather than an a simple, obvious, easy-to-understand API. I realized a while ago that a more straightforward set of declares on IAdvancedBus would be a far more obvious and intentional design. To this end, I’ve refactored the advanced bus to separate declares from publishing and consuming. I just pushed the changes to NuGet and have also updated the Advanced Bus documentation. Note that these are breaking changes, so please be careful if you are upgrading to the latest version, 0.12, and upwards. Here is a taste of how it works: Declare a queue, exchange and binding, and consume raw message bytes: var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced; var queue = advancedBus.QueueDeclare("my_queue"); var exchange = advancedBus.ExchangeDeclare("my_exchange", ExchangeType.Direct); advancedBus.Bind(exchange, queue, "routing_key"); advancedBus.Consume(queue, (body, properties, info) => Task.Factory.StartNew(() => { var message = Encoding.UTF8.GetString(body); Console.Out.WriteLine("Got message: '{0}'", message); })); Note that I’ve renamed ‘Subscribe’ to ‘Consume’ to better reflect the underlying AMQP method. Declare an exchange and publish a message: var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced; var exchange = advancedBus.ExchangeDeclare("my_exchange", ExchangeType.Direct); using (var channel = advancedBus.OpenPublishChannel()) { var body = Encoding.UTF8.GetBytes("Hello World!"); channel.Publish(exchange, "routing_key", new MessageProperties(), body); } You can also delete exchanges, queues and bindings: var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced; // declare some objects var queue = advancedBus.QueueDeclare("my_queue"); var exchange = advancedBus.ExchangeDeclare("my_exchange", ExchangeType.Direct); var binding = advancedBus.Bind(exchange, queue, "routing_key"); // and then delete them advancedBus.BindingDelete(binding); advancedBus.ExchangeDelete(exchange); advancedBus.QueueDelete(queue); advancedBus.Dispose(); I think these changes make for a much better advanced API. Have a look at the documentation for the details.

A file poller is a very useful mechanism to solve common IT problems. Camel’s built-in file component is extremely flexible, and there are many options available for configuration. Let’s cover few common usages here. Polling a Directory for Input Files Here is a typical Camel Route used to poll a directory for input files every second. import org.slf4j.*; import org.apache.camel.*; import org.apache.camel.builder.*; import java.io.*; public class FileRouteBuilder extends RouteBuilder { static Logger LOG = LoggerFactory.getLogger(FileRouteBuilder.class); public void configure() { from("file://target/input?delay=1000") .process(new Processor() { public void process(Exchange msg) { File file = msg.getIn().getBody(File.class); LOG.info("Processing file: " + file); } }); } } Run this with following: mvn compile exec:java -Dexec.mainClass=org.apache.camel.main.Main -Dexec.args='-r camelcoredemo.FileRouteBuilder' The program will begin to poll your target/input folder under your current directory and wait for incoming files. To test with input files, you would need to open another terminal and then create some files like this: echo 'Hello 1' > target/input/test1.txt echo 'Hello 2' > target/input/test2.txt You should now see the first prompt window start picking up the files and passing them to the next Processor step. In the Processor, we obtain the File object from the message body. It then simply logs its file name. You may hit CTRL+C when you are done. There many configurable options from the file component. You may use this in the URL, but most of the default settings are enough to get you going as the simple case above shows us. Some of these default behaviors are such that if the input folder doesn’t exist, it will create it. And when the file is done processing by the Route, it will be moved into a .camel folder. If you don’t want the file at all after processing, then set delete=true in the URL. Reading in the File Content and Converting to Different Types By default, the file component will create a org.apache.camel.component.file.GenericFile object for each file found and pass it down your Route as message body. You may retrieve all your file information through this object. Alternatively, you may also use the Exchange API to auto convert the message body object to a type you expect to receive (eg: as with msg.getIn().getBody(File.class)). In the example above, the File is a type you expect to get from the message body, and Camel will try to convert it for you. Camel uses the context’s registry space to pre-register many TypeConverter's that can handle the conversion of most of the common data types (like Java primitives). These TypeConverters are a powerful way to make your Route and Processor more flexible and portable. Camel will not only convert just your File object from a message body, but it can also read the file content. If your files are character text based, then you can simply do this. from("file://target/input?charset=UTF-8") .process(new Processor() { public void process(Exchange msg) { String text = msg.getIn().getBody(String.class); LOG.info("Processing text: " + text); } }); That’s it! Simply specify that it is a String type, and Camel will read your file and pass in the entire file text content as a body message. You may even use the charset to change the encoding. If you are dealing with a binary file, then simply try byte[] bytes =msg.getIn().getBody(byte[].class); conversion instead. Pretty cool huh? Polling and Processing Large Files When working with large files, there are a few options in the file component that you might want to use to ensure proper handling. For example, you might want to move the input file into a staging folder before the Route starts the processing; and when it’s done, move it to a .completed folder. from("file://target/input?preMove=staging&move=.completed") .process(new Processor() { public void process(Exchange msg) { File file = msg.getIn().getBody(File.class); LOG.info("Processing file: " + file); } }); To feed input files properly into the polling folder, it’s best if the sender generates the input files in a temporary folder first, and only when it’s ready then move it into the polling folder. This will minimize reading an incomplete file by the Route if the input file might take time to generate. Another solution to this is to configure the file endpoint to only read the polling folder when there is a signal or when a ready-marker file exists. For example: from("file://target/input?preMove=staging&move=.completed&doneFileName=ReadyFile.txt") .process(new Processor() { public void process(Exchange msg) { File file = msg.getIn().getBody(File.class); LOG.info("Processing file: " + file); } }); The code above will only read the target/input folder when a ReadyFile.txt file exists. The marker file can just be an empty file, and it will be removed by Camel after polling. This solution would allow the sender to generate input files in no matter how long it takes. Another concern with large file processing is avoiding loading a file's entire content into memory for processing. To be more practical, you want to split the file into records (eg: per line) and process it one by one (this is called "streaming"). Here is how you would do that using Camel. from("file://target/input?preMove=staging&move=.completed") .split(body().tokenize("\n")) .streaming() .process(new Processor() { public void process(Exchange msg) { String line = msg.getIn().getBody(String.class); LOG.info("Processing line: " + line); } }); This Route will allow you to process large size file without consuming too much memory, and it will process it line-by-line very efficiently. Writing Messages Back into File The file component can also be used to write messages into files. Recall that we may use dataset components to generate sample messages. We will use that to feed the Route and send it to the file component so you can see that each message generated will be saved into a file. package camelcoredemo; import org.slf4j.*; import org.apache.camel.*; import org.apache.camel.builder.*; import org.apache.camel.main.Main; import org.apache.camel.component.dataset.*; public class FileDemoCamel extends Main { static Logger LOG = LoggerFactory.getLogger(FileDemoCamel.class); public static void main(String[] args) throws Exception { FileDemoCamel main = new FileDemoCamel(); main.enableHangupSupport(); main.addRouteBuilder(createRouteBuilder()); main.bind("sampleGenerator", createDataSet()); main.run(args); } static RouteBuilder createRouteBuilder() { return new RouteBuilder() { public void configure() { from("dataset://sampleGenerator") .to("file://target/output"); } }; } static DataSet createDataSet() { return new SimpleDataSet(); } } Compile and run it. mvn compile exec:java -Dexec.mainClass=camelcoredemo.FileDemoCamel Upon completion, you will see that 10 files would be generated in the target/output folder with the file name in ID--- format. There are more options availabe from File component that you may explore. Try it out with a Route and see for yourself.

I’ll leave it to the experts to tell how awesome OSGi is. Among the many benefits, I could tell you why we picked up OSGi for a pet project - Modularity, avoiding JAR hell and dynamic updates (hey, why not?) We chose Apache Felix (an OSGi framework specification implementation) and Apache Karaf (ummm, how do I put this - something like an app server for OSGi applications). Besides serving as an OSGi container, Karaf has a lot of awesome features (pun intended). And we like the idea of managing multiple Karaf instances managed through Zookeeper. Enough talk, let’s see some code. This is a rudimentary tutorial on how to run a basic OSGi CXF-JAX Rest Service on Karaf. Given a REST parameter (a name), the service would just return Hello, (name). That’s it !! The entire project could be downloaded here So, if the request is http://localhost:8181/cxf/karafsimple/say/hello/arun, the response would be Hello, arun Please note that this project does not have any domain models and therefore has only two projects - one for the REST (Controller) and the other for the actual service implementation. The project structure looks like this : Step 1 - Service Implementation This project just has two ‘useful’ things - the HelloService interface and the HelloServiceImpl class. HelloService package me.rerun.karafcxf.service.impl; public interface HelloService { public String sayHello(String name); } HelloServiceImpl package me.rerun.karafcxf.service.impl; public class HelloServiceImpl implements HelloService { @Override public String sayHello(String name) { return "Hello, "+name; } } Stupid right? Step 2 - REST project Similar to the Service project, this project also has just two notable things - the HelloRestService interface and the HelloRestServiceImpl class. HelloRestService HelloRestService package me.rerun.karafcxf.rest; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.PathParam; //Maps for the `say` in the URL @Path("say") public interface HelloRestService { @GET @Path("hello/{name}") //Maps for the `hello/John` in the URL public String handleGet(@PathParam("name") String name); } HelloRestServiceImpl The HelloRestServiceImpl does nothing but calls the HelloService which gets injected through Blueprint Dependency Injection. Hey, does the inject look very familiar to Spring DI? Exactly !! The Blueprint DI is heavily influenced by Spring DI. In fact, the original work for blueprint is done by Spring. HelloRestServiceImpl package me.rerun.karafcxf.rest; import me.rerun.karafcxf.service.impl.HelloService; public class HelloRestServiceImpl implements HelloRestService{ //Just like Spring. Please add Getters/Setters. Blueprint annotations are still work in progress private HelloService helloService; public String handleGet(String name){ return helloService.sayHello(name); } /* Constructor */ public HelloRestServiceImpl(){ } /* Getters and Setters */ public HelloService getHelloService() { return helloService; } public void setHelloService(HelloService helloService) { this.helloService = helloService; } } Step 3 - Injections XMLs (of any name) inside OSGI-INF/blueprint folder will get picked up for DI scanning. serviceimpl.xml Does two things in one tag : Registers the HelloService into the service registry for lookup Says that its implementation is HelloServiceImpl resources/OSGI-INF/blueprint/serviceimpl.xml In fact, you could do this in two separate steps. More on that here. resources/OSGI-INF/blueprint/serviceimpl.xml rest.xml resources/OSGI-INF/blueprint/rest.xml 1) cxf-bus is the bus configuration for CXF. It is like the manager for all CXF services. The most common use as far as I know is to configure custom interceptors (for auditing, request/response manipulation, headers manipulation etc) 2) the jaxrs:server initiates a server which would start listening to the URLs that we mapped for. Of course, we would want to map the handlers for the URLs and those go under the serviceBeans. 3) The third note in the XML is the just the restServiceImpl configuration and the injection of the helloService as a property inside the HelloRestServiceImpl 4) The reference tag will lookup the service registry for a bean with the same id. Step 4 - KAR - Karaf Archive (Optional but easier this way) Technically, you could just start throwing the bundles generated through the Maven into to the deploy directory of karaf. However, as the project goes big and your dependencies are becoming many, it is advisable to use the .kar archive. Picture .kar as your .war or .ear bundle. A .kar archive, internally, looks something like this : Building .kar Building the .kar is composed of two steps : Step 1 feature.xml Similar to your web descriptor or application descriptor, we have the features descriptor in Karaf to represent the entire repository that we are bundling together to compose our application. In our case, we don’t have any external .jar dependencies except for the cxf and the http service for which we are using the in-built features available inside karaf. feature.xml ${project.description} http cxf this-project-dependants mvn:karafcxf/karafcxf.service.impl/1.0-SNAPSHOT mvn:karafcxf/karafcxf.rest/1.0-SNAPSHOT Step 2 Maven plugin configuration to create .kar, which indicates where your feature.xml is located (Note that this file is located inside your karaf project) pom.xml org.apache.karaf.tooling features-maven-plugin 2.3.2 create-kar create-kar ${project.basedir}/src/main/resources/feature.xml Wraps Notice the wrap protocol in front of the mvn protocol in few bundles as in wrap:mvn:org.apache.httpcomponents/httpmime/4.2.5 Not all Jars are OSGi ready but they would obviously be used as a dependency in our project. In those cases, Karaf notices the wrapprotocol and bundles the JAR into an OSGi bundle. In case of doubt, just drop off the wrap and Karaf would complain that the jar is not OSGi compatible. (Alternatively, you could open up each of the dependant jars and check their manifest files) 013-08-28 01:38:48,669 | WARN | raf-2.3.2/deploy | KarArtifactInstaller | eployer.kar.KarArtifactInstaller 192 | 24 - org.apache.karaf.deployer.kar - 2.3.2 | Unable to install Kar feature xx-xxx-xxxxxx/0.0.0 org.osgi.framework.BundleException: Jar is not a bundle, no Bundle-SymbolicName mvn:org.apache.httpcomponents/httpcore/4.2.4 at org.apache.karaf.features.internal.FeaturesServiceImpl.installBundleIfNeeded(FeaturesServiceImpl.java:836)[26:org.apache.karaf.features.core:2.3.2] at org.apache.karaf.features.internal.FeaturesServiceImpl.doInstallFeature(FeaturesServiceImpl.java:618)[26:org.apache.karaf.features.core:2.3.2] at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:414)[26:org.apache.karaf.features.core:2.3.2] at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:402)[26:org.apache.karaf.features.core:2.3.2] at Proxy508d2419_d21e_4a93_b7fb_26e28d2f03a6.installFeature(Unknown Source)[:] at org.apache.karaf.deployer.kar.KarArtifactInstaller.installFeatures(KarArtifactInstaller.java:189)[24:org.apache.karaf.deployer.kar:2.3.2] at org.apache.karaf.deployer.kar.KarArtifactInstaller.install(KarArtifactInstaller.java:134)[24:org.apache.karaf.deployer.kar:2.3.2] at org.apache.karaf.deployer.kar.KarArtifactInstaller.update(KarArtifactInstaller.java:348)[24:org.apache.karaf.deployer.kar:2.3.2] at org.apache.felix.fileinstall.internal.DirectoryWatcher.update(DirectoryWatcher.java:1103)[6:org.apache.felix.fileinstall:3.2.6] at org.apache.felix.fileinstall.internal.DirectoryWatcher.update(DirectoryWatcher.java:898)[6:org.apache.felix.fileinstall:3.2.6] at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:482)[6:org.apache.felix.fileinstall:3.2.6] at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:291)[6:org.apache.felix.fileinstall:3.2.6] Changing Log Levels For all our development environment, we would want to increase our log level to get more feedback from Karaf. This could be achieved by modifying the org.ops4j.pax.logging.cfg file located in your /etc Step 5 - Other Maven configuration Not technically a step because we would have covered this from the beginning anyway. And nothing fancy here. Parent pom.xml Has the rest of the sub-modules configured. The various library dependencies configured pom.xml 4.0.0 karafcxf karafcxf pom 1.0-SNAPSHOT 2.7.5 karafcxf.rest karafcxf.service.impl karafcxf.kar … … Rest and ServiceImpl pom.xml The other poms.xmls aren’t interesting They just affiliate themselves to the parent pom with the parent tag as in pom.xml org.apache.felix maven-bundle-plugin 2.3.7 true ${project.artifactId} ${project.version} com.nutraspace.coreservices.search.rest.Activator com.nutraspace.coreservices.search.rest*;version=${project.version} * Step 6 - Bring up Karaf My Karaf installation is located at : /Users/Gabriel/apps/apache-karaf-2.3.2 Start Karaf : /bin/./karaf Installing CXF and HTTP services features:chooseurl cxf 2.7.5 features:install http cxf Checking whether your bundle is installed and your service running osgi:list dxf:list-endpoints Stop Karaf Ctrl +D (Please note that Ctrl +C closes the connection abruptly instead of stopping Karaf) In case of accidental Ctrl+C and if Karaf isn’t starting properly, do a rm -rf /data/cache/* URL Mapping Like I mentioned earlier, the target request URL will be something like http://localhost:8181/cxf/karafsimple/say/hello/arun and the response would be Hello, arun 1) The HelloRestService interface has all the JAX RS annotations for the URL mapping. Well, technically, the interfaces just maps for say/hello/(name) 2) The karafsimple in the URL is derived from the the JAX RS address in blueprint.xml (about that later in Step 3 - rest.xml). 3) The cxf is a default if you deploy a CXF service on Karaf which obviously you could change.

Camel Timer is a simple and yet useful component. It brings the JDK’s timer functionality into your camel Route with very simple config. from("timer://mytimer?period=1000") .process(new Processor() { public void process(Exchange msg) { LOG.info("Processing {}", msg); } }); That will generate a timer event message every second. You may short hand 1000 with 1s instead. It supports mfor minutes, or h for hours as well. Pretty handy. Another useful timer feature is that it can limit (stop) the number of timer messages after a certain count. You simply need to add repeatCount option toward the url. Couple of properties from the event message would be useful when handling the timer message. Here is an example how to read them. from("timer://mytimer?period=1s&repeatCount=5") .process(new Processor() { public void process(Exchange msg) { java.util.Date fireTime = msg.getProperty(Exchange.TIMER_FIRED_TIME, java.util.Date.class); int eventCount = msg.getProperty(Exchange.TIMER_COUNTER, Integer.class); LOG.info("We received {}th timer event that was fired on {}", eventCount, fireTime); } }); There are more options availabe from Timer component that you may explore. Try it out with a Route and see it for yourself.

API Management solutions must combine an API Portal (for signing up developers) with an API Gateway (to link back to the enterprise). But where do these come from, and what is the relationship with SOA? To answer these questions, first let's look at a bit of history: In the 2000's, we had the SOA Gateway and the SOA Registry, working hand-in-hand. This was "SOA Governance". The SOA Registry (with a Repository) was intended to be the "central store of truth" for information about Web Services. It was often the public face of SOA Governance, the part which people could see. Usually the services in the registry took the form of heavyweight SOAP services, defined by WSDLs. The problem was that developers were often forced to register their SOAP services in the registry, rather than feeling that it was something beneficial to them. Browsing the registry was also a chore, involving the use of UDDI, also a heavyweight protocol (in fact, it was built on SOAP). Fast-forward to the current decade, and we find that the SOA Registry has been replaced by the API Portal. An API portal is also the "central store of truth", but now it includes REST APIs definitions (usually expressed using a Swagger-type format) as well as SOAP services. The API Portal is designed to be useful and helpful to developers who wish to build apps, rather than feeling like a chore to use. The lesson of SOA was that an attitude of "If we build it, they will come" (or "If we put it in the SOA Registry, people will use it") does not work. You have to make it into a pleasant experience for developers. API portals work for the very reason that SOA registries did not work: usability. Just like the SOA Gateway worked with the SOA Registry, so the API Gateway works hand-in-hand with the API Portal. Together, the combination of the API Portal with the API Gateway constitutes "API Management". The API Portal is for developers to sign up to use APIs, receive API Keys and quotas, and the API Gateway operates at runtime, managing the API Key usage and enforcing the API usage quotas. The API Gateway also performs the very important task of bridging from the technologies used by API clients (REST, OAuth) to the technologies used in the enterprise (Kerberos, SAML, or proprietary identity tokens such as CA SiteMinder smsession tokens). For more on this bridging, check out my webinar with Jason Cardinal from Identica tomorrow on "Bridging APIs to Enterprise Infrastructure". Gartner defines the combination of SOA Governance and API Management as "Application Services Governance". I'm proud to say that Axway (which acquired Vordel in 2012) is recognized by Gartner as a Leader in the category of Application Services Governance. We've seen an evolution of technologies (SOAP to REST) and approach (the UDDI registry to the web-based API Portal) in the journey from SOA Governance to API Management. From 30,000 feet, SOA Governance and API Management might look similar, but the new approach of API Management has already outshone SOA. The API Gateway and API Portal are key to this.

Apache Wicket is known for its capability of transparently handling the state of web applications on server side, and can be easily adopted to create RESTful services.