You can easily tell Maven to include the version of the artifact and its Git/SVN/… revision in the JAR manifest file and then access that information at runtime via getClass().getPackage.getImplementationVersion(). (All credit goes to Markus Krüger and other colleagues.) Include Maven artifact version in the manifest (Note: You will actually not want to use it, if you also want to include a SCM revision; see below.) pom.xml: ... org.apache.maven.plugins maven-jar-plugin ... true true ... ... The resulting MANIFEST.MF of the JAR file will then include the following entries, with values from the indicated properties: Built-By: ${user.name} Build-Jdk: ${java.version} Specification-Title: ${project.name} Specification-Version: ${project.version} Specification-Vendor: ${project.organization.name Implementation-Title: ${project.name} Implementation-Version: ${project.version} Implementation-Vendor-Id: ${project.groupId} Implementation-Vendor: ${project.organization.name} (Specification-Vendor and Implementation-Vendor come from the POM’s organization/name.) Include SCM revision For this you can either use the Build Number Maven plugin that produces the property ${buildNumber}, or retrieve it from environment variables passed by Jenkinsor Hudson (SVN_REVISION for Subversion, GIT_COMMIT for Git). For git alone, you could also use the maven-git-commit-id-plugin that can either replace strings such as ${git.commit.id} in existing resource files (using maven’s resource filtering, which you must enable) with the actual values or output all of them into a git.properties file. Let’s use the buildnumber-maven-plugin and create the manifest entries explicitely, containing the build number (i.e. revision) org.codehaus.mojo buildnumber-maven-plugin 1.2 validate create false false org.apache.maven.plugins maven-jar-plugin 2.4 ${project.name} ${project.version} ${buildNumber} ... Accessing the version & revision As mentioned above, you can access the manifest entries from your code via getClass().getPackage.getImplementationVersion() andgetClass().getPackage.getImplementationTitle(). References SO: How to get Maven Artifact version at runtime? Maven Archiver documentation

Have you ever tried to update specific fields of an entity using Entity Framework? If yes, then I can assume that System.Data.Entity.Validation.DbEntityValidationResult – Validation failed for one or more entities. See ‘EntityValidationErrors’ property for more details. Exception is not something new to you. While working recently on a project for which I was using Entity Framework 5.x CF (the part with code first is not relevant) I bumped into an interesting piece of code. The main idea behind it is to perform updates for a single property and ignore the rest even if they don't respect the constraints defined in the model. Please note that I have found PartialEntityValidation class somewhere on the internet and unfortunately I do not remember where, I was not able to find it again. The only changes made by me are related to code formatting. I will present the code first and explain the logic behind it afterwards. using System.Collections.Generic; using System.Data.Entity.Infrastructure; using System.Data.Entity.Validation; namespace MyProject.Data.Contexts.Concrete { public class PartialEntityValidation { private readonly IDictionary _store; public PartialEntityValidation() { _store = new Dictionary(); } public void Register(DbEntityEntry entry, string[] properties) { if (_store.ContainsKey(entry)) _store[entry] = properties; else _store.Add(entry, properties); } public void Unregister(DbEntityEntry entry) { _store.Remove(entry); } public bool IsResponsibleFor(DbEntityEntry entry) { return _store.ContainsKey(entry); } public void Validate(DbEntityValidationResult validationResult) { var entry = validationResult.Entry; foreach (var property in _store[entry]) { var validationErrors = entry.Property(property).GetValidationErrors(); foreach (var validationError in validationErrors) validationResult.ValidationErrors.Add(validationError); } } } } PartialEntityValidation class has a simple constructor which initializes a new dictionary store where the key of it is an DbEntityEntry and the value (currently) an empty string array. The string array will hold later on the properties which should be validated. The class has four methods: Register, Unregister, IsResponsibleFor and Validate. Now let me explain a bit what each method does. Register adds a new DbEntityEntry with the properties that have to be validated to the existing store. It basically registers a new entity entry for validation. It is quite obvious what Unregister method does; it removes an entity entry from the existing store. IsResponsibleFor method determines if an entity entry is present in the existing store and if it will be part of the validation process. The last method is Validate which accepts a single parameter of DbEntityValidationResult type, from which it takes the entity entry associated with the validation result and validates each property of the entity entry present in the store. If there are any validation errors those are added on ValidationErrors collection of the validation result. Now it is time to see how to use PartialEntityValidation class. In My case I have exposed it as a public read only property of the context and had ValidateEntity method overridden at context level. using System; using System.Collections.Generic; using System.Configuration; using System.Data; using System.Data.Entity; using System.Data.Entity.Infrastructure; using System.Data.Entity.Validation; namespace MyProject.Data.Contexts.Concrete { public class MyContext : DbContext { private readonly PartialEntityValidation _partialEntityValidation; static MyContext() { Database.SetInitializer(null); } protected MyContext() : base("MyDb") { _partialEntityValidation = new PartialEntityValidation(); Configuration.LazyLoadingEnabled = false; } public PartialEntityValidation PartialValidation { get { return _partialEntityValidation; } } protected override DbEntityValidationResult ValidateEntity(DbEntityEntry entityEntry, IDictionary items) { if (PartialValidation.IsResponsibleFor(entityEntry)) { var validationResult = new DbEntityValidationResult(entityEntry, new List()); PartialValidation.Validate(validationResult); return validationResult; } return base.ValidateEntity(entityEntry, items); } } } I am sure that I do not have to explain the logic behind ValidateEntity method from the db context. The final piece is the repository where the partial update is invoked. using System; using System.Linq; using System.Linq.Expressions; namespace MyProject.Data.Repositories.Concrete { public class MyRepository { private readonly MyContext _context; public MyRepository(MyContext context) { _context = context; } public void DeleteProduct(Guid productId) { var product = new Product { ProductId = productId, IsDeleted = true }; _context.PartialValidation.Register(_context.Entry(product), new[] { "IsDeleted" }); Update(workspace, p => p.IsDeleted); _context.SaveChanges(); ((IObjectContextAdapter)_context).ObjectContext.Detach(product); } public void Update(T entity, Expression> property) where T : class { var entry = _context.Entry(entity); _context.Set().Attach(entity); entry.Property(property).IsModified = true; } } } Please take into consideration that this is a demo repository which doesn't do much at all except to present how a partial update is performed. The repository has two important methods. First one is DeleteProduct which performs a soft delete by updating "IsDeleted" property of product entity to true. This is achieved by creating a new product object of which primary key is "ProductId" with "IsDeleted" property set to true. The next step is to register the entity entry along with the "IsDeleted" property (this is the only property we want to have it updated, all other properties of Product entity will be ignored) into PartialValidation store. Update method is called which attaches the entity entry to the context and marks it as modified in order to have it updated by entity framework. That is all, I really hope that you got the idea behind this even if I did not provided an actual sample application for download.

In this blog post, I will present a simple tutorial on uploading a large file to Amazon S3 as fast as the network supports. Amazon S3 is clustered storage service of Amazon. It is designed to make web-scale computing easier. Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers. For using Amazon services, you'll need your AWS access key identifiers, which AWS assigned you when you created your AWS account. The following are the AWS access key identifiers: Access Key ID (a 20-character, alphanumeric sequence) For example: 022QF06E7MXBSH9DHM02 Secret Access Key (a 40-character sequence) For example: kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct Caution Your Secret Access Key is a secret, which only you and AWS should know. It is important to keep it confidential to protect your account. Store it securely in a safe place. Never include it in your requests to AWS, and never e-mail it to anyone. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your Secret Access Key. The Access Key ID is associated with your AWS account. You include it in AWS service requests to identify yourself as the sender of the request. The Access Key ID is not a secret, and anyone could use your Access Key ID in requests to AWS. To provide proof that you truly are the sender of the request, you also include a digital signature calculated using your Secret Access Key. The sample code handles this for you. Your Access Key ID and Secret Access Key are displayed to you when you create your AWS account. They are not e-mailed to you. If you need to see them again, you can view them at any time from your AWS account. To get your AWS access key identifiers Go to the Amazon Web Services web site at http://aws.amazon.com. Point to Your Account and click Security Credentials. Log in to your AWS account. The Security Credentials page is displayed. Your Access Key ID is displayed in the Access Identifiers section of the page. To display your Secret Access Key, click Show in the Secret Access Key column. You can use your Amazon keys from a properties file in your application. Here is a sample for properties file containing Amazon keys: # Fill in your AWS Access Key ID and Secret Access Key # http://aws.amazon.com/security-credentials accessKey = secretKey = Here is sample AmazonUtil class for getting AWS Credentials from properties file. public class AmazonUtil { private static final Logger logger = LogUtil.getLogger(); private static final String AWS_CREDENTIALS_CONFIG_FILE_PATH = ConfigUtil.CONFIG_DIRECTORY_PATH + File.separator + "aws-credentials.properties"; private static AWSCredentials awsCredentials; static { init(); } private AmazonUtil() { } private static void init() { try { awsCredentials = new PropertiesCredentials(IOUtil.getResourceAsStream(AWS_CREDENTIALS_CONFIG_FILE_PATH)); } catch (IOException e) { logger.error("Unable to initialize AWS Credentials from " + AWS_CREDENTIALS_CONFIG_FILE_PATH); } } public static AWSCredentials getAwsCredentials() { return awsCredentials; } } Amazon S3 has Multipart Upload service which allows faster, more flexible uploads into Amazon S3. Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size. For more information on Multipart Upload, review the Amazon S3 Developer Guide In this tutorial, my sample application uploads each file parts to Amazon S3 with different threads for using network throughput as possible as much. Each file part is associated with a thread and each thread uploads its associated part with Amazon S3 API. Figure 1. Amazon S3 Parallel Multi-Part File Upload Mechanism Amazon S3 API suppots MultiPart File Upload in this way: 1. Send a MultipartUploadRequest to Amazon. 2. Get a response containing a unique id for this upload operation. 3. For i in ${partCount} 3.1. Calculate size and offset of split-i in whole file. 3.2. Build a UploadPartRequest with file offset, size of current split and unique upload id. 3.3. Give this request to a thread and starts upload by running thread. 3.3.1. Send associated UploadPartRequest to Amazon. 3.3.2. Get response after successful upload and save ETag property of response. 4. Wait all threads to terminate 5. Get ETags (ETag is an identifier for successfully completed uploads) of all terminated threads. 6. Send a CompleteMultipartUploadRequest to Amazon with unique upload id and all ETags. So Amazon joins all file parts as target objects. Here is implementation: public class AmazonS3Util { private static final Logger logger = LogUtil.getLogger(); public static final long DEFAULT_FILE_PART_SIZE = 5 * 1024 * 1024; // 5MB public static long FILE_PART_SIZE = DEFAULT_FILE_PART_SIZE; private static AmazonS3 s3Client; private static TransferManager transferManager; static { init(); } private AmazonS3Util() { } private static void init() { // ... s3Client = new AmazonS3Client(AmazonUtil.getAwsCredentials()); transferManager = new TransferManager(AmazonUtil.getAwsCredentials()); } // ... public static void putObjectAsMultiPart(String bucketName, File file) { putObjectAsMultiPart(bucketName, file, FILE_PART_SIZE); } public static void putObjectAsMultiPart(String bucketName, File file, long partSize) { List partETags = new ArrayList(); List uploaders = new ArrayList(); // Step 1: Initialize. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, file.getName()); InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest); long contentLength = file.length(); try { // Step 2: Upload parts. long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { // Last part can be less than part size. Adjust part size. partSize = Math.min(partSize, (contentLength - filePosition)); // Create request to upload a part. UploadPartRequest uploadRequest = new UploadPartRequest(). withBucketName(bucketName).withKey(file.getName()). withUploadId(initResponse.getUploadId()).withPartNumber(i). withFileOffset(filePosition). withFile(file). withPartSize(partSize); uploadRequest.setProgressListener(new UploadProgressListener(file, i, partSize)); // Upload part and add response to our list. MultiPartFileUploader uploader = new MultiPartFileUploader(uploadRequest); uploaders.add(uploader); uploader.upload(); filePosition += partSize; } for (MultiPartFileUploader uploader : uploaders) { uploader.join(); partETags.add(uploader.getPartETag()); } // Step 3: complete. CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, file.getName(), initResponse.getUploadId(), partETags); s3Client.completeMultipartUpload(compRequest); } catch (Throwable t) { logger.error("Unable to put object as multipart to Amazon S3 for file " + file.getName(), t); s3Client.abortMultipartUpload( new AbortMultipartUploadRequest( bucketName, file.getName(), initResponse.getUploadId())); } } // ... private static class UploadProgressListener implements ProgressListener { File file; int partNo; long partLength; UploadProgressListener(File file) { this.file = file; } @SuppressWarnings("unused") UploadProgressListener(File file, int partNo) { this(file, partNo, 0); } UploadProgressListener(File file, int partNo, long partLength) { this.file = file; this.partNo = partNo; this.partLength = partLength; } @Override public void progressChanged(ProgressEvent progressEvent) { switch (progressEvent.getEventCode()) { case ProgressEvent.STARTED_EVENT_CODE: logger.info("Upload started for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.COMPLETED_EVENT_CODE: logger.info("Upload completed for file " + "\"" + file.getName() + "\"" + ", " + file.length() + " bytes data has been transferred"); break; case ProgressEvent.FAILED_EVENT_CODE: logger.info("Upload failed for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.CANCELED_EVENT_CODE: logger.info("Upload cancelled for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.PART_STARTED_EVENT_CODE: logger.info("Upload started at " + partNo + ". part for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.PART_COMPLETED_EVENT_CODE: logger.info("Upload completed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + (partLength > 0 ? partLength : progressEvent.getBytesTransfered()) + " bytes data has been transferred"); break; case ProgressEvent.PART_FAILED_EVENT_CODE: logger.info("Upload failed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; } } } private static class MultiPartFileUploader extends Thread { private UploadPartRequest uploadRequest; private PartETag partETag; MultiPartFileUploader(UploadPartRequest uploadRequest) { this.s3Client = s3Client; this.uploadRequest = uploadRequest; } @Override public void run() { partETag = s3Client.uploadPart(uploadRequest).getPartETag(); } private PartETag getPartETag() { return partETag; } private void upload() { start(); } } }

I happen to read through a chapter on XML parsing and building APIs in Java. And I tried out the different parsers on a sample XML. Then I thought of sharing it on my blog so that I can have a reference to the code as well as a reference for anyone reading this. In this post I parse the same XML in different parsers to perform the same operation of populating the XML content into objects and then adding the objects to a list. The sample XML considered in the examples is: Rakesh Mishra Bangalore John Davis Chennai Rajesh Sharma Pune And the obejct into which the XML content is to be extracted is defined as below: class Employee{ String id; String firstName; String lastName; String location; @Override public String toString() { return firstName+" "+lastName+"("+id+")"+location; } } There are 3 main parsers for which I have given sample code: DOM Parser SAX Parser StAX Parser Using DOM Parser I am making use of the DOM parser implementation that comes with the JDK and in my example I am using JDK 7. The DOM Parser loads the complete XML content into a Tree structure. And we iterate through the Node and NodeList to get the content of the XML. The code for XML parsing using DOM parser is given below. public class DOMParserDemo { public static void main(String[] args) throws Exception { //Get the DOM Builder Factory DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); //Get the DOM Builder DocumentBuilder builder = factory.newDocumentBuilder(); //Load and Parse the XML document //document contains the complete XML as a Tree. Document document = builder.parse( ClassLoader.getSystemResourceAsStream("xml/employee.xml")); List empList = new ArrayList<>(); //Iterating through the nodes and extracting the data. NodeList nodeList = document.getDocumentElement().getChildNodes(); for (int i = 0; i < nodeList.getLength(); i++) { //We have encountered an tag. Node node = nodeList.item(i); if (node instanceof Element) { Employee emp = new Employee(); emp.id = node.getAttributes(). getNamedItem("id").getNodeValue(); NodeList childNodes = node.getChildNodes(); for (int j = 0; j < childNodes.getLength(); j++) { Node cNode = childNodes.item(j); //Identifying the child tag of employee encountered. if (cNode instanceof Element) { String content = cNode.getLastChild(). getTextContent().trim(); switch (cNode.getNodeName()) { case "firstName": emp.firstName = content; break; case "lastName": emp.lastName = content; break; case "location": emp.location = content; break; } } } empList.add(emp); } } //Printing the Employee list populated. for (Employee emp : empList) { System.out.println(emp); } } } class Employee{ String id; String firstName; String lastName; String location; @Override public String toString() { return firstName+" "+lastName+"("+id+")"+location; } } The output for the above will be: Rakesh Mishra(111)Bangalore John Davis(112)Chennai Rajesh Sharma(113)Pune Using SAX Parser SAX Parser is different from the DOM Parser where SAX parser doesn’t load the complete XML into the memory, instead it parses the XML line by line triggering different events as and when it encounters different elements like: opening tag, closing tag, character data, comments and so on. This is the reason why SAX Parser is called an event based parser. Along with the XML source file, we also register a handler which extends the DefaultHandler class. The DefaultHandler class provides different callbacks out of which we would be interested in: startElement() – triggers this event when the start of the tag is encountered. endElement() – triggers this event when the end of the tag is encountered. characters() – triggers this event when it encounters some text data. The code for parsing the XML using SAX Parser is given below: import java.util.ArrayList; import java.util.List; import javax.xml.parsers.SAXParser; import javax.xml.parsers.SAXParserFactory; import org.xml.sax.Attributes; import org.xml.sax.SAXException; import org.xml.sax.helpers.DefaultHandler; public class SAXParserDemo { public static void main(String[] args) throws Exception { SAXParserFactory parserFactor = SAXParserFactory.newInstance(); SAXParser parser = parserFactor.newSAXParser(); SAXHandler handler = new SAXHandler(); parser.parse(ClassLoader.getSystemResourceAsStream("xml/employee.xml"), handler); //Printing the list of employees obtained from XML for ( Employee emp : handler.empList){ System.out.println(emp); } } } /** * The Handler for SAX Events. */ class SAXHandler extends DefaultHandler { List empList = new ArrayList<>(); Employee emp = null; String content = null; @Override //Triggered when the start of tag is found. public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException { switch(qName){ //Create a new Employee object when the start tag is found case "employee": emp = new Employee(); emp.id = attributes.getValue("id"); break; } } @Override public void endElement(String uri, String localName, String qName) throws SAXException { switch(qName){ //Add the employee to list once end tag is found case "employee": empList.add(emp); break; //For all other end tags the employee has to be updated. case "firstName": emp.firstName = content; break; case "lastName": emp.lastName = content; break; case "location": emp.location = content; break; } } @Override public void characters(char[] ch, int start, int length) throws SAXException { content = String.copyValueOf(ch, start, length).trim(); } } class Employee { String id; String firstName; String lastName; String location; @Override public String toString() { return firstName + " " + lastName + "(" + id + ")" + location; } } The output for the above would be: Rakesh Mishra(111)Bangalore John Davis(112)Chennai Rajesh Sharma(113)Pune Using StAX Parser StAX stands for Streaming API for XML and StAX Parser is different from DOM in the same way SAX Parser is. StAX parser is also in a subtle way different from SAX parser. The SAX Parser pushes the data but StAX parser pulls the required data from the XML. The StAX parser maintains a cursor at the current position in the document allows to extract the content available at the cursor whereas SAX parser issues events as and when certain data is encountered. XMLInputFactory and XMLStreamReader are the two class which can be used to load an XML file. And as we read through the XML file using XMLStreamReader, events are generated in the form of integer values and these are then compared with the constants in XMLStreamConstants. The below code shows how to parse XML using StAX parser: import java.util.ArrayList; import java.util.List; import javax.xml.stream.XMLInputFactory; import javax.xml.stream.XMLStreamConstants; import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLStreamReader; public class StaxParserDemo { public static void main(String[] args) throws XMLStreamException { List empList = null; Employee currEmp = null; String tagContent = null; XMLInputFactory factory = XMLInputFactory.newInstance(); XMLStreamReader reader = factory.createXMLStreamReader( ClassLoader.getSystemResourceAsStream("xml/employee.xml")); while(reader.hasNext()){ int event = reader.next(); switch(event){ case XMLStreamConstants.START_ELEMENT: if ("employee".equals(reader.getLocalName())){ currEmp = new Employee(); currEmp.id = reader.getAttributeValue(0); } if("employees".equals(reader.getLocalName())){ empList = new ArrayList<>(); } break; case XMLStreamConstants.CHARACTERS: tagContent = reader.getText().trim(); break; case XMLStreamConstants.END_ELEMENT: switch(reader.getLocalName()){ case "employee": empList.add(currEmp); break; case "firstName": currEmp.firstName = tagContent; break; case "lastName": currEmp.lastName = tagContent; break; case "location": currEmp.location = tagContent; break; } break; case XMLStreamConstants.START_DOCUMENT: empList = new ArrayList<>(); break; } } //Print the employee list populated from XML for ( Employee emp : empList){ System.out.println(emp); } } } class Employee{ String id; String firstName; String lastName; String location; @Override public String toString(){ return firstName+" "+lastName+"("+id+") "+location; } } The output for the above is: Rakesh Mishra(111) Bangalore John Davis(112) Chennai Rajesh Sharma(113) Pune With this I have covered parsing the same XML document and performing the same task of populating the list of Employee objects using all the three parsers namely: DOM Parser SAX Parser StAX Parser

Avro has the ability to generate Java code from Avro schema, IDL and protocol files. Avro also has a plugin which allows you to generate these Java sources directly from Maven, which is a good idea as it avoids issues that can arise if your schema/protocol files stray from the checked-in code generated equivalents. Today I created a simple GitHub project called avro-maven because I had to fiddle a bit to get Avro and Maven to play nice. The GitHub project is self-contained and also has a README which goes over the basics. In this post I’ll go over how to use Maven to generate code for schema, IDL and protocol files. pom.xml updates to support the Avro plugin Avro schema files only define types, whereas IDL and protocol files model types as well as RPC semantics such as messages. The only difference between IDL and protocol files is that IDL files are Avro’s DSL for specifying RPC, versus protocol files are the same in JSON form. Each type of file has an entry that can be used in the goals element as can be seen below. All three can be used together, or if you only have schema files you can safely remove the protocol and idl-protocol entries (and vice-versa). org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema protocol idl-protocol ... org.apache.avro avro ${avro.version} org.apache.avro avro-maven-plugin ${avro.version} org.apache.avro avro-compiler ${avro.version} org.apache.avro avro-ipc ${avro.version} By default the plugin assumes that your Avro sources are located in ${basedir}/src/main/avro, and that you want your generated sources to be written to ${project.build.directory}/generated-sources/avro, where ${project.build.directory} is typically the target directory. Keep reading if you want to change any of these settings. Avro configurables Luckily Avro’s Maven plugin offers the ability to customize various code generation settings. The following table shows the configurables that can be used for any of the schema, IDL and protocol code generators. Configurable Default value Description sourceDirectory ${basedir}/src/main/avro The Avro source directory for schema, protocol and IDL files. outputDirectory ${project.build.directory}/generated-sources/avro The directory where Avro writes code-generated sources. testSourceDirectory ${basedir}/src/test/avro The input directory containing any Avro files used in testing. testOutputDirectory ${project.build.directory}/generated-test-sources/avro The output directory where Avro writes code-generated files for your testing purposes. fieldVisibility PUBLIC_DEPRECATED Determines the accessibility of fields (e.g. whether they are public or private). Must be one of PUBLIC, PUBLIC_DEPRECATED or PRIVATE. PUBLIC_DEPRECATED merely adds a deprecated annotation to each field, e.g. "@Deprecated public long time". In addition, the includes and testIncludes configurables can also be used to specify alternative file extensions to the defaults, which are **/*.avsc, **/*.avpr and **/*.avdl for schema, protocol and IDL files respectively. Let’s look at an example of how we can specify all of these options for schema compilation. org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema ${project.basedir}/src/main/myavro/ ${project.basedir}/src/main/java/ ${project.basedir}/src/main/myavro/ ${project.basedir}/src/test/java/ PRIVATE **/*.avro **/*.test As a reminder everything covered in this blog article can be seen in action in the GitHub repo at https://github.com/alexholmes/avro-maven.

In our previous article we have explained on how to protect the data while it is in transit through Transport Layer Security (TLS)/Secured Socket Layer (SSL). Now let us try to understand how to apply security mechanism for a JEE 6 based web application using LDAP server for authentication. Objective: • Configure a LDAP realm in the JEE Application Server • Apply JEE security to a sample web application. Products used: IDE: Netbeans 7.2 Java Development Kit (JDK): Version 6 Glassfish server: 3.1 Authentication Mechanism: Form Based authentication Authentication server: LDAP OpenDS v2.2 Apply JEE security to the sample web application: The JEE web applications can be secured either through Declarative security or Programmatic security. Declarative security can be implemented in JEE applications by using annotations or through deployment descriptor. This type of security mechanism is used when the roles and authentication process is simple, when it can make use of existing security providers (even external like LDAP, Kerberos). Programmatic security provides additional security mechanism when declarative security is not sufficient for the application in context. It is used when we require custom made security and when rich set of roles, authentication is required. Configure Realm in the Glassfish Application Server Before we configure a realm in the Glassfish Application server you will need to install and configure an LDAP server which we will be using for our project. You can get the complete instructions in the following article: “How to install and configure LDAP server”. Once the installation is successful start your Glassfish server and go to the admin console. Create a new LDAP Realm. Create new LDAP Realm Add the configuration settings as per the configurations set up done for the LDAP server. Glassfish Web App LDAP Realm JAAS Context – identifier which will be used in the application module to connect with the LDAP server. (e.g. ldapRealm) Directory – LDAP server URL path (e.g. ldap://localhost:389) Base DN: Distinguished name in the LDAP directory identifying the location of the user data. Applying JEE security to the web application Create a sample web application as per the following structure: SampleWebApp Directory Form based authentication mechanism will be used for authentication of the users. JEE Login and Authentication Let us explain the whole process with help of above diagram and the code. Set up a sample web application in Netbeans IDE. SampleWebApp in Netbeans IDE SampleWebApp Configuration Step 1: As explained in the above diagram a client browser tries to request for a protected resource from the websitehttp://{samplewebsite.com}/{contextroot}/index.jsp. The webserver goes into the web configuration file and figures out that the requested resource is protected. web.xml Code SecurityConstraint Secured resources /* GeneralUser Administrator NONE Step 2: The webserver presents the Login.jsp as a part of the Form based authentication mechanism to the client. These configurations are checked from the web configuration file. web.xml FORM ldapRealm /Login.jsp /LoginError.jsp Step 3: The client submits the login form to the web server. When the servers finds that the form action is “j_security_check” it processes the request to authenticate the client’s credential. The jsp form must contain the login elements j_username and j_password which will allow the web server to invoke the login authentication mechanism. Login.jsp username: password: While processing the request the webserver will send the authentication request to the LDAP server since LDAP realm is used in the login-config. The LDAP server will authenticate the user based on the username and password stored in the LDAP repository. Step 4: If the authentication is successful the secured resource (in this case index.jsp) is returned to the client and the container uses a session id to identify a login session for the client. The container maintains the login session with a cookie containing the session-id. The server sends this cookie back to the client, and as long as the client is able to show this cookie for subsequent requests, then the container easily recognize the client and hence maintains the session for this client. Step 5: Only if the authentication is unsuccessful the user will be redirected to the LoginError.jsp as per the configuration in the web.xml. /LoginError.jsp This shows how to apply form based security authentication to a sample web application. Now let us get a brief look on the secured resource which is used for this project. In this project the secured resource is index.jsp which accepts a username and forwards the request to LoginServlet. Login servlet dispatches the request to Success.jsp which then prints the username to the client. index.jsp Please type your name LoginServlet.java protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try { RequestDispatcher requestDispatcher = getServletConfig().getServletContext(). getRequestDispatcher("/Success.jsp"); requestDispatcher.forward(request, response); } finally { out.close(); } } Success.jsp You have been successfully logged in as ${param.username} web.xml LoginServlet com.login.LoginServlet LoginServlet /LoginServlet You can download the complete working code from the below link. SampleWebApp-Code Download Hope our readers have enjoyed this article. Keep watching this space for more articles on JEE security.

With the release of ASP.NET MVC 4 one of the exciting features packed in the release was ASP.NET Web API.

Merge sort is a classical "divide and conquer" sorting algorithm. You should have to never write one because you'd be silly to do that when a standard library class already will already do it for you. But, it is useful to demonstrate a few characteristics of programming techniques in Scala. Firstly a quick recap on the merge sort. It is a divide and conquer algorithm. A list of elements is split up into smaller and smaller lists. When a list has one element it is considered sorted. It is then merged with the list beside it. When there are no more lists to merged the original data set is considered sorted. Now let's take a look how to do that using an imperative approach in Java. public void sort(int[] values) { int[] numbers = values; int[] auxillaryNumbers = new int[values.length]; mergesort(numbers, auxillaryNumbers, 0, values.length - 1); } private void mergesort(int [] numbers, int [] auxillaryNumbers, int low, int high) { // Check if low is smaller then high, if not then the array is sorted if (low < high) { // Get the index of the element which is in the middle int middle = low + (high - low) / 2; // Sort the left side of the array mergesort(numbers, auxillaryNumbers, low, middle); // Sort the right side of the array mergesort(numbers, auxillaryNumbers, middle + 1, high); // Combine them both // Alex: the first time we hit this when there is min difference between high and low. merge(numbers, auxillaryNumbers, low, middle, high); } } /** * Merges a[low .. middle] with a[middle..high]. * This method assumes a[low .. middle] and a[middle..high] are sorted. It returns * a[low..high] as an sorted array. */ private void merge(int [] a, int[] aux, int low, int middle, int high) { // Copy both parts into the aux array for (int k = low; k <= high; k++) { aux[k] = a[k]; } int i = low, j = middle + 1; for (int k = low; k <= high; k++) { if (i > middle) a[k] = aux[j++]; else if (j > high) a[k] = aux[i++]; else if (aux[j] < aux[i]) a[k] = aux[j++]; else a[k] = aux[i++]; } } public static void main(String args[]){ ... ms.sort(new int[] {5, 3, 1, 17, 2, 8, 19, 11}); ... } } Discussion... An auxillary array is used to achieve the sort. Elements to be sorted are copied into it and then once sorted copied back. It is important this array is only created once otherwise there can be a performance hit from extensive array created. The merge method does not have to create an auxiliary array however since it changes an object it means the merge method has side effects. Merge sort big(O) performance is N log N. Now let's have a go at a Scala solution. def mergeSort(xs: List[Int]): List[Int] = { val n = xs.length / 2 if (n == 0) xs else { def merge(xs: List[Int], ys: List[Int]): List[Int] = (xs, ys) match { case(Nil, ys) => ys case(xs, Nil) => xs case(x :: xs1, y :: ys1) => if (x < y) x::merge(xs1, ys) else y :: merge(xs, ys1) } val (left, right) = xs splitAt(n) merge(mergeSort(left), mergeSort(right)) } } Key discussion points: It is the same divide and conquer idea. The splitAt function is used to divide up the data up each time into a tuple. For every recursion this will new a new tuple. The local function merge is then used to perform the merging. Local functions are a useful feature as they help promote encapsulation and prevent code bloat. Neiher the mergeSort() or merge() functions have any side effects. They don't change any object. They create (and throw away) objects. Because the data is not been passed across iterations of the merging, there is no need to pass beginning and ending pointers which can get very buggy. This merge recursion uses pattern matching to great effect here. Not only is there matching for data lists but when a match happens the data lists are assigned to variables: x meaning the top element in the left list xs1 the rest of the left list y meaning the top element in the right list ys1 meaning the rest of the data in the right list This makes it very easy to compare the top elements and to pass around the rest of the date to compare. Would the iterative approach be possible in Java? Of course. But it would be much more complex. You don't have any pattern matching and you don't get a nudge to declare objects as immutable as Scala does with making you make something val or var. In Java, it would always be easier to read the code for this problem if it was done in an imperative style where objects are being changed across iterations of a loop. But Scala a functional recursive approach can be quite neat. So here we see an example of how Scala makes it easier to achieve good, clean, concise recursion and a make a functional approach much more possible.

As people know, I'm a huge fan of PhoneGap and what it allows me to do with JavaScript, HTML, and CSS. But I think it is crucial to remember that you don't always need PhoneGap. A great example of that is camera access. Did you know that recent mobile browsers support accessing the camera directly from HTML and JavaScript? Let's look at an example. Over a year ago I wrote a blog post where I created an application called "Color Thief." This application made use of PhoneGap's Camera API and a third party JavaScript library called Color Thief. I loved this example because it demonstrated how you could combine the extra power that PhoneGap provides along with existing JavaScript libraries. This morning I watched an excellent Google IO presentation (https://www.youtube.com/watch?v=EPYnGFEcis4&feature=youtube_gdata_player) on Mobile HTML. It was an overview of some of the exciting stuff you can now do with mobile HTML and JavaScript. To be clear, this was all without using wrappers like PhoneGap. In one of the examples the presenters discussed the new "capture" support for the input/file field type. This is rather simple to implement: If supported (recent Android and latest iOS), the user can then use their camera to select a picture. I decided to rebuild my old demo to skip PhoneGap completely and just make use of this feature. Here's the code: For the most part, this is pretty similar to the last version. I no longer wait for the deviceready event, but instead just listen for the document itself to load. Instead of listening for a button click, I've switched to a input field using type=file. I now listen for the change event, and on that, I see if I have access to a file. If I do, I can then use the URL object to create a pointer to the source and then simply add it to my DOM. After that, Color Thief takes over. The only tricky part I ran into was that in iOS the URL object is still prefixed. You can see how I get around that in the startup code. To be fair, this isn't 100% backwards compatible, I could add a few checks in here to ensure that things will work and gracefully let people on older phones know they can't use this feature. But the end result is nearly the exact same functionality in a web page - no PhoneGap, no native code. <br>

A few days ago I was going through a project's Maven dependencies, removing unused junk, checking jar file version numbers adding a little dependency management and generally tidying up (yes, I know that this isn't something we often get time to do, but even Maven dependencies can be a form of technical debt). After recompiling and running the unit tests I ran some end to end tests only to find that the whole thing fell apart... Big time. The exception I got was the usual one that all Spring developers get, a java.lang.IllegalStateException: Failed to load ApplicationContext ...exception. This is nothing new and as a Spring developer you find the problem, which is usually a missing bean definition and move on. Only this time it was something different, and that's because the cause was: java.lang.NoSuchFieldError: NULL ...which gives you no clues about what's going wrong. Now I knew that I'd been messing around with the project's dependencies, so I must have broken something somewhere. It turned out that it was a transient dependency problem. I was using Spring Security version 3.1.1-RELEASE, which is built using version 3.0.7-RELEASE of the Spring core libraries and not as you'd expect version 3.1.1-RELEASE. This meant that I'd ended up with different and incompatible versions of some of the Spring libraries on my classpath. You may well wonder why the Guys at Spring Security build their code with version 3.0.7-RELEASE and they say that this is intentional and that it's to do with backwards compatibility issues. As Rob Winch, Spring Security Lead at SpringSource, says: "Spring Security uses 3.0.x (intentionally to support users that require it). For this reason, if you build with Maven and want to use Spring 3.1 you must either exclude the Spring dependencies in your maven pom, explicitly add the Spring 3.1 dependencies to your pom, or add a dependency management section to your pom. This is not a bug. Even if Spring Security was changed to use Spring 3.1 by default, the users using Spring 3.0 would encounter the same problem. The reason this occurs is due to the algorithm that Maven uses to resolve transitive dependency versions [1]" Once you know how, the problem is easy to spot. If you're using STS/eclipse you can easily examine Maven dependencies using the POM editor. The fix is simple too, all you need to do is to explicitly define the wayward Spring libraries in your POM. For example: org.springframework spring-core 3.1.1-RELEASE Finally, you can check that it's fixed using STS/eclipse's POM file editor, where you'll see that the unwanted version is now labelled as "omitted". [1] http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies

A common feature that you will come across in websites is the ability to resize an image to fit an exact size so that it will be displayed correctly on your design. If you have a very large image and you are going to place it on your website in a space that is only 100px x 100px then you will want to be able to resize this image to fit in the space. One option is to just set the width and height attributes on the image tag in your HTML, this will force the image to be displayed at this size. This will work perfectly fine and the image will fit in the 100px x 100px space but the problem is that when the browser loads the image it will not resize the image but will just display it in the limited size. This means that the image will still need to be downloaded at full size, if the image is very large it can take some time for you to download a large image just to be displayed in a small space. A better solution would be to resize the image to 100px x 100px which will reduce it's size so that the browser doesn't need to download a large image just to display it in a small space. In this tutorial we are going to create a PHP class that will allow you to resize an image to any dimension you want, it will allow you to resize while keeping the aspect ratio of the image. When the class has resized the image you can either save the image on the server or download the image. Class Methods First lets plan out the class we are going to create: We need to pass an existing image to the class, this is the image that we will use to resize. We need to pass in the desired image dimensions so the class can work out what's the new size of the image. Then we need to be able to save the image to a location on the server, and choose to download the image. The Constructor This class relies on an original image being found and set on the class, without this the class will not work correctly. Because of this we should pass the image filename into the constructor of the class. This will then check if the file exists on the server, if it does then we can call the set image method where we can get the image and create a resource of the image and store this in a class variable. /** * Class constructor requires to send through the image filename * * @param string $filename - Filename of the image you want to resize */ public function __construct( $filename ) { if(file_exists($filename)) { $this->setImage( $filename ); } else { throw new Exception('Image ' . $filename . ' can not be found, try another image.'); } } Set The Image The set image method is used to create a image resource based on the image given to the class this uses the PHP functions imagecreatefromjpeg, imagecreatefromgif, imagecreatefrompng to create the image resource from the given image. We can then use this with the functions imagesx and imagesy to return the current width and height of the image. This will allow us to resize the image easier later in the script. /** * Set the image variable by using image create * * @param string $filename - The image filename */ private function setImage( $filename ) { $size = getimagesize($filename); $this->ext = $size['mime']; switch($this->ext) { // Image is a JPG case 'image/jpg': case 'image/jpeg': // create a jpeg extension $this->image = imagecreatefromjpeg($filename); break; // Image is a GIF case 'image/gif': $this->image = @imagecreatefromgif($filename); break; // Image is a PNG case 'image/png': $this->image = @imagecreatefrompng($filename); break; // Mime type not found default: throw new Exception("File is not an image, please use another file type.", 1); } $this->origWidth = imagesx($this->image); $this->origHeight = imagesy($this->image); } Resize The Image The resize function is what we will use to calculate the new values for the image width and height. This takes 3 parameters the new width, new height and the resize option, this will allow us to resize the image to exact dimensions, use the defined width with the height and keep aspect ratio, use the define height keeping the aspect ratio or let the class decide the best way of resizing the image. Once we have the new width and height of the image we can create a new image resource by using the PHP function imagecreatetruecolor(). Now we can create the new image from the old image resizing it to the new dimensions by using the imagecopyresampled() function. /** * Resize the image to these set dimensions * * @param int $width - Max width of the image * @param int $height - Max height of the image * @param string $resizeOption - Scale option for the image * * @return Save new image */ public function resizeTo( $width, $height, $resizeOption = 'default' ) { switch(strtolower($resizeOption)) { case 'exact': $this->resizeWidth = $width; $this->resizeHeight = $height; break; case 'maxwidth': $this->resizeWidth = $width; $this->resizeHeight = $this->resizeHeightByWidth($width); break; case 'maxheight': $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; break; default: if($this->origWidth > $width || $this->origHeight > $height) { if ( $this->origWidth > $this->origHeight ) { $this->resizeHeight = $this->resizeHeightByWidth($width); $this->resizeWidth = $width; } else if( $this->origWidth < $this->origHeight ) { $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; } } else { $this->resizeWidth = $width; $this->resizeHeight = $height; } break; } $this->newImage = imagecreatetruecolor($this->resizeWidth, $this->resizeHeight); imagecopyresampled($this->newImage, $this->image, 0, 0, 0, 0, $this->resizeWidth, $this->resizeHeight, $this->origWidth, $this->origHeight); } /** * Get the resized height from the width keeping the aspect ratio * * @param int $width - Max image width * * @return Height keeping aspect ratio */ private function resizeHeightByWidth($width) { return floor(($this->origHeight / $this->origWidth) * $width); } /** * Get the resized width from the height keeping the aspect ratio * * @param int $height - Max image height * * @return Width keeping aspect ratio */ private function resizeWidthByHeight($height) { return floor(($this->origWidth / $this->origHeight) * $height); } Save The Image With the new image now set in a class variable we can now use this to save the image on the server. This function will take 3 parameters the save path, the image quality and if we want to download the image. For each mime type PHP has a function imagejpeg(), imagegif(), imagepng() that will allow you to save the image by passing in the new image resource and the path the image is going to be saved. Once this image is saved on the server and we decided to download it we can change the headers to allow the browser to download the image on the clients machine. /** * Save the image as the image type the original image was * * @param String[type] $savePath - The path to store the new image * @param string $imageQuality - The qulaity level of image to create * * @return Saves the image */ public function saveImage($savePath, $imageQuality="100", $download = false) { switch($this->ext) { case 'image/jpg': case 'image/jpeg': // Check PHP supports this file type if (imagetypes() & IMG_JPG) { imagejpeg($this->newImage, $savePath, $imageQuality); } break; case 'image/gif': // Check PHP supports this file type if (imagetypes() & IMG_GIF) { imagegif($this->newImage, $savePath); } break; case 'image/png': $invertScaleQuality = 9 - round(($imageQuality/100) * 9); // Check PHP supports this file type if (imagetypes() & IMG_PNG) { imagepng($this->newImage, $savePath, $invertScaleQuality); } break; } if($download) { header('Content-Description: File Transfer'); header("Content-type: application/octet-stream"); header("Content-disposition: attachment; filename= ".$savePath.""); readfile($savePath); } imagedestroy($this->newImage); } Full Resize Image Class setImage( $filename ); } else { throw new Exception('Image ' . $filename . ' can not be found, try another image.'); } } /** * Set the image variable by using image create * * @param string $filename - The image filename */ private function setImage( $filename ) { $size = getimagesize($filename); $this->ext = $size['mime']; switch($this->ext) { // Image is a JPG case 'image/jpg': case 'image/jpeg': // create a jpeg extension $this->image = imagecreatefromjpeg($filename); break; // Image is a GIF case 'image/gif': $this->image = @imagecreatefromgif($filename); break; // Image is a PNG case 'image/png': $this->image = @imagecreatefrompng($filename); break; // Mime type not found default: throw new Exception("File is not an image, please use another file type.", 1); } $this->origWidth = imagesx($this->image); $this->origHeight = imagesy($this->image); } /** * Save the image as the image type the original image was * * @param String[type] $savePath - The path to store the new image * @param string $imageQuality - The qulaity level of image to create * * @return Saves the image */ public function saveImage($savePath, $imageQuality="100", $download = false) { switch($this->ext) { case 'image/jpg': case 'image/jpeg': // Check PHP supports this file type if (imagetypes() & IMG_JPG) { imagejpeg($this->newImage, $savePath, $imageQuality); } break; case 'image/gif': // Check PHP supports this file type if (imagetypes() & IMG_GIF) { imagegif($this->newImage, $savePath); } break; case 'image/png': $invertScaleQuality = 9 - round(($imageQuality/100) * 9); // Check PHP supports this file type if (imagetypes() & IMG_PNG) { imagepng($this->newImage, $savePath, $invertScaleQuality); } break; } if($download) { header('Content-Description: File Transfer'); header("Content-type: application/octet-stream"); header("Content-disposition: attachment; filename= ".$savePath.""); readfile($savePath); } imagedestroy($this->newImage); } /** * Resize the image to these set dimensions * * @param int $width - Max width of the image * @param int $height - Max height of the image * @param string $resizeOption - Scale option for the image * * @return Save new image */ public function resizeTo( $width, $height, $resizeOption = 'default' ) { switch(strtolower($resizeOption)) { case 'exact': $this->resizeWidth = $width; $this->resizeHeight = $height; break; case 'maxwidth': $this->resizeWidth = $width; $this->resizeHeight = $this->resizeHeightByWidth($width); break; case 'maxheight': $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; break; default: if($this->origWidth > $width || $this->origHeight > $height) { if ( $this->origWidth > $this->origHeight ) { $this->resizeHeight = $this->resizeHeightByWidth($width); $this->resizeWidth = $width; } else if( $this->origWidth < $this->origHeight ) { $this->resizeWidth = $this->resizeWidthByHeight($height); $this->resizeHeight = $height; } } else { $this->resizeWidth = $width; $this->resizeHeight = $height; } break; } $this->newImage = imagecreatetruecolor($this->resizeWidth, $this->resizeHeight); imagecopyresampled($this->newImage, $this->image, 0, 0, 0, 0, $this->resizeWidth, $this->resizeHeight, $this->origWidth, $this->origHeight); } /** * Get the resized height from the width keeping the aspect ratio * * @param int $width - Max image width * * @return Height keeping aspect ratio */ private function resizeHeightByWidth($width) { return floor(($this->origHeight/$this->origWidth)*$width); } /** * Get the resized width from the height keeping the aspect ratio * * @param int $height - Max image height * * @return Width keeping aspect ratio */ private function resizeWidthByHeight($height) { return floor(($this->origWidth/$this->origHeight)*$height); } } ?> Using The Resize Image PHP Class Because we have created this to allow you to resize the image in multiple ways it means that there are different ways of using the class. Resize the image to an exact size. Resize the image to a max width size keeping aspect ratio of the image. Resize the image to a max height size keeping aspect ratio of the image. Resize the image to a given width and height and allow the code to work out which way of resizing is best keeping the aspect ratio. You can save the created resize image on the server. You can download the created resize image on the server. Resize Exact Size To resize an image to an exact size you can use the following code. First pass in the image we want to resize in the class constructor, then define the width and height with the scale option of exact. The class will now have the create dimensions to create the new image, now call the function saveImage() and pass in the new file location to the new image. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'exact'); $resize->saveImage('images/be-original-exact.jpg'); Resize Max Width Size If you choose to set the image to be an exact size then when the image is resized it could lose it's aspect ratio, which means the image could look stretched. But if you know the max width that you want the image to be you can resize the image to a max width, this will keep the aspect ratio of the image. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'maxWidth'); $resize->saveImage('images/be-original-maxWidth.jpg'); Resize Max Height Size Just as you can select a max width for the image while keeping aspect ratio you can also select a max height while keeping aspect ratio. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'maxHeight'); $resize->saveImage('images/be-original-maxHeight.jpg'); Resize Auto Size From Given Width And Height You can also allow the code to work out the best way to resize the image, so if the image height is larger than the width then it will resize the image by using the height and keeping aspect ratio. If the image width is larger than the height then the image will be resized using the width and keeping the aspect ratio. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100); $resize->saveImage('images/be-original-default.jpg'); Download The Resized Image The default behaviour for this class is to save the image on the server, but you can easily change this to download by passing in a true parameter to the saveImage method. $resize = new ResizeImage('images/Be-Original.jpg'); $resize->resizeTo(100, 100, 'exact'); $resize->saveImage('images/be-original-exact.jpg', "100", true);

Github has a nice API for inspecting repositories – it lets you read gists, issues, commit history, files and so on. Git repository data lends itself to demonstrating the power of combining full text and faceted search, as there is a mix of free text fields (commit messages, code) and enumerable fields (committers, dates, committer employers). Github APIs return JSON, which has the nice property of resembling a tree structure – results can be recursed over without fear of infinite loops. Note that to download the entire commit history for a repository, you need to page through it by sha hash. The API I use here lacks diffs, which must be retrieved elsewhere. To test this, access a URL like so. The configurable arguments are the repository owner and name fields. https://api.github.com/repos/torvalds/linux/commits This is what a commit looks like: { "sha": "7638417db6d59f3c431d3e1f261cc637155684cd", "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/7638417db6d59f3c431d3e1f261cc637155684cd", "author": { "date": "2008-07-09T16:13:30+12:00", "name": "Scott Chacon", "email": "[email protected]" }, "committer": { "date": "2008-07-09T16:13:30+12:00", "name": "Scott Chacon", "email": "[email protected]" }, "message": "my commit message", "tree": { "url": "https://api.github.com/repos/octocat/Hello-World/git/trees/827efc6d56897b048c772eb4087f854f46256132", "sha": "827efc6d56897b048c772eb4087f854f46256132" }, "parents": [ { "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/7d1b31e74ee336d15cbd21741bc88a537ed063a0", "sha": "7d1b31e74ee336d15cbd21741bc88a537ed063a0" } ] } To make the test simple, I download these as JSON locally, then start a python webserver. Were I to make many such calls on a public site, I’d set up a proxy to the github APIs. python -m SimpleHTTPServer This data has a number of nested objects and must be flattened to fit into the lunr.jsfull-text index. This example uses the commit number (0, 1, 2..N) as the location in the index, but a real environment should use the commit hash to allow partitioning the ingestion process. Nested objects are flattened by joining subsequent keys with underscores in between. A production-worthy solution needs to escape these to prevent collisions. var documents = []; function recurse(doc_num, base, obj, value) { if ($.isPlainObject(value)) { $.each(value, function (k, v) { recurse(doc_num, base + obj + "_", k, v); }); } else { process(doc_num, base + obj, value); } } function process(doc_num, key, value) { if (documents.length <= doc_num) documents[doc_num] = {}; if (value !== null) documents[doc_num][key] = value + ''; } $.each(data, function(doc_num, commit) { $.each(commit, function(k, v) { recurse(doc_num, '', k, v) }); }); Normally, one sets up a lunr full-text index by specifying all the fields, much like Solr’s numerous XML config files. Lunr doesn’t have nearly as many configuration options, since you only specify the ‘boost’ parameter to increase the value of certain fields in ranking. I imagine this will change as the project grows, at the very least to include type hints. Given the simplicity of field objects, you can infer infer the field list from JSON payloads. The code below provides two modes, one where you inspect the entire JSON payload, or one where you limit how many commits you check, a good option when JSON data is consistent. The function accepts configuration objects resembling ExtJS config objects, which lets you override as desired. If fields derived from existing data are required, they can be inserted after any documents are inserted. function inferIndex(documents, config) { return lunr(function() { this.ref('id'); var found = {}; var idx = this; $.each(documents, function(doc_num, doc) { if (config && config.limit && config.limit < doc_num) return; $.each(doc, function(k, v) { if (!found[k]) { if (config && config[k]) { idx.field(k, config[k]); } else { idx.field(k); } found[k] = true; } }); }); }); } var index = inferIndex(documents, {limit: 1, 'commit_author_name':{boost:10}); Inserting flattened documents into the index becomes simple. The method below provides a callback, should you desire to add calculated fields fields. $.each(documents, function(doc_num, attrs, doc_cb) { var doc = $.extend( {id: doc_num}, attrs); if (doc_cb) { doc = doc_cb(doc); } index.add(doc); }); At this point we’ve indexed the entire commit history from a git repository, which lets us search for commits by topic. While this is useful, it’d be really nice to be able to facet on fields, which would return the number of documents in a category, like a SQL group by. I’ve found it particularly convenient to facet on author, date, or author’s company. If you have access to the original documents, you can easily construct facets based on the results of a lunr search: function facet(index, query, data, field) { var results = index.search(query); var facets = {}; $.each(results, function(index, searchResult) { var doc = data[searchResult.ref]; facets[doc[field]] = (facets[doc[field]] === undefined ? 0 : facets[doc[field]]) + 1; } ); return facets; } Commit messages in repositories where I work often contain names of clients who requested a feature or bug fix. Consequently doing a search faceted by author provides a list of who worked with each client the most – this can also tell you who has worked with various pieces of technology. The following query demonstrates this technique: var facets = facet(index, 'driver', documents, 'commit_author_name'); {"Wolfram Sang":24,"Linus Torvalds":3} The approach shown here works well, but requires retrieving results requires access to the original document data. If we want to filter the results to a category, we need a richer search API than lunr currently provides, as well as callback options within the search API. In Solr there are also options to skip lower-casing data, as that may be inappropriate for category titles. Mitigating these issues will be explored further in future essays.

If you’re uploading blobs by splitting blobs into blocks and you get the error – The specified blob or block content is invalid, then this post is for you. Short Version If you’re uploading blobs by splitting blobs into blocks and you get the above mentioned error, ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error. Long Version Now for the longer version of this post . A few days back I was working with storage client library especially around uploading blobs in chunks and with one particular blob I was constantly getting the error – The specified blob or block content is invalid. I tried numerous combinations even resorting to REST API directly but to no avail. It only happened with just one blob. Furthermore if I uploaded the same blob without splitting it into blocks, all was well. I was at my wits’ end. Tried searching the Internet for this error but could not find a conclusive answer to my problem. After much trial and error, I was able to simulate the same problem on other blobs as well. Here’s how you can recreate it: Start uploading the blob by splitting it into blocks. For block id, let’s do a 7 character long string e.g. intValue.ToString(“d7”). This will ensure that my block ids would be “0000001”, “0000002”, …, ”0000010” ….. After one or two blocks are uploaded, cancel the operation. Now re-upload the blob by splitting it into blocks. However this time for block id, let’s do a 6 character long string e.g. intValue.ToString(“d6”). You’ll get the error as soon as you try to upload the 1st block. Possible Solutions Now that we know the root cause of this problem, let’s look at some of the possible solutions to solve this problem. Wait out One possible solution is to wait out. I know its lame but still a possible solution. We know that Windows Azure Blob Storage Service keeps all uncommitted blocks for a duration of 7 days and if within 7 days those uncommitted blocks are not committed, the storage service purges them. I wish storage service provided some mechanism to purge uncommitted blocks programmatically. Commit uncommitted blocks You could possibly commit the blocks which are in uncommitted state so that at least you get a blob (which would not be the blob we wanted to upload in the first place). You can then delete that blob and re-upload the blob by specifying block ids which are of same length. To fetch the list of uncommitted blocks, if you’re using REST API directly you can perform “Get Block List” operation and pass “blocklisttype=uncommitted” as one of the query string parameters. If you’re using storage client library (assuming you’re using the version 2.x of .Net storage client library), you can do something like the code below: private static List GetUncommittedBlockIds(CloudBlockBlob blob) { var sasUri = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy() { SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5), Permissions = SharedAccessBlobPermissions.Read, }); var blobUri = new Uri(string.Format("{0}{1}", blob.Uri, sasUri)); List uncommittedBlockIds = new List(); var request = BlobHttpWebRequestFactory.GetBlockList(blobUri, null, null, BlockListingFilter.Uncommitted, null, null); //request.Headers.Add("Authorization", using (var resp = (HttpWebResponse)request.GetResponse()) { using (var stream = resp.GetResponseStream()) { var getBlockListResponse = new GetBlockListResponse(stream); var blocks = getBlockListResponse.Blocks; foreach (var block in blocks.Where(b => !b.Committed)) { uncommittedBlockIds.Add(Encoding.UTF8.GetString(Convert.FromBase64String(block.Name))); } } } return uncommittedBlockIds; } A few things to keep in mind here: Microsoft.WindowsAzure.Storage.Blob namespace does not have the capability to get the list of uncommitted blocks. You would need to make use ofMicrosoft.WindowsAzure.Storage.Blob.Protocol namespace. Because we’re kind of invoking the REST API by executing an HttpWebRequest, I created a shared access signature on the blob so that I don’t have to create “Authorization” header. Fetch uncommitted blocks to see block id length You could fetch the list of uncommitted blocks just to find out the length of the block id used. You could then use that block id length for your new upload session and do the upload. Please see the code snippet above to find this information. Upload another blob with same name without splitting it into blocks You could also upload another blob with the same name without splitting it into blocks. It could very well be a zero byte blob. That way your uncommitted block list will be wiped clean. Then you could delete that dummy blob and re-upload the actual blob. A Few Words About Blocks Since we’re talking about blocks, I thought it might be useful to mention a few points about them: Blocks and block related operations are only applicable for “Block Blobs”. Duh!! You’ll get an error if you’re trying to do these operations on a “Page Blob”. For uploading large blobs, it is recommended that you split your blob into blocks. In fact if your blob size is more than 64 MB, then you have to split it into blocks. Minimum size of a block is 1 Byte and the maximum size of a block is 4 MB. It is recommended that you choose a block size based on your internet connectivity and number of parallel threads you want use to upload these blocks. A blob can be split into a maximum of 50000 blocks. It’s important to remember this limitation because you are reminded of this limit when you’re trying to upload 50001st block. The length of all the block ids must be same. So if you’re using an integer value to denote block id, you make sure that you pad that integer value with “0” so that you get same length. So you could do something likeint.ToString(“d6”). When passing the block id as a parameter, it must be Base64 encoded. While the order in which the blocks are uploaded is not important, the order is important when you commit the block list because that’s when the blob is constructed by the service. For example, let’s say you’re uploading a blob by splitting it into 5 blocks (with ids “000001”, “000002”, “000003”, “000004”, and “000005”). You could upload these blocks in any order – 000004, 000001, 000003, 000005, 000002 however when you commit the block list, ensure that the block ids are passed in proper order i.e. 000001, 000002, 000003, 000004, 000005. Summary That’s it for this post. I hope you’ve found this information useful. I spent considerable amount of time trying to fix this problem so I hope it will help some folks out. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

this is the first blog post in a series which looks at some data organization patterns in mapreduce. we’ll look at how to bucket output across multiple files in a single task, how to multiplex data across multiple files, and also how to coalesce data. these are all common patterns that are useful to have in your mapreduce toolkit. we’ll kick things off with a look at bucketing data outputs in your map or reduce tasks. by default when using a fileoutputformat-derived outputformat (such as textoutputformat), all the outputs for a reduce task (or a map task in a map-only job) are written to a single file in hdfs. imagine a situation where you have user activity logs being streamed into hdfs, and you want to write a mapreduce job to better organize the incoming data. as an example a large organization with multiple products may want to bucket the logs based on the product. to do this you’ll need the ability to write to multiple output files in a single task. let’s take a look at how we can make that happen. multipleoutputformat there are a few ways you can achieve your goal, and the first option we’ll look at is the multipleoutputformat class in hadoop. this is an abstract class that lets you do the following: define the output path for each and every key/value output record being emitted by a task. incorporate the input paths into the output directory for map-only jobs. redefine the key and value that are used to write to the underlying recordwriter . this is useful in situations where you want to remove data from the outputs as it duplicates data in the filename. for each output path, define the recordwriter that should be used to write the outputs. ok enough with the words - let’s look at some data and code. first up is the simple data we’ll use in our example - imagine you work at a fruit market with locations in multiple cities, and you have a purchase transaction stream which contains the store location along with the fruit that was purchased. cupertino apple sunnyvale banana cupertino pear to help bucket your data for future analysis, you want to bin each record into city-specific files. for the simple data set above you don’t want to filter, project or transform your data, just bucket it out, so a simple identity map-only job will do the job. to force more than one mapper, we’ll write the data to two separate files. $ tab="$(printf '\t')" $ hdfs -put - file1.txt << eof cupertino${tab}apple sunnyvale${tab}banana eof $ hdfs -put - file2.txt << eof cupertino${tab}pear eof here’s the code which will let you write city-specific output files. import org.apache.commons.lang.stringutils; import org.apache.hadoop.conf.configuration; import org.apache.hadoop.conf.configured; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.text; import org.apache.hadoop.mapred.*; import org.apache.hadoop.mapred.lib.identitymapper; import org.apache.hadoop.mapred.lib.multipletextoutputformat; import org.apache.hadoop.util.progressable; import org.apache.hadoop.util.tool; import org.apache.hadoop.util.toolrunner; import java.io.ioexception; import java.util.arrays; /** * an example of how to use {@link org.apache.hadoop.mapred.lib.multipleoutputformat}. */ public class mofexample extends configured implements tool { /** * create output files based on the output record's key name. */ static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } } /** * the main job driver. */ public int run(final string[] args) throws exception { string csvinputs = stringutils.join(arrays.copyofrange(args, 0, args.length - 1), ","); path outputdir = new path(args[args.length - 1]); jobconf jobconf = new jobconf(super.getconf()); jobconf.setjarbyclass(mofexample.class); jobconf.setnumreducetasks(0); jobconf.setmapperclass(identitymapper.class); jobconf.setinputformat(keyvaluetextinputformat.class); jobconf.setoutputformat(keybasedmultipletextoutputformat.class); fileinputformat.setinputpaths(jobconf, csvinputs); fileoutputformat.setoutputpath(jobconf, outputdir); return jobclient.runjob(jobconf).issuccessful() ? 0 : 1; } /** * main entry point for the utility. * * @param args arguments * @throws exception when something goes wrong */ public static void main(final string[] args) throws exception { int res = toolrunner.run(new configuration(), new mofexample(), args); system.exit(res); } } run this code and you’ll see the following files in hdfs, where /output is the job output directory: $ hadoop fs -lsr /output /output/cupertino/part-00000 /output/cupertino/part-00001 /output/sunnyvale/part-00000 if you look at the output files you’ll see that the files contain the correct buckets. $ hadoop fs -lsr /output/cupertino/* cupertino apple cupertino pear $ hadoop fs -lsr /output/sunnyvale/* sunnyvale banana awesome, you have your data bucketed by store. now that we have everything working, let’s look at what we did to get there. we had to do two things to get this working: extend multipletextoutputformat this is where the magic happened - let’s look at that class again. static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } } you are working with text, which is why you extended multipletextoutputformat , a class that in turn extends multipleoutputformat . multipletextoutputformat is a simple class which instructs the multipleoutputformat to use textoutputformat as the underlying output format for writing out the records. if you were to use multipleoutputformat as-is it behaves as if you were using the regular textoutputformat , which is to say that it’ll only write to a single output file. to write data to multiple files you had to extend it, as with the example above. the generatefilenameforkeyvalue method allows you to return the output path for an input record. the third argument, name , is the original fileoutputformat -created filename, which is in the form “part-nnnnn”, where “nnnnn” is the task index, to ensure uniqueness. to avoid file collisions, it’s a good idea to make sure your generated output paths are unique, and leveraging the original output file is certainly a good way of doing this. in our example we’re using the key as the directory name, and then writing to the original fileoutputformat filename within that directory. specify the outputformat the next step was easy - specify that this output format should be used for your job: jobconf.setoutputformat(keybasedmultipletextoutputformat.class); earlier we also mentioned that you can use the input path as part of the output path, which we will look at next. using the input filename as part of the output filename in map-only jobs what if we wanted to keep the input filename as part of the output filename? this only works for map-only jobs, and can be accomplished by overriding the getinputfilebasedoutputfilename method. let’s look at the following code to understand how this method fits into the overall sequence of actions that the multipleoutputformat class performs: public void write(k key, v value) throws ioexception { // get the file name based on the key string keybasedpath = generatefilenameforkeyvalue(key, value, myname); // get the file name based on the input file name string finalpath = getinputfilebasedoutputfilename(myjob, keybasedpath); // get the actual key k actualkey = generateactualkey(key, value); v actualvalue = generateactualvalue(key, value); recordwriter rw = this.recordwriters.get(finalpath); if (rw == null) { // if we don't have the record writer yet for the final path, create // one // and add it to the cache rw = getbaserecordwriter(myfs, myjob, finalpath, myprogressable); this.recordwriters.put(finalpath, rw); } rw.write(actualkey, actualvalue); }; the getinputfilebasedoutputfilename method is called with the output of generatefilenameforkeyvalue , which contains our already-customized output file. our new keybasedmultipletextoutputformat can now be updated to override getinputfilebasedoutputfilename and append the original input filename to the output filename: static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(object key, object value, string name) { return key.tostring() + "/" + name; } @override protected string getinputfilebasedoutputfilename(jobconf job, string name) { string infilename = new path(job.get("map.input.file")).getname(); return name + "-" + infilename; } if you run with your modified outputformat class you’ll see the following files in hdfs, confirming that the input filenames are now concatenated to the end of each output file. $ hadoop fs -lsr /output /output/cupertino/part-00000-file1.txt /output/cupertino/part-00001-file2.txt /output/sunnyvale/part-00000-file1.txt the implementation of getinputfilebasedoutputfilename in multipleoutputformat doesn’t do anything interesting by default, but if you set the value of the mapred.outputformat.numoftrailinglegs configurable to an integer greater than 0, then the getinputfilebasedoutputfilename will use part of the input path as the output path. let’s see what happens when we set the value to 1: jobconf.setint("mapred.outputformat.numoftrailinglegs", 1); the output files in hdfs now exactly mirror the input files used for the job: $ hadoop fs -lsr /output /output/file1.txt /output/file2.txt if we set mapred.outputformat.numoftrailinglegs to 2, and our input files exist in the /inputs directory, then our output directory looks like this: $ hadoop fs -lsr /output /output/input/file1.txt /output/input/file2.txt basically as you keep incrementing mapred.outputformat.numoftrailinglegs , then multipleoutputformat will continue to go up the parent directories of the input file and use them in the output path. modifying the output key and value it’s very possible that the actual key and value you want to emit are different from those that were used to determine the output file. in our example, we took the output key and wrote to a directory using the key name. if you do that keeping the key in the output file may be redundant. how would we modify the output record so that the key isn’t written? multipleoutputformat has your back with the generateactualkey method. class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } @override protected text generateactualkey(text key, text value) { return null; } } the returned value from this method replaces the key that’s supplied to the underlying recordwriter , so if you return null as in the above example, no key will be written to the file. $ hadoop fs -lsr /output/cupertino/* apple pear $ hadoop fs -lsr /output/sunnyvale/* banana you can achieve the same result for the output value by overriding the generateactualvalue method. changing the recordwriter in our final step we’ll look at how you can leverage multiple recordwriter classes for different output files. this is accomplished by overriding the getrecordwriter method. in the example below we’re leveraging the same textoutputformat for all the files, but it gives you a sense of what can be accomplished. static class keybasedmultipletextoutputformat extends multipletextoutputformat { @override protected string generatefilenameforkeyvalue(text key, text value, string name) { return key.tostring() + "/" + name; } @override public recordwriter getrecordwriter(filesystem fs, jobconf job, string name, progressable prog) throws ioexception { if (name.startswith("apple")) { return new textoutputformat().getrecordwriter(fs, job, name, prog); } else if (name.startswith("banana")) { return new textoutputformat().getrecordwriter(fs, job, name, prog); } return super.getrecordwriter(fs, job, name, prog); } } conclusion when using multipleoutputformat , give some thought to the number of distinct files that each reducer will create. it would be prudent to plan your bucketing so that you have a relatively small number of files. in this post we extended multipletextoutputformat , which is a simple extension of multipleoutputformat that supports text outputs. multiplesequencefileoutputformat also exists to support sequencefiles in a similar fashion. so what are the shortcomings with the multipleoutputformat class? if you have a job that uses both map and reduce phases, then multipleoutputformat can’t be used in the map-side to write outputs. of course, multipleoutputformat works fine in map-only jobs. all recordwriter classes must support exactly the same output record types. for example, you wouldn’t be able to support a recordwriter that emitted for one output file, and have another recordwriter that emitted . multipleoutputformat exists in the mapred package, so it won’t work with a job that requires use of the mapreduce package. all is not lost if you bump into either one of these issues, as you’ll discover in the next blog post.

When building websites, you’ll often want users to be able to search for something by name. On LinerNotes, users can search for bands, albums, genres etc from a search bar that appears on the homepage and in the omnipresent nav bar. And we need a way to match those queries to entities in our Postgres database. At first, this might seem like a simple problem with a simple solution, especially if you’re using the ORM; just jam the user input into an ORM filter and retrieve every matching string. But there’s a problem: if you do Bands.objects.filter(name="beatles") You’ll probably get nothing back, because the name column in your “bands” table probably says “The Beatles” and as far as Postgres is concerned if it’s not exactly the same string, it’s not a match. Users are naturally terrible at spelling, and even if they weren’t they’d be bad at guessing exactly how the name is formatted in your database. Of course you can use the LIKE keyword in SQL (or the equivalent ‘__contains’ suffix in the ORM) to give yourself a little flexibility and make sure that “Beatles” returns “The Beatles”. But 1) the LIKE keyword requires you to evaluate a regex against every row in your table, or hope that you’ve configured your indices to support LIKE (a quick Google doesn’t tell me whether Django does that by default in the ORM) and 2) what if the user types “Beetles”? Well, then you’ve got a bit of a problem. No matter how obvious it is to human you that “beatles” is close to “beetles”[1], to the computer they’re just two non-identical byte sequences. If you want the computer to understand them as similar you’re going to have to give it a metric for similarity and a method to make the comparison. There are a few ways to do that. You can do what I did initially and whip out the power tools, i.e. a dedicated search system like Solr or ElasticSearch. These guys have notions of fuzziness built right in (Solr more automatically than ES). But they’re designed for full-text indexing of documents (e.g. full web pages) and they’re rather complex to set up and administer. ES has been enough of a hassle to keep running smoothly that I took the time to see if I could push the search workload to Postgres, and hence this article. Unless you need to do something real fancy, it’s probably overkill to use them for just matching names. Instead, we’re going to follow Starr Horne’s advice and use a Postgres EXTENSION that lets us build fuzziness into our query in a fast and fairly simple way. Specifically, we’re going to use an extension called pg_trgm (i.e. “Postgres Trigram”) which gives Postgres a “similarity” function that can evaluate how many three-character subsequences (i.e. “trigrams”) two strings share. This is actually a pretty good metric for fuzzy matching short strings like names. To use pg_trgm, you’ll need to install the “Postgres Contrib” package. On ubuntu: sudo apt-get install postgres-contrib **WARNING: THIS WILL TRY TO RESTART YOUR DATABASE** then pop open psql and install pg_trgm (NB: this only works on Postgres 9.1+; Google for the instructions if you’re on a lower version.) psql CREATE EXTENSION pg_trgm; \dx # to check it's installed Now you can do SELECT * FROM types_and_labels_view WHERE label %'Mountain Goats' ORDER BY similarity(label,'Mountain Goats') DESC LIMIT 100; And out will pop the 100 most similar names. This will still take a long time if your table is large, but we can improve that with a special type of index provided by pg_trgm: CREATE INDEX labels_trigram_index ON types_and_labels_table USING gist (label gist_trgm_ops); or CREATE INDEX labels_trigram_index ON types_and_labels_table USING gin (label gin_trgm_ops); (GIN is slower than GIST to build, but answers queries faster. That’ll take a while to build (possibly quite a while), but once it does you should be able to fuzzy search with ease and speed. If you’re using Django, you will have to drop into writing SQL to use this (until someone, maybe you, writes a Django extension to do this in the ORM.) And as a frustrating finishing note, my attempt to implement this on LinerNotes was not ultimately succesful. It seems that that index query performance is at least O(n) and with 50 million entities in my database queries take at least 10 seconds. I’ve read that performance is great up to about 100k records then drops off sharply from there. There are some apparently additional options for improving query performance, but I’ll be sticking with ElasticSearch for now. [1] Sorry, Googlebot! Not sorry, Bingbot.

Let's say you have a NetBeans Platform application that puts a premium on vertical space. Maybe a Heads Up Display on a Touch Screen? Wouldn't it be great to have the menu slide out from the edge of the screen only when you need it? Well the NetBeans Platform provides slide-in TopComponents, of course, but a JMenu just isn't going to work out so well inside one. We can use JavaFX as part of the solution as it provides some capabilities that the base Swing components available in the NetBeans Platform do not. Let's say we take all of our root MenuBar items and place them within an Accordion type pane. Each collapsible TitledPane of the Accordion control could then contain the sub-menu items, maybe represented by a JavaFX MenuButton. This would allow for a recursive Menu like effect but the overall container could be placed anywhere. Something like the screenshot below: What we see here is the described effect sliding out and overlayed on top of the Favorites tab. I sprinkled in some transparency for good measure. Notice how we are able to completely eliminate the Menu Bar and Tool Bar gaining potentially valuable real estate? The rest of this tutorial will explain the steps necessary to achieve something like this. That article was written by Geertjan Wielenga and it will become clear that much of the base code to accomplish this article was extended from Geertjan's example. Thanks again Geertjan! Similar articles to this that may be helpful are below: https://dzone.com/articles/javafx-fxml-meets-netbeans https://dzone.com/articles/how-embed-javafx-chart-visual All these articles are loosely coupled in a tutorial arc towards explaining and demonstrating the advantages of integrating JavaFX into the NetBeans Platform. The following two steps are borrowed exactly as found from Geertjan's tutorial: Step 1. Remove the default menubar and replace with your own: import org.openide.modules.ModuleInstall; public class Installer extends ModuleInstall { @Override public void restored() { System.setProperty("netbeans.winsys.menu_bar.path", "LookAndFeel/MenuBar.instance"); } } Step 2: In the layer.xml file define your Swing replacement menubar. I have also taken the liberty to hide the Toolbars as well. Now why are we replacing the old MenuBar with a new MenuBar if we intend to hide it? Well if you hide the MenuBar via the layer.xml as I did the Toolbars the filesystem folder tree will not be instantiated. That means we won't be able to dynamically determine the Menu Folder tree to rebuild our custom AccordionMenu. The solution? Make an empty Menubar. package polaris.javafxwizard.jfxmenu; import javax.swing.JMenuBar; /** * * @author SPhillips (King of Australia) */ public class HiddenMenuBar extends JMenuBar { public HiddenMenuBar() { super(); } } Step 3: Build an "AccordionMenu" using JavaFX This is where the tutorials diverge and this process gets a bit more complicated. Our task is to use the JavaFX/Swing Interop pattern to create a component that extends JFXPanel yet can give the user access to all the items that were once in the Menu Bar. The basic algorithm is as such: Create a component that extends JFXPanel Implement the standard Platform.runLater() pattern for creating a JavaFX scene Loop through each top level file object in the Menu folder of the application file system: Create a JavaFX Flow Pane for each file object Recursively create JavaFX ButtonMenu items for submenus Add ButtonMenu items to FlowPanes Add FlowPane to JavaFX TitledPane Add TitledPane to JavaFX Accordion component Add Accordion to scene So instead of Menus and SubMenus, we are using MenuButtons which can be recursively added to other MenuButtons and MenuItems. The Accordion control gives us a space saving collapsible view with some nice animation. The FlowPane makes it easy to layout the MenuButtons horizontally in a way that maximizes space. Below is the code for my AccordionMenu class. You will see where I borrowed heavily from Geertjan's example: polaris.javafxwizard.jfxmenu; import java.lang.reflect.InvocationTargetException; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import javafx.application.Platform; import javafx.embed.swing.JFXPanel; import javafx.event.ActionEvent; import javafx.event.EventHandler; import javafx.geometry.Orientation; import javafx.scene.Group; import javafx.scene.Scene; import javafx.scene.control.Accordion; import javafx.scene.control.Button; import javafx.scene.control.MenuButton; import javafx.scene.control.MenuItem; import javafx.scene.control.TitledPane; import javafx.scene.effect.DropShadow; import javafx.scene.layout.FlowPane; import javafx.scene.paint.Color; import javax.swing.Action; import javax.swing.SwingUtilities; import org.openide.awt.Actions; import org.openide.filesystems.FileObject; import org.openide.filesystems.FileUtil; import org.openide.loaders.DataFolder; import org.openide.loaders.DataObject; import org.openide.util.Exceptions; /** * * @author SPhillips (King of Australia) */ public class AccordionMenu extends JFXPanel{ public Accordion accordionPane; public String transparentCSS = "-fx-background-color: rgba(0,100,100,0.1);"; public AccordionMenu() { super(); // create JavaFX scene Platform.setImplicitExit(false); Platform.runLater(new Runnable() { @Override public void run() { createScene(); //Standard Swing Interop Pattern } }); } private void createScene() { FileObject menuFolder = FileUtil.getConfigFile("Menu"); FileObject[] menuKids = menuFolder.getChildren(); //for each Menu folder need to create a TilePane and add it to an Accordion List titledPaneList = new ArrayList<>(); for (FileObject menuKid : FileUtil.getOrder(Arrays.asList(menuKids), true)) { //Build a Flow pane based on menu children //TOP level menu items should all be flow panes FlowPane flowPane = buildFlowPane(menuKid); flowPane.setStyle(transparentCSS); TitledPane newTitledPaneFromFileObject = new TitledPane(menuKid.getName(), flowPane); newTitledPaneFromFileObject.setAnimated(true); newTitledPaneFromFileObject.autosize(); newTitledPaneFromFileObject.setStyle(transparentCSS); titledPaneList.add(newTitledPaneFromFileObject); } Group g = new Group(); Scene scene = new Scene(g, 400, 400,new Color(0.0,0.0,0.0,0.0)); scene.setFill(null); g.setStyle(transparentCSS); accordionPane = new Accordion(); accordionPane.setStyle(transparentCSS); accordionPane.getPanes().addAll(titledPaneList); g.getChildren().add(accordionPane); setScene(scene); validate(); this.setOpaque(true); this.setBackground(new java.awt.Color(0.0f, 0.0f, 0.0f, 0.0f)); } private FlowPane buildFlowPane(FileObject fo) { //FlowPanes are made up of Buttons and MenuButtons built from actions and sub menus FlowPane flowPane = new FlowPane(Orientation.HORIZONTAL,5,5); flowPane.setStyle(transparentCSS); //If anything at the Flow Pane level is an action we need to add it as a button //otherwise we can recursively build it as a MenuButton DataFolder df = DataFolder.findFolder(fo); DataObject[] childs = df.getChildren(); for (DataObject oneChild : childs) { //If child is folder we need to build recursively if (oneChild.getPrimaryFile().isFolder()) { FileObject childFo = oneChild.getPrimaryFile(); MenuButton newMenuButton = new MenuButton(childFo.getName()); buildMenuButton(childFo, newMenuButton); flowPane.getChildren().add(newMenuButton); } else { Object instanceObj = FileUtil.getConfigObject(oneChild.getPrimaryFile().getPath(), Object.class); if (instanceObj instanceof Action) { //If it is an Action we have reached an endpoint final Action a = (Action) instanceObj; String name = (String) a.getValue(Action.NAME); String cutAmpersand = Actions.cutAmpersand(name); Button buttonItem = new Button(cutAmpersand); MenuEventHandler meh = new MenuEventHandler(a); buttonItem.setOnAction(meh); buttonItem.setEffect(new DropShadow()); flowPane.getChildren().add(buttonItem); } } } return flowPane; } private void buildMenuButton(FileObject fo, MenuButton menuButton) { DataFolder df = DataFolder.findFolder(fo); DataObject[] childs = df.getChildren(); for (DataObject oneChild : childs) { //If child is folder we need to build recursively if (oneChild.getPrimaryFile().isFolder()) { FileObject childFo = oneChild.getPrimaryFile(); //Menu newMenu = new Menu(childFo.getName()); MenuButton newMenuButton = new MenuButton(childFo.getName()); //menu.getItems().add(newMenu); buildMenuButton(childFo, newMenuButton); } else { Object instanceObj = FileUtil.getConfigObject(oneChild.getPrimaryFile().getPath(), Object.class); if (instanceObj instanceof Action) { //If it is an Action we have reached an endpoint final Action a = (Action) instanceObj; String name = (String) a.getValue(Action.NAME); String cutAmpersand = Actions.cutAmpersand(name); MenuItem menuItem = new MenuItem(cutAmpersand); MenuEventHandler meh = new MenuEventHandler(a); menuItem.setOnAction(meh); menuButton.getItems().add(menuItem); } } } } private class MenuEventHandler implements EventHandler { public Action theAction; public MenuEventHandler(Action action) { super(); theAction = action; } @Override public void handle(final ActionEvent t) { try { SwingUtilities.invokeAndWait(new Runnable() { @Override public void run() { java.awt.event.ActionEvent event = new java.awt.event.ActionEvent( t.getSource(), t.hashCode(), t.toString()); theAction.actionPerformed(event); } }); } catch ( InterruptedException | InvocationTargetException ex) { Exceptions.printStackTrace(ex); } } } } I took the liberty of placing a few CSS stylings here and there, trying to play with the transparency. Also I found that it looked better if a JavaFX Button was used for any Actions found at the very top level, instead of a MenuButton with a single item. Step 4: Build a Slide in TopComponent for the new AccordionMenu Now that you have a JFXPanel Swing Interop component, your NetBeans Platform TopComponent doesn't need to know about JavaFX. However in this scenario the Platform also is contributing via its wonderful docking framework. Use the Window wizard and select Left Sliding In as a mode. I would also advise making this component not closable, otherwise the user could lose the ability to use the menu. Here are the annotations and constructor code in my TopComponent: @ConvertAsProperties( dtd = "-//polaris.javafxwizard.jfxmenu//SlidingAccordion//EN", autostore = false) @TopComponent.Description( preferredID = "SlidingAccordionTopComponent", iconBase="polaris/javafxwizard/jfxmenu/categories.png", persistenceType = TopComponent.PERSISTENCE_ALWAYS) @TopComponent.Registration(mode = "leftSlidingSide", openAtStartup = true) @ActionID(category = "Window", id = "polaris.javafxwizard.jfxmenu.SlidingAccordionTopComponent") @ActionReference(path = "Menu/JavaFX" /*, position = 333 */) @TopComponent.OpenActionRegistration( displayName = "#CTL_SlidingAccordionAction", preferredID = "SlidingAccordionTopComponent") @Messages({ "CTL_SlidingAccordionAction=SlidingAccordion", "CTL_SlidingAccordionTopComponent=SlidingAccordion Window", "HINT_SlidingAccordionTopComponent=This is a SlidingAccordion window" }) public final class SlidingAccordionTopComponent extends TopComponent { public AccordionMenu accordionMenu; public SlidingAccordionTopComponent() { initComponents(); setName(Bundle.CTL_SlidingAccordionTopComponent()); setToolTipText(Bundle.HINT_SlidingAccordionTopComponent()); putClientProperty(TopComponent.PROP_CLOSING_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_DRAGGING_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_MAXIMIZATION_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_UNDOCKING_DISABLED, Boolean.TRUE); putClientProperty(TopComponent.PROP_KEEP_PREFERRED_SIZE_WHEN_SLIDED_IN, Boolean.TRUE); setLayout(new BorderLayout()); //Standard JFXPanel Swing Interop Pattern accordionMenu = new AccordionMenu(); //transparency Color transparent = new Color(0.0f, 0.0f, 0.0f, 0.0f); accordionMenu.setOpaque(true); accordionMenu.setBackground(transparent); this.add(accordionMenu); this.setOpaque(true); this.setBackground(transparent); } Step 5. See how great it looks We now have a slide out collapsible application menu provided by JavaFX components. These components can be "skinned" using CSS stylings and as such the menu can be crafted differently for different applications. (By the way if anyone reading this has some ideas please contact me because I am not a CSS guy at all) Best of all we have adapted our application to work nicely with a Heads Up Display or Kiosk view that typically run on touchscreen computers. This is because we have saved real estate and implemented an interface that is more condusive to single touches versus mouse drag events. Hey let's see how it might look with an application that needs all the space it can get?

The Reportlab library is a great way to generate PDFs in Python. Recently, I noticed that it has the ability to do barcodes. I had heard about it being able to generate QR codes, but I hadn’t really dug under the covers to see what else it could do. In this tutorial, we’ll take a look at some of the barcodes that Reportlab can generate. If you don’t already have Reportlab, go to their website and get it before jumping into the article. Reportlab’s Barcode Library Reportlab provides for several different types of bar codes: code39 (i.e. code 3 of 9), code93, code 128, EANBC, QR, and USPS. I saw one called “fourstate” as well, but I couldn’t figure out how to get it to work. Underneath some of these types, there are sub-types such as Standard, Extended or MultiWidth. I didn’t have much luck getting the MultiWidth one to work for the code128 bar code as it kept giving me an attribute error, so we’ll just ignore that one. If you know how to do it, ping me in the comments or via my contact form and let me know. I’ll update the article if anyone can show me how to add that or the fourstate barcode. Anyway, the best way to learn is to just write some code. Here’s a pretty straight-forward example: from reportlab.graphics.barcode import code39, code128, code93 from reportlab.graphics.barcode import eanbc, qr, usps from reportlab.graphics.shapes import Drawing from reportlab.lib.pagesizes import letter from reportlab.lib.units import mm from reportlab.pdfgen import canvas from reportlab.graphics import renderPDF #---------------------------------------------------------------------- def createBarCodes(): """ Create barcode examples and embed in a PDF """ c = canvas.Canvas("barcodes.pdf", pagesize=letter) barcode_value = "1234567890" barcode39 = code39.Extended39(barcode_value) barcode39Std = code39.Standard39(barcode_value, barHeight=20, stop=1) # code93 also has an Extended and MultiWidth version barcode93 = code93.Standard93(barcode_value) barcode128 = code128.Code128(barcode_value) # the multiwidth barcode appears to be broken #barcode128Multi = code128.MultiWidthBarcode(barcode_value) barcode_usps = usps.POSTNET("50158-9999") codes = [barcode39, barcode39Std, barcode93, barcode128, barcode_usps] x = 1 * mm y = 285 * mm x1 = 6.4 * mm for code in codes: code.drawOn(c, x, y) y = y - 15 * mm # draw the eanbc8 code barcode_eanbc8 = eanbc.Ean8BarcodeWidget(barcode_value) bounds = barcode_eanbc8.getBounds() width = bounds[2] - bounds[0] height = bounds[3] - bounds[1] d = Drawing(50, 10) d.add(barcode_eanbc8) renderPDF.draw(d, c, 15, 555) # draw the eanbc13 code barcode_eanbc13 = eanbc.Ean13BarcodeWidget(barcode_value) bounds = barcode_eanbc13.getBounds() width = bounds[2] - bounds[0] height = bounds[3] - bounds[1] d = Drawing(50, 10) d.add(barcode_eanbc13) renderPDF.draw(d, c, 15, 465) # draw a QR code qr_code = qr.QrCodeWidget('www.mousevspython.com') bounds = qr_code.getBounds() width = bounds[2] - bounds[0] height = bounds[3] - bounds[1] d = Drawing(45, 45, transform=[45./width,0,0,45./height,0,0]) d.add(qr_code) renderPDF.draw(d, c, 15, 405) c.save() if __name__ == "__main__": createBarCodes() Let’s break this down a bit. The code39.Extended39 doesn’t really accept much beyond the value itself. On the other hand, code39.Standard39, code93.Standard93 and code128.Code128 all have basically the same API. You can change the barWidth, barHeight, turn on the start/stop symbols and add “quiet” zones. The usps bar code module provides two types of bar code: FIM and POSTNET. FIM or Facing ID Marks only encodes one letter (A-D) which I personally didn’t find it very interesting. So I just show the POSTNET version which should be pretty familiar to people in the United States as it appears on the bottom of most envelopes. POSTNET encodes the zip code! The next three bar codes use a different API to draw them on the PDF that I discovered viaStackOverflow. Basically you create a Drawing object of a certain size and then add the bar code to the drawing. Finally you use the renderPDF module to place the drawing on the PDF. It’s pretty convoluted, but it works pretty well. The EANBC codes are ones you’ll see on some manufactured products, such as tissue boxes. If you’d like to see the result of the code above, you can download the PDF here. Wrapping Up At this point you should be able to go forth and create your own bar codes in your PDFs. Reportlab is pretty handy and I hope you’ll find this additional tool helpful in your endeavors. Additional Reading A step by step Reportlab tutoral Reportlab: Mixing Fixed Content and Flowables Reportlab Tables – Creating Tables in PDFs with Python Creating QR Codes with Python StackOverflow question on Python barcode generation StackOverflow question on reportlab, QR codes and django Get the Source! barcodes.tar

After thoroughly exploring CompletableFuture API in Java 8 we are prepared to write a simplistic web crawler. We solved similar problem already using ExecutorCompletionService, Guava ListenableFuture and Scala/Akka. I choose the same problem so that it's easy to compare approaches and implementation techniques. First we shall define a simple, blocking method to download the contents of a single URL private String downloadSite(final String site) { try { log.debug("Downloading {}", site); final String res = IOUtils.toString(new URL("http://" + site), UTF_8); log.debug("Done {}", site); return res; } catch (IOException e) { throw Throwables.propagate(e); } } Nothing fancy. This method will be later invoked for different sites inside thread pool. Another method parses the Stringinto an XML Document (let me leave out the implementation, no one wants to look at it): private Document parse(String xml) //... Finally the core of our algorithm, function computing relevance of each website taking Document as input. Just as above we don't care about the implementation, only the signature is important: private CompletableFuture calculateRelevance(Document doc) //... Let's put all the pieces together. Having a list of websites our crawler shall start downloading the contents of each web site asynchronously and concurrently. Then each downloaded HTML string will be parsed to XML Document and laterrelevance will be computed. As a last step we take all computed relevance metrics and find the biggest one. This sounds pretty straightforward to the moment when you realize that both downloading content and computing relevance is asynchronous (returns CompletableFuture) and we definitely don't want to block or busy wait. Here is the first piece: ExecutorService executor = Executors.newFixedThreadPool(4); List topSites = Arrays.asList( "www.google.com", "www.youtube.com", "www.yahoo.com", "www.msn.com" ); List> relevanceFutures = topSites.stream(). map(site -> CompletableFuture.supplyAsync(() -> downloadSite(site), executor)). map(contentFuture -> contentFuture.thenApply(this::parse)). map(docFuture -> docFuture.thenCompose(this::calculateRelevance)). collect(Collectors.>toList()); There is actually a lot going on here. Defining thread pool and sites to crawl is obvious. But there is this chained expression computing relevanceFutures. The sequence of map() and collect() in the end is quite descriptive. Starting from a list of web sites we transform each site (String) into CompletableFuture by submitting asynchronous task (downloadSite()) into thread pool. So we have a list of CompletableFuture. We continue transforming it, this time applying parse() method on each of them. Remember that thenApply() will invoke supplied lambda when underlying future completes and returnsCompletableFuture immediately. Third and last transformation step composes eachCompletableFuture in the input list with calculateRelevance(). Note that calculateRelevance()returns CompletableFuture instead of Double, thus we use thenCompose() rather than thenApply(). After that many stages we finally collect() a list of CompletableFuture. Now we would like to run some computations on all results. We have a list of futures and we would like to know when all of them (last one) complete. Of course we can register completion callback on each future and use CountDownLatch to block until all callbacks are invoked. I am too lazy for that, let us utilize existing CompletableFuture.allOf(). Unfortunately it has two minor drawbacks - takes vararg instead of Collection and doesn't return a future of aggregated results but Void instead. By aggregated results I mean: if we provide List> such method should return CompletableFuture>, not CompletableFuture! Luckily it's easy to fix with a bit of glue code: private static CompletableFuture> sequence(List> futures) { CompletableFuture allDoneFuture = CompletableFuture.allOf(futures.toArray(new CompletableFuture[futures.size()])); return allDoneFuture.thenApply(v -> futures.stream(). map(future -> future.join()). collect(Collectors.toList()) ); } Watch carefully sequence() argument and return types. The implementation is surprisingly simple, the trick is to use existing allOf() but when allDoneFuture completes (which means all underlying futures are done), simply iterate over all futures and join() (blocking wait) on each. However this call is guaranteed not to block because by now all futures completed! Equipped with such utility method we can finally complete our task: CompletableFuture> allDone = sequence(relevanceFutures); CompletableFuture maxRelevance = allDone.thenApply(relevances -> relevances.stream(). mapToDouble(Double::valueOf). max() ); This one is easy - when allDone completes, apply our function that counts maximal relevance in whole set.maxRelevance is still a future. By the time your JVM reaches this line, probably none of the websites are yet downloaded. But we encapsulated business logic on top of futures, stacking them in an event-driven manner. Code remains readable (version without lambda and with ordinary Futures would be at least twice as long) but avoids blocking main thread. Of course allDone can as well be an intermediate step, we can further transform it, not really having the result yet. Shortcomings CompletableFuture in Java 8 is a huge step forward. From tiny, thin abstraction over asynchronous task to full-blown, functional, feature rich utility. However after few days of playing with it I found few minor disadvantages: CompletableFuture.allOf() returning CompletableFuture discussed earlier. I think it's fair to say that if I pass a collection of futures and want to wait for all of them, I would also like to extract the results when they arrive easily. It's even worse with CompletableFuture.anyOf(). If I am waiting for any of the futures to complete, I can't imagine passing futures of different types, say CompletableFuture andCompletableFuture. If I don't care which one completes first, how am I suppose to handle return type? Typically you will pass a collection of homogeneous futures (e.g. CompletableFuture) and thenanyOf() can simply return future of that type (instead of CompletableFuture again). Mixing settable and listenable abstractions. In Guava there is ListenableFuture and SettableFuture extending it. ListenableFuture allows registering callbacks while SettableFuture adds possibility to set value of the future (resolve it) from arbitrary thread and context. CompletableFuture is equivalent to SettableFuture but there is no limited version equivalent to ListenableFuture. Why is it a problem? If API returns CompletableFuture and then two threads wait for it to complete (nothing wrong with that), one of these threads can resolve this future and wake up other thread, while it's only the API implementation that should do it. But when API tries to resolve the future later, call to complete() is ignored. It can lead to really nasty bugs which are avoided in Guava by separating these two responsibilities. CompletableFuture is ignored in JDK. ExecutorService was not retrofitted to return CompletableFuture. Literally CompletableFuture is not referenced anywhere in JDK. It's a really useful class, backward compatible withFuture, but not really promoted in standard library. Bloated API (?) Fifty methods in total, most in three variants. Splitting settable and listenable (see above) would help. Also some methods like runAfterBoth() or runAfterEither() IMHO do not really belong to anyCompletableFuture. Is there a difference between fast.runAfterBoth(predictable, ...) andpredictable.runAfterBoth(fast, ...)? No, but API favours one or the other. Actually I believerunAfterBoth(fast, predictable, ...) much better expresses my intention. CompletableFuture.getNow(T) should take Supplier instead of raw reference. In the example belowexpensiveAlternative() is always code, irrespective to whether future finished or not: future.getNow(expensiveAlternative()); However we can easily tweak this behaviour (I know, there is a small race condition here, but the original getNow()works this way as well): public static T getNow( CompletableFuture future, Supplier valueIfAbsent) throws ExecutionException, InterruptedException { if (future.isDone()) { return future.get(); } else { return valueIfAbsent.get(); } } With this utility method we can avoid calling expensiveAlternative() when it's not needed: getNow(future, () -> expensiveAlternative()); //or: getNow(future, this::expensiveAlternative); In overall CompletableFuture is a wonderful new tool in our JDK belt. Minor API issues and sometimes too verbose syntax due to limited type inference shouldn't stop you from using it. At least it's a solid foundation for better abstractions and more robust code.

Check out a list of 256 programming languages, from ABC to Z shell.

Introduction: I am from Pakistan. Like most of Pakistanis, I am also a huge fan of Cricket. Yesterday, I got some time from my busy job. I thought to create a simple Cricket Live Score feature using ASP.NET SignalR. It is also Interesting to note that the guys(David Fowler from West Indies and Damian Edwards from Australia) that are heavily involved in SignalR development are also cricket fans(correct me if I am wrong). In this article, I will show you how to create a simple 'Live Cricket Score' feature using SignalR. For a quick demo, just open this link in different browsers and in one browser just append ?admin= query-string(which will become the admin who update the score), then just click Update Score button many times until 2 overs finish(the match will be limited to 2 overs). For each click, the live score will be updated in each browser. Description: If you are new to SignalR then Getting Started will be great for you. Assuming that you have configured your application with ASP.NET SignalR. Let create a simple html page with following lines, Pakistan v/s India / Overs In the above page, we have referenced jquery, signalr and knockout scripts. We have also define a simple template which will show the runs, wickets and overs of both teams. For demo purpose, we have added a Update Score button in the page. This button when clicked will record a single ball of an over as well as runs scored in this ball . After each ball, I will tell SignalR to broadcast teams score-card across all the connected client. When the match finished(after 24 balls), I will again tell the SignalR to show the final result to all users. Here are the scripts which are required for our work, $(function () { var chat = $.connection.cricketHub, viewModel = { teams: ko.observableArray([ { team: 'india', runs: 0, wickets: 0, balls: 0, oversText: '0.0' }, { team: 'pakistan', runs: 0, wickets: 0, balls: 0, oversText: '0.0'}]) }, index = 0;// index represant the position of the current batting team chat.client.updateScore = function (t) { viewModel.teams(JSON.parse(t)); }; chat.client.matchFinished = function (result) { alert(result); }; $.connection.hub.start().done(function () { $('#updateScoreLink').click(function () { var teams = viewModel.teams(), team1 = teams[index], team2 = teams[(index + 1) % 2], wicketFalled = getRandomInt(0, 5) == 3 ? true : false; // Assuming that the wicket will only fall when a 0-5 random number is 3 team1.runs += getRandomInt(0, 6); // Get a random score between 0 to 6 team1.wickets += wicketFalled ? 1 : 0; team1.balls += 1; team1.oversText = truncate(team1.balls / 6).toString() + '.' + (team1.balls % 6).toString(); chat.server.updateScore(JSON.stringify(teams)); if (team1.balls === 12 && team2.balls === 12) { $('#updateScoreLink').hide(); var message = (team1.runs > team2.runs ? team1.team : team2.team) + ' won'; message = team1.runs === team2.runs ? 'match tied' : message; message += '\n' + team1.team + ': ' + team1.runs + '/' + team1.wickets + ' overs: ' + team1.oversText; message += '\n' + team2.team + ': ' + team2.runs + '/' + team2.wickets + ' overs: ' + team2.oversText; chat.server.matchFinished(message); } else if (team1.balls === 12) { index = 1;// second team batting } viewModel.teams([team1,team2]); }); }); ko.applyBindings(viewModel); if (isAdmin()) { $('#updateScore').show(); } function getRandomInt(min, max) { return Math.floor(Math.random() * (max - min + 1)) + min; } function truncate(n) { return Math[n > 0 ? "floor" : "ceil"](n); } function isAdmin(){ // For demo, assuming that only admin can update the score // and the admin will be user which have admin querystring. return location.href.indexOf('admin=') > -1 } }); In the above scripts, we are initializing SignalR hub cricketHub object(we will see this hub class shortly), viewModel object which include team information(runs, wickets, etc) and index which represent the current batting team. Next, we have two client functions(updateScore and matchFinished) which will be invoked when the server broadcast a message to all clients. When the hub start, we are registering event handler Update Score click event where we are getting a random run between 0-6, a wicket or not depending upon a random number, updating our view-model and then telling SignalR to broadcast the view-model across all the connected clients so that all clients automatically update their scorecard. Finally, when both teams completed their 12 balls, we will send the results to all the connected clients using SignalR. Note that for demo purpose we are only showing the Update Score button to the user which have admin= query-string. Finally, here is our CricketHub class, public class CricketHub : Hub { public void UpdateScore(string teams) { Clients.All.UpdateScore(teams); } public void MatchFinished(string result) { Clients.All.MatchFinished(result); } } Summary: In this article, I showed you one of the usage of ASP.NET SignalR. You have seen that how easily we can create a real-time Live Cricket Score using the power of ASP.NET SignalR which works in every browser . Hopefully you will enjoy my this article too.