My team and I are creating a services platform based on a set of RESTful JSON services where each service contributes to the platform by providing distinct feature(s) and/or data. With logs being generated all over the place, we thought it was a good idea to centralize logging and perhaps also provide a rudimentary log viewer that allowed us to view, filter, sort and search our logs. We also wanted our logging to be asynchronous as we didn’t want our services to be held up while trying to write logs say maybe directly to a database. The strategy for achieving this was straight forward. Setup ActiveMQ Create a log4j appender that writes logs to the queue (log4j ships with one such appender but lets write our own. Write a message listener that reads logs from a JMS queue setup on an MQ server and persists them Let’s take a look one by one. Setup ActiveMQ Setting up an external ActiveMQ server is simple enough. A great tutorial is available at http://servicebus.blogspot.com/2011/02/installing-apache-active-mq-on-ubuntu.html to set it up on Ubuntu. You can also choose to embed a message broker within your application. Spring makes this easy. We will see how later. Creating a Lo4j JMS appender First, we create a log4j JMS appender. log4j ships with one such appender (that writes to a JMS topic instead of a queue) import javax.jms.DeliveryMode; import javax.jms.Destination; import javax.jms.MessageProducer; import javax.jms.ObjectMessage; import javax.jms.Session; import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.log4j.Appender; import org.apache.log4j.AppenderSkeleton; import org.apache.log4j.Logger; import org.apache.log4j.PatternLayout; import org.apache.log4j.spi.LoggingEvent; /** * JMSQueue appender is a log4j appender that writes LoggingEvent to a queue. * @author faheem * */ public class JMSQueueAppender extends AppenderSkeleton implements Appender{ private static Logger logger = Logger.getLogger("JMSQueueAppender"); private String brokerUri; private String queueName; @Override public void close() { } @Override public boolean requiresLayout() { return false; } @Override protected synchronized void append(LoggingEvent event) { try { ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory( this.brokerUri); // Create a Connection javax.jms.Connection connection = connectionFactory.createConnection(); connection.start();np // Create a Session Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE); // Create the destination (Topic or Queue) Destination destination = session.createQueue(this.queueName); // Create a MessageProducer from the Session to the Topic or Queue MessageProducer producer = session.createProducer(destination); producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT); ObjectMessage message = session.createObjectMessage(new LoggingEventWrapper(event)); // Tell the producer to send the message producer.send(message); // Clean up session.close(); connection.close(); } catch (Exception e) { e.printStackTrace(); } } public void setBrokerUri(String brokerUri) { this.brokerUri = brokerUri; } public String getBrokerUri() { return brokerUri; } public void setQueueName(String queueName) { this.queueName = queueName; } public String getQueueName() { return queueName; } } Lets see whats happening here. Line 19: We implement the Log4J appender interface that asks us to implement three methods. requiresLayout, close and append. We will keep things simple for the moment and implement the append method which gets called whenever a method call to the logger is made. Line 37: log4j calls the append method and passes a LoggingEvent object as a parameter which represents a call to a logger. A LoggingEvent object encapsulates all information about every log item. Line 41 & 42: Create a new connection factory by providing it with a uri of a JMS, in our case activemq, server Line 45, 46 and 49: We establish a connection and a session to the JMS server. A Session can be opened in several modes. An Auto_Acknowledge session is one in which the acknowledgment of message happens automatically. Other modes include Client_Acknowledge in which a client has to explicitly acknowledge receipt and/or processing of a message and two other modes. For details, refer to the docs at http://download.oracle.com/javaee/1.4/api/javax/jms/Session.html Line 52: Create a queue. Send the queue name to connect to as a parameter. Line 56: We set the delivery mode to Non_Persistent. The other option is Persistent where the message is persisted to a persistent store. Persistent mode slows down but adds reliability to the message transfer. Line 58: We are doing multiple things. First of all I am wrapping the LoggingEvent object into a LoggingEventWrapper. This is because there are some properties within the LoggingEvent object that are not serializeable and also because I want to capture some additional information such as IP address and host name. Next, using the JMS session object, I prepare an object (the wrapper) for transport. Line 61: I send the object to the queue. Below is the code for the wrapper. import java.io.Serializable; import java.net.InetAddress; import java.net.UnknownHostException; import org.apache.log4j.EnhancedPatternLayout; import org.apache.log4j.spi.LoggingEvent; /** * Logging Event Wraps a log4j LoggingEvent object. Wrapping is required by some information is lost * when the LoggingEvent is serialized. The idea is to extract all information required from the LoggingEvent * object, place it in the wrapper and then serialize the LoggingEventWrapper. This way all required data remains * available to us. * @author faheem * */ public class LoggingEventWrapper implements Serializable{ private static final String ENHANCED_PATTERN_LAYOUT = "%throwable"; private static final long serialVersionUID = 3281981073249085474L; private LoggingEvent loggingEvent; private Long timeStamp; private String level; private String logger; private String message; private String detail; private String ipAddress; private String hostName; public LoggingEventWrapper(LoggingEvent loggingEvent){ this.loggingEvent = loggingEvent; //Format event and set detail field EnhancedPatternLayout layout = new EnhancedPatternLayout(); layout.setConversionPattern(ENHANCED_PATTERN_LAYOUT); this.detail = layout.format(this.loggingEvent); } public Long getTimeStamp() { return this.loggingEvent.timeStamp; } public String getLevel() { return this.loggingEvent.getLevel().toString(); } public String getLogger() { return this.loggingEvent.getLoggerName(); } public String getMessage() { return this.loggingEvent.getRenderedMessage(); } public String getDetail() { return this.detail; } public LoggingEvent getLoggingEvent() { return loggingEvent; } public String getIpAddress() { try { return InetAddress.getLocalHost().getHostAddress(); } catch (UnknownHostException e) { return "Could not determine IP"; } } public String getHostName() { try { return InetAddress.getLocalHost().getHostName(); } catch (UnknownHostException e) { return "Could not determine Host Name"; } } } The Message Listener The message listener “listens” to the queue (or topic). Whenever a new message is added to the queue, the onMessage method is called. import javax.jms.JMSException; import javax.jms.Message; import javax.jms.MessageListener; import javax.jms.ObjectMessage; import org.apache.log4j.Logger; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class LogQueueListener implements MessageListener { public static Logger logger = Logger.getLogger(LogQueueListener.class); @Autowired private ILoggingService loggingService; public void onMessage( final Message message ) { if ( message instanceof ObjectMessage ) { try{ final LoggingEventWrapper loggingEventWrapper = (LoggingEventWrapper)((ObjectMessage) message).getObject(); loggingService.saveLog(loggingEventWrapper); } catch (final JMSException e) { logger.error(e.getMessage(), e); } catch (Exception e) { logger.error(e.getMessage(),e); } } } } Line 23: Checking if the object being picked off the queue is an instance of ObjectMessage Line 26: Extracting LoggingEventWrapper from the Message Line 27: Call a service method to persist the log Wiring up in Spring Lines 5-9: Use the broker tag to setup an embedded message broker. Since I am using an external one, I don’t need it. Line 12: Mention the name of the queue you want to connect to. Line 14: URI of the Broker Server. Line 15-19: Connection Factory setup Line 26-28: Message Listener Setup where we specify the number of concurrent threads that can consume messages off the queue. Of course, the above example will not work out of the box. You still have to include all JMS dependencies and implement the service that persists logs. But I hope it gives you a decent idea.

Oracle offers two certifications for web component developers one for Java EE 5 and another one for Java EE 6.

Serialization is a form of persistence: serialized data survives the process and the RAM where it was created and can be reconstituted inside different processes and machines that live in a different time or place. Sometimes serialization is a poor form of persistence in fact, one that confuses the boundary between the different schemas the data can fit in. However, what I found useful in the last years of development is to institute a strict separation: serialize Value Objects, Entities, and everything that represents the state of the application. Meanwhile, use Dependency Injection over services that are part of a larger object graph and never serialize this second kind of objects. In the discussion that follows, I make the assumption that serialization and deserialization occur on the same machine (e.g. like for web-oriented sessions.) The problem with serialization, which work transparently most of the time, is the need to serialize service objects instead of limiting the procedure to data structures. How can you store such objects? Not options Some options to solve this problems are really not options. Serialization by itself will fail because of the staleness of the references contained in these objects. For example, in PHP trying to serialize a database connections composed by a Repository or DAO object will rightly fail with an exception. Whenever an object represents a resource of the current machine, it cannot usually be serialized except in the case when the only resource involved is RAM. If the resource is disk space or other running processes such as a database daemon, the reconstitution of the object in another place and time will fail and it's best to just stop the developer immediately during storage. Quasi-options Some solutions to the problem try to avoid the staleness problem by serializing objects without their resources, and make them regrab a new version of them on deserialization. In PHP for example, this can be done with the __sleep() and __wakeup() magic methods, called automatically during serialization and deserializaton respectively. This deserialization mechanism introduces a dependency from the serialized Entity to external services: such a dependency is already in place when building the object the first time (passing the XService in the constructor) but it is aggravated when deserializing (depending on a XServiceFactory instead of just an XService). An improvement, from the dependencies point of view, is to reattach collaborators to deserialized objects like you would for other persistence-related tasks. For example, EntityRepository can inject the missing pieces of Entity every time its find() method is called. However, there is still another option, which is the most resilient from the modelling point of view and not only that of dependency management: injecting non-serializable collaborators through the stack. Objects can collaborate even without keeping field references to each other, and injecting dependencies as parameters move the dependency starting point from the server to the client object (which may or may not be desirable). What is most important is that Entities are relieved of having to manage external references in any context, not only that of persistence and in particular serialization. The metaphor for the 3rd option Misko Hevery likes to say: have you ever seen a credit card able to charge itself? If a CreditCard is an Entity in your domain, it would be very strange to keeping a wire attached to your wallet wherever you go. With the first option, you have the card spring a wire when it is taken out of the wallet, like in horror movies. This intelligent cable tries as its best to attach to the nearest Point of Sale (a bad case of bluetooth I think). With Repositories in mind, you're not dealing with automated wires anymore, but you're still attaching cables between cards and fixed devices. In reality, cards collaborate with the PoS in a fast process that does not last more than a few seconds. Actually, sometimes they don't touch it at all, as in all Internet-based purchases. Keeping services around to deal with external dependencies does not mean the API of your Domain Model has to be biased towards service objects: pos.charge(creditCard); // can equivalently be: creditCard.chargeOn(pos); This is a form of Double Dispatch since there are two objects collaborating and you can dispatch (send messages) to both, being polimorphic by substituting both objects. The sequence of calls is: client -> creditCard -> pos The client object still looks at CreditCard as a behaviorally complete object, but it is clear which dependency is necessary to run each use case (CreditCard method). You can persist a CreditCard easily and send it over the wire to caches or databases. When it comes the time to charge, it is the client that has to bring forward a service able to connect to a bank.

A quick Java snippet for log scraping: package com.agilemobiledeveloper.logcheck; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import com.jcraft.jsch.Channel; import com.jcraft.jsch.ChannelSftp; import com.jcraft.jsch.JSch; import com.jcraft.jsch.Session; /** * * @author spannt * */ public class LogScraper { /** * @param args */ public static void main(String[] args) { String SFTPHOST = "myunixsite.com"; int SFTPPORT = 22; String SFTPUSER = "myunixid"; String SFTPPASS = "myunixpassword"; String SFTPWORKINGDIR = "/some/unix/directory"; String SERRORFILE = "SystemErr.log"; String SOUTFILE = "SystemOut.log"; Session session = null; Channel channel = null; ChannelSftp channelSftp = null; StringBuilder out = new StringBuilder(); try { JSch jsch = new JSch(); session = jsch.getSession(SFTPUSER, SFTPHOST, SFTPPORT); session.setPassword(SFTPPASS); java.util.Properties config = new java.util.Properties(); config.put("StrictHostKeyChecking", "no"); session.setConfig(config); session.connect(); channel = session.openChannel("sftp"); channel.connect(); channelSftp = (ChannelSftp) channel; channelSftp.cd(SFTPWORKINGDIR); System.out.println("Error File"); out.append("Error File:").append( LogScraper.parseStream(channelSftp.get(SERRORFILE))); System.out.println("Output File"); out.append("Output File:").append( LogScraper.parseStream(channelSftp.get(SOUTFILE))); } catch (Exception ex) { ex.printStackTrace(); out.append(ex.getLocalizedMessage()); } System.out.println("Logs=" + out.toString()); } /** * * line.contains("Exception") || * * @param file * @return String of error data */ public static String parseStream(InputStream inputFileStream) { if ( null == inputFileStream ) { return "Log Empty"; } StringBuilder out = new StringBuilder(); BufferedReader br = new BufferedReader(new InputStreamReader( inputFileStream)); String line = null; try { while ((line = br.readLine()) != null) { if (line.contains("OutOfMemoryError")) { out.append(line).append(System.lineSeparator()); } } } catch (IOException e) { e.printStackTrace(); out.append(e.getLocalizedMessage()); } return out.toString(); } }

Adopting Martin Fowler's approach to domain-specific language.

Apache Hive is a high level SQL-like interface to Hadoop. It lets you execute mostly unadulterated SQL, like this: CREATE TABLE test_table(key string, stats map); The map column type is the only thing that doesn’t look like vanilla SQL here. Hive can actually use different backends for a given table. Map is used to interface with column oriented backends like HBase. Essentially, because we won’t know ahead of time all the column names that could be in the HBase table, Hive will just return them all as a key/value dictionary. There are then helpers to access individual columns by key, or even pivot the map into one key per logical row. As part of the Hadoop family, Hive is focused on bulk loading and processing. So it’s not a surprise that Hive does not support inserting raw values like the following SQL: INSERT INTO suppliers (supplier_id, supplier_name) VALUES (24553, 'IBM'); However, for unit testing Hive scripts, it would be nice to be able to insert a few records manually. Then you could run your map reduce HQL, and validate the output. Luckily, Hive can load CSV files, so it’s relatively easy to insert a handful or records that way. CREATE TABLE foobar(key string, stats map) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY '|' MAP KEYS TERMINATED BY ':' ; LOAD DATA LOCAL INPATH '/tmp/foobar.csv' INTO TABLE foobar; This will load a CSV file with the following data, where c4ca4-0000001-79879483-000000000124 is the key, and comments and likesare columns in a map. c4ca4-0000001-79879483-000000000124,comments:0|likes:0 c4ca4-0000001-79879483-000000000124,comments:0|likes:0 Because I’ve been doing this quite a bit in my unit tests, I wrote a quick Python helper to dump a list of key/map tuples to a temporary CSV file, and then load it into Hive. This uses hiver to talk to Hive over thrift. import hiver from django.core.files.temp import NamedTemporaryFile def _hql(self, hql): client = hiver.connect(settings.HIVE_HOST, settings.HIVE_PORT) try: client.execute(hql) finally: client.shutdown() def insert(self, table_name, rows): ''' cannot insert single rows via hive, need to save to a temp file and bulk load that ''' csv_file = NamedTemporaryFile(delete=True) for row in rows: map_repr = '|'.join('%s:%s' % (key, value) for key, value in row[1].items()) csv_file.write(row[0] + "," + map_repr + "\n") csv_file.flush() try: _hql('DROP TABLE IF EXISTS %s' % table_name) _hql(""" CREATE TABLE %s ( key string, map ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY '|' MAP KEYS TERMINATED BY ':' """ % (table_name)) _hql(""" LOAD DATA LOCAL INPATH '%s' INTO TABLE %s """ % (csv_file.name, table_name) finally: csv_file.close() You can call it like this: insert('test_table', [ ('c4ca4-0000001-79879483-000000000124', {'comments': 1, 'likes': 2}), ('c4ca4-0000001-79879483-000000000124', {'comments': 1, 'likes': 2}), ('c4ca4-0000001-79879496-000000000124', {'comments': 1, 'likes': 2}), ('b4aed-0000002-79879783-000000000768', {'comments': 1, 'likes': 2}), ('b4aed-0000002-79879783-000000000768', {'comments': 1, 'likes': 2}), ])

[This blog was syndicated from http://blog.grallandco.com] Introduction When I was looking for a more effective way to create my cluster I asked some sysadmins which tools I should use to do it. The answer I got during OSDC was not Puppet, nor Chef, but wasAnsible. This article shows you how you can easily configure and create a Couchbase cluster deployed and many linux boxes...and the only thing you need on these boxes is an SSH Server! Thanks to Jan-Piet Mens that was one of the person that convinced me to use Ansible and answered questions I had about Ansible. You can watch the demonstration below, and/or look at all the details in the next paragraph. Ansible Ansible is an open-source software that allows administrator to configure and manage many computers over SSH. I won't go in all the details about the installation, just follow the steps documented in the Getting Started Guide. As you can see from this guide, you just need Python and few other libraries and clone Ansible project from Github. So I am expecting that you have Ansible working with your various servers on which you want to deploy Couchbase. Also for this first scripts I am using root on my server to do all the operations. So be sure you have register the root ssh keys to your administration server, from where you are running the Ansible scripts. Create a Couchbase Cluster So before going into the details of the Ansible script it is interesting to explain how you create a Couchbase Cluster. So here are the 5 steps to create and configure a cluster: Install Couchbase on each nodes of the cluster, as documented here. Take one of the node and "initialize" the cluster, using cluster-init command. Add the other nodes to the cluster, using server-add command. Rebalance, using rebalance command. Create a Bucket, using bucket-create command. So the goal now is to create an Ansible Playbook that executes these steps for you. Ansible Playbook for Couchbase The first think you need is to have the list of hosts you want to target, so I have create a hosts file that contains all my server organized in 2 groups: [couchbase-main] vm1.grallandco.com [couchbase-nodes] vm2.grallandco.com vm3.grallandco.com The group [couchbase-main] group is just one of the node that will drive the installation and configuration, as you probably already know, Couchbase does not have any master... All nodes in the cluster are identical. To ease the configuration of the cluster, I have create another file that contains all parameters that must be sent to all the various commands. This file is located in the group_vars/all see the section Splitting Out Host and Group Specific Data in the documentation. # Adminisrator user and password admin_user: Administrator admin_password: password # ram quota for the cluster cluster_ram_quota: 1024 # bucket and replicas bucket_name: ansible bucket_ram_quota: 512 num_replicas: 2 Use this file to configure your cluster. Let's describe the playbook file : - name: Couchbase Installation hosts: all user: root tasks: - name: download Couchbase package get_url: url=http://packages.couchbase.com/releases/2.0.1/couchbase-server-enterprise_x86_64_2.0.1.deb dest=~/. - name: Install dependencies apt: pkg=libssl0.9.8 state=present - name: Install Couchbase .deb file on all machines shell: dpkg -i ~/couchbase-server-enterprise_x86_64_2.0.1.deb As expected, the installation has to be done on all servers as root then we need to execute 3 tasks: Download the product, the get_url command will only download the file if not already present Install the dependencies with the apt command, the state=present allows the system to only install this package if not already present Install Couchbase with a simple shell command. (here I am not checking if Couchbase is already installed) So we have now installed Couchbase on all the nodes. Let's now configure the first node and add the others: - name: Initialize the cluster and add the nodes to the cluster hosts: couchbase-main user: root tasks: - name: Configure main node shell: /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-init-username=${admin_user} --cluster-init-password=${admin_password} --cluster-init-port=8091 --cluster-init-ramsize=${cluster_ram_quota} - name: Create shell script for configuring main node action: template src=couchbase-add-node.j2 dest=/tmp/addnodes.sh mode=750 - name: Launch config script action: shell /tmp/addnodes.sh - name: Rebalance the cluster shell: /opt/couchbase/bin/couchbase-cli rebalance -c 127.0.0.1:8091 -u ${admin_user} -p ${admin_password} - name: create bucket ${bucket_name} with ${num_replicas} replicas shell: /opt/couchbase/bin/couchbase-cli bucket-create -c 127.0.0.1:8091 --bucket=${bucket_name} --bucket-type=couchbase --bucket-port=11211 --bucket-ramsize=${bucket_ram_quota} --bucket-replica=${num_replicas} -u ${admin_user} -p ${admin_password} Now we need to execute specific taks on the "main" server: Initialization of the cluster using the Couchbase CLI, on line 06 and 07 Then the system needs to ask all other server to join the cluster. For this the system needs to get the various IP and for each IP address execute the add-server command with the IP address. As far as I know it is not possible to get the IP address from the main playbook YAML file, so I ask the system to generate a shell script to add each node and execute the script. This is done from the line 09 to 13. To generate the shell script, I use Ansible Template, the template is available in the couchbase-add-node.j2 file. {% for host in groups['couchbase-nodes'] %} /opt/couchbase/bin/couchbase-cli server-add -c 127.0.0.1:8091 -u ${admin_user} -p ${admin_password} --server-add={{ hostvars[host]['ansible_eth0']['ipv4']['address'] }:8091 --server-add-username=${admin_user} --server-add-password=${admin_password} {% endfor %} As you can see this script loop on each server in the [couchbase-nodes] group and use its IP address to add the node to the cluster. Finally the script rebalance the cluster (line 16) and add a new bucket (line 19). You are now ready to execute the playbook using the following command : ./bin/ansible-playbook -i ./couchbase/hosts ./couchbase/couchbase.yml -vv I am adding the -vv parameter to allow you to see more information about what's happening during the execution of the script. This will execute all the commands described in the playbook, and after few seconds you will have a new cluster ready to be used! You can for example open a browser and go to the Couchase Administration Console and check that your cluster is configured as expected. As you can see it is really easy and fast to create a new cluster using Ansible. I have also create a script to uninstall properly the cluster.. just launch ./bin/ansible-playbook -i ./couchbase/hosts ./couchbase/couchbase-uninstall.yml

Hadoop provides a Java native API to support file system operations..

I'd like to show how to instantiate Spring's component with non-default constructor (constructor with parameters) from other Spring driven component because it seems to me that a lot of Java developers doesn't know that this is possible in Spring. First of all, simple annotation based Spring configuration. It's it is scanning target package for components. @Configuration @ComponentScan("sk.lkrnac.blog.componentwithparams") public class SpringContext { } Here is component with non-default constructor. This component needs to be specified as prototype, because we will be creating new instances of it. It also needs to be named (will clarify why below). @Component(SpringConstants.COMPONENT_WITH_PARAMS) @Scope("prototype") public class ComponentWithParams { private String param1; private int param2; public ComponentWithParams(String param1, int param2) { this.param1 = param1; this.param2 = param2; } @Override public String toString() { return "ComponentWithParams [param1=" + param1 + ", param2=" + param2 + "]"; } } Next component instantiates ComponentWithParams. It needs to be aware of Spring's context instance to be able to pass arguments into non-default constructor. getBean(String name, Object... args) method is used for that purpose. First parameter of this method is name of the instantiating component (that is why component needed to be named). Constructor parameters follow. @Component public class CreatorComponent implements ApplicationContextAware{ ApplicationContext context; public void createComponentWithParam(String param1, int param2){ ComponentWithParams component = (ComponentWithParams) context.getBean(SpringConstants.COMPONENT_WITH_PARAMS, param1, param2); System.out.println(component.toString() + " instanciated..."); } @Override public void setApplicationContext(ApplicationContext context) throws BeansException { this.context = context; } } Last code snippet is main class which loads Spring context and runs test. public class Main { public static void main(String[] args) { AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(SpringContext.class); CreatorComponent invoker = context.getBean(CreatorComponent.class); invoker.createComponentWithParam("Fisrt", 1); invoker.createComponentWithParam("Second", 2); invoker.createComponentWithParam("Third", 3); context.close(); } } Console output: ComponentWithParams instanciated... ComponentWithParams instanciated... ComponentWithParams instanciated... And that's it. You can download souce code on Github

In this tutorial we are going to investigate ways you can tell what events a certain element has bound to it. This is really useful when you need to debug the on method to see how it is working. There are a number of ways to bind events to an element, there is bind(), live() and delegate(). Bind - Used to attach events to current elements on the page. If new elements are added with the same selector the event will not be added to the element. Live - This is used to attach an events to all elements on the page and future elements with the same selector. Delegate - This is used to attach events to all elements on the page and future elements based on a specific root element. As of jQuery 1.7 the on method will give you all the functionality that you need to bound events to elements. It will allow you to add an event to all elements on the page and all future elements. $('.element').on('click', function(){ // stuff }); $('.element').on('hover', function(){ // stuff }); $('.element').on('mouseenter', function(){ // stuff }); $('form').on('submit', function(){ // stuff }); When you are assigning events to future elements you might want to find out what events are currently assigned to these elements. Using developer tools in your browser you can see what events are currently bound to the element by using these 2 methods. First you can display the events by using the console, the second method is to view the events in the event listener window in the developer tools. Display Events In Console In your browser if you press F12 a new window called developer tools will open, this allows you to view the entire HTML DOM in more detail, this also has a console feature which will allow you to type in any Javascript to run at this current time on the page. You can use the console to display a list of events currently assigned to an element by using the following code. $._data( $('.element')[0], 'events' ); This will display all the events that are currently assigned to the element. Event Listener Window The other option to use to view what events are currently assigned to an element is to use the event listener window in your browser's developer tools. All you have to do is press F12 to open the developer tools, select the element in the HTML DOM that you want to investigate, on the right side of the window you will see an option called Event Listeners. When you expand this menu you will see all the current events assigned to the element including all click events bound from jQuery.

avro has a little-known gem of a feature which allows you to control which fields in an avro record are used for partitioning , sorting and grouping in mapreduce. the following figure gives a quick refresher as to what these terms mean. oh, and don’t take the placement of the “sorting” literally - sorting actually occurs on both the map and reduce side - but it’s always performed in the context of a specific partition (i.e. for a specific reducer). by default all the fields in an avro map output key are used for partitioning, sorting and grouping in mapreduce. let’s walk through an example and see how this works. you’ll begin with a simple schema github source : {"type": "record", "name": "com.alexholmes.avro.weathernoignore", "doc": "a weather reading.", "fields": [ {"name": "station", "type": "string"}, {"name": "time", "type": "long"}, {"name": "temp", "type": "int"}, {"name": "counter", "type": "int", "default": 0} ] } we’re going to see what happens when we run this code against a small sample data set, which we’ll generate using avro code github source : file input = tmpfolder.newfile("input.txt"); avrofiles.createfile(input, weathernoignore.schema$, arrays.aslist( weathernoignore.newbuilder().setstation("sfo").settime(1).settemp(3).build(), weathernoignore.newbuilder().setstation("iad").settime(1).settemp(1).build(), weathernoignore.newbuilder().setstation("sfo").settime(2).settemp(1).build(), weathernoignore.newbuilder().setstation("sfo").settime(1).settemp(2).build(), weathernoignore.newbuilder().setstation("sfo").settime(1).settemp(1).build() ).toarray()); to understand how avro is partitioning, sorting and grouping the data, we’ll write an identity mapper and reducer, with a small enhancement to the reducer to increment the counter field for each record we see in an individual reducer instance github source : package com.alexholmes.avro.sort.basic; import com.alexholmes.avro.weathernoignore; import org.apache.avro.mapred.avrokey; import org.apache.avro.mapred.avrovalue; import org.apache.avro.mapreduce.avrojob; import org.apache.avro.mapreduce.avrokeyinputformat; import org.apache.avro.mapreduce.avrokeyoutputformat; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.nullwritable; import org.apache.hadoop.mapreduce.job; import org.apache.hadoop.mapreduce.mapper; import org.apache.hadoop.mapreduce.reducer; import org.apache.hadoop.mapreduce.lib.input.fileinputformat; import org.apache.hadoop.mapreduce.lib.output.fileoutputformat; import java.io.ioexception; public class avrosort { private static class sortmapper extends mapper, nullwritable, avrokey, avrovalue> { @override protected void map(avrokey key, nullwritable value, context context) throws ioexception, interruptedexception { context.write(key, new avrovalue(key.datum())); } } private static class sortreducer extends reducer, avrovalue, avrokey, nullwritable> { @override protected void reduce(avrokey key, iterable> values, context context) throws ioexception, interruptedexception { int counter = 1; for (avrovalue weathernoignore : values) { weathernoignore.datum().setcounter(counter++); context.write(new avrokey(weathernoignore.datum()), nullwritable.get()); } } } public boolean runmapreduce(final job job, path inputpath, path outputpath) throws exception { fileinputformat.setinputpaths(job, inputpath); job.setinputformatclass(avrokeyinputformat.class); avrojob.setinputkeyschema(job, weathernoignore.schema$); job.setmapperclass(sortmapper.class); avrojob.setmapoutputkeyschema(job, weathernoignore.schema$); avrojob.setmapoutputvalueschema(job, weathernoignore.schema$); job.setreducerclass(sortreducer.class); avrojob.setoutputkeyschema(job, weathernoignore.schema$); job.setoutputformatclass(avrokeyoutputformat.class); fileoutputformat.setoutputpath(job, outputpath); return job.waitforcompletion(true); } } if you look at the output of the job below, you’ll see that the output is sorted across all the fields, and that the sorting is in field ordinal order. what this means is that when mapreduce is sorting these records, it compares the station field first, then the time field second, and so on according to the ordering of the fields in the avro schema. this is pretty much what you’d expect if you write your own complex writable type, and your comparator compared all the fields in order. {"station": "iad", "time": 1, "temp": 1, "counter": 1} {"station": "sfo", "time": 1, "temp": 1, "counter": 1} {"station": "sfo", "time": 1, "temp": 2, "counter": 1} {"station": "sfo", "time": 1, "temp": 3, "counter": 1} {"station": "sfo", "time": 2, "temp": 1, "counter": 1} oh, and before we move on notice that the value for the counter field is always 1 , meaning that each reducer was only fed a single key/vaue pair, which makes sense since our identity mapper only emitted a single value for each key, the keys are unique, and the mapreduce partitioner, sorter and grouper were using all the fields in the record. excluding fields for sorting avro gives us the ability to indicate that specific fields should be ignored when performing ordering functions. in mapreduce these fields are ignored for sorting/partitioning and grouping in mapreduce, which basically means that we have the ability to perform secondary sorting. let’s examine the following schema github source : {"type": "record", "name": "com.alexholmes.avro.weather", "doc": "a weather reading.", "fields": [ {"name": "station", "type": "string"}, {"name": "time", "type": "long"}, {"name": "temp", "type": "int", "order": "ignore"}, {"name": "counter", "type": "int", "order": "ignore", "default": 0} ] } it’s pretty much identical to the first schema, the only difference being that the last two fields are flagged as being “ignored” for sorting/partitioning/grouping. let’s run the same (other than modified to work with the different schema) mapreduce code github source as above against this new schema and examine the outputs. {"station": "iad", "time": 1, "temp": 1, "counter": 1} {"station": "sfo", "time": 1, "temp": 3, "counter": 1} {"station": "sfo", "time": 1, "temp": 2, "counter": 2} {"station": "sfo", "time": 1, "temp": 1, "counter": 3} {"station": "sfo", "time": 2, "temp": 1, "counter": 1} there are a couple of notable differences between this output, and the output from the previous schema which didn’t have any ignored fields. first, it’s clear that the temp field isn’t being used in the sorting, which makes sense since we specified that it should be ignored in the schema. however, more interestingly, note the value of the counter field. all records that had identical station and time values went to the same reducer invocation, evidenced by the increasing value of counter . this is essentially secondary sort! now, all of this greatness isn’t without some limitations: you can’t support two mapreduce jobs that use the same avro key, but have different sorting/partitioning/grouping requirements. although it’s conceivable that you could create a new instance of the avro schema and set the ignored flags for these fields yourself. the partitioner, sorter and grouping functions in mapreduce all work off of the same fields (i.e. they all ignore fields that set as ignored in the schema). this means that your options for secondary sorting are limited. for example, you wouldn’t be able to partition all stations to the same reducer, and then group by station and time. ordering uses a field’s ordinal position to determine its order within the overall set of fields to be ordered. in other words, in a two-field record, the first field is always compared before the second. there’s no way to change this behavior other than flipping the order of the fields in the record. having said all of that - the “ignoring fields” feature for sorting is pretty awesome, and something that will no doubt come in handy in my future mapreduce work.

Question: Do you need to debug maven plugin: fornax-oaw-m2-plugin ? If so, go on reading. You can find the basic info at their home page: usage - http://fornax.itemis.de/confluence/display/fornax/Usage+%28TOM%29 advanced configuration - http://fornax.itemis.de/confluence/display/fornax/Configuration+%28TOM%29 However remote debugging topic is not covered there. The problem with maven plugins might be, that sometimes it's not enough to debug maven run itself (as described in my previous post). However merging information from the previous links with common remote debugging options I came to solution that simply works for me: org.fornax.toolsupport fornax-oaw-m2-plugin 2.1.1 pom -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8100 -Xnoagent -Djava.compiler=NONE generate-sources run-workflow Please note that port to remotely connect to with debugger is 8100 in my sample. Feel free to adapt it based on your setup. After using the configuration in your run, you should get to point where plugin run gets suspended and waits for remote debugger to connect (to find out how to remotelly debug, see some existing tutorial, like this one: https://dzone.com/articles/how-debug-remote-java-applicat). That's it. Enjoy your debugging session. This post is from pb's blog.

In my previous post I showed different ways in which we can parse XML document in Java. You must have noticed the code being too much verbose. Other JVM languages like Groovy, Scala provide much better support for parsing XML documents. In this post I give you the code to parse the XML document in Groovy and one can compare the ease with which we can parse XML documents. I make use of XmlSlurper API in Groovy which loads the complete XML into a tree and this tree can be then navigated using Groovy’s version of XPath called GPath. The XML document I am using is the same one used here and also the intent is to parse the XML and create a list of Employee object. class XmlParserDemo { static void main(args){ def empList = new ArrayList(); def emp; def employees = new XmlSlurper().parse(ClassLoader. getSystemResourceAsStream("xml/employee.xml")); employees.employee.each{ node -> emp = new Employee(); emp.firstName = node.firstName emp.lastName = node.lastName emp.id = node.@id emp.location = node.location empList.add(emp) } empList.each{ empT -> println(empT)} } } class Employee{ String firstName String lastName String id String location @Override public String toString(){ return "${firstName} ${lastName}(${id}) in ${location}" } } The output is: Rakesh Mishra(111) in Bangalore John Davis(112) in Chennai Rajesh Sharma(113) in Pune And the XML is present in the “xml” package and is available in the classpath of the application. I am using the ClassLoader to load the XML resource.

You can easily tell Maven to include the version of the artifact and its Git/SVN/… revision in the JAR manifest file and then access that information at runtime via getClass().getPackage.getImplementationVersion(). (All credit goes to Markus Krüger and other colleagues.) Include Maven artifact version in the manifest (Note: You will actually not want to use it, if you also want to include a SCM revision; see below.) pom.xml: ... org.apache.maven.plugins maven-jar-plugin ... true true ... ... The resulting MANIFEST.MF of the JAR file will then include the following entries, with values from the indicated properties: Built-By: ${user.name} Build-Jdk: ${java.version} Specification-Title: ${project.name} Specification-Version: ${project.version} Specification-Vendor: ${project.organization.name Implementation-Title: ${project.name} Implementation-Version: ${project.version} Implementation-Vendor-Id: ${project.groupId} Implementation-Vendor: ${project.organization.name} (Specification-Vendor and Implementation-Vendor come from the POM’s organization/name.) Include SCM revision For this you can either use the Build Number Maven plugin that produces the property ${buildNumber}, or retrieve it from environment variables passed by Jenkinsor Hudson (SVN_REVISION for Subversion, GIT_COMMIT for Git). For git alone, you could also use the maven-git-commit-id-plugin that can either replace strings such as ${git.commit.id} in existing resource files (using maven’s resource filtering, which you must enable) with the actual values or output all of them into a git.properties file. Let’s use the buildnumber-maven-plugin and create the manifest entries explicitely, containing the build number (i.e. revision) org.codehaus.mojo buildnumber-maven-plugin 1.2 validate create false false org.apache.maven.plugins maven-jar-plugin 2.4 ${project.name} ${project.version} ${buildNumber} ... Accessing the version & revision As mentioned above, you can access the manifest entries from your code via getClass().getPackage.getImplementationVersion() andgetClass().getPackage.getImplementationTitle(). References SO: How to get Maven Artifact version at runtime? Maven Archiver documentation

Have you ever tried to update specific fields of an entity using Entity Framework? If yes, then I can assume that System.Data.Entity.Validation.DbEntityValidationResult – Validation failed for one or more entities. See ‘EntityValidationErrors’ property for more details. Exception is not something new to you. While working recently on a project for which I was using Entity Framework 5.x CF (the part with code first is not relevant) I bumped into an interesting piece of code. The main idea behind it is to perform updates for a single property and ignore the rest even if they don't respect the constraints defined in the model. Please note that I have found PartialEntityValidation class somewhere on the internet and unfortunately I do not remember where, I was not able to find it again. The only changes made by me are related to code formatting. I will present the code first and explain the logic behind it afterwards. using System.Collections.Generic; using System.Data.Entity.Infrastructure; using System.Data.Entity.Validation; namespace MyProject.Data.Contexts.Concrete { public class PartialEntityValidation { private readonly IDictionary _store; public PartialEntityValidation() { _store = new Dictionary(); } public void Register(DbEntityEntry entry, string[] properties) { if (_store.ContainsKey(entry)) _store[entry] = properties; else _store.Add(entry, properties); } public void Unregister(DbEntityEntry entry) { _store.Remove(entry); } public bool IsResponsibleFor(DbEntityEntry entry) { return _store.ContainsKey(entry); } public void Validate(DbEntityValidationResult validationResult) { var entry = validationResult.Entry; foreach (var property in _store[entry]) { var validationErrors = entry.Property(property).GetValidationErrors(); foreach (var validationError in validationErrors) validationResult.ValidationErrors.Add(validationError); } } } } PartialEntityValidation class has a simple constructor which initializes a new dictionary store where the key of it is an DbEntityEntry and the value (currently) an empty string array. The string array will hold later on the properties which should be validated. The class has four methods: Register, Unregister, IsResponsibleFor and Validate. Now let me explain a bit what each method does. Register adds a new DbEntityEntry with the properties that have to be validated to the existing store. It basically registers a new entity entry for validation. It is quite obvious what Unregister method does; it removes an entity entry from the existing store. IsResponsibleFor method determines if an entity entry is present in the existing store and if it will be part of the validation process. The last method is Validate which accepts a single parameter of DbEntityValidationResult type, from which it takes the entity entry associated with the validation result and validates each property of the entity entry present in the store. If there are any validation errors those are added on ValidationErrors collection of the validation result. Now it is time to see how to use PartialEntityValidation class. In My case I have exposed it as a public read only property of the context and had ValidateEntity method overridden at context level. using System; using System.Collections.Generic; using System.Configuration; using System.Data; using System.Data.Entity; using System.Data.Entity.Infrastructure; using System.Data.Entity.Validation; namespace MyProject.Data.Contexts.Concrete { public class MyContext : DbContext { private readonly PartialEntityValidation _partialEntityValidation; static MyContext() { Database.SetInitializer(null); } protected MyContext() : base("MyDb") { _partialEntityValidation = new PartialEntityValidation(); Configuration.LazyLoadingEnabled = false; } public PartialEntityValidation PartialValidation { get { return _partialEntityValidation; } } protected override DbEntityValidationResult ValidateEntity(DbEntityEntry entityEntry, IDictionary items) { if (PartialValidation.IsResponsibleFor(entityEntry)) { var validationResult = new DbEntityValidationResult(entityEntry, new List()); PartialValidation.Validate(validationResult); return validationResult; } return base.ValidateEntity(entityEntry, items); } } } I am sure that I do not have to explain the logic behind ValidateEntity method from the db context. The final piece is the repository where the partial update is invoked. using System; using System.Linq; using System.Linq.Expressions; namespace MyProject.Data.Repositories.Concrete { public class MyRepository { private readonly MyContext _context; public MyRepository(MyContext context) { _context = context; } public void DeleteProduct(Guid productId) { var product = new Product { ProductId = productId, IsDeleted = true }; _context.PartialValidation.Register(_context.Entry(product), new[] { "IsDeleted" }); Update(workspace, p => p.IsDeleted); _context.SaveChanges(); ((IObjectContextAdapter)_context).ObjectContext.Detach(product); } public void Update(T entity, Expression> property) where T : class { var entry = _context.Entry(entity); _context.Set().Attach(entity); entry.Property(property).IsModified = true; } } } Please take into consideration that this is a demo repository which doesn't do much at all except to present how a partial update is performed. The repository has two important methods. First one is DeleteProduct which performs a soft delete by updating "IsDeleted" property of product entity to true. This is achieved by creating a new product object of which primary key is "ProductId" with "IsDeleted" property set to true. The next step is to register the entity entry along with the "IsDeleted" property (this is the only property we want to have it updated, all other properties of Product entity will be ignored) into PartialValidation store. Update method is called which attaches the entity entry to the context and marks it as modified in order to have it updated by entity framework. That is all, I really hope that you got the idea behind this even if I did not provided an actual sample application for download.

In this blog post, I will present a simple tutorial on uploading a large file to Amazon S3 as fast as the network supports. Amazon S3 is clustered storage service of Amazon. It is designed to make web-scale computing easier. Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers. For using Amazon services, you'll need your AWS access key identifiers, which AWS assigned you when you created your AWS account. The following are the AWS access key identifiers: Access Key ID (a 20-character, alphanumeric sequence) For example: 022QF06E7MXBSH9DHM02 Secret Access Key (a 40-character sequence) For example: kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct Caution Your Secret Access Key is a secret, which only you and AWS should know. It is important to keep it confidential to protect your account. Store it securely in a safe place. Never include it in your requests to AWS, and never e-mail it to anyone. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your Secret Access Key. The Access Key ID is associated with your AWS account. You include it in AWS service requests to identify yourself as the sender of the request. The Access Key ID is not a secret, and anyone could use your Access Key ID in requests to AWS. To provide proof that you truly are the sender of the request, you also include a digital signature calculated using your Secret Access Key. The sample code handles this for you. Your Access Key ID and Secret Access Key are displayed to you when you create your AWS account. They are not e-mailed to you. If you need to see them again, you can view them at any time from your AWS account. To get your AWS access key identifiers Go to the Amazon Web Services web site at http://aws.amazon.com. Point to Your Account and click Security Credentials. Log in to your AWS account. The Security Credentials page is displayed. Your Access Key ID is displayed in the Access Identifiers section of the page. To display your Secret Access Key, click Show in the Secret Access Key column. You can use your Amazon keys from a properties file in your application. Here is a sample for properties file containing Amazon keys: # Fill in your AWS Access Key ID and Secret Access Key # http://aws.amazon.com/security-credentials accessKey = secretKey = Here is sample AmazonUtil class for getting AWS Credentials from properties file. public class AmazonUtil { private static final Logger logger = LogUtil.getLogger(); private static final String AWS_CREDENTIALS_CONFIG_FILE_PATH = ConfigUtil.CONFIG_DIRECTORY_PATH + File.separator + "aws-credentials.properties"; private static AWSCredentials awsCredentials; static { init(); } private AmazonUtil() { } private static void init() { try { awsCredentials = new PropertiesCredentials(IOUtil.getResourceAsStream(AWS_CREDENTIALS_CONFIG_FILE_PATH)); } catch (IOException e) { logger.error("Unable to initialize AWS Credentials from " + AWS_CREDENTIALS_CONFIG_FILE_PATH); } } public static AWSCredentials getAwsCredentials() { return awsCredentials; } } Amazon S3 has Multipart Upload service which allows faster, more flexible uploads into Amazon S3. Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size. For more information on Multipart Upload, review the Amazon S3 Developer Guide In this tutorial, my sample application uploads each file parts to Amazon S3 with different threads for using network throughput as possible as much. Each file part is associated with a thread and each thread uploads its associated part with Amazon S3 API. Figure 1. Amazon S3 Parallel Multi-Part File Upload Mechanism Amazon S3 API suppots MultiPart File Upload in this way: 1. Send a MultipartUploadRequest to Amazon. 2. Get a response containing a unique id for this upload operation. 3. For i in ${partCount} 3.1. Calculate size and offset of split-i in whole file. 3.2. Build a UploadPartRequest with file offset, size of current split and unique upload id. 3.3. Give this request to a thread and starts upload by running thread. 3.3.1. Send associated UploadPartRequest to Amazon. 3.3.2. Get response after successful upload and save ETag property of response. 4. Wait all threads to terminate 5. Get ETags (ETag is an identifier for successfully completed uploads) of all terminated threads. 6. Send a CompleteMultipartUploadRequest to Amazon with unique upload id and all ETags. So Amazon joins all file parts as target objects. Here is implementation: public class AmazonS3Util { private static final Logger logger = LogUtil.getLogger(); public static final long DEFAULT_FILE_PART_SIZE = 5 * 1024 * 1024; // 5MB public static long FILE_PART_SIZE = DEFAULT_FILE_PART_SIZE; private static AmazonS3 s3Client; private static TransferManager transferManager; static { init(); } private AmazonS3Util() { } private static void init() { // ... s3Client = new AmazonS3Client(AmazonUtil.getAwsCredentials()); transferManager = new TransferManager(AmazonUtil.getAwsCredentials()); } // ... public static void putObjectAsMultiPart(String bucketName, File file) { putObjectAsMultiPart(bucketName, file, FILE_PART_SIZE); } public static void putObjectAsMultiPart(String bucketName, File file, long partSize) { List partETags = new ArrayList(); List uploaders = new ArrayList(); // Step 1: Initialize. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, file.getName()); InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest); long contentLength = file.length(); try { // Step 2: Upload parts. long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { // Last part can be less than part size. Adjust part size. partSize = Math.min(partSize, (contentLength - filePosition)); // Create request to upload a part. UploadPartRequest uploadRequest = new UploadPartRequest(). withBucketName(bucketName).withKey(file.getName()). withUploadId(initResponse.getUploadId()).withPartNumber(i). withFileOffset(filePosition). withFile(file). withPartSize(partSize); uploadRequest.setProgressListener(new UploadProgressListener(file, i, partSize)); // Upload part and add response to our list. MultiPartFileUploader uploader = new MultiPartFileUploader(uploadRequest); uploaders.add(uploader); uploader.upload(); filePosition += partSize; } for (MultiPartFileUploader uploader : uploaders) { uploader.join(); partETags.add(uploader.getPartETag()); } // Step 3: complete. CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, file.getName(), initResponse.getUploadId(), partETags); s3Client.completeMultipartUpload(compRequest); } catch (Throwable t) { logger.error("Unable to put object as multipart to Amazon S3 for file " + file.getName(), t); s3Client.abortMultipartUpload( new AbortMultipartUploadRequest( bucketName, file.getName(), initResponse.getUploadId())); } } // ... private static class UploadProgressListener implements ProgressListener { File file; int partNo; long partLength; UploadProgressListener(File file) { this.file = file; } @SuppressWarnings("unused") UploadProgressListener(File file, int partNo) { this(file, partNo, 0); } UploadProgressListener(File file, int partNo, long partLength) { this.file = file; this.partNo = partNo; this.partLength = partLength; } @Override public void progressChanged(ProgressEvent progressEvent) { switch (progressEvent.getEventCode()) { case ProgressEvent.STARTED_EVENT_CODE: logger.info("Upload started for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.COMPLETED_EVENT_CODE: logger.info("Upload completed for file " + "\"" + file.getName() + "\"" + ", " + file.length() + " bytes data has been transferred"); break; case ProgressEvent.FAILED_EVENT_CODE: logger.info("Upload failed for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.CANCELED_EVENT_CODE: logger.info("Upload cancelled for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.PART_STARTED_EVENT_CODE: logger.info("Upload started at " + partNo + ". part for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.PART_COMPLETED_EVENT_CODE: logger.info("Upload completed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + (partLength > 0 ? partLength : progressEvent.getBytesTransfered()) + " bytes data has been transferred"); break; case ProgressEvent.PART_FAILED_EVENT_CODE: logger.info("Upload failed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; } } } private static class MultiPartFileUploader extends Thread { private UploadPartRequest uploadRequest; private PartETag partETag; MultiPartFileUploader(UploadPartRequest uploadRequest) { this.s3Client = s3Client; this.uploadRequest = uploadRequest; } @Override public void run() { partETag = s3Client.uploadPart(uploadRequest).getPartETag(); } private PartETag getPartETag() { return partETag; } private void upload() { start(); } } }

I happen to read through a chapter on XML parsing and building APIs in Java. And I tried out the different parsers on a sample XML. Then I thought of sharing it on my blog so that I can have a reference to the code as well as a reference for anyone reading this. In this post I parse the same XML in different parsers to perform the same operation of populating the XML content into objects and then adding the objects to a list. The sample XML considered in the examples is: Rakesh Mishra Bangalore John Davis Chennai Rajesh Sharma Pune And the obejct into which the XML content is to be extracted is defined as below: class Employee{ String id; String firstName; String lastName; String location; @Override public String toString() { return firstName+" "+lastName+"("+id+")"+location; } } There are 3 main parsers for which I have given sample code: DOM Parser SAX Parser StAX Parser Using DOM Parser I am making use of the DOM parser implementation that comes with the JDK and in my example I am using JDK 7. The DOM Parser loads the complete XML content into a Tree structure. And we iterate through the Node and NodeList to get the content of the XML. The code for XML parsing using DOM parser is given below. public class DOMParserDemo { public static void main(String[] args) throws Exception { //Get the DOM Builder Factory DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); //Get the DOM Builder DocumentBuilder builder = factory.newDocumentBuilder(); //Load and Parse the XML document //document contains the complete XML as a Tree. Document document = builder.parse( ClassLoader.getSystemResourceAsStream("xml/employee.xml")); List empList = new ArrayList<>(); //Iterating through the nodes and extracting the data. NodeList nodeList = document.getDocumentElement().getChildNodes(); for (int i = 0; i < nodeList.getLength(); i++) { //We have encountered an tag. Node node = nodeList.item(i); if (node instanceof Element) { Employee emp = new Employee(); emp.id = node.getAttributes(). getNamedItem("id").getNodeValue(); NodeList childNodes = node.getChildNodes(); for (int j = 0; j < childNodes.getLength(); j++) { Node cNode = childNodes.item(j); //Identifying the child tag of employee encountered. if (cNode instanceof Element) { String content = cNode.getLastChild(). getTextContent().trim(); switch (cNode.getNodeName()) { case "firstName": emp.firstName = content; break; case "lastName": emp.lastName = content; break; case "location": emp.location = content; break; } } } empList.add(emp); } } //Printing the Employee list populated. for (Employee emp : empList) { System.out.println(emp); } } } class Employee{ String id; String firstName; String lastName; String location; @Override public String toString() { return firstName+" "+lastName+"("+id+")"+location; } } The output for the above will be: Rakesh Mishra(111)Bangalore John Davis(112)Chennai Rajesh Sharma(113)Pune Using SAX Parser SAX Parser is different from the DOM Parser where SAX parser doesn’t load the complete XML into the memory, instead it parses the XML line by line triggering different events as and when it encounters different elements like: opening tag, closing tag, character data, comments and so on. This is the reason why SAX Parser is called an event based parser. Along with the XML source file, we also register a handler which extends the DefaultHandler class. The DefaultHandler class provides different callbacks out of which we would be interested in: startElement() – triggers this event when the start of the tag is encountered. endElement() – triggers this event when the end of the tag is encountered. characters() – triggers this event when it encounters some text data. The code for parsing the XML using SAX Parser is given below: import java.util.ArrayList; import java.util.List; import javax.xml.parsers.SAXParser; import javax.xml.parsers.SAXParserFactory; import org.xml.sax.Attributes; import org.xml.sax.SAXException; import org.xml.sax.helpers.DefaultHandler; public class SAXParserDemo { public static void main(String[] args) throws Exception { SAXParserFactory parserFactor = SAXParserFactory.newInstance(); SAXParser parser = parserFactor.newSAXParser(); SAXHandler handler = new SAXHandler(); parser.parse(ClassLoader.getSystemResourceAsStream("xml/employee.xml"), handler); //Printing the list of employees obtained from XML for ( Employee emp : handler.empList){ System.out.println(emp); } } } /** * The Handler for SAX Events. */ class SAXHandler extends DefaultHandler { List empList = new ArrayList<>(); Employee emp = null; String content = null; @Override //Triggered when the start of tag is found. public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException { switch(qName){ //Create a new Employee object when the start tag is found case "employee": emp = new Employee(); emp.id = attributes.getValue("id"); break; } } @Override public void endElement(String uri, String localName, String qName) throws SAXException { switch(qName){ //Add the employee to list once end tag is found case "employee": empList.add(emp); break; //For all other end tags the employee has to be updated. case "firstName": emp.firstName = content; break; case "lastName": emp.lastName = content; break; case "location": emp.location = content; break; } } @Override public void characters(char[] ch, int start, int length) throws SAXException { content = String.copyValueOf(ch, start, length).trim(); } } class Employee { String id; String firstName; String lastName; String location; @Override public String toString() { return firstName + " " + lastName + "(" + id + ")" + location; } } The output for the above would be: Rakesh Mishra(111)Bangalore John Davis(112)Chennai Rajesh Sharma(113)Pune Using StAX Parser StAX stands for Streaming API for XML and StAX Parser is different from DOM in the same way SAX Parser is. StAX parser is also in a subtle way different from SAX parser. The SAX Parser pushes the data but StAX parser pulls the required data from the XML. The StAX parser maintains a cursor at the current position in the document allows to extract the content available at the cursor whereas SAX parser issues events as and when certain data is encountered. XMLInputFactory and XMLStreamReader are the two class which can be used to load an XML file. And as we read through the XML file using XMLStreamReader, events are generated in the form of integer values and these are then compared with the constants in XMLStreamConstants. The below code shows how to parse XML using StAX parser: import java.util.ArrayList; import java.util.List; import javax.xml.stream.XMLInputFactory; import javax.xml.stream.XMLStreamConstants; import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLStreamReader; public class StaxParserDemo { public static void main(String[] args) throws XMLStreamException { List empList = null; Employee currEmp = null; String tagContent = null; XMLInputFactory factory = XMLInputFactory.newInstance(); XMLStreamReader reader = factory.createXMLStreamReader( ClassLoader.getSystemResourceAsStream("xml/employee.xml")); while(reader.hasNext()){ int event = reader.next(); switch(event){ case XMLStreamConstants.START_ELEMENT: if ("employee".equals(reader.getLocalName())){ currEmp = new Employee(); currEmp.id = reader.getAttributeValue(0); } if("employees".equals(reader.getLocalName())){ empList = new ArrayList<>(); } break; case XMLStreamConstants.CHARACTERS: tagContent = reader.getText().trim(); break; case XMLStreamConstants.END_ELEMENT: switch(reader.getLocalName()){ case "employee": empList.add(currEmp); break; case "firstName": currEmp.firstName = tagContent; break; case "lastName": currEmp.lastName = tagContent; break; case "location": currEmp.location = tagContent; break; } break; case XMLStreamConstants.START_DOCUMENT: empList = new ArrayList<>(); break; } } //Print the employee list populated from XML for ( Employee emp : empList){ System.out.println(emp); } } } class Employee{ String id; String firstName; String lastName; String location; @Override public String toString(){ return firstName+" "+lastName+"("+id+") "+location; } } The output for the above is: Rakesh Mishra(111) Bangalore John Davis(112) Chennai Rajesh Sharma(113) Pune With this I have covered parsing the same XML document and performing the same task of populating the list of Employee objects using all the three parsers namely: DOM Parser SAX Parser StAX Parser

Avro has the ability to generate Java code from Avro schema, IDL and protocol files. Avro also has a plugin which allows you to generate these Java sources directly from Maven, which is a good idea as it avoids issues that can arise if your schema/protocol files stray from the checked-in code generated equivalents. Today I created a simple GitHub project called avro-maven because I had to fiddle a bit to get Avro and Maven to play nice. The GitHub project is self-contained and also has a README which goes over the basics. In this post I’ll go over how to use Maven to generate code for schema, IDL and protocol files. pom.xml updates to support the Avro plugin Avro schema files only define types, whereas IDL and protocol files model types as well as RPC semantics such as messages. The only difference between IDL and protocol files is that IDL files are Avro’s DSL for specifying RPC, versus protocol files are the same in JSON form. Each type of file has an entry that can be used in the goals element as can be seen below. All three can be used together, or if you only have schema files you can safely remove the protocol and idl-protocol entries (and vice-versa). org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema protocol idl-protocol ... org.apache.avro avro ${avro.version} org.apache.avro avro-maven-plugin ${avro.version} org.apache.avro avro-compiler ${avro.version} org.apache.avro avro-ipc ${avro.version} By default the plugin assumes that your Avro sources are located in ${basedir}/src/main/avro, and that you want your generated sources to be written to ${project.build.directory}/generated-sources/avro, where ${project.build.directory} is typically the target directory. Keep reading if you want to change any of these settings. Avro configurables Luckily Avro’s Maven plugin offers the ability to customize various code generation settings. The following table shows the configurables that can be used for any of the schema, IDL and protocol code generators. Configurable Default value Description sourceDirectory ${basedir}/src/main/avro The Avro source directory for schema, protocol and IDL files. outputDirectory ${project.build.directory}/generated-sources/avro The directory where Avro writes code-generated sources. testSourceDirectory ${basedir}/src/test/avro The input directory containing any Avro files used in testing. testOutputDirectory ${project.build.directory}/generated-test-sources/avro The output directory where Avro writes code-generated files for your testing purposes. fieldVisibility PUBLIC_DEPRECATED Determines the accessibility of fields (e.g. whether they are public or private). Must be one of PUBLIC, PUBLIC_DEPRECATED or PRIVATE. PUBLIC_DEPRECATED merely adds a deprecated annotation to each field, e.g. "@Deprecated public long time". In addition, the includes and testIncludes configurables can also be used to specify alternative file extensions to the defaults, which are **/*.avsc, **/*.avpr and **/*.avdl for schema, protocol and IDL files respectively. Let’s look at an example of how we can specify all of these options for schema compilation. org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema ${project.basedir}/src/main/myavro/ ${project.basedir}/src/main/java/ ${project.basedir}/src/main/myavro/ ${project.basedir}/src/test/java/ PRIVATE **/*.avro **/*.test As a reminder everything covered in this blog article can be seen in action in the GitHub repo at https://github.com/alexholmes/avro-maven.

In our previous article we have explained on how to protect the data while it is in transit through Transport Layer Security (TLS)/Secured Socket Layer (SSL). Now let us try to understand how to apply security mechanism for a JEE 6 based web application using LDAP server for authentication. Objective: • Configure a LDAP realm in the JEE Application Server • Apply JEE security to a sample web application. Products used: IDE: Netbeans 7.2 Java Development Kit (JDK): Version 6 Glassfish server: 3.1 Authentication Mechanism: Form Based authentication Authentication server: LDAP OpenDS v2.2 Apply JEE security to the sample web application: The JEE web applications can be secured either through Declarative security or Programmatic security. Declarative security can be implemented in JEE applications by using annotations or through deployment descriptor. This type of security mechanism is used when the roles and authentication process is simple, when it can make use of existing security providers (even external like LDAP, Kerberos). Programmatic security provides additional security mechanism when declarative security is not sufficient for the application in context. It is used when we require custom made security and when rich set of roles, authentication is required. Configure Realm in the Glassfish Application Server Before we configure a realm in the Glassfish Application server you will need to install and configure an LDAP server which we will be using for our project. You can get the complete instructions in the following article: “How to install and configure LDAP server”. Once the installation is successful start your Glassfish server and go to the admin console. Create a new LDAP Realm. Create new LDAP Realm Add the configuration settings as per the configurations set up done for the LDAP server. Glassfish Web App LDAP Realm JAAS Context – identifier which will be used in the application module to connect with the LDAP server. (e.g. ldapRealm) Directory – LDAP server URL path (e.g. ldap://localhost:389) Base DN: Distinguished name in the LDAP directory identifying the location of the user data. Applying JEE security to the web application Create a sample web application as per the following structure: SampleWebApp Directory Form based authentication mechanism will be used for authentication of the users. JEE Login and Authentication Let us explain the whole process with help of above diagram and the code. Set up a sample web application in Netbeans IDE. SampleWebApp in Netbeans IDE SampleWebApp Configuration Step 1: As explained in the above diagram a client browser tries to request for a protected resource from the websitehttp://{samplewebsite.com}/{contextroot}/index.jsp. The webserver goes into the web configuration file and figures out that the requested resource is protected. web.xml Code SecurityConstraint Secured resources /* GeneralUser Administrator NONE Step 2: The webserver presents the Login.jsp as a part of the Form based authentication mechanism to the client. These configurations are checked from the web configuration file. web.xml FORM ldapRealm /Login.jsp /LoginError.jsp Step 3: The client submits the login form to the web server. When the servers finds that the form action is “j_security_check” it processes the request to authenticate the client’s credential. The jsp form must contain the login elements j_username and j_password which will allow the web server to invoke the login authentication mechanism. Login.jsp username: password: While processing the request the webserver will send the authentication request to the LDAP server since LDAP realm is used in the login-config. The LDAP server will authenticate the user based on the username and password stored in the LDAP repository. Step 4: If the authentication is successful the secured resource (in this case index.jsp) is returned to the client and the container uses a session id to identify a login session for the client. The container maintains the login session with a cookie containing the session-id. The server sends this cookie back to the client, and as long as the client is able to show this cookie for subsequent requests, then the container easily recognize the client and hence maintains the session for this client. Step 5: Only if the authentication is unsuccessful the user will be redirected to the LoginError.jsp as per the configuration in the web.xml. /LoginError.jsp This shows how to apply form based security authentication to a sample web application. Now let us get a brief look on the secured resource which is used for this project. In this project the secured resource is index.jsp which accepts a username and forwards the request to LoginServlet. Login servlet dispatches the request to Success.jsp which then prints the username to the client. index.jsp Please type your name LoginServlet.java protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try { RequestDispatcher requestDispatcher = getServletConfig().getServletContext(). getRequestDispatcher("/Success.jsp"); requestDispatcher.forward(request, response); } finally { out.close(); } } Success.jsp You have been successfully logged in as ${param.username} web.xml LoginServlet com.login.LoginServlet LoginServlet /LoginServlet You can download the complete working code from the below link. SampleWebApp-Code Download Hope our readers have enjoyed this article. Keep watching this space for more articles on JEE security.

With the release of ASP.NET MVC 4 one of the exciting features packed in the release was ASP.NET Web API.