RPC/encoded is a vestige from before SOAP objects were defined with XML Schema. It’s not widely supported anymore. You will need to generate the stubs using Apache Axis 1.0, which is from the same era. java org.apache.axis.wsdl.WSDL2Java http://someurl?WSDL You will need the following jars or equivalents in the -cp classpath param: axis-1.4.jar commons-logging-1.1.ja commons-discovery-0.2.jar jaxrpc-1.1.jar saaj-1.1.jar wsdl4j-1.4.jar activation-1.1.jar mail-1.4.jar This will generate similar stubs to wsimport. Alternatively, if you are not using the parts of the schema that require rpc/encoded, you can download a copy of the WSDL and comment out those bits. Then run wsimport against the local file. If you look at the WSDL, the following bits are using rpc/encoded: Sources 1. http://bitkickers.blogspot.com/2008/12/rpcencoded-web-services-on-java-16.html 2. http://stackoverflow.com/questions/412772/java-rpc-encoded-wsdls-are-not-supported-in-jaxws-2-0

Sorry for not having written for some time but I was busy with writing the JBoss Drools Refcard for DZone and I am in the middle of writing a book about Mockito so I don't have too much time left for blogging... Anyway quite recently on my current project I had an interesting situation regarding unit testing with Mockito and JAXB structures. We have very deeply nested JAXB structures generated from schemas that are provided for us which means that we can't change it in anyway. Let's take a look at the project structure: The project structure is pretty simple - there is a Player.xsd schema file that thanks to using the jaxb2-maven-plugin produces the generated JAXB Java classes corresponding to the schema in the target/jaxb/ folder in the appropriate package that is defined in the pom.xml. Speaking of which let's take a look at the pom.xml file. The pom.xml : 4.0.0 com.blogspot.toomuchcoding mockito-deep_stubs 0.0.1-SNAPSHOT UTF-8 1.6 1.6 spring-release http://maven.springframework.org/release maven-us-nuxeo https://maven-us.nuxeo.org/nexus/content/groups/public junit junit 4.10 org.mockito mockito-all 1.9.5 test org.apache.maven.plugins maven-compiler-plugin 2.5.1 org.codehaus.mojo jaxb2-maven-plugin 1.5 xjc xjc com.blogspot.toomuchcoding.model ${project.basedir}/src/main/resources/xsd Apart from the previously defined project dependencies, as mentioned previously in the jaxb2-maven-plugin in the configuration node you can define the packageName value that defines to which package should the JAXB classes be generated basing on the schemaDirectory value where the plugin can find the proper schema files. Speaking of which let's check the Player.xsd schema file (simillar to the one that was present in the Spring JMS automatic message conversion article of mine): As you can see I'm defining some complex types that even though might have no business sense but you can find such examples in the real life :) Let's find out how the method that we would like to test looks like. Here we have the PlayerServiceImpl that implements the PlayerService interface: package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. Anyway let's see how you could test the method: @Test public void shouldReturnTrueIfCountryCodeIsTheSame() throws Exception { //given PlayerDetails playerDetails = new PlayerDetails(); ClubDetails clubDetails = new ClubDetails(); CountryDetails countryDetails = new CountryDetails(); CountryCodeDetails countryCodeDetails = new CountryCodeDetails(); playerDetails.setClubDetails(clubDetails); clubDetails.setCountry(countryDetails); countryDetails.setCountryCode(countryCodeDetails); countryCodeDetails.setCountryCode(CountryCodeType.ENG); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } The function checks if, once you have the same Country Code, you get a true boolean from the method. The only problem is the amount of sets and instantiations that take place when you want to create the input message. In our projects we have twice as many nested elements so you can only imagine the number of code that we would have to produce to create the input object... So what can be done to improve this code? Mockito comes to the rescue to together with the RETURN_DEEP_STUBS default answer to the Mockito.mock(...) method: @Test public void shouldReturnTrueIfCountryCodeIsTheSameUsingMockitoReturnDeepStubs() throws Exception { //given PlayerDetails playerDetailsMock = mock(PlayerDetails.class, RETURNS_DEEP_STUBS); CountryCodeType countryCodeType = CountryCodeType.ENG; when(playerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode()).thenReturn(countryCodeType); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetailsMock, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } So what happened here is that you use the Mockito.mock(...) method and provide the RETURNS_DEEP_STUBS answer that will create mocks automatically for you. Mind you that Enums can't be mocked that's why you can't write in the Mockito.when(...) functionplayerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode().getValue(). Summing it up you can compare the readability of both tests and see how clearer it is to work with JAXB structures by using Mockito RETURNS_DEEP_STUBS default answer. Naturally sources for this example are available at BitBucket and GitHub.

Elasticsearch is a flexible and powerful open source, distributed real-time search and analytics engine for the cloud based on Apache Lucene which provides full text search capabilities. It is document oriented and schema free. Asciidoctor is a pure Ruby processor for converting AsciiDoc source files and strings into HTML 5, DocBook 4.5 and other formats. Apart of Asciidoctor Ruby part, there is an Asciidoctor-java-integration project which let us call Asciidoctor functions from Java without noticing that Ruby code is being executed. In this post we are going to see how we can use Elasticsearch over AsciiDocdocuments to make them searchable by their header information or by their content. Let's add required dependencies: junit junit 4.11 test com.googlecode.lambdaj lambdaj 2.3.3 org.elasticsearch elasticsearch 0.90.1 org.asciidoctor asciidoctor-java-integration 0.1.3 Lambdaj library is used to convert AsciiDoc files to a json documents. Now we can start an Elasticsearch instance which in our case it is going to be an embedded instance. node = nodeBuilder().local(true).node(); Next step is parse AsciiDoc document header, read its content and convert them into a json document. An example of json document stored in Elasticsearch can be: { "title":"Asciidoctor Maven plugin 0.1.2 released!", "authors":[ { "author":"Jason Porter", "email":"[email protected]" } ], "version":null, "content":"= Asciidoctor Maven plugin 0.1.2 released!.....", "tags":[ "release", "plugin" ] } And for converting an AsciiDoc File to a json document we are going to useXContentBuilder class which is provided by ElasticsearchJava API to create jsondocuments programmatically. package com.lordofthejars.asciidoctor; import static org.elasticsearch.common.xcontent.XContentFactory.*; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; import java.util.List; import org.asciidoctor.Asciidoctor; import org.asciidoctor.Author; import org.asciidoctor.DocumentHeader; import org.asciidoctor.internal.IOUtils; import org.elasticsearch.common.xcontent.XContentBuilder; import ch.lambdaj.function.convert.Converter; public class AsciidoctorFileJsonConverter implements Converter { private Asciidoctor asciidoctor; public AsciidoctorFileJsonConverter() { this.asciidoctor = Asciidoctor.Factory.create(); } public XContentBuilder convert(File asciidoctor) { DocumentHeader documentHeader = this.asciidoctor.readDocumentHeader(asciidoctor); XContentBuilder jsonContent = null; try { jsonContent = jsonBuilder() .startObject() .field("title", documentHeader.getDocumentTitle()) .startArray("authors"); Author mainAuthor = documentHeader.getAuthor(); jsonContent.startObject() .field("author", mainAuthor.getFullName()) .field("email", mainAuthor.getEmail()) .endObject(); List authors = documentHeader.getAuthors(); for (Author author : authors) { jsonContent.startObject() .field("author", author.getFullName()) .field("email", author.getEmail()) .endObject(); } jsonContent.endArray() .field("version", documentHeader.getRevisionInfo().getNumber()) .field("content", readContent(asciidoctor)) .array("tags", parseTags((String)documentHeader.getAttributes().get("tags"))) .endObject(); } catch (IOException e) { throw new IllegalArgumentException(e); } return jsonContent; } private String[] parseTags(String tags) { tags = tags.substring(1, tags.length()-1); return tags.split(", "); } private String readContent(File content) throws FileNotFoundException { return IOUtils.readFull(new FileInputStream(content)); } } Basically we are building the json document by calling startObject methods to start a new object, field method to add new fields, and startArray to start an array. Then this builder will be used to render the equivalent object in json format. Notice that we are using readDocumentHeader method from Asciidoctor class which returns header attributes from AsciiDoc file without reading and rendering the whole document. And finally content field is set with all document content. And now we are ready to start indexing documents. Note that populateData method receives as parameter a Client object. This object is from Elasticsearch Java APIand represents a connection to Elasticsearch database. import static ch.lambdaj.Lambda.convert; //.... private void populateData(Client client) throws IOException { List asciidoctorFiles = new ArrayList() {{ add(new File("target/test-classes/java_release.adoc")); add(new File("target/test-classes/maven_release.adoc")); }; List jsonDocuments = convertAsciidoctorFilesToJson(asciidoctorFiles); for (int i=0; i < jsonDocuments.size(); i++) { client.prepareIndex("docs", "asciidoctor", Integer.toString(i)).setSource(jsonDocuments.get(i)).execute().actionGet(); } client.admin().indices().refresh(new RefreshRequest("docs")).actionGet(); } private List convertAsciidoctorFilesToJson(List asciidoctorFiles) { return convert(asciidoctorFiles, new AsciidoctorFileJsonConverter()); } It is important to note that the first part of the algorithm is converting all our AsciiDocfiles (in our case two) to XContentBuilder instances by using previous converter class and the method convert of Lambdaj project. If you want you can take a look to both documents used in this example in https://github.com/asciidoctor/asciidoctor.github.com/blob/develop/news/asciidoctor-java-integration-0-1-3-released.adoc and https://github.com/asciidoctor/asciidoctor.github.com/blob/develop/news/asciidoctor-maven-plugin-0-1-2-released.adoc. Next part is inserting documents inside one index. This is done by using prepareIndexmethod, which requires an index name (docs), an index type (asciidoctor), and the idof the document being inserted. Then we call setSource method which transforms theXContentBuilder object to json, and finally by calling execute().actionGet(), data is sent to database. The final step is only required because we are using an embedded instance ofElasticsearch (in production this part should not be required), which refresh the indexes by calling refresh method. After that point we can start querying Elasticsearch for retrieving information from our AsciiDoc documents. Let's start with very simple example, which returns all documents inserted: SearchResponse response = client.prepareSearch().execute().actionGet(); Next we are going to search for all documents that has been written by Alex Sotowhich in our case is one. import static org.elasticsearch.index.query.QueryBuilders.matchQuery; //.... QueryBuilder matchQuery = matchQuery("author", "Alex Soto"); QueryBuilder matchQuery = matchQuery("author", "Alexander Soto"); Note that I am searching for field author the string Alex Soto, which returns only one. The other document is written by Jason. But it is interesting to say that if you search for Alexander Soto, the same document will be returned; Elasticsearch is smart enough to know that Alex and Alexander are very similar names so it returns the document too. More queries, how about finding documents written by someone who is called Alex, but not Soto. import static org.elasticsearch.index.query.QueryBuilders.fieldQuery; //.... QueryBuilder matchQuery = fieldQuery("author", "+Alex -Soto"); And of course no results are returned in this case. See that in this case we are using afield query instead of a term query, and we use +, and - symbols to exclude and include words. Also you can find all documents which contains the word released on title. import static org.elasticsearch.index.query.QueryBuilders.matchQuery; //.... QueryBuilder matchQuery = matchQuery("title", "released"); And finally let's find all documents that talks about 0.1.2 release, in this case only one document talks about it, the other one talks about 0.1.3. QueryBuilder matchQuery = matchQuery("content", "0.1.2"); Now we only have to send the query to Elasticsearch database, which is done by using prepareSearch method. SearchResponse response = client.prepareSearch("docs") .setTypes("asciidoctor") .setQuery(matchQuery) .execute() .actionGet(); SearchHits hits = response.getHits(); for (SearchHit searchHit : hits) { System.out.println(searchHit.getSource().get("content")); } Note that in this case we are printing the AsciiDoc content through console, but you could use asciidoctor.render(String content, Options options) method to render the content into required format. So in this post we have seen how to index documents using Elasticsearch, how to get some important information from AsciiDoc files using Asciidoctor-java-integration project, and finally how to execute some queries to inserted documents. Of course there are more kind of queries in Elasticsearch, but the intend of this post wasn't to explore all possibilities of Elasticsearch. Also as corollary, note how important it is using AsciiDoc format for writing your documents. Without much effort you can build a search engine for your documentation. On the other side, imagine all code that would be required to implement the same using any proprietary binary format like Microsoft Word. So we have shown another reason to use AsciiDoc instead of other formats.

I’ve recently had the need to automate configuration of Nginx on an Ubuntu server. Of course, in UNIX land we like to use SSH (Secure Shell) to log into our servers and manage them remotely. Wouldn’t it be nice, I thought, if there was a managed SSH library somewhere so that I could automate logging onto my Ubuntu server, run various commands and transfer files. A short Google turned up SSH.NET by the somewhat mysterious Olegkap (at least I couldn’t find out anything else about them) which turned out to be just what I wanted. Here’s the blurb on the CodePlex site: “This project was inspired by Sharp.SSH library which was ported from java and it seems like was not supported for quite some time. This library is complete rewrite using .NET 4.0, without any third party dependencies and to utilize the parallelism as much as possible to allow best performance I can get.” It does exactly what it says on the tin. It’s on NuGet, so you can grab it with: PM> Install-Package SSH.NET Here’s how you run a remote command. First you need to build a ConnectionInfo object: public ConnectionInfo CreateConnectionInfo() { const string privateKeyFilePath = @"C:\some\private\key.pem"; ConnectionInfo connectionInfo; using (var stream = new FileStream(privateKeyFilePath, FileMode.Open, FileAccess.Read)) { var privateKeyFile = new PrivateKeyFile(stream); AuthenticationMethod authenticationMethod = new PrivateKeyAuthenticationMethod("ubuntu", privateKeyFile); connectionInfo = new ConnectionInfo( "my.server.com", "ubuntu", authenticationMethod); } return connectionInfo; } Then you simply create an SshClient instance and run commands: public void Connect() { using (var ssh = new SshClient(CreateConnectionInfo())) { ssh.Connect(); var command = ssh.CreateCommand("uptime"); var result = command.Execute(); Console.Out.WriteLine(result); ssh.Disconnect(); } } Here I’m running the ‘uptime’ command which output this when I ran it just now: 14:37:46 up 22 days, 3:59, 0 users, load average: 0.08, 0.03, 0.05 To transfer a file, just use the ScpClient: public void GetConfigurationFiles() { using (var scp = new ScpClient(CreateNginxServerConnectionInfo())) { scp.Connect(); scp.Download("/etc/nginx/", new DirectoryInfo(@"D:\Temp\ScpDownloadTest")); scp.Disconnect(); } } Which grabs all my Nginx configuration and transfers it to a directory tree on my windows machine. All in all a very nice little library that’s been working well for me so far. Give it a try if you need to interact with a UNIX-like machine from .NET code.

about an hour ago i gave a presentation on indexeddb. one of the attendees asked about dates and being able to filter based on a date range. i told him that my assumption was that you would need to convert the dates into numbers and use a number-based range. turns out i was wrong. here is an example. i began by creating an objectstore that used an index on the created field. since our intent is to search via a date field, i decided "created" would be a good name. i also named my objectstore as "data". boring, but it works. var openrequest = indexeddb.open("idbpreso_date1",1); openrequest.onupgradeneeded = function(e) { var thisdb = e.target.result; if(!thisdb.objectstorenames.contains("data")) { var os = thisdb.createobjectstore("data", {autoincrement:true}); os.createindex("created", "created", {unique:false}); } } next - i built a simple way to seed data. i based on a button click event to add 10 objects. each object will have one property, created, and the date object will be based on a random date from now till 7 days in the future. function doseed() { var now = new date(); for(var i=0; i<10; i++) { var daydiff = getrandomint(1, 7); var thisdate = new date(); thisdate.setdate(now.getdate() + daydiff); db.transaction(["data"],"readwrite").objectstore("data").add({created:thisdate}); } } //credit: mozilla developer center function getrandomint (min, max) { return math.floor(math.random() * (max - min + 1)) + min; } note that since indexeddb calls are asynchronous, my code should handle updating the user to let them know when the operation is done. since this is just a quick demo though, and since that add operation will complete incredibly fast, i decided to not worry about it. so at this point we'd have an application that lets us add data containing a created property with a valid javascript date. note i didn't change it to milliseconds. i just passed it in as is. for the final portion i added two date fields on my page. in chrome this is rendered nicely: based on these, i can then create an indexeddb range of either bounds, lowerbounds, or upperbounds. i.e., give me crap either after a date, before a date, or inside a date range. function dosearch() { var fromdate = document.queryselector("#fromdate").value; var todate = document.queryselector("#todate").value; var range; if(fromdate == "" && todate == "") return; var transaction = db.transaction(["data"],"readonly"); var store = transaction.objectstore("data"); var index = store.index("created"); if(fromdate != "") fromdate = new date(fromdate); if(todate != "") todate = new date(todate); if(fromdate != "" && todate != "") { range = idbkeyrange.bound(fromdate, todate); } else if(fromdate == "") { range = idbkeyrange.upperbound(todate); } else { range = idbkeyrange.lowerbound(fromdate); } var s = ""; index.opencursor(range).onsuccess = function(e) { var cursor = e.target.result; if(cursor) { s += "key "+cursor.key+""; for(var field in cursor.value) { s+= field+"="+cursor.value[field]+""; } s+=""; cursor.continue(); } document.queryselector("#status").innerhtml = s; } } the only conversion required here was to take the user input and turn it into "real" date objects. once done, everything works great: you can run the full demo below.

My team and I are creating a services platform based on a set of RESTful JSON services where each service contributes to the platform by providing distinct feature(s) and/or data. With logs being generated all over the place, we thought it was a good idea to centralize logging and perhaps also provide a rudimentary log viewer that allowed us to view, filter, sort and search our logs. We also wanted our logging to be asynchronous as we didn’t want our services to be held up while trying to write logs say maybe directly to a database. The strategy for achieving this was straight forward. Setup ActiveMQ Create a log4j appender that writes logs to the queue (log4j ships with one such appender but lets write our own. Write a message listener that reads logs from a JMS queue setup on an MQ server and persists them Let’s take a look one by one. Setup ActiveMQ Setting up an external ActiveMQ server is simple enough. A great tutorial is available at http://servicebus.blogspot.com/2011/02/installing-apache-active-mq-on-ubuntu.html to set it up on Ubuntu. You can also choose to embed a message broker within your application. Spring makes this easy. We will see how later. Creating a Lo4j JMS appender First, we create a log4j JMS appender. log4j ships with one such appender (that writes to a JMS topic instead of a queue) import javax.jms.DeliveryMode; import javax.jms.Destination; import javax.jms.MessageProducer; import javax.jms.ObjectMessage; import javax.jms.Session; import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.log4j.Appender; import org.apache.log4j.AppenderSkeleton; import org.apache.log4j.Logger; import org.apache.log4j.PatternLayout; import org.apache.log4j.spi.LoggingEvent; /** * JMSQueue appender is a log4j appender that writes LoggingEvent to a queue. * @author faheem * */ public class JMSQueueAppender extends AppenderSkeleton implements Appender{ private static Logger logger = Logger.getLogger("JMSQueueAppender"); private String brokerUri; private String queueName; @Override public void close() { } @Override public boolean requiresLayout() { return false; } @Override protected synchronized void append(LoggingEvent event) { try { ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory( this.brokerUri); // Create a Connection javax.jms.Connection connection = connectionFactory.createConnection(); connection.start();np // Create a Session Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE); // Create the destination (Topic or Queue) Destination destination = session.createQueue(this.queueName); // Create a MessageProducer from the Session to the Topic or Queue MessageProducer producer = session.createProducer(destination); producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT); ObjectMessage message = session.createObjectMessage(new LoggingEventWrapper(event)); // Tell the producer to send the message producer.send(message); // Clean up session.close(); connection.close(); } catch (Exception e) { e.printStackTrace(); } } public void setBrokerUri(String brokerUri) { this.brokerUri = brokerUri; } public String getBrokerUri() { return brokerUri; } public void setQueueName(String queueName) { this.queueName = queueName; } public String getQueueName() { return queueName; } } Lets see whats happening here. Line 19: We implement the Log4J appender interface that asks us to implement three methods. requiresLayout, close and append. We will keep things simple for the moment and implement the append method which gets called whenever a method call to the logger is made. Line 37: log4j calls the append method and passes a LoggingEvent object as a parameter which represents a call to a logger. A LoggingEvent object encapsulates all information about every log item. Line 41 & 42: Create a new connection factory by providing it with a uri of a JMS, in our case activemq, server Line 45, 46 and 49: We establish a connection and a session to the JMS server. A Session can be opened in several modes. An Auto_Acknowledge session is one in which the acknowledgment of message happens automatically. Other modes include Client_Acknowledge in which a client has to explicitly acknowledge receipt and/or processing of a message and two other modes. For details, refer to the docs at http://download.oracle.com/javaee/1.4/api/javax/jms/Session.html Line 52: Create a queue. Send the queue name to connect to as a parameter. Line 56: We set the delivery mode to Non_Persistent. The other option is Persistent where the message is persisted to a persistent store. Persistent mode slows down but adds reliability to the message transfer. Line 58: We are doing multiple things. First of all I am wrapping the LoggingEvent object into a LoggingEventWrapper. This is because there are some properties within the LoggingEvent object that are not serializeable and also because I want to capture some additional information such as IP address and host name. Next, using the JMS session object, I prepare an object (the wrapper) for transport. Line 61: I send the object to the queue. Below is the code for the wrapper. import java.io.Serializable; import java.net.InetAddress; import java.net.UnknownHostException; import org.apache.log4j.EnhancedPatternLayout; import org.apache.log4j.spi.LoggingEvent; /** * Logging Event Wraps a log4j LoggingEvent object. Wrapping is required by some information is lost * when the LoggingEvent is serialized. The idea is to extract all information required from the LoggingEvent * object, place it in the wrapper and then serialize the LoggingEventWrapper. This way all required data remains * available to us. * @author faheem * */ public class LoggingEventWrapper implements Serializable{ private static final String ENHANCED_PATTERN_LAYOUT = "%throwable"; private static final long serialVersionUID = 3281981073249085474L; private LoggingEvent loggingEvent; private Long timeStamp; private String level; private String logger; private String message; private String detail; private String ipAddress; private String hostName; public LoggingEventWrapper(LoggingEvent loggingEvent){ this.loggingEvent = loggingEvent; //Format event and set detail field EnhancedPatternLayout layout = new EnhancedPatternLayout(); layout.setConversionPattern(ENHANCED_PATTERN_LAYOUT); this.detail = layout.format(this.loggingEvent); } public Long getTimeStamp() { return this.loggingEvent.timeStamp; } public String getLevel() { return this.loggingEvent.getLevel().toString(); } public String getLogger() { return this.loggingEvent.getLoggerName(); } public String getMessage() { return this.loggingEvent.getRenderedMessage(); } public String getDetail() { return this.detail; } public LoggingEvent getLoggingEvent() { return loggingEvent; } public String getIpAddress() { try { return InetAddress.getLocalHost().getHostAddress(); } catch (UnknownHostException e) { return "Could not determine IP"; } } public String getHostName() { try { return InetAddress.getLocalHost().getHostName(); } catch (UnknownHostException e) { return "Could not determine Host Name"; } } } The Message Listener The message listener “listens” to the queue (or topic). Whenever a new message is added to the queue, the onMessage method is called. import javax.jms.JMSException; import javax.jms.Message; import javax.jms.MessageListener; import javax.jms.ObjectMessage; import org.apache.log4j.Logger; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class LogQueueListener implements MessageListener { public static Logger logger = Logger.getLogger(LogQueueListener.class); @Autowired private ILoggingService loggingService; public void onMessage( final Message message ) { if ( message instanceof ObjectMessage ) { try{ final LoggingEventWrapper loggingEventWrapper = (LoggingEventWrapper)((ObjectMessage) message).getObject(); loggingService.saveLog(loggingEventWrapper); } catch (final JMSException e) { logger.error(e.getMessage(), e); } catch (Exception e) { logger.error(e.getMessage(),e); } } } } Line 23: Checking if the object being picked off the queue is an instance of ObjectMessage Line 26: Extracting LoggingEventWrapper from the Message Line 27: Call a service method to persist the log Wiring up in Spring Lines 5-9: Use the broker tag to setup an embedded message broker. Since I am using an external one, I don’t need it. Line 12: Mention the name of the queue you want to connect to. Line 14: URI of the Broker Server. Line 15-19: Connection Factory setup Line 26-28: Message Listener Setup where we specify the number of concurrent threads that can consume messages off the queue. Of course, the above example will not work out of the box. You still have to include all JMS dependencies and implement the service that persists logs. But I hope it gives you a decent idea.

Oracle offers two certifications for web component developers one for Java EE 5 and another one for Java EE 6.

Serialization is a form of persistence: serialized data survives the process and the RAM where it was created and can be reconstituted inside different processes and machines that live in a different time or place. Sometimes serialization is a poor form of persistence in fact, one that confuses the boundary between the different schemas the data can fit in. However, what I found useful in the last years of development is to institute a strict separation: serialize Value Objects, Entities, and everything that represents the state of the application. Meanwhile, use Dependency Injection over services that are part of a larger object graph and never serialize this second kind of objects. In the discussion that follows, I make the assumption that serialization and deserialization occur on the same machine (e.g. like for web-oriented sessions.) The problem with serialization, which work transparently most of the time, is the need to serialize service objects instead of limiting the procedure to data structures. How can you store such objects? Not options Some options to solve this problems are really not options. Serialization by itself will fail because of the staleness of the references contained in these objects. For example, in PHP trying to serialize a database connections composed by a Repository or DAO object will rightly fail with an exception. Whenever an object represents a resource of the current machine, it cannot usually be serialized except in the case when the only resource involved is RAM. If the resource is disk space or other running processes such as a database daemon, the reconstitution of the object in another place and time will fail and it's best to just stop the developer immediately during storage. Quasi-options Some solutions to the problem try to avoid the staleness problem by serializing objects without their resources, and make them regrab a new version of them on deserialization. In PHP for example, this can be done with the __sleep() and __wakeup() magic methods, called automatically during serialization and deserializaton respectively. This deserialization mechanism introduces a dependency from the serialized Entity to external services: such a dependency is already in place when building the object the first time (passing the XService in the constructor) but it is aggravated when deserializing (depending on a XServiceFactory instead of just an XService). An improvement, from the dependencies point of view, is to reattach collaborators to deserialized objects like you would for other persistence-related tasks. For example, EntityRepository can inject the missing pieces of Entity every time its find() method is called. However, there is still another option, which is the most resilient from the modelling point of view and not only that of dependency management: injecting non-serializable collaborators through the stack. Objects can collaborate even without keeping field references to each other, and injecting dependencies as parameters move the dependency starting point from the server to the client object (which may or may not be desirable). What is most important is that Entities are relieved of having to manage external references in any context, not only that of persistence and in particular serialization. The metaphor for the 3rd option Misko Hevery likes to say: have you ever seen a credit card able to charge itself? If a CreditCard is an Entity in your domain, it would be very strange to keeping a wire attached to your wallet wherever you go. With the first option, you have the card spring a wire when it is taken out of the wallet, like in horror movies. This intelligent cable tries as its best to attach to the nearest Point of Sale (a bad case of bluetooth I think). With Repositories in mind, you're not dealing with automated wires anymore, but you're still attaching cables between cards and fixed devices. In reality, cards collaborate with the PoS in a fast process that does not last more than a few seconds. Actually, sometimes they don't touch it at all, as in all Internet-based purchases. Keeping services around to deal with external dependencies does not mean the API of your Domain Model has to be biased towards service objects: pos.charge(creditCard); // can equivalently be: creditCard.chargeOn(pos); This is a form of Double Dispatch since there are two objects collaborating and you can dispatch (send messages) to both, being polimorphic by substituting both objects. The sequence of calls is: client -> creditCard -> pos The client object still looks at CreditCard as a behaviorally complete object, but it is clear which dependency is necessary to run each use case (CreditCard method). You can persist a CreditCard easily and send it over the wire to caches or databases. When it comes the time to charge, it is the client that has to bring forward a service able to connect to a bank.

A quick Java snippet for log scraping: package com.agilemobiledeveloper.logcheck; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import com.jcraft.jsch.Channel; import com.jcraft.jsch.ChannelSftp; import com.jcraft.jsch.JSch; import com.jcraft.jsch.Session; /** * * @author spannt * */ public class LogScraper { /** * @param args */ public static void main(String[] args) { String SFTPHOST = "myunixsite.com"; int SFTPPORT = 22; String SFTPUSER = "myunixid"; String SFTPPASS = "myunixpassword"; String SFTPWORKINGDIR = "/some/unix/directory"; String SERRORFILE = "SystemErr.log"; String SOUTFILE = "SystemOut.log"; Session session = null; Channel channel = null; ChannelSftp channelSftp = null; StringBuilder out = new StringBuilder(); try { JSch jsch = new JSch(); session = jsch.getSession(SFTPUSER, SFTPHOST, SFTPPORT); session.setPassword(SFTPPASS); java.util.Properties config = new java.util.Properties(); config.put("StrictHostKeyChecking", "no"); session.setConfig(config); session.connect(); channel = session.openChannel("sftp"); channel.connect(); channelSftp = (ChannelSftp) channel; channelSftp.cd(SFTPWORKINGDIR); System.out.println("Error File"); out.append("Error File:").append( LogScraper.parseStream(channelSftp.get(SERRORFILE))); System.out.println("Output File"); out.append("Output File:").append( LogScraper.parseStream(channelSftp.get(SOUTFILE))); } catch (Exception ex) { ex.printStackTrace(); out.append(ex.getLocalizedMessage()); } System.out.println("Logs=" + out.toString()); } /** * * line.contains("Exception") || * * @param file * @return String of error data */ public static String parseStream(InputStream inputFileStream) { if ( null == inputFileStream ) { return "Log Empty"; } StringBuilder out = new StringBuilder(); BufferedReader br = new BufferedReader(new InputStreamReader( inputFileStream)); String line = null; try { while ((line = br.readLine()) != null) { if (line.contains("OutOfMemoryError")) { out.append(line).append(System.lineSeparator()); } } } catch (IOException e) { e.printStackTrace(); out.append(e.getLocalizedMessage()); } return out.toString(); } }

Adopting Martin Fowler's approach to domain-specific language.

Apache Hive is a high level SQL-like interface to Hadoop. It lets you execute mostly unadulterated SQL, like this: CREATE TABLE test_table(key string, stats map); The map column type is the only thing that doesn’t look like vanilla SQL here. Hive can actually use different backends for a given table. Map is used to interface with column oriented backends like HBase. Essentially, because we won’t know ahead of time all the column names that could be in the HBase table, Hive will just return them all as a key/value dictionary. There are then helpers to access individual columns by key, or even pivot the map into one key per logical row. As part of the Hadoop family, Hive is focused on bulk loading and processing. So it’s not a surprise that Hive does not support inserting raw values like the following SQL: INSERT INTO suppliers (supplier_id, supplier_name) VALUES (24553, 'IBM'); However, for unit testing Hive scripts, it would be nice to be able to insert a few records manually. Then you could run your map reduce HQL, and validate the output. Luckily, Hive can load CSV files, so it’s relatively easy to insert a handful or records that way. CREATE TABLE foobar(key string, stats map) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY '|' MAP KEYS TERMINATED BY ':' ; LOAD DATA LOCAL INPATH '/tmp/foobar.csv' INTO TABLE foobar; This will load a CSV file with the following data, where c4ca4-0000001-79879483-000000000124 is the key, and comments and likesare columns in a map. c4ca4-0000001-79879483-000000000124,comments:0|likes:0 c4ca4-0000001-79879483-000000000124,comments:0|likes:0 Because I’ve been doing this quite a bit in my unit tests, I wrote a quick Python helper to dump a list of key/map tuples to a temporary CSV file, and then load it into Hive. This uses hiver to talk to Hive over thrift. import hiver from django.core.files.temp import NamedTemporaryFile def _hql(self, hql): client = hiver.connect(settings.HIVE_HOST, settings.HIVE_PORT) try: client.execute(hql) finally: client.shutdown() def insert(self, table_name, rows): ''' cannot insert single rows via hive, need to save to a temp file and bulk load that ''' csv_file = NamedTemporaryFile(delete=True) for row in rows: map_repr = '|'.join('%s:%s' % (key, value) for key, value in row[1].items()) csv_file.write(row[0] + "," + map_repr + "\n") csv_file.flush() try: _hql('DROP TABLE IF EXISTS %s' % table_name) _hql(""" CREATE TABLE %s ( key string, map ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY '|' MAP KEYS TERMINATED BY ':' """ % (table_name)) _hql(""" LOAD DATA LOCAL INPATH '%s' INTO TABLE %s """ % (csv_file.name, table_name) finally: csv_file.close() You can call it like this: insert('test_table', [ ('c4ca4-0000001-79879483-000000000124', {'comments': 1, 'likes': 2}), ('c4ca4-0000001-79879483-000000000124', {'comments': 1, 'likes': 2}), ('c4ca4-0000001-79879496-000000000124', {'comments': 1, 'likes': 2}), ('b4aed-0000002-79879783-000000000768', {'comments': 1, 'likes': 2}), ('b4aed-0000002-79879783-000000000768', {'comments': 1, 'likes': 2}), ])

[This blog was syndicated from http://blog.grallandco.com] Introduction When I was looking for a more effective way to create my cluster I asked some sysadmins which tools I should use to do it. The answer I got during OSDC was not Puppet, nor Chef, but wasAnsible. This article shows you how you can easily configure and create a Couchbase cluster deployed and many linux boxes...and the only thing you need on these boxes is an SSH Server! Thanks to Jan-Piet Mens that was one of the person that convinced me to use Ansible and answered questions I had about Ansible. You can watch the demonstration below, and/or look at all the details in the next paragraph. Ansible Ansible is an open-source software that allows administrator to configure and manage many computers over SSH. I won't go in all the details about the installation, just follow the steps documented in the Getting Started Guide. As you can see from this guide, you just need Python and few other libraries and clone Ansible project from Github. So I am expecting that you have Ansible working with your various servers on which you want to deploy Couchbase. Also for this first scripts I am using root on my server to do all the operations. So be sure you have register the root ssh keys to your administration server, from where you are running the Ansible scripts. Create a Couchbase Cluster So before going into the details of the Ansible script it is interesting to explain how you create a Couchbase Cluster. So here are the 5 steps to create and configure a cluster: Install Couchbase on each nodes of the cluster, as documented here. Take one of the node and "initialize" the cluster, using cluster-init command. Add the other nodes to the cluster, using server-add command. Rebalance, using rebalance command. Create a Bucket, using bucket-create command. So the goal now is to create an Ansible Playbook that executes these steps for you. Ansible Playbook for Couchbase The first think you need is to have the list of hosts you want to target, so I have create a hosts file that contains all my server organized in 2 groups: [couchbase-main] vm1.grallandco.com [couchbase-nodes] vm2.grallandco.com vm3.grallandco.com The group [couchbase-main] group is just one of the node that will drive the installation and configuration, as you probably already know, Couchbase does not have any master... All nodes in the cluster are identical. To ease the configuration of the cluster, I have create another file that contains all parameters that must be sent to all the various commands. This file is located in the group_vars/all see the section Splitting Out Host and Group Specific Data in the documentation. # Adminisrator user and password admin_user: Administrator admin_password: password # ram quota for the cluster cluster_ram_quota: 1024 # bucket and replicas bucket_name: ansible bucket_ram_quota: 512 num_replicas: 2 Use this file to configure your cluster. Let's describe the playbook file : - name: Couchbase Installation hosts: all user: root tasks: - name: download Couchbase package get_url: url=http://packages.couchbase.com/releases/2.0.1/couchbase-server-enterprise_x86_64_2.0.1.deb dest=~/. - name: Install dependencies apt: pkg=libssl0.9.8 state=present - name: Install Couchbase .deb file on all machines shell: dpkg -i ~/couchbase-server-enterprise_x86_64_2.0.1.deb As expected, the installation has to be done on all servers as root then we need to execute 3 tasks: Download the product, the get_url command will only download the file if not already present Install the dependencies with the apt command, the state=present allows the system to only install this package if not already present Install Couchbase with a simple shell command. (here I am not checking if Couchbase is already installed) So we have now installed Couchbase on all the nodes. Let's now configure the first node and add the others: - name: Initialize the cluster and add the nodes to the cluster hosts: couchbase-main user: root tasks: - name: Configure main node shell: /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-init-username=${admin_user} --cluster-init-password=${admin_password} --cluster-init-port=8091 --cluster-init-ramsize=${cluster_ram_quota} - name: Create shell script for configuring main node action: template src=couchbase-add-node.j2 dest=/tmp/addnodes.sh mode=750 - name: Launch config script action: shell /tmp/addnodes.sh - name: Rebalance the cluster shell: /opt/couchbase/bin/couchbase-cli rebalance -c 127.0.0.1:8091 -u ${admin_user} -p ${admin_password} - name: create bucket ${bucket_name} with ${num_replicas} replicas shell: /opt/couchbase/bin/couchbase-cli bucket-create -c 127.0.0.1:8091 --bucket=${bucket_name} --bucket-type=couchbase --bucket-port=11211 --bucket-ramsize=${bucket_ram_quota} --bucket-replica=${num_replicas} -u ${admin_user} -p ${admin_password} Now we need to execute specific taks on the "main" server: Initialization of the cluster using the Couchbase CLI, on line 06 and 07 Then the system needs to ask all other server to join the cluster. For this the system needs to get the various IP and for each IP address execute the add-server command with the IP address. As far as I know it is not possible to get the IP address from the main playbook YAML file, so I ask the system to generate a shell script to add each node and execute the script. This is done from the line 09 to 13. To generate the shell script, I use Ansible Template, the template is available in the couchbase-add-node.j2 file. {% for host in groups['couchbase-nodes'] %} /opt/couchbase/bin/couchbase-cli server-add -c 127.0.0.1:8091 -u ${admin_user} -p ${admin_password} --server-add={{ hostvars[host]['ansible_eth0']['ipv4']['address'] }:8091 --server-add-username=${admin_user} --server-add-password=${admin_password} {% endfor %} As you can see this script loop on each server in the [couchbase-nodes] group and use its IP address to add the node to the cluster. Finally the script rebalance the cluster (line 16) and add a new bucket (line 19). You are now ready to execute the playbook using the following command : ./bin/ansible-playbook -i ./couchbase/hosts ./couchbase/couchbase.yml -vv I am adding the -vv parameter to allow you to see more information about what's happening during the execution of the script. This will execute all the commands described in the playbook, and after few seconds you will have a new cluster ready to be used! You can for example open a browser and go to the Couchase Administration Console and check that your cluster is configured as expected. As you can see it is really easy and fast to create a new cluster using Ansible. I have also create a script to uninstall properly the cluster.. just launch ./bin/ansible-playbook -i ./couchbase/hosts ./couchbase/couchbase-uninstall.yml

Hadoop provides a Java native API to support file system operations..

I'd like to show how to instantiate Spring's component with non-default constructor (constructor with parameters) from other Spring driven component because it seems to me that a lot of Java developers doesn't know that this is possible in Spring. First of all, simple annotation based Spring configuration. It's it is scanning target package for components. @Configuration @ComponentScan("sk.lkrnac.blog.componentwithparams") public class SpringContext { } Here is component with non-default constructor. This component needs to be specified as prototype, because we will be creating new instances of it. It also needs to be named (will clarify why below). @Component(SpringConstants.COMPONENT_WITH_PARAMS) @Scope("prototype") public class ComponentWithParams { private String param1; private int param2; public ComponentWithParams(String param1, int param2) { this.param1 = param1; this.param2 = param2; } @Override public String toString() { return "ComponentWithParams [param1=" + param1 + ", param2=" + param2 + "]"; } } Next component instantiates ComponentWithParams. It needs to be aware of Spring's context instance to be able to pass arguments into non-default constructor. getBean(String name, Object... args) method is used for that purpose. First parameter of this method is name of the instantiating component (that is why component needed to be named). Constructor parameters follow. @Component public class CreatorComponent implements ApplicationContextAware{ ApplicationContext context; public void createComponentWithParam(String param1, int param2){ ComponentWithParams component = (ComponentWithParams) context.getBean(SpringConstants.COMPONENT_WITH_PARAMS, param1, param2); System.out.println(component.toString() + " instanciated..."); } @Override public void setApplicationContext(ApplicationContext context) throws BeansException { this.context = context; } } Last code snippet is main class which loads Spring context and runs test. public class Main { public static void main(String[] args) { AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(SpringContext.class); CreatorComponent invoker = context.getBean(CreatorComponent.class); invoker.createComponentWithParam("Fisrt", 1); invoker.createComponentWithParam("Second", 2); invoker.createComponentWithParam("Third", 3); context.close(); } } Console output: ComponentWithParams instanciated... ComponentWithParams instanciated... ComponentWithParams instanciated... And that's it. You can download souce code on Github

In this tutorial we are going to investigate ways you can tell what events a certain element has bound to it. This is really useful when you need to debug the on method to see how it is working. There are a number of ways to bind events to an element, there is bind(), live() and delegate(). Bind - Used to attach events to current elements on the page. If new elements are added with the same selector the event will not be added to the element. Live - This is used to attach an events to all elements on the page and future elements with the same selector. Delegate - This is used to attach events to all elements on the page and future elements based on a specific root element. As of jQuery 1.7 the on method will give you all the functionality that you need to bound events to elements. It will allow you to add an event to all elements on the page and all future elements. $('.element').on('click', function(){ // stuff }); $('.element').on('hover', function(){ // stuff }); $('.element').on('mouseenter', function(){ // stuff }); $('form').on('submit', function(){ // stuff }); When you are assigning events to future elements you might want to find out what events are currently assigned to these elements. Using developer tools in your browser you can see what events are currently bound to the element by using these 2 methods. First you can display the events by using the console, the second method is to view the events in the event listener window in the developer tools. Display Events In Console In your browser if you press F12 a new window called developer tools will open, this allows you to view the entire HTML DOM in more detail, this also has a console feature which will allow you to type in any Javascript to run at this current time on the page. You can use the console to display a list of events currently assigned to an element by using the following code. $._data( $('.element')[0], 'events' ); This will display all the events that are currently assigned to the element. Event Listener Window The other option to use to view what events are currently assigned to an element is to use the event listener window in your browser's developer tools. All you have to do is press F12 to open the developer tools, select the element in the HTML DOM that you want to investigate, on the right side of the window you will see an option called Event Listeners. When you expand this menu you will see all the current events assigned to the element including all click events bound from jQuery.

avro has a little-known gem of a feature which allows you to control which fields in an avro record are used for partitioning , sorting and grouping in mapreduce. the following figure gives a quick refresher as to what these terms mean. oh, and don’t take the placement of the “sorting” literally - sorting actually occurs on both the map and reduce side - but it’s always performed in the context of a specific partition (i.e. for a specific reducer). by default all the fields in an avro map output key are used for partitioning, sorting and grouping in mapreduce. let’s walk through an example and see how this works. you’ll begin with a simple schema github source : {"type": "record", "name": "com.alexholmes.avro.weathernoignore", "doc": "a weather reading.", "fields": [ {"name": "station", "type": "string"}, {"name": "time", "type": "long"}, {"name": "temp", "type": "int"}, {"name": "counter", "type": "int", "default": 0} ] } we’re going to see what happens when we run this code against a small sample data set, which we’ll generate using avro code github source : file input = tmpfolder.newfile("input.txt"); avrofiles.createfile(input, weathernoignore.schema$, arrays.aslist( weathernoignore.newbuilder().setstation("sfo").settime(1).settemp(3).build(), weathernoignore.newbuilder().setstation("iad").settime(1).settemp(1).build(), weathernoignore.newbuilder().setstation("sfo").settime(2).settemp(1).build(), weathernoignore.newbuilder().setstation("sfo").settime(1).settemp(2).build(), weathernoignore.newbuilder().setstation("sfo").settime(1).settemp(1).build() ).toarray()); to understand how avro is partitioning, sorting and grouping the data, we’ll write an identity mapper and reducer, with a small enhancement to the reducer to increment the counter field for each record we see in an individual reducer instance github source : package com.alexholmes.avro.sort.basic; import com.alexholmes.avro.weathernoignore; import org.apache.avro.mapred.avrokey; import org.apache.avro.mapred.avrovalue; import org.apache.avro.mapreduce.avrojob; import org.apache.avro.mapreduce.avrokeyinputformat; import org.apache.avro.mapreduce.avrokeyoutputformat; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.nullwritable; import org.apache.hadoop.mapreduce.job; import org.apache.hadoop.mapreduce.mapper; import org.apache.hadoop.mapreduce.reducer; import org.apache.hadoop.mapreduce.lib.input.fileinputformat; import org.apache.hadoop.mapreduce.lib.output.fileoutputformat; import java.io.ioexception; public class avrosort { private static class sortmapper extends mapper, nullwritable, avrokey, avrovalue> { @override protected void map(avrokey key, nullwritable value, context context) throws ioexception, interruptedexception { context.write(key, new avrovalue(key.datum())); } } private static class sortreducer extends reducer, avrovalue, avrokey, nullwritable> { @override protected void reduce(avrokey key, iterable> values, context context) throws ioexception, interruptedexception { int counter = 1; for (avrovalue weathernoignore : values) { weathernoignore.datum().setcounter(counter++); context.write(new avrokey(weathernoignore.datum()), nullwritable.get()); } } } public boolean runmapreduce(final job job, path inputpath, path outputpath) throws exception { fileinputformat.setinputpaths(job, inputpath); job.setinputformatclass(avrokeyinputformat.class); avrojob.setinputkeyschema(job, weathernoignore.schema$); job.setmapperclass(sortmapper.class); avrojob.setmapoutputkeyschema(job, weathernoignore.schema$); avrojob.setmapoutputvalueschema(job, weathernoignore.schema$); job.setreducerclass(sortreducer.class); avrojob.setoutputkeyschema(job, weathernoignore.schema$); job.setoutputformatclass(avrokeyoutputformat.class); fileoutputformat.setoutputpath(job, outputpath); return job.waitforcompletion(true); } } if you look at the output of the job below, you’ll see that the output is sorted across all the fields, and that the sorting is in field ordinal order. what this means is that when mapreduce is sorting these records, it compares the station field first, then the time field second, and so on according to the ordering of the fields in the avro schema. this is pretty much what you’d expect if you write your own complex writable type, and your comparator compared all the fields in order. {"station": "iad", "time": 1, "temp": 1, "counter": 1} {"station": "sfo", "time": 1, "temp": 1, "counter": 1} {"station": "sfo", "time": 1, "temp": 2, "counter": 1} {"station": "sfo", "time": 1, "temp": 3, "counter": 1} {"station": "sfo", "time": 2, "temp": 1, "counter": 1} oh, and before we move on notice that the value for the counter field is always 1 , meaning that each reducer was only fed a single key/vaue pair, which makes sense since our identity mapper only emitted a single value for each key, the keys are unique, and the mapreduce partitioner, sorter and grouper were using all the fields in the record. excluding fields for sorting avro gives us the ability to indicate that specific fields should be ignored when performing ordering functions. in mapreduce these fields are ignored for sorting/partitioning and grouping in mapreduce, which basically means that we have the ability to perform secondary sorting. let’s examine the following schema github source : {"type": "record", "name": "com.alexholmes.avro.weather", "doc": "a weather reading.", "fields": [ {"name": "station", "type": "string"}, {"name": "time", "type": "long"}, {"name": "temp", "type": "int", "order": "ignore"}, {"name": "counter", "type": "int", "order": "ignore", "default": 0} ] } it’s pretty much identical to the first schema, the only difference being that the last two fields are flagged as being “ignored” for sorting/partitioning/grouping. let’s run the same (other than modified to work with the different schema) mapreduce code github source as above against this new schema and examine the outputs. {"station": "iad", "time": 1, "temp": 1, "counter": 1} {"station": "sfo", "time": 1, "temp": 3, "counter": 1} {"station": "sfo", "time": 1, "temp": 2, "counter": 2} {"station": "sfo", "time": 1, "temp": 1, "counter": 3} {"station": "sfo", "time": 2, "temp": 1, "counter": 1} there are a couple of notable differences between this output, and the output from the previous schema which didn’t have any ignored fields. first, it’s clear that the temp field isn’t being used in the sorting, which makes sense since we specified that it should be ignored in the schema. however, more interestingly, note the value of the counter field. all records that had identical station and time values went to the same reducer invocation, evidenced by the increasing value of counter . this is essentially secondary sort! now, all of this greatness isn’t without some limitations: you can’t support two mapreduce jobs that use the same avro key, but have different sorting/partitioning/grouping requirements. although it’s conceivable that you could create a new instance of the avro schema and set the ignored flags for these fields yourself. the partitioner, sorter and grouping functions in mapreduce all work off of the same fields (i.e. they all ignore fields that set as ignored in the schema). this means that your options for secondary sorting are limited. for example, you wouldn’t be able to partition all stations to the same reducer, and then group by station and time. ordering uses a field’s ordinal position to determine its order within the overall set of fields to be ordered. in other words, in a two-field record, the first field is always compared before the second. there’s no way to change this behavior other than flipping the order of the fields in the record. having said all of that - the “ignoring fields” feature for sorting is pretty awesome, and something that will no doubt come in handy in my future mapreduce work.

Question: Do you need to debug maven plugin: fornax-oaw-m2-plugin ? If so, go on reading. You can find the basic info at their home page: usage - http://fornax.itemis.de/confluence/display/fornax/Usage+%28TOM%29 advanced configuration - http://fornax.itemis.de/confluence/display/fornax/Configuration+%28TOM%29 However remote debugging topic is not covered there. The problem with maven plugins might be, that sometimes it's not enough to debug maven run itself (as described in my previous post). However merging information from the previous links with common remote debugging options I came to solution that simply works for me: org.fornax.toolsupport fornax-oaw-m2-plugin 2.1.1 pom -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8100 -Xnoagent -Djava.compiler=NONE generate-sources run-workflow Please note that port to remotely connect to with debugger is 8100 in my sample. Feel free to adapt it based on your setup. After using the configuration in your run, you should get to point where plugin run gets suspended and waits for remote debugger to connect (to find out how to remotelly debug, see some existing tutorial, like this one: https://dzone.com/articles/how-debug-remote-java-applicat). That's it. Enjoy your debugging session. This post is from pb's blog.

In my previous post I showed different ways in which we can parse XML document in Java. You must have noticed the code being too much verbose. Other JVM languages like Groovy, Scala provide much better support for parsing XML documents. In this post I give you the code to parse the XML document in Groovy and one can compare the ease with which we can parse XML documents. I make use of XmlSlurper API in Groovy which loads the complete XML into a tree and this tree can be then navigated using Groovy’s version of XPath called GPath. The XML document I am using is the same one used here and also the intent is to parse the XML and create a list of Employee object. class XmlParserDemo { static void main(args){ def empList = new ArrayList(); def emp; def employees = new XmlSlurper().parse(ClassLoader. getSystemResourceAsStream("xml/employee.xml")); employees.employee.each{ node -> emp = new Employee(); emp.firstName = node.firstName emp.lastName = node.lastName emp.id = node.@id emp.location = node.location empList.add(emp) } empList.each{ empT -> println(empT)} } } class Employee{ String firstName String lastName String id String location @Override public String toString(){ return "${firstName} ${lastName}(${id}) in ${location}" } } The output is: Rakesh Mishra(111) in Bangalore John Davis(112) in Chennai Rajesh Sharma(113) in Pune And the XML is present in the “xml” package and is available in the classpath of the application. I am using the ClassLoader to load the XML resource.

You can easily tell Maven to include the version of the artifact and its Git/SVN/… revision in the JAR manifest file and then access that information at runtime via getClass().getPackage.getImplementationVersion(). (All credit goes to Markus Krüger and other colleagues.) Include Maven artifact version in the manifest (Note: You will actually not want to use it, if you also want to include a SCM revision; see below.) pom.xml: ... org.apache.maven.plugins maven-jar-plugin ... true true ... ... The resulting MANIFEST.MF of the JAR file will then include the following entries, with values from the indicated properties: Built-By: ${user.name} Build-Jdk: ${java.version} Specification-Title: ${project.name} Specification-Version: ${project.version} Specification-Vendor: ${project.organization.name Implementation-Title: ${project.name} Implementation-Version: ${project.version} Implementation-Vendor-Id: ${project.groupId} Implementation-Vendor: ${project.organization.name} (Specification-Vendor and Implementation-Vendor come from the POM’s organization/name.) Include SCM revision For this you can either use the Build Number Maven plugin that produces the property ${buildNumber}, or retrieve it from environment variables passed by Jenkinsor Hudson (SVN_REVISION for Subversion, GIT_COMMIT for Git). For git alone, you could also use the maven-git-commit-id-plugin that can either replace strings such as ${git.commit.id} in existing resource files (using maven’s resource filtering, which you must enable) with the actual values or output all of them into a git.properties file. Let’s use the buildnumber-maven-plugin and create the manifest entries explicitely, containing the build number (i.e. revision) org.codehaus.mojo buildnumber-maven-plugin 1.2 validate create false false org.apache.maven.plugins maven-jar-plugin 2.4 ${project.name} ${project.version} ${buildNumber} ... Accessing the version & revision As mentioned above, you can access the manifest entries from your code via getClass().getPackage.getImplementationVersion() andgetClass().getPackage.getImplementationTitle(). References SO: How to get Maven Artifact version at runtime? Maven Archiver documentation

Have you ever tried to update specific fields of an entity using Entity Framework? If yes, then I can assume that System.Data.Entity.Validation.DbEntityValidationResult – Validation failed for one or more entities. See ‘EntityValidationErrors’ property for more details. Exception is not something new to you. While working recently on a project for which I was using Entity Framework 5.x CF (the part with code first is not relevant) I bumped into an interesting piece of code. The main idea behind it is to perform updates for a single property and ignore the rest even if they don't respect the constraints defined in the model. Please note that I have found PartialEntityValidation class somewhere on the internet and unfortunately I do not remember where, I was not able to find it again. The only changes made by me are related to code formatting. I will present the code first and explain the logic behind it afterwards. using System.Collections.Generic; using System.Data.Entity.Infrastructure; using System.Data.Entity.Validation; namespace MyProject.Data.Contexts.Concrete { public class PartialEntityValidation { private readonly IDictionary _store; public PartialEntityValidation() { _store = new Dictionary(); } public void Register(DbEntityEntry entry, string[] properties) { if (_store.ContainsKey(entry)) _store[entry] = properties; else _store.Add(entry, properties); } public void Unregister(DbEntityEntry entry) { _store.Remove(entry); } public bool IsResponsibleFor(DbEntityEntry entry) { return _store.ContainsKey(entry); } public void Validate(DbEntityValidationResult validationResult) { var entry = validationResult.Entry; foreach (var property in _store[entry]) { var validationErrors = entry.Property(property).GetValidationErrors(); foreach (var validationError in validationErrors) validationResult.ValidationErrors.Add(validationError); } } } } PartialEntityValidation class has a simple constructor which initializes a new dictionary store where the key of it is an DbEntityEntry and the value (currently) an empty string array. The string array will hold later on the properties which should be validated. The class has four methods: Register, Unregister, IsResponsibleFor and Validate. Now let me explain a bit what each method does. Register adds a new DbEntityEntry with the properties that have to be validated to the existing store. It basically registers a new entity entry for validation. It is quite obvious what Unregister method does; it removes an entity entry from the existing store. IsResponsibleFor method determines if an entity entry is present in the existing store and if it will be part of the validation process. The last method is Validate which accepts a single parameter of DbEntityValidationResult type, from which it takes the entity entry associated with the validation result and validates each property of the entity entry present in the store. If there are any validation errors those are added on ValidationErrors collection of the validation result. Now it is time to see how to use PartialEntityValidation class. In My case I have exposed it as a public read only property of the context and had ValidateEntity method overridden at context level. using System; using System.Collections.Generic; using System.Configuration; using System.Data; using System.Data.Entity; using System.Data.Entity.Infrastructure; using System.Data.Entity.Validation; namespace MyProject.Data.Contexts.Concrete { public class MyContext : DbContext { private readonly PartialEntityValidation _partialEntityValidation; static MyContext() { Database.SetInitializer(null); } protected MyContext() : base("MyDb") { _partialEntityValidation = new PartialEntityValidation(); Configuration.LazyLoadingEnabled = false; } public PartialEntityValidation PartialValidation { get { return _partialEntityValidation; } } protected override DbEntityValidationResult ValidateEntity(DbEntityEntry entityEntry, IDictionary items) { if (PartialValidation.IsResponsibleFor(entityEntry)) { var validationResult = new DbEntityValidationResult(entityEntry, new List()); PartialValidation.Validate(validationResult); return validationResult; } return base.ValidateEntity(entityEntry, items); } } } I am sure that I do not have to explain the logic behind ValidateEntity method from the db context. The final piece is the repository where the partial update is invoked. using System; using System.Linq; using System.Linq.Expressions; namespace MyProject.Data.Repositories.Concrete { public class MyRepository { private readonly MyContext _context; public MyRepository(MyContext context) { _context = context; } public void DeleteProduct(Guid productId) { var product = new Product { ProductId = productId, IsDeleted = true }; _context.PartialValidation.Register(_context.Entry(product), new[] { "IsDeleted" }); Update(workspace, p => p.IsDeleted); _context.SaveChanges(); ((IObjectContextAdapter)_context).ObjectContext.Detach(product); } public void Update(T entity, Expression> property) where T : class { var entry = _context.Entry(entity); _context.Set().Attach(entity); entry.Property(property).IsModified = true; } } } Please take into consideration that this is a demo repository which doesn't do much at all except to present how a partial update is performed. The repository has two important methods. First one is DeleteProduct which performs a soft delete by updating "IsDeleted" property of product entity to true. This is achieved by creating a new product object of which primary key is "ProductId" with "IsDeleted" property set to true. The next step is to register the entity entry along with the "IsDeleted" property (this is the only property we want to have it updated, all other properties of Product entity will be ignored) into PartialValidation store. Update method is called which attaches the entity entry to the context and marks it as modified in order to have it updated by entity framework. That is all, I really hope that you got the idea behind this even if I did not provided an actual sample application for download.