Working Example on Github There's a small, self contained mavenised example project over on Github to accompany this post - check it out here:https://github.com/adrianmilne/jpa-lucene-spring-demo Running the Demo See the README file over on GitHub for details of running the demo. Essentially - it's just running the Unit Tests, with the usual maven build and test results output to the console - example below. This is the result of running the DBUnit test, which inserts Book data into the HSQL database using JPA, and then uses Lucene to query the data, testing that the expected Books are returned (i.e. only those int he SCI-FI category, containing the word 'Space', and ensuring that any with 'Space' in the title appear before those with 'Space' only in the description. The Book Entity Our simple example stores Books. The Book entity class below is a standard JPA Entity with a few additional annotations to identify it to Lucene: @Indexed - this identifies that the class will be added to the Lucene index. You can define a specific index by adding the 'index' attribute to the annotation. We're just choosing the simplest, minimal configuration for this example. In addition to this - you also need to specify which properties on the entity are to be indexed, and how they are to be indexed. For our example we are again going for the default option by just adding an @Field annotation with no extra parameters. We are adding one other annotation to the 'title' field - @Boost - this is just telling Lucene to give more weight to search term matches that appear in this field (than the same term appearing in the description field). This example is purposefully kept minimal in terms of the ins-and-outs of Lucene (I may cover that in a later post) - we're really just concentrating on the integration with JPA and Spring for now. package com.cor.demo.jpa.entity; import javax.persistence.Entity; import javax.persistence.EnumType; import javax.persistence.Enumerated; import javax.persistence.GeneratedValue; import javax.persistence.Id; import javax.persistence.Lob; import org.hibernate.search.annotations.Boost; import org.hibernate.search.annotations.Field; import org.hibernate.search.annotations.Indexed; /** * Book JPA Entity. */ @Entity @Indexed public class Book { @Id @GeneratedValue private Long id; @Field @Boost(value = 1.5f) private String title; @Field @Lob private String description; @Field @Enumerated(EnumType.STRING) private BookCategory category; public Book(){ } public Book(String title, BookCategory category, String description){ this.title = title; this.category = category; this.description = description; } public Long getId() { return id; } public void setId(Long id) { this.id = id; } public String getTitle() { return title; } public void setTitle(String title) { this.title = title; } public BookCategory getCategory() { return category; } public void setCategory(BookCategory category) { this.category = category; } public String getDescription() { return description; } public void setDescription(String description) { this.description = description; } @Override public String toString() { return "Book [id=" + id + ", title=" + title + ", description=" + description + ", category=" + category + "]"; } } The Book Manager The BookManager class acts as a simple service layer for the Book operations - used for adding books and searching books. As you can see, the JPA database resources are autowired in by Spring from the application-context.xml. We are just using an in-memory hsql database in this example. package com.cor.demo.jpa.manager; import java.util.List; import javax.persistence.EntityManager; import javax.persistence.PersistenceContext; import javax.persistence.PersistenceContextType; import javax.persistence.Query; import org.hibernate.search.jpa.FullTextEntityManager; import org.hibernate.search.jpa.Search; import org.hibernate.search.query.dsl.QueryBuilder; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.context.annotation.Scope; import org.springframework.stereotype.Component; import org.springframework.transaction.annotation.Transactional; import com.cor.demo.jpa.entity.Book; import com.cor.demo.jpa.entity.BookCategory; /** * Manager for persisting and searching on Books. Uses JPA and Lucene. */ @Component @Scope(value = "singleton") public class BookManager { /** Logger. */ private static Logger LOG = LoggerFactory.getLogger(BookManager.class); /** JPA Persistence Unit. */ @PersistenceContext(type = PersistenceContextType.EXTENDED, name = "booksPU") private EntityManager em; /** Hibernate Full Text Entity Manager. */ private FullTextEntityManager ftem; /** * Method to manually update the Full Text Index. This is not required if inserting entities * using this Manager as they will automatically be indexed. Useful though if you need to index * data inserted using a different method (e.g. pre-existing data, or test data inserted via * scripts or DbUnit). */ public void updateFullTextIndex() throws Exception { LOG.info("Updating Index"); getFullTextEntityManager().createIndexer().startAndWait(); } /** * Add a Book to the Database. */ @Transactional public Book addBook(Book book) { LOG.info("Adding Book : " + book); em.persist(book); return book; } /** * Delete All Books. */ @SuppressWarnings("unchecked") @Transactional public void deleteAllBooks() { LOG.info("Delete All Books"); Query allBooks = em.createQuery("select b from Book b"); List books = allBooks.getResultList(); // We need to delete individually (rather than a bulk delete) to ensure they are removed // from the Lucene index correctly for (Book b : books) { em.remove(b); } } @SuppressWarnings("unchecked") @Transactional public void listAllBooks() { LOG.info("List All Books"); LOG.info("------------------------------------------"); Query allBooks = em.createQuery("select b from Book b"); List books = allBooks.getResultList(); for (Book b : books) { LOG.info(b.toString()); getFullTextEntityManager().index(b); } } /** * Search for a Book. */ @SuppressWarnings("unchecked") @Transactional public List search(BookCategory category, String searchString) { LOG.info("------------------------------------------"); LOG.info("Searching Books in category '" + category + "' for phrase '" + searchString + "'"); // Create a Query Builder QueryBuilder qb = getFullTextEntityManager().getSearchFactory().buildQueryBuilder().forEntity(Book.class).get(); // Create a Lucene Full Text Query org.apache.lucene.search.Query luceneQuery = qb.bool() .must(qb.keyword().onFields("title", "description").matching(searchString).createQuery()) .must(qb.keyword().onField("category").matching(category).createQuery()).createQuery(); Query fullTextQuery = getFullTextEntityManager().createFullTextQuery(luceneQuery, Book.class); // Run Query and print out results to console List result = (List) fullTextQuery.getResultList(); // Log the Results LOG.info("Found Matching Books :" + result.size()); for (Book b : result) { LOG.info(" - " + b); } return result; } /** * Convenience method to get Full Test Entity Manager. Protected scope to assist mocking in Unit * Tests. * @return Full Text Entity Manager. */ protected FullTextEntityManager getFullTextEntityManager() { if (ftem == null) { ftem = Search.getFullTextEntityManager(em); } return ftem; } /** * Get the JPA Entity Manager (required for the DBUnit Tests). * @return Entity manager */ protected EntityManager getEntityManager() { return em; } /** * Sets the JPA Entity Manager (required to assist with mocking in Unit Test) * @param em EntityManager */ protected void setEntityManager(EntityManager em) { this.em = em; } } application-context.xml This is the Spring configuration file. You can see in the JPA Entity Manager configuration the key for 'hibernate.search.default.indexBase' is added to the jpaPropertyMap to tell Lucene where to create the index. We have also externalised the database login credentials to a properties file (as you may wish to change these for different environments), for example by updating the propertyConfigurer to look for and use a different external properties if it finds one on the file system). classpath:/system.properties Testing Using DBUnit In the project is an example of using DBUnit with Spring to test adding and searching against the database using DBUnit to populate the database with test data, exercise the Book Manager search operations and then clean the database down. This is a great way to test database functionality and can be easily integrated into maven and continuous build environments. Because DBUnit bypasses the standard JPA insertion calls - the data does not get automatically added to the Lucene index. We have a method exposed on the service interface to update the Full Text index 'updateFullTextIndex()' - calling this causes Lucene to update the index with the current data in the database. This can be useful when you are adding search to pre-populated databases to index the existing content. package com.cor.demo.jpa.manager; import java.io.InputStream; import java.util.List; import org.dbunit.DBTestCase; import org.dbunit.database.DatabaseConnection; import org.dbunit.database.IDatabaseConnection; import org.dbunit.dataset.IDataSet; import org.dbunit.dataset.xml.FlatXmlDataSetBuilder; import org.dbunit.operation.DatabaseOperation; import org.hibernate.impl.SessionImpl; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import com.cor.demo.jpa.entity.Book; import com.cor.demo.jpa.entity.BookCategory; /** * DBUnit Test - loads data defined in 'test-data-set.xml' into the database to run tests against the * BookManager. More thorough (and ultimately easier in this context) than using mocks. */ @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = { "classpath:/application-context.xml" }) public class BookManagerDBUnitTest extends DBTestCase { /** Logger. */ private static Logger LOG = LoggerFactory.getLogger(BookManagerDBUnitTest.class); /** Book Manager Under Test. */ @Autowired private BookManager bookManager; @Before public void setup() throws Exception { DatabaseOperation.CLEAN_INSERT.execute(getDatabaseConnection(), getDataSet()); } @After public void tearDown() { deleteBooks(); } @Override protected IDataSet getDataSet() throws Exception { InputStream inputStream = this.getClass().getClassLoader().getResourceAsStream("test-data-set.xml"); FlatXmlDataSetBuilder builder = new FlatXmlDataSetBuilder(); return builder.build(inputStream); } /** * Get the underlying database connection from the JPA Entity Manager (DBUnit needs this connection). * @return Database Connection * @throws Exception */ private IDatabaseConnection getDatabaseConnection() throws Exception { return new DatabaseConnection(((SessionImpl) (bookManager.getEntityManager().getDelegate())).connection()); } /** * Tests the expected results for searching for 'Space' in SCF-FI books. */ @Test public void testSciFiBookSearch() throws Exception { bookManager.listAllBooks(); bookManager.updateFullTextIndex(); List results = bookManager.search(BookCategory.SCIFI, "Space"); assertEquals("Expected 2 results for SCI FI search for 'Space'", 2, results.size()); assertEquals("Expected 1st result to be '2001: A Space Oddysey'", "2001: A Space Oddysey", results.get(0).getTitle()); assertEquals("Expected 2nd result to be 'Apollo 13'", "Apollo 13", results.get(1).getTitle()); } private void deleteBooks() { LOG.info("Deleting Books...-"); bookManager.deleteAllBooks(); } } The source data for the test is defined in an xml file.

Remembrance and reflection, how allied; What thin partitions sense from thought divide. - Alexander Pope Retrospectives, and why you need them A couple of months ago we looked at how to conduct a Sprint Review. We saw that a Review considers what work was done, and distinguished it from a Sprint Retrospective which reflects upon how work is being done. The distinction between the two can appear to be rather academic, and slurring a Review and a Retrospective together is a mistake that is often made by immature teams. After all, both take a reflective view of a Sprint that has just finished, and both can be said to fulfill a remit of historical inquiry. Yet while the separation of concerns might seem to be a narrow one, it is nonetheless quite precise, and there is great value to be had in maintaining the appropriate focus. A Review looks candidly at what has been achieved, and soberly at what remains to be achieved, with regard to product completion. A Retrospective on the other hand is an opportunity for the Scrum Team to inspect and adapt their actual implementation of the Scrum process. The rationale behind this inspection is methodological but it is in no sense abstract. It is grounded firmly in the desire to achieve worthwhile and practical reform. Perhaps there are certain working practices which the team can make more efficient, or which can otherwise be improved upon. If so, a Retrospective presents the ideal opportunity for those improvements to be discussed and brought into action. Failing to inspect and adapt in this manner will condemn a team to perpetual infancy and the repetition of past mistakes. Sprint Retrospectives help keep a Scrum team on the road to continual improvement. When these sessions are done well, team members will not be afraid to challenge the status quo, and will do so in a constructive and informed manner. The result will be an improved delivery of value – in fact, the sort of productivity gain that might well be identified in the Sprint Review we considered earlier. In this article we’ll switch our attention fully to Retrospectives, and examine the matter of how they should be conducted. Setting up a Retrospective As any event manager will tell you, the key to a successful gig lies in the preparation. Okay…I’ll concede that a holding a Retrospective isn’t as mammoth an undertaking as hosting the Thinking Digital conference, nor can it be said to demand the organizational skills of Bruce Springsteen’s road manager. Nevertheless it’s still important to get a few ducks in a row. Let’s start by lining them up and giving them some admittedly rather unimaginative names: Why, Who, Where, When, and What. We’ve just covered the issue of why a Retrospective needs to be held…that duck’s down. Let’s pop the rest. Who should attend a Sprint Retrospective? The invitation list for a Sprint Retrospective should be simple and uncontroversial. According to the Scrum Guide all Scrum Team members are expected to attend. That’s the Developers, the Scrum Master (who may facilitate the session), and the Product Owner. No others are expected. In fact, it would be quite irregular to extend the invitation to other people, even if they consider themselves to be important players or stakeholders. That’s because it is the Scrum Team who are responsible for the way they have implemented the Scrum Framework. Only they are in a position to inspect and adapt their very own ways of working. For this reason, all members have a duty to be present, to contribute, and to help make each Retrospective a success. Some teams exclude the Product Owner from this activity, arguing that if he or she was present, the team would not be able to have an open and frank discussion. This is a common issue and we’ll return to it later. For now though, just take it as read that a good Retrospective must include all Scrum Team members, and will give each a voice in molding the processes and working environment that they collectively own. Where should a Retrospective be held? Let’s answer this one with another question. If all of the Scrum Team members are co-located, and if they have the necessary equipment to hand (such as their Scrum board, plus a whiteboard for notes), why not hold the retrospective in situ? In other words, why not just hold the session at the team’s desks? Well, although this might sound like a capital idea, there can be problems. Perhaps it would create too much of a disturbance and annoy other teams within earshot? Then again, perhaps the physical layout of the working area is simply not conducive to holding a meeting. Perhaps the team is not entirely co-located in the first place. Any one of these things can tip the balance in favour of booking an actual meeting room, and getting everyone to decamp there for a Sprint Retrospective. If so, remember to book such a room in advance…if possible as a recurring appointment for the anticipated duration of the project. Make sure it has sufficient capacity and the resources needed. When should a Retrospective happen? The glib answer is to say that a Retrospective should happen “at the end of each Sprint”. A more useful answer would say whether or not it should precede or follow the Sprint Review. In my experience it is generally better to do the Review first, because that helps to establish a context within which the Retrospective can happen. The next thing to consider is how long to allow for the session. As with all Scrum events, a Sprint Retrospective is time-boxed. This means that it isn’t allowed to exceed a set length. The rules of Scrum are exact: for a one month Sprint the limit for a Retrospective is 3 hours, which is reduced to one-and-a-half hours for a two week Sprint. You should adjust this value by the same ratio if needed. Note that if a Retrospective finishes before the time-box expires, that’s fine and dandy. You aren’t obliged to use all of the available time. The rule is simply that the time-box must never be exceeded. Scrum is not a philosophy in which matters are allowed to drag on. What topics should the Retrospective cover? This is the biggest duck in the row, and it’ll take a few pings to knock it down. What we have to do is to establish a suitable agenda for a Sprint Retrospective. We have to formulate it in such a way that the inspection of the team’s Scrum implementation does indeed happen. We also have to make sure that any recommendations for its adaptation are elicited, agreed, and turned into achievable action items. The Scrum Guide provides us with something of a starting point. It isn’t much, but I reckon that if you look at it through a beer glass with your head sideways and one eye closed, you can just about discern a notional agenda for holding a Sprint Retrospective. A notional agenda The Scrum Guide is sparing in the advice it gives on how to conduct a Retrospective. We are told that a Scrum Team must: Inspect how the last Sprint went with regards to people, relationships, process, and tools; Identify and order the major items that went well and potential improvements; and, Create a plan for implementing improvements to the way the Scrum Team does its work…[including]…ways to increase product quality by adapting the Definition of “Done” as appropriate. Yes, I know that’s not much to go on, but each of these items is clearly significant. They seem to address the very rubric of agile practice; we can recognize in them a succinct appeal to the three legs of Transparency, Inspection, and Adaptation. In them, we can see not only a notional agenda, but also how critical a Sprint Retrospective is to the Scrum process. A Retrospective is arguably the most important time-boxed event that any agile process can have. If we want to turn these points into a more formal agenda for the session, we’ll have to make sure that each of them is addressed carefully. Towards a canonical format Scrum has been around for well over a decade now, and a fairly standard agenda for conducting a Sprint Retrospective has emerged. Here’s what it looks like. Set the scene. Ways to do this can include any or all of the following: Sketching out a timeline of significant events that occurred in the Sprint, so its historical context can be established Holding the Sprint Review shortly beforehand, so the project context is fresh in attendees’ minds Declaring the Prime Directive in order to define a professional context of mutual respect and openness Assess prior action items. Unless this is the first sprint, there will have been an earlier retrospective in which some improvements will have been proposed. Look back over each of them. Have they been followed through? In short, has the process actually been adapted following that earlier inspection? If any action items remain undone, make a note of them. They’ll have to be considered when determining actions for the future. Set up a Retrospective Board. This can be a whiteboard, or even a large sheet of paper stuck to a wall. Divide it into four quadrants and label each in the following manner. The precise terminology does tend to vary a bit. There can be subtle and not-so-subtle differences in meaning (consider the difference between “good points” and “things to continue doing”). Be aware of these differences, as they will shape the responses and ultimately the results. “What went well” (or “good points”, or “things to continue doing”) “What didn’t go well” (or “bad points”, or “things to stop doing”) “Ideas for improvement” (or things to “start doing”) “Shout-outs” (i.e. recognition of noteworthy individual contributions) Storm the Board. There are several ways in which this can be done. Here are some of the more common ones: Sticky notes. This method is fairly democratic in that each attendee gets a clear say by putting sticky notes on a board. Assertive individuals are therefore less able to dominate others. However, it can be disjointed as attention shifts from one person’s topics to another person’s. As such, it can be hard to develop a line of thought for group discussion. Here’s the process: Blocks of notes are distributed to the attendees. They are given a small time-box (5 or 10 minutes) to jot down their ideas…good points, bad points, improvements, and shouts. Each attendee should write one point per sticky note. There is no limit to the number of points they can make. After the time is up, attendees take it in turn to put their notes on the board and in the relevant quadrants As an attendee puts their sticky note on the board, they briefly state what the point is to the rest of the team Once the last attendee has finished, duplicate points will be identified by the group and removed. Facilitator-as-arbitrator. In this approach a facilitator will act as a scribe for the group, and write their ideas on the board. Group discussion of ideas is encouraged, and the facilitator can arbitrate in the event of disagreement. The downside is that it can favor the more assertive type of individual who ends up doing most of the talking. Here’s how it’s done: The facilitator stands in front of the board with a marker pen Any attendee who has a suggestion to make will make it – a good point, bad point, idea, or shout-out The facilitator writes each suggestion into the appropriate quadrant, disallowing any duplicates. The group discuss the merits of each suggestion The facilitator will erase, keep, or revise each suggestion according to group opinion Hybrid. This uses a mix of techniques, such as a facilitated session for identifying good points and bad points, and a sticky-note approach in order to elicit ideas for improvement. Changing the techniques used in a Retrospective every now and then can help keep the sessions fresh, and is certainly a good idea if you reckon they are getting a bit stale. Propose actions. I have five rules that I apply when “storming the board” with a team: For every bad point there must be an idea for improvement. In other words, for everything that people are being asked to stop doing an alternative and better course of action must be proposed. This rule helps to keep attendees focused on the need to adapt the process constructively, and not to use the session for mere complaint. If you have been storming for “good points” rather than for things to “start doing”, make sure that each of those points is matched with an idea for further improvement. It isn’t enough to look back appreciatively whenever something positive has happened. Your challenge is to translate that observation into an even bigger future win. Re-assess undone action items from the previous Retrospective. If any remain undone, ask if they are worth bringing forward. Ask why they weren’t implemented, with a view to finding out what really needs to happen to expedite them. If these outstanding actions are impractical, or are no longer relevant, jettison them and concentrate on those improvements which are valuable and achievable. Ask the “Five Whys”. For each action item you produce, you need to be sure that you have understood the root cause and that the action will be appropriate. A shallow retrospective is no retrospective at all. It has to be deep and probing. Improve the Definition of Done. The Scrum Guide doesn’t provide much advice about holding Retrospectives, but it is quite clear about the need to revisit the Definition of Done. This is something that many teams, including some quite experienced ones, forget or otherwise fail to do. So be careful to identify any room for improvement in the team’s understanding of what “done” means, and what it should take for work to be considered potentially releasable. Vote. It’s quite possible that the list of proposed actions will be extensive. In aggregate they could amount to too much change if all were to be implemented in the forthcoming Sprint. You can resolve this by getting team members to vote on action items, so that only the most important ones are taken forward. For example, if the team can take forward five items, allow each attendee to vote for five of them. The most popular can then be actioned. Other observations Here are some other things to consider when conducting a Sprint Retrospective. Decide whether or not to precede it with the Scrum “Prime Directive”. This is an assertion which is meant to be said, in earnest, before each and every Retrospective. It isn’t mentioned in the Scrum Guide, but it is widely recognized and some teams do choose to recite it. “Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand” We considered the significance of this assertion in an earlier article on Agile Teamwork in Practice, so I’m not going to say much more about it here. However, Martin Fowler has expressed his thoughts on the Prime Directive, and I suggest you read his opinion piece in full. All I’ll add is that I am in agreement with his observations and that I share his sense of revulsion. Determine what to do about Product Owner representation. According to the Scrum Primer the Product Owner may attend a Sprint Retrospective. Only “Development Team” members are actually required to be there. Yet according to the Scrum Guide, all “Scrum Team” members must attend. The Scrum Team is a wider group than the Development team and includes the Product Owner. The reason for this discrepancy probably lies in the interpretation of process ownership. If we see the Development Team as owning the process through which iterative and incremental value will be delivered to a Product Owner, then the PO would not indeed have a say in the adaptation of that process. He or she would merely be a consumer of its outputs, and would therefore be a stakeholder in a Sprint Review but not in a Sprint Retrospective. However, if we view the process as a more collaborative one, in which the Development Team works with the Product Owner to deliver potentially releasable increments of value every Sprint, then the PO would indeed be a stakeholder in how that process is managed, and must therefore attend. It’s therefore important to determine what relationship the Development Team has, or should have, with the Product Owner. It’s unquestionably best if a Product Owner is on-side as a team player, and can handle root cause analysis and the exposure of potentially uncomfortable truths. Whether or not that is the case though is only something that the team can decide. Remember they’re human. Bring snacks and drinks to keep attendees refreshed, and allow enough time for breaks – at least 10 minutes every hour. Consider wrapping up the session with a “touchy feely graph” of some sort, which captures the mood and confidence of the team. Allow everyone to mark a dot or cross on a chart to show how positive or negative they feel about things, and then see how the mood changes…hopefully for the better…from one Sprint to the next. Conclusion A Sprint Retrospective is arguably the most important event that a team can hold. It provides the means to inspect and adapt the team’s actual implementation of the Scrum framework. In this article we’ve looked at how to create an agenda for the session and how to facilitate it, and at the issues of when and where it should be held, and who should attend. Those who cannot remember the past are condemned to repeat it. - George Santayana

Dan McCreary and Ann Kelly, authors of 'Making Sense of NoSQL,' discuss the business drivers and motivations that make NoSQL so popular to organizations today.

Jim and I have been doing a bit of work over the last week which involved calling neo4j’s HA status URI to check whether or not an instance was a master/slave and we’ve been using jersey-client. The code looked roughly like this: class Neo4jInstance { private Client httpClient; private URI hostname; public Neo4jInstance(Client httpClient, URI hostname) { this.httpClient = httpClient; this.hostname = hostname; } public Boolean isSlave() { String slaveURI = hostname.toString() + ":7474/db/manage/server/ha/slave"; ClientResponse response = httpClient.resource(slaveURI).accept(TEXT_PLAIN).get(ClientResponse.class); return Boolean.parseBoolean(response.getEntity(String.class)); } } While writing some tests against this code we wanted to stub out the actual calls to the HA slave URI so we could simulate both conditions and a brief search suggested that mockito was the way to go. We ended up with a test that looked like this: @Test public void shouldIndicateInstanceIsSlave() { Client client = mock( Client.class ); WebResource webResource = mock( WebResource.class ); WebResource.Builder builder = mock( WebResource.Builder.class ); ClientResponse clientResponse = mock( ClientResponse.class ); when( builder.get( ClientResponse.class ) ).thenReturn( clientResponse ); when( clientResponse.getEntity( String.class ) ).thenReturn( "true" ); when( webResource.accept( anyString() ) ).thenReturn( builder ); when( client.resource( anyString() ) ).thenReturn( webResource ); Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave(); assertTrue(isSlave); } which is pretty gnarly but does the job. I thought there must be a better way so I continued searching and eventually came across this post on the mailing list which suggested creating a custom ClientHandler and stubbing out requests/responses there. I had a go at doing that and wrapped it with a little DSL that only covers our very specific use case: private static ClientBuilder client() { return new ClientBuilder(); } static class ClientBuilder { private String uri; private int statusCode; private String content; public ClientBuilder requestFor(String uri) { this.uri = uri; return this; } public ClientBuilder returns(int statusCode) { this.statusCode = statusCode; return this; } public Client create() { return new Client() { public ClientResponse handle(ClientRequest request) throws ClientHandlerException { if (request.getURI().toString().equals(uri)) { InBoundHeaders headers = new InBoundHeaders(); headers.put("Content-Type", asList("text/plain")); return createDummyResponse(headers); } throw new RuntimeException("No stub defined for " + request.getURI()); } }; } private ClientResponse createDummyResponse(InBoundHeaders headers) { return new ClientResponse(statusCode, headers, new ByteArrayInputStream(content.getBytes()), messageBodyWorkers()); } private MessageBodyWorkers messageBodyWorkers() { return new MessageBodyWorkers() { public Map> getReaders(MediaType mediaType) { return null; } public Map> getWriters(MediaType mediaType) { return null; } public String readersToString(Map> mediaTypeListMap) { return null; } public String writersToString(Map> mediaTypeListMap) { return null; } public MessageBodyReader getMessageBodyReader(Class tClass, Type type, Annotation[] annotations, MediaType mediaType) { return (MessageBodyReader) new StringProvider(); } public MessageBodyWriter getMessageBodyWriter(Class tClass, Type type, Annotation[] annotations, MediaType mediaType) { return null; } public List getMessageBodyWriterMediaTypes(Class tClass, Type type, Annotation[] annotations) { return null; } public MediaType getMessageBodyWriterMediaType(Class tClass, Type type, Annotation[] annotations, List mediaTypes) { return null; } }; } public ClientBuilder content(String content) { this.content = content; return this; } } If we change our test to use this code it now looks like this: @Test public void shouldIndicateInstanceIsSlave() { Client client = client().requestFor("http://localhost:7474/db/manage/server/ha/slave"). returns(200). content("true"). create(); Boolean isSlave = new Neo4jInstance(client, URI.create("http://localhost")).isSlave(); assertTrue(isSlave); } Is there a better way? In Ruby I’ve used WebMock to achieve this and Ashok pointed me towards WebStub which looks nice except I’d need to pass in the hostname + port rather than constructing that in the code.

I recently wanted to attach an EBS volume to an existing EC2 instance that I had running and since it was for a one off tasks (famous last words) I decided to configure it manually. I created the EBS volume through the AWS console and one thing that initially caught me out is that the EC2 instance and EBS volume need to be in the same region and zone. Therefore if I create my EC2 instance in ‘eu-west-1b’ then I need to create my EBS volume in ‘eu-west-1b’ as well otherwise I won’t be able to attach it to that instance. I attached the device as /dev/sdf although the UI gives the following warning: Linux Devices: /dev/sdf through /dev/sdp Note: Newer linux kernels may rename your devices to /dev/xvdf through /dev/xvdp internally, even when the device name entered here (and shown in the details) is /dev/sdf through /dev/sdp. After attaching the EBS volume to the EC2 instance my next step was to SSH onto my EC2 instance and make the EBS volume available. The first step is to create a file system on the volume: $ sudo mkfs -t ext3 /dev/sdf mke2fs 1.42 (29-Nov-2011) Could not stat /dev/sdf --- No such file or directory The device apparently does not exist; did you specify it correctly? It turns out that warning was handy and the device has in fact been renamed. We can confirm this by callingfdisk: $ sudo fdisk -l Disk /dev/xvda1: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvda1 doesn't contain a valid partition table Disk /dev/xvdf: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/xvdf doesn't contain a valid partition table /dev/xvdf is the one we’re interested in so I re-ran the previous command: $ sudo mkfs -t ext3 /dev/xvdf mke2fs 1.42 (29-Nov-2011) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 3276800 inodes, 13107200 blocks 655360 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 400 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done Once I’d done that I needed to create a mount point for the volume and I thought the best place was probably a directory under /mnt: $ sudo mkdir /mnt/ebs The final step is to mount the volume: $ sudo mount /dev/xvdf /mnt/ebs And if we run df we can see that it’s ready to go: $ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.9G 883M 6.7G 12% / udev 288M 8.0K 288M 1% /dev tmpfs 119M 164K 118M 1% /run none 5.0M 0 5.0M 0% /run/lock none 296M 0 296M 0% /run/shm /dev/xvdf 50G 180M 47G 1% /mnt/ebs

OLAP (Online Analytical Processing) is a very common way to analyze raw transaction data by aggregating along different combinations of dimensions. This is a well-established field in Business Intelligence / Reporting. In this post, I will highlight the key ideas in OLAP operation and illustrate how to do this in R. Facts and Dimensions The core part of OLAP is a so-called "multi-dimensional data model", which contains two types of tables; "Fact" table and "Dimension" table A Fact table contains records each describe an instance of a transaction. Each transaction records contains categorical attributes (which describes contextual aspects of the transaction, such as space, time, user) as well as numeric attributes (called "measures" which describes quantitative aspects of the transaction, such as no of items sold, dollar amount). A Dimension table contain records that further elaborates the contextual attributes, such as user profile data, location details ... etc. In a typical setting of Multi-dimensional model ... Each fact table contains foreign keys that references the primary key of multiple dimension tables. In the most simple form, it is called a STAR schema. Dimension tables can contain foreign keys that references other dimensional tables. This provides a sophisticated detail breakdown of the contextual aspects. This is also called a SNOWFLAKE schema. Also this is not a hard rule, Fact table tends to be independent of other Fact table and usually doesn't contain reference pointer among each other. However, different Fact table usually share the same set of dimension tables. This is also called GALAXY schema. But it is a hard rule that Dimension table NEVER points / references Fact table A simple STAR schema is shown in following diagram. Each dimension can also be hierarchical so that the analysis can be done at different degree of granularity. For example, the time dimension can be broken down into days, weeks, months, quarter and annual; Similarly, location dimension can be broken down into countries, states, cities ... etc. Here we first create a sales fact table that records each sales transaction. # Setup the dimension tables state_table <- data.frame(key=c("CA", "NY", "WA", "ON", "QU"), name=c("California", "new York", "Washington", "Ontario", "Quebec"), country=c("USA", "USA", "USA", "Canada", "Canada")) month_table <- data.frame(key=1:12, desc=c("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"), quarter=c("Q1","Q1","Q1","Q2","Q2","Q2","Q3","Q3","Q3","Q4","Q4","Q4")) prod_table <- data.frame(key=c("Printer", "Tablet", "Laptop"), price=c(225, 570, 1120)) # Function to generate the Sales table gen_sales <- function(no_of_recs) { # Generate transaction data randomly loc <- sample(state_table$key, no_of_recs, replace=T, prob=c(2,2,1,1,1)) time_month <- sample(month_table$key, no_of_recs, replace=T) time_year <- sample(c(2012, 2013), no_of_recs, replace=T) prod <- sample(prod_table$key, no_of_recs, replace=T, prob=c(1, 3, 2)) unit <- sample(c(1,2), no_of_recs, replace=T, prob=c(10, 3)) amount <- unit*prod_table[prod,]$price sales <- data.frame(month=time_month, year=time_year, loc=loc, prod=prod, unit=unit, amount=amount) # Sort the records by time order sales <- sales[order(sales$year, sales$month),] row.names(sales) <- NULL return(sales) } # Now create the sales fact table sales_fact <- gen_sales(500) # Look at a few records head(sales_fact) month year loc prod unit amount 1 1 2012 NY Laptop 1 225 2 1 2012 CA Laptop 2 450 3 1 2012 ON Tablet 2 2240 4 1 2012 NY Tablet 1 1120 5 1 2012 NY Tablet 2 2240 6 1 2012 CA Laptop 1 225 Multi-dimensional Cube Now, we turn this fact table into a hypercube with multiple dimensions. Each cell in the cube represents an aggregate value for a unique combination of each dimension. # Build up a cube revenue_cube <- tapply(sales_fact$amount, sales_fact[,c("prod", "month", "year", "loc")], FUN=function(x){return(sum(x))}) # Showing the cells of the cude revenue_cube , , year = 2012, loc = CA month prod 1 2 3 4 5 6 7 8 9 10 11 12 Laptop 1350 225 900 675 675 NA 675 1350 NA 1575 900 1350 Printer NA 2280 NA NA 1140 570 570 570 NA 570 1710 NA Tablet 2240 4480 12320 3360 2240 4480 3360 3360 5600 2240 2240 3360 , , year = 2013, loc = CA month prod 1 2 3 4 5 6 7 8 9 10 11 12 Laptop 225 225 450 675 225 900 900 450 675 225 675 1125 Printer NA 1140 NA 1140 570 NA NA 570 NA 1140 1710 1710 Tablet 3360 3360 1120 4480 2240 1120 7840 3360 3360 1120 5600 4480 , , year = 2012, loc = NY month prod 1 2 3 4 5 6 7 8 9 10 11 12 Laptop 450 450 NA NA 675 450 675 NA 225 225 NA 450 Printer NA 2280 NA 2850 570 NA NA 1710 1140 NA 570 NA Tablet 3360 13440 2240 2240 2240 5600 5600 3360 4480 3360 4480 3360 , , year = 2013, loc = NY ..... dimnames(revenue_cube) $prod [1] "Laptop" "Printer" "Tablet" $month [1] "1" "2" "3" "4" "5" "6" "7" "8" "9" "10" "11" "12" $year [1] "2012" "2013" $loc [1] "CA" "NY" "ON" "QU" "WA" OLAP Operations Here are some common operations of OLAP Slice Dice Rollup Drilldown Pivot "Slice" is about fixing certain dimensions to analyze the remaining dimensions. For example, we can focus in the sales happening in "2012", "Jan", or we can focus in the sales happening in "2012", "Jan", "Tablet". # Slice # cube data in Jan, 2012 revenue_cube[, "1", "2012",] loc prod CA NY ON QU WA Laptop 1350 450 NA 225 225 Printer NA NA NA 1140 NA Tablet 2240 3360 5600 1120 2240 # cube data in Jan, 2012 revenue_cube["Tablet", "1", "2012",] CA NY ON QU WA 2240 3360 5600 1120 2240 "Dice" is about limited each dimension to a certain range of values, while keeping the number of dimensions the same in the resulting cube. For example, we can focus in sales happening in [Jan/ Feb/Mar, Laptop/Tablet, CA/NY]. revenue_cube[c("Tablet","Laptop"), c("1","2","3"), , c("CA","NY")] , , year = 2012, loc = CA month prod 1 2 3 Tablet 2240 4480 12320 Laptop 1350 225 900 , , year = 2013, loc = CA month prod 1 2 3 Tablet 3360 3360 1120 Laptop 225 225 450 , , year = 2012, loc = NY month prod 1 2 3 Tablet 3360 13440 2240 Laptop 450 450 NA , , year = 2013, loc = NY month prod 1 2 3 Tablet 3360 4480 6720 Laptop 450 NA 225 "Rollup" is about applying an aggregation function to collapse a number of dimensions. For example, we want to focus in the annual revenue for each product and collapse the location dimension (ie: we don't care where we sold our product). apply(revenue_cube, c("year", "prod"), FUN=function(x) {return(sum(x, na.rm=TRUE))}) prod year Laptop Printer Tablet 2012 22275 31350 179200 2013 25200 33060 166880 "Drilldown" is the reverse of "rollup" and applying an aggregation function to a finer level of granularity. For example, we want to focus in the annual and monthly revenue for each product and collapse the location dimension (ie: we don't care where we sold our product). apply(revenue_cube, c("year", "month", "prod"), FUN=function(x) {return(sum(x, na.rm=TRUE))}) , , prod = Laptop month year 1 2 3 4 5 6 7 8 9 10 11 12 2012 2250 2475 1575 1575 2250 1800 1575 1800 900 2250 1350 2475 2013 2250 900 1575 1575 2250 2475 2025 1800 2025 2250 3825 2250 , , prod = Printer month year 1 2 3 4 5 6 7 8 9 10 11 12 2012 1140 5700 570 3990 4560 2850 1140 2850 2850 1710 3420 570 2013 1140 4560 3420 4560 2850 1140 570 3420 1140 3420 3990 2850 , , prod = Tablet month year 1 2 3 4 5 6 7 8 9 10 11 12 2012 14560 23520 17920 12320 10080 14560 13440 15680 25760 12320 11200 7840 2013 8960 11200 10080 7840 14560 10080 29120 15680 15680 8960 12320 22400 "Pivot" is about analyzing the combination of a pair of selected dimensions. For example, we want to analyze the revenue by year and month. Or we want to analyze the revenue by product and location. apply(revenue_cube, c("year", "month"), FUN=function(x) {return(sum(x, na.rm=TRUE))}) month year 1 2 3 4 5 6 7 8 9 10 11 12 2012 17950 31695 20065 17885 16890 19210 16155 20330 29510 16280 15970 10885 2013 12350 16660 15075 13975 19660 13695 31715 20900 18845 14630 20135 27500 apply(revenue_cube, c("prod", "loc"), FUN=function(x) {return(sum(x, na.rm=TRUE))}) loc prod CA NY ON QU WA Laptop 16425 9450 7650 7425 6525 Printer 15390 19950 7980 10830 10260 Tablet 90720 117600 45920 34720 57120 I hope you can get a taste of the richness of data processing model in R. However, since R is doing all the processing in RAM. This requires your data to be small enough so it can fit into the local memory in a single machine.

Definition : JMS : Java Message Service is an API that is part of Java EE for sending messages between two or more clients. There are many JMS providers such as OpenMQ (glassfish’s default), HornetQ(Jboss), and ActiveMQ. RabbitMQ: is an open source message broker software which uses the AMQP standard and is written by Erlang. Messaging Model: JMS supports two models: one to one and publish/subscriber. RabbitMQ supports the AMQP model which has 4 models : direct, fanout, topic, headers. Data types: JMS supports 5 different data types but RabbitMQ supports only the binary data type. Workflow strategy: In AMQP, producers send to the exchange then the queue, but in JMS, producers send to the queue or topic directly. Technology compatibility: JMS is specific for java users only, but RabbitMQ supports many technologies. Performance: If you would like to know more about their performance, this benchmark is a good place to start, but look for others as well.

This is a continuation of an introductory discussion on generics, previous parts of which can be found here. In this post I will focus on type parameter bounds and their usage. Bounded Type Parameters When a generic type is compiled, all occurrences of a type parameter are removed by the compiler and replaced by a concrete type. The compiler also generates appropriate casting needed for type safety by itself during this procedure. This concrete type is typically Object, but compiler can use other types as well. This process is called Type Erasure and will be explained in a future post. For the time being, all we need to understand is that the type information of generic types are lost once they are compiled. For this reason, if we want to access a method/property using a type parameter, we’ll typically be able to access those ones that are defined in the class Object (I am oversimplifying here as we’ll be able to access other methods/properties as well if we use a bound, which we will discuss here in this post). For example, take a look at the following code snippet – public class MyGenericClass { private E prop; public MyGenericClass(E prop) { this.prop = prop; } public E getProp() { return this.prop; } public void printProp() { //OK, because toString is defined within Object System.out.println(this.prop.toString()); } public int getValue() { /** * NOT OK, because Object doesn’t have * compareTo method. Compile-time Error. */ return this.prop.intValue(); } } After compiling the above class, I get the following message – MyGenericClass.java:37: error: cannot find symbol return this.prop.intValue(); ^ symbol: method intValue(E) location: variable prop of type E where E is a type-variable: E extends Object declared in class MyGenericClass 1 error Although the message seems cryptic, the reason behind this is the one that I’ve stated above – when this type is compiled the type parameter E here will be replaced by Object by the compiler, and it doesn’t have intValue method. So the problem occurs here because the compiler is using Object to replace the type parameters. If I could somehow tell the compiler to use other types during this erasure which has an intValue (Number, for example) method, then this error would have been resolved. This is exactly what parameter bounds do. By using a type as a bound on a type parameter, I can instruct compiler to use that type during the erasure in place of Object, and then I can easily access the methods/properties defined in that type. The general syntax for specifying a type parameter bound is as follows – public class MyGenericClass This also tells the compiler that when a type argument is passed during an instance creation of this generic type, it will be a subtype of MyBoundType, so it can safely let us access the methods that are defined in that type using the type parameter E. If any other type is passed, then the compiler will issue a compile-time error. The extends keyword specify the bound relation between E and MyBoundType. We will use the same keyword even if E is bounded by an interface type, that is, if MyBoundType is an interface. Here extends means both classical extends and implements. So, if we use Number as our parameter bound for our last example, the error message will be gone because now the compiler will use Number to erase type parameter E, and it has an intValue method defined in it - public class MyGenericClass { private E prop; public MyGenericClass(E prop) { this.prop = prop; } public E getProp() { return this.prop; } public void printProp() { // OK, because toString is defined within Object System.out.println(this.prop.toString()); } public int getValue() { // Now it compiles just fine! return this.prop.intValue(); } } This code will now compile just fine. Remember our generic Insertion Sort algorithm from the first part of the series? We had declared it like this – public class InsertionSort> This told the compiler that the type arguments that will be passed here will all implement the Comparable interface, so they will have a compareTo method. As a result, compiler allowed us to do this inside the sort method – for (int i = 1; i <= elements.length - 1; i++) { E valueToInsert = elements[i]; int holePos = i; /** * See how we are calling compareTo method * on the type parameter? */ while (holePos > 0 && valueToInsert.compareTo(elements[holePos - 1]) < 0) { elements[holePos] = elements[holePos - 1]; holePos--; } elements[holePos] = valueToInsert; } This example also shows that we can pass another generic type as a type parameter bound. In fact we can use all Classes, Interfaces and Enums and even another type parameter as a bound. Only primitive and array types are not allowed as a bound. Multiple and Recursive Bounds We can define multiple bounds on a single parameter. In this case we use the & operator to list them in the following way - public class MyGenericClass This tells the compiler that the type argument that is passed will be a subtype of MyBoundClass and implements MyBoundInterface. So, we will be able to access all the methods/properties defined in those types. The Java Language Specification requires us to list the class bound first, otherwise the compiler will throw an error. For example, the following will throw an error – /** * Will throw an error because we are not * listing the class bound first. */ public class MyGenericClass We can also declare recursive bounds, so that a bound can depend on itself too. Consider our sorting example from first part of the series. We declared it like this – public class InsertionSort> Here, the bound is recursive because E itself depends upon E (the one that is supplied to Comparator). If we passInteger as a type argument when creating an instance of InsertionSort, then the type argument to Comparable will beInteger too. We can also declare mutually recursive bounds like this – public class MyGenericClass , U extends SecondType> Java Enum Declaration Let us now consider an example from the Java API itself. We all know that enumerations in Java are all objects of a class, and that class extends the Enum class. The declaration of that class looks like this – public abstract class Enum> implements Comparable, Serializable Beginners in Java Generics find the first line very confusing. Before explaining the reasoning behind this weird declaration, let us explore an example. Suppose that we are going to build a software system which will have various types of vehicles (a vehicle simulation system, perhaps?). The vehicles will have a name and a length. We also want to compare vehicles with each other based on their lengths. An approach for building the vehicle classes might be something like this – public abstract class Vehicle { private String name; private double length; public String getName() { return name; } public void setName(String name) { this.name = name; } public double getLength() { return length; } public void setLength(double length) { this.length = length; } } public class Car extends Vehicle implements Comparable { public int compareTo(Car anotherCar) { double thisLength = this.getLength(); double thatLength = anotherCar.getLength(); if (thisLength > thatLength) return 1; else if (thisLength < thatLength) return -1; return 0; } // other methods and properties } public class Bus extends Vehicle implements Comparable { public int compareTo(Bus anotherBus) { double thisLength = this.getLength(); double thatLength = anotherBus.getLength(); if (thisLength > thatLength) return 1; else if (thisLength < thatLength) return -1; return 0; } // other methods and properties } The problem of the above implementation is pretty obvious – even though the comparing logics are almost the same among all the subtypes of Vehicle, it’s duplicated in all of them. This creates a maintenance problem as now changing the comparison logic forces us to change the code in many places. To remedy this, we can remove the comparison from the subtypes and move it up in the Vehicle. To do this, we will rewrite those classes as follows – public abstract class Vehicle implements Comparable { private String name; private double length; public String getName() { return name; } public void setName(String name) { this.name = name; } public double getLength() { return length; } public void setLength(double length) { this.length = length; } public int compareTo(Vehicle anotherVehicle) { double thisLength = this.getLength(); double thatLength = anotherVehicle.getLength(); if (thisLength > thatLength) return 1; else if (thisLength < thatLength) return -1; return 0; } } public class Car extends Vehicle { // car-specific methods and properties } public class Bus extends Vehicle { // bus-specific methods and properties } This approach has also a problem. The above implementation will allow us to compare a car with a bus without any errors – Car car = new Car(); car.setName("Toyota"); car.setLength(2); Bus bus = new Bus(); bus.setName("Volvo"); bus.setLength(4); car.compareTo(bus); // No error This is certainly a problem, since comparing a bus with a car should not be done using the same logic that is used to compare a car with a car. How can we solve this? How can we reuse the comparison logic among all the subtypes, while at the same time raising error flags at compile time whenever someone tries to compare two incompatible types? In the last example the problem occurred because the compareTo method has a parameter which is of type Vehicle. As a result we were able to pass any subtypes of Vehicle to it, like the way we passed a bus to compare with a car. Let’s try to change the type of this parameter so that now this kind of mixing generates an error. If we want to allow the comparison of a car only with a car, then the argument to the compareTo method must be of type Car. If we change it to Car, we will also need to change the type argument that we are passing to Comparable in the Vehicle class declaration – public abstract class Vehicle implements Comparable { // other methods and properties public int compareTo(Car car) { // method implementation } } But then this will not allow us to compare any other types. We will not be able to compare a bus with another bus. To allow this, we will need to change the parameter to be of type Bus. If we declare a new subtype named Cycle, we will also need this method to support this type too! So we can see that the parameter type of this compare method should vary if we need to enforce compatible comparison. From the above discussion it’s clear that we need to parameterize the parameter type of the compareTo method, and in turn, parameterize the Vehicle class itself. If we do this, we will then be able to pass Car, Bus, and Cycle etc. as its type argument, which in turn will be used as the parameter type of the compare method. In general, after we declare Vehicleas a generic type, all of its subtypes will pass themselves as a type argument while extending from it, so that the parameter type of this compareTo method matches their type – public abstract class Vehicle implements Comparable { // other methods and properties public int compareTo(E vehicle) { // method implementation } } /** * Now this class’s compareTo version will take a Car type * as its argument. */ public class Car extends Vehicle { // car specific method and properties } /** * Now this class’s compareTo version will take a Bus type * as its argument. */ public class Bus extends Vehicle { // bus specific method and properties } /** * Doing something like this will now generate a * compile-time error. */ car.compareTo(bus); This approach solves our last problem that we were facing, but introduces a new one. After converting Vehicle a generic type and using the type parameter as the parameter type of the compare method, it looks like this – public int compareTo(E anotherVehicle) { double thisLength = this.getLength(); // Now the following line is an error. double thatLength = anotherVehicle.getLength(); if (thisLength > thatLength) return 1; else if (thisLength < thatLength) return -1; return 0; } Since we didn’t put any bound on the type parameter, and Object class doesn’t have a getLength method, compiler will generate an error. We get to call this method on an object of type E only if it’s bounded by Vehicle itself, because then compiler will know that objects of this type will have this method. So our compare method will work only if E is bounded by Vehicle itself! After this modification, the classes look like below – public abstract class Vehicle> implements Comparable { private String name; private double length; public String getName() { return name; } public void setName(String name) { this.name = name; } public double getLength() { return length; } public void setLength(double length) { this.length = length; } public int compareTo(E anotherVehicle) { double thisLength = this.getLength(); double thatLength = anotherVehicle.getLength(); if (thisLength > thatLength) return 1; else if (thisLength < thatLength) return -1; return 0; } } public class Car extends Vehicle { // Car-specific properties and methods } public class Bus extends Vehicle { // Bus-specific properties and methods } // and in main Car car = new Car(); car.setName("Toyota"); car.setLength(2); Bus bus = new Bus(); bus.setName("Volvo"); bus.setLength(4); car.compareTo(car); // Works as expected car.compareTo(bus); // compile-time error Even with the above example, a certain kind of type mixing is possible. Rather than discussing it here, I am going to leave it to you to figure it out. If you can’t, check out the next post of this series! I guess now you know why the Enum class is declared in that way. This kind of recursive bound allows us to write methods in a supertype which will take its subtypes as its arguments, or return them as return value. I encourage you to check out the source code of the Enum class to find out these methods. That’s it for today. Stay tuned for the next post! Resources Java Generics and Collections Java Generics FAQs by Angelika Langer

this is an old yet still popular question. there are multiple reasons that string is designed to be immutable in java. a good answer depends on good understanding of memory, synchronization, data structures, etc. in the following, i will summarize some answers. 1. requirement of string pool string pool (string intern pool) is a special storage area in java heap. when a string is created and if the string already exists in the pool, the reference of the existing string will be returned, instead of creating a new object and returning its reference. the following code will create only one string object in the heap. string string1 = "abcd"; string string2 = "abcd"; here is how it looks: if string is not immutable, changing the string with one reference will lead to the wrong value for the other references. 2. allow string to cache its hashcode the hashcode of string is frequently used in java. for example, in a hashmap. being immutable guarantees that hashcode will always the same, so that it can be cashed without worrying the changes.that means, there is no need to calculate hashcode every time it is used. this is more efficient. 3. security string is widely used as parameter for many java classes, e.g. network connection, opening files, etc. were string not immutable, a connection or file would be changed and lead to serious security threat. the method thought it was connecting to one machine, but was not. mutable strings could cause security problem in reflection too, as the parameters are strings. here is a code example: boolean connect(string s){ if (!issecure(s)) { throw new securityexception(); } //here will cause problem, if s is changed before this by using other references. causeproblem(s); } in summary, the reasons include design, efficiency, and security. actually, this is also true for many other “why” questions in a java interview.

Node.js https module used to make a remote call to a remote server using https and BASIC authentication: var options = { host: 'test.example.com', port: 443, path: '/api/service/'+servicename, // authentication headers headers: { 'Authorization': 'Basic ' + new Buffer(username + ':' + passw).toString('base64') } }; //this is the call request = https.get(options, function(res){ var body = ""; res.on('data', function(data) { body += data; }); res.on('end', function() { //here we have the full response, html or json object console.log(body); }) res.on('error', function(e) { onsole.log("Got error: " + e.message); }); }); }

okay, i know that i have been critical about the lmdb codebase so far. but one thing that i really want to point out for it is that it was pretty easy to actually get things working on windows. it wasn’t smooth, in the sense that i had to muck around with the source a bit (hack endianess, remove a bunch of unix specific header files, etc). but that took less than an hour, and it was pretty much it. since i am by no means an experienced c developer, i consider this a major win. compare that to leveldb, which flat out won’t run on windows no matter how much time i spent trying, and it is a pleasure . also, stepping through the code i am starting to get a sense of how it works that is much different than the one i had when i just read the code. it is like one of those 3d images, you suddenly see something. the first thing that became obvious is that i totally missed the significance of the lock file. lmdb actually create two files: lock.mdb data.mdb lock.mdb is used to synchronized data between different readers. it seems to mostly be there if you want to have multiple writers using different processes. that is a very interesting model for an embedded database, i’ve to admit. not something that i think other embedded databases are offering. in order to do that, it create two named mutexes (one for read and one for write). a side note on windows support: lmdb supports windows, but it is very much a 2nd class citizen. you can see it in things like path not found error turning into a no such process error (because it try to use getlasterror() codes as c codes), or when it doesn’t create a directory even though not creating it would fail. i am currently debugging through the code and fixing such issues as i go along (but no, i am doing heavy handed magic fixes, just to get past this stage to the next one, otherwise i would have sent a pull request). here is one such example. here is the original code: but readfile in win32 will return false if the file is empty, so you actually need to write something like this to make the code work: past that hurdle, i think that i get a lot more about what is going on with the way lmdb works than before. let us start with the way data.mdb works. it is important to note that for pretty much everything in lmdb we use the system page size. by default, that is 4kb. the data file starts with 2 pages allocated. those page contain the following information: looking back at how couchdb did things, i am pretty sure that those two pages are going to be pretty important . i am guess that they would always contain the root of the data in the file. there is also the last transaction on them, which is what i imagine determine how something gets committed. i don’t know yet, as i said, guessing based on how couchdb works. i’ll continue this review in another time. next time, transactions…

This time we’re up for a bigger challenge. We want to automatically display and possibly search and filter std::map objects in WinDbg. The script for std::vectors was relatively easy because of the flat structure of the data in a vector; maps are more complex beasts. Specifically, an map in the Visual C++ STL is implemented as a red-black tree. Each tree node has three important pointers: _Left, _Right, and _Parent. Additionally, each node has a _Myval field that contains the std::pair with the key and value represented by the node. Iterating a tree structure requires recursion, and WinDbg scripts don’t have any syntax to define functions. However, we can invoke a script recursively – a script is allowed to contain the $$>a< command that invokes it again with a different set of arguments. The path to the script is also readily available in ${$arg0}. Before I show you the script, there’s just one little challenge I had to deal with. When you call a script recursively, the values of the pseudo-registers (like $t0) will be clobbered by the recursive invocation. I was on the verge of allocating memory dynamically or calling into a shell process to store and load variables, when I stumbled upon the .push and .pop commands, which store the register context and load it, respectively. These are a must for recursive WinDbg scripts. OK, so suppose you want to display values from an std::map where the key is less than or equal to 2. Here we go: 0:000> $$>a< traverse_map.script my_map -c ".block { .if (@@(@$t9.first) <= 2) { .echo ----; ?? @$t9.second } }" size = 10 ---- struct point +0x000 x : 0n1 +0x004 y : 0n2 +0x008 data : extra_data ---- struct point +0x000 x : 0n0 +0x004 y : 0n1 +0x008 data : extra_data ---- struct point +0x000 x : 0n2 +0x004 y : 0n3 +0x008 data : extra_data For each pair (stored in the $t9 pseudo-register), the block checks if the first component is less than or equal to 2, and if it is, outputs the second component. Next, here’s the script. Note it’s considerably more complex that what we had to with vectors, because it essentially invokes itself with a different set of parameters and then repeats recursively. .if ($sicmp("${$arg1}", "-n") == 0) { .if (@@(@$t0->_Isnil) == 0) { .if (@$t2 == 1) { .printf /D "%p\n", @$t0, @$t0 .printf "key = " ?? @$t0->_Myval.first .printf "value = " ?? @$t0->_Myval.second } .else { r? $t9 = @$t0->_Myval command } } $$ Recurse into _Left, _Right unless they point to the root of the tree .if (@@(@$t0->_Left) != @@(@$t1)) { .push /r /q r? $t0 = @$t0->_Left $$>a< ${$arg0} -n .pop /r /q } .if (@@(@$t0->_Right) != @@(@$t1)) { .push /r /q r? $t0 = @$t0->_Right $$>a< ${$arg0} -n .pop /r /q } } .else { r? $t0 = ${$arg1} .if (${/d:$arg2}) { .if ($sicmp("${$arg2}", "-c") == 0) { r $t2 = 0 aS ${/v:command} "${$arg3}" } } .else { r $t2 = 1 aS ${/v:command} " " } .printf "size = %d\n", @@(@$t0._Mysize) r? $t0 = @$t0._Myhead->_Parent r? $t1 = @$t0->_Parent $$>a< ${$arg0} -n ad command } Of particular note are the aS command which configures an alias that is then used by the recursive invocation to invoke a command block for each of the map’s elements; the $sicmp function which compares strings; and the .printf /D function, which outputs a chunk of DML. Finally, the recursion terminates when _Left or _Right are equal to the root of the tree (that’s just how the tree is implemented in this case).

originally written by paul chapman there are two ways to generate output using spring mvc: you can use the restful @responsebody approach and http message converters, typically to return data-formats like json or xml. programmatic clients, mobile apps and ajax enabled browsers are the usual clients. alternatively you may use view resolution . although views are perfectly capable of generating json and xml if you wish (more on that in my next post), views are normally used to generate presentation formats like html for a traditional web-application. actually there is a third possibility – some applications require both, and spring mvc supports such combinations easily. we will come back to that right at the end. in either case you'll need to deal with multiple representations (or views) of the same data returned by the controller. working out which data format to return is called content negotiation . there are three situations where we need to know what type of data-format to send in the http response: httpmessageconverters: determine the right converter to use. request mappings: map an incoming http request to different methods that return different formats. view resolution: pick the right view to use. determining what format the user has requested relies on a contentnegotationstrategy . there are default implementations available out of the box, but you can also implement your own if you wish. in this post i want to discuss how to configure and use content negotiation with spring, mostly in terms of restful controllers using http message converters. in a later post i will show how to setup content negotiation specifically for use with views using spring's contentnegotiatingviewresolver . how does content negotiation work? getting the right content when making a request via http it is possible to specify what type of response you would like by setting the accept header property. web browsers have this preset to request html (among other things). in fact, if you look, you will see that browsers actually send very confusing accept headers, which makes relying on them impractical. see http://www.gethifi.com/blog/browser-rest-http-accept-headers for a nice discussion of this problem. bottom-line: accept headers are messed up and you can't normally change them either (unless you use javascript and ajax). so, for those situations where the accept header property is not desirable, spring offers some conventions to use instead. (this was one of the nice changes in spring 3.2 making a flexible content selection strategy available across all of spring mvc not just when using views). you can configure a content negotiation strategy centrally once and it will apply wherever different formats (media types) need to be determined. enabling content negotiation in spring mvc spring supports a couple of conventions for selecting the format required: url suffixes and/or a url parameter. these work alongside the use of accept headers. as a result, the content-type can be requested in any of three ways. by default they are checked in this order: add a path extension (suffix) in the url. so, if the incoming url is something like http://myserver/myapp/accounts/list.html then html is required. for a spreadsheet the url should be http://myserver/myapp/accounts/list.xls . the suffix to media-type mapping is automatically defined via the javabeans activation framework or jaf (so activation.jar must be on the class path). a url parameter like this: http://myserver/myapp/accounts/list?format=xls . the name of the parameter is format by default, but this may be changed. using a parameter is disabled by default, but when enabled, it is checked second. finally the accept http header property is checked. this is how http is actually defined to work, but, as previously mentioned, it can be problematic to use. the java configuration to set this up, looks like this. simply customize the predefined content negotiation manager via its configurer. note the mediatype helper class has predefined constants for most well-known media-types. @configuration @enablewebmvc public class webconfig extends webmvcconfigureradapter { /** * setup a simple strategy: use all the defaults and return xml by default when not sure. */ @override public void configurecontentnegotiation(contentnegotiationconfigurer configurer) { configurer.defaultcontenttype(mediatype.application_xml); } } when using xml configuration, the content negotiation strategy is most easily setup via the contentnegotiationmanagerfactorybean : the contentnegotiationmanager created by either setup is an implementation of contentnegotationstrategy that implements the ppa strategy (path extension, then parameter, then accept header) described above. additional configuration options in java configuration, the strategy can be fully customized using methods on the configurer: @configuration @enablewebmvc public class webconfig extends webmvcconfigureradapter { /** * total customization - see below for explanation. */ @override public void configurecontentnegotiation(contentnegotiationconfigurer configurer) { configurer.favorpathextension(false). favorparameter(true). parametername("mediatype"). ignoreacceptheader(true). usejaf(false). defaultcontenttype(mediatype.application_json). mediatype("xml", mediatype.application_xml). mediatype("json", mediatype.application_json); } } in xml, the strategy can be configured using methods on the factory bean: what we did, in both cases: disabled path extension. note that favor does not mean use one approach in preference to another, it just enables or disables it. the order of checking is always path extension, parameter, accept header. enable the use of the url parameter but instead of using the default parameter, format , we will use mediatype instead. ignore the accept header completely. this is often the best approach if most of your clients are actually web-browsers (typically making rest calls via ajax). don't use the jaf, instead specify the media type mappings manually – we only wish to support json and xml. listing user accounts example to demonstrate, i have put together a simple account listing application as our worked example – the screenshot shows a typical list of accounts in html. the complete code can be found at github: https://github.com/paulc4/mvc-content-neg . to return a list of accounts in json or xml, i need a controller like this. we will ignore the html generating methods for now. @controller class accountcontroller { @requestmapping(value="/accounts", method=requestmethod.get) @responsestatus(httpstatus.ok) public @responsebody list list(model model, principal principal) { return accountmanager.getaccounts(principal) ); } // other methods ... } here is the content-negotiation strategy setup: or, using java configuration, the code looks like this: @override public void configurecontentnegotiation( contentnegotiationconfigurer configurer) { // simple strategy: only path extension is taken into account configurer.favorpathextension(true). ignoreacceptheader(true). usejaf(false). defaultcontenttype(mediatype.text_html). mediatype("html", mediatype.text_html). mediatype("xml", mediatype.application_xml). mediatype("json", mediatype.application_json); } provided i have jaxb2 and jackson on my classpath, spring mvc will automatically setup the necessary httpmessageconverters . my domain classes must also be marked up with jaxb2 and jackson annotations to enable conversion (otherwise the message converters don't know what to do). in response to comments (below), the annotated account class is shown below . here is the json output from our accounts application (note path-extension in url). how does the system know whether to convert to xml or json? because of content negotiation – any one of the three ( ppa strategy ) options discussed above will be used depending on how the contentnegotiationmanager is configured. in this case the url ends in accounts.json because the path-extension is the only strategy enabled. in the sample code you can switch between xml or java configuration of mvc by setting an active profile in the web.xml . the profiles are "xml" and "javaconfig" respectively. combining data and presentation formats spring mvc's rest support builds on the existing mvc controller framework. so it is possible to have the same web-applications return information both as raw data (like json) and using a presentation format (like html). both techniques can easily be used side by side in the same controller, like this: @controller class accountcontroller { // restful method @requestmapping(value="/accounts", produces={"application/xml", "application/json"}) @responsestatus(httpstatus.ok) public @responsebody list listwithmarshalling(principal principal) { return accountmanager.getaccounts(principal); } // view-based method @requestmapping("/accounts") public string listwithview(model model, principal principal) { // call restful method to avoid repeating account lookup logic model.addattribute( listwithmarshalling(principal) ); // return the view to use for rendering the response return ¨accounts/list¨; } } there is a simple pattern here: the @responsebody method handles all data access and integration with the underlying service layer (the accountmanager ). the second method calls the first and sets up the response in the model for use by a view. this avoids duplicated logic. to determine which of the two @requestmapping methods to pick, we are again using our ppa content negotiation strategy. it allows the produces option to work. urls ending with accounts.xml or accounts.json map to the first method, any other urls ending in accounts.anything map to the second. another approach alternatively we could do the whole thing with just one method if we used views to generate all possible content-types. this is where the contentnegotiatingviewresolver comes in and that will be the subject of my next post . acknoweldgements i would like to thank rossen stoyanchev for his help in writing this post. any errors are my own. addendum: the annotated account class added 2 june 2013 . since there were some questions on how to annotate a class for jaxb, here is part of the account class. for brevity i have omitted the data-members, and all methods except the annotated getters. i could annotate the data-members directly if preferred (just like jpa annotations in fact). remember that jackson can marshal objects to json using these same annotations. /** * represents an account for a member of a financial institution. an account has * zero or more {@link transaction}s and belongs to a {@link customer}. an aggregate entity. */ @entity @table(name = "t_account") @xmlrootelement public class account { // data-members omitted ... public account(customer owner, string number, string type) { this.owner = owner; this.number = number; this.type = type; } /** * returns the number used to uniquely identify this account. */ @xmlattribute public string getnumber() { return number; } /** * get the account type. * * @return one of "credit", "savings", "check". */ @xmlattribute public string gettype() { return type; } /** * get the credit-card, if any, associated with this account. * * @return the credit-card number or null if there isn't one. */ @xmlattribute public string getcreditcardnumber() { return stringutils.hastext(creditcardnumber) ? creditcardnumber : null; } /** * get the balance of this account in local currency. * * @return current account balance. */ @xmlattribute public monetaryamount getbalance() { return balance; } /** * returns a single account transaction. callers should not attempt to hold * on or modify the returned object. this method should only be used * transitively; for example, called to facilitate reporting or testing. * * @param name * the name of the transaction account e.g "fred smith" * @return the beneficiary object */ @xmlelement // make these a nested element public set gettransactions() { return transactions; } // setters and other methods ... }

Just about every 6 months or so an article appears cursing Maven, attracting both proponents as opponents to Maven and Ant. While it’s real fun to watch (I really get a laugh when people start to advocate the return to Ant), most of the time it’s always the same arguments. Maven lacks flexibility, the plugin system sucks (when will people learn to use plugin versions…), you can’t use scripting and the all time favorite: the release plugin sucks. Well, I am a Maven addict and I’m happy to say: yes, I agree, the release plugin sucks. Big time. But here’s something you may have forgotten: you don’t need it! Even more: you shouldn’t use it. The Maven release plugin tries to make releasing software a breeze. That’s where the plugin authors got it wrong to start with. Releases are not something done on a whim. They are carefully planned and orchestrated actions, preceded by countless rules and followed by more rules. Assuming you can bundle all that in a simple mvn release:release is just plain naive. Even Maven’s most fierce supporters agree on this. The Maven release plugin just tries to do too much stuff at once: build your software, tag it, build it again, deploy it, build the site (triggering yet another build in the process) and deploy the site. And whilst doing that, running the tests x times. Most of the time, you’re making candidate releases, so building the complete documentation is a complete waste of time. Now, if you break down the release plugin into sensible steps, you’ll really save yourself a whole lot of trouble. I use these steps to release something. As a sidenote: I use git and git-flow standards (as described here). Assume the POM’s version’s currently on 1.0-SNAPSHOT. Announce the release process Very important. As I said, you don’t release on a whim. Make sure everyone on your team knows a release is pending and has all their stuff pushed to the development branch that needs to be included. Branch the development branch into a release branch. Following git-flow rules, I make a release branch 1.0. Update the POM version of the development branch. Update the version to the next release version. For example mvn versions:set -DnewVersion=2.0-SNAPSHOT. Commit and push. Now you can put resources developing towards the next release version. Update the POM version of the release branch. Update the version to the standard CR version. For example mvn versions:set -DnewVersion=1.0.CR-SNAPSHOT. Commit and push. Run tests on the release branch. Run all the tests. If one or more fail, fix them first. Create a candidate release from the release branch. Use the Maven version plugin to update your POM’s versions. For example mvn versions:set -DnewVersion=1.0.CR1. Commit and push. Make a tag on git. Use the Maven version plugin to update your POM’s versions back to the standard CR version. For example mvn versions:set -DnewVersion=1.0.CR-SNAPSHOT. Commit and push. Checkout the new tag. Do a deployment build (mvn clean deploy). Since you’ve just run your tests and fixed any failing ones, this shouldn’t fail. Put deployment on QA environment. Iterate until QA gives a green light on the candidate release. Fix bugs. Fix bugs reported on the CR releases on the release branch. Merge into development branch on regular intervals (or even better, continuous). Run tests continuously, making bug reports on failures and fixing them as you go. Create a candidate release. Use the Maven version plugin to update your POM’s versions. For example mvn versions:set -DnewVersion=1.0.CRx. Commit and push. Make a tag on git. Use the Maven version plugin to update your POM’s versions back to the standard CR version. For example mvn versions:set -DnewVersion=1.0.CR-SNAPSHOT. Commit and push. Checkout the new tag. Do a deployment build (mvn clean deploy). Since you’ve run your tests continuously, this shouldn’t fail. Put deployment on QA environment. Once QA has signed off on the release, create a final release. Check whether there are no new commits since the last release tag (if there are, slap developers as they have done stuff that wasn’t needed or asked for). Use the Maven version plugin to update your POM’s versions. For example mvn versions:set -DnewVersion=1.0. Commit and push. Tag the release branch. Merge into the master branch. Checkout the master branch. Do a deployment build (mvn clean deploy). Start production release and deployment process (in most companies, not a small feat). This can involve building the site and doing other stuff, some not even Maven related. There’s no way in hell Maven can automate this process and if you try, you’ll bump into the many pitfalls the release plugin has to offer. The release plugin is just a combination of the versions, scm, deploy and site plugin that seriously violates the single responsibility principle. The release plugin is one of the reasons Maven has gotten a bad reputation with some people. It’s long due for an overhaul, but if you ask me, they should just remove it altogether. Releasing software is a process, not a single command on the command line. The process I just described isn’t perfect in any way, but it works and I avoid using the release plugin as it just does too much stuff. Have fun bashing Maven, but please, keep it clean :) .

Recently I’ve been doing some x64 assembly hacking, and something I had to Google a bit and collect from a few places is how to go over all command-line arguments (colloquially known as argv from C) and do something with them. I already discussed how arguments get passed into a program in the past (not the C main, mind you, but rather the real entry point of a program – _start), so what was left is just a small matter of implementation. Here it is, in GNU Assembly (gas) syntax for Linux. This is pure assembly code – it does not use the C standard library or runtime at all. It demonstrates a lot of interesting concepts such as reading command-line arguments, issuing Linux system calls and string processing. #---------------- DATA ----------------# .data # We need buf_for_itoa to be large enough to contain a 64-bit integer. # endbuf_for_itoa will point to the end of buf_for_itoa and is useful # for passing to itoa. .set BUFLEN, 32 buf_for_itoa: .space BUFLEN, 0x0 .set endbuf_for_itoa, buf_for_itoa + BUFLEN - 1 newline_str: .asciz "\n" argc_str: .asciz "argc: " #---------------- CODE ----------------# .globl _start .text _start: # On entry to _start, argc is in (%rsp), argv[0] in 8(%rsp), # argv[1] in 16(%rsp) and so on. lea argc_str, %rdi call print_cstring mov (%rsp), %r12 # save argc in r12 # Convert the argc value to a string and print it out mov %r12, %rdi lea endbuf_for_itoa, %rsi call itoa mov %rax, %rdi call print_cstring lea newline_str, %rdi call print_cstring # In a loop, pick argv[n] for 0 <= n < argc and print it out, # followed by a newline. r13 holds n. xor %r13, %r13 .L_argv_loop: mov 8(%rsp, %r13, 8), %rdi # argv[n] is in (rsp + 8 + 8*n) call print_cstring lea newline_str, %rdi call print_cstring inc %r13 cmp %r12, %r13 jl .L_argv_loop # exit(0) mov $60, %rax mov $0, %rdi syscall This code uses a couple of support functions. The first is print_cstring: # Function print_cstring # Print a null-terminated string to stdout. # Arguments: # rdi address of string # Returns: void print_cstring: # Find the terminating null mov %rdi, %r10 .L_find_null: cmpb $0, (%r10) je .L_end_find_null inc %r10 jmp .L_find_null .L_end_find_null: # r10 points to the terminating null. so r10-rdi is the length sub %rdi, %r10 # Now that we have the length, we can call sys_write # sys_write(unsigned fd, char* buf, size_t count) mov $1, %rax # Populate address of string into rsi first, because the later # assignment of fd clobbers rdi. mov %rdi, %rsi mov $1, %rdi mov %r10, %rdx syscall ret More interestingly, here is itoa. It’s a bit more general than what I actually use in the main program because it also supports negative numbers. It can convert any number that fits into a 64-bit register. Note the unusual API for receiving and returning the place where the actual string is written. Since it’s very natural for an itoa implementation to emit the digits in reverse, I wanted to avoid actual string reversing by writing the digits into a buffer from the end towards the beginning. # Function itoa # Convert an integer to a null-terminated string in memory. # Assumes that there is enough space allocated in the target # buffer for the representation of the integer. Since the number itself # is accepted in the register, its value is bounded. # Arguments: # rdi: the integer # rsi: address of the *last* byte in the target buffer # Returns: # rax: address of the first byte in the target string that # contains valid information. itoa: movb $0, (%rsi) # Write the terminating null and advance. dec %rsi # If the input number is negative, we mark it by placing 1 into r9 # and negate it. In the end we check if r9 is 1 and add a '-' in front. mov $0, %r9 cmp $0, %rdi jge .L_input_positive neg %rdi mov $1, %r9 .L_input_positive: mov %rdi, %rax # Place the number into rax for the division. mov $10, %r8 # The base is in r8 .L_next_digit: # Prepare rdx:rax for division by clearing rdx. rax remains from the # previous div. rax will be rax / 10, rdx will be the next digit to # write out. xor %rdx, %rdx div %r8 # Write the digit to the buffer, in ascii dec %rsi add $0x30, %dl movb %dl, (%rsi) cmp $0, %rax # We're done when the quotient is 0. jne .L_next_digit # If we marked in r9 that the input is negative, it's time to add that # '-' in front of the output. cmp $1, %r9 jne .L_itoa_done dec %rsi movb $0x2d, (%rsi) .L_itoa_done: mov %rsi, %rax # rsi points to the first byte now; return it. ret Some notes about the code: GAS vs. Intel syntax: I used to believe the Intel syntax is better looking, but grew to tolerate GAS because it’s the default used by tools on Linux. After a very short time you get used to it and don’t really mind it any longer. Yes, even the weird indirect addressing syntax (mov 8(%rsp, %r13, 8), %rdi) grows on you. In other words, focus on the code, not syntax. I could pick any representation for strings, but ended up going with the C-like null-terminated strings. If you look carefully at print_cstring you’ll notice that a length-prefix representation could be better since the write system call doesn’t care about the null and wants the length passed explicitly. However, since real life assembly code often does have to inter-operate with C, null-terminated strings make more sense. Even though my own functions could use any calling convention, I’m sticking with the System V AMD64 ABI. It’s natural because system calls use it as well w.r.t. argument and return value passing. AFAIU they can also clobber scratch registers so care must be taken to preserve information in registers around system calls. Related posts: Creating a tiny ‘Hello World’ executable in assembly

The topic of how to deal with a slow consumer in ActiveMQ comes up on the mail list from time to time. In the current releases there is a strategy class that allows you to configure a time interval after which a slow consumer is closed (some nice documentation can be found here). The AbortSlowConsumerStrategy detects a consumer is slow by looking at its prefetch buffer and checking if its been to full for to long. When the consumer is aborted the messages that were in its prefetch buffer can then be sent on to other consumers on the same destination. The AbortSlowConsumerStrategy works well when you have consumers whose prefetch limit is set to a value greater than one, but for those cases where you've set a value of zero or one the consumer may never get marked as slow by the broker so the strategy might never kick in to abort the slow consumers. To solve the problem of slow consumers with small prefetch values we've introduced the AbortSlowAckConsumerStrategy. This strategy works by checking the time since a consumer last acknowledged a message as apposed to the time that the consumer has had a full prefetch buffer. The strategy will poll all the consumers of destination at a configurable rate to determine if the consumer is slow and if the consumer meets the configured criteria for abort then it will be given the boot and any prefetched messages will be redispatched. The AbortSlowAckConsumerStrategy is configured in the XML configuration file as follows: ... ... This configuration adds the AbortSlowAckConsumerStrategy with default options and applies it only to Topic consumers, for Queues you just need to replace the word 'topic' with 'queue' in the policy entry. There are various options that can be set on the strategy to determine when it aborts a slow consumer. Table 2. Settings for Abort Slow Consumer Strategy Attribute Default Description maxTimeSinceLastAck 30000 Specifies the amount of time that must elapse since the last message acknowledge before a consumer is marked as slow. ignoreIdleConsumers true Specifies whether the time since last acknowledge is applied to consumer that has no dispatched messages. If set false this strategy can be used to abort any consumer after the time since last acknowledge interval has expired. maxSlowCount -1 Specifies the number of times a consumer can be considered slow before it is aborted. -1 specifies that a consumer can be considered slow an infinite number of times. maxSlowDuration 30000 Specifies the maximum amount of time, in milliseconds, that a consumer can be continuously slow before it is aborted. checkPeriod 30000 Specifies, in milliseconds, the time between checks for slow consumers. abortConnection false Specifies whether the broker forces the consumer connection to close. The default value specifies that the broker will send a message to the consumer requesting it to close its connection. true specifies that the broker will automatically close the consumer's connection. If you read through the options above you should notice that it's possible to use this strategy not only to abort slow consumers that are receiving messages from the broker but also you can use this as a way to abort any consumer after a specified time interval. To accomplish this all you need to do is set the ignoreIdleConsumers option to false and the strategy will treat any consumer that hasn't acknowledged a message since the configured maxTimeSinceLastAck regardless of whether its had a message dispatched to it or not. Lets look at an example of how to abort any topic consumer that hasn't acknowledged a message in the last 60 seconds: ... ...

Circular dependencies and JSON have always been a pain. But it’s not just JSON, the problem also exists when you’re trying to serialize a graph which contains circular dependencies (parent/child with bidirectional relationships). Some time ago, we were considering exposing our JPA datamodel through REST services. Off course, a lot of JPA model classes contain bidirectional relationships, which was a real pain to get working. We ended up with a separate data model consisting of DTO’s (yuck!) and a mapping between the two models. But after a while we had to abandon our REST quest due to the fact that the JPA data model was getting to complicated. So we let go of the loose coupling between the client and the server, which made the issue go away completely. REST services where built when the need for external communication arose, but for client-server communication a more direct dependency was used (CDI/EJB or Spring injection). Recently, I once again looked at Jackson. My reasons now where a bit different. Our data model has grown to a point where finding out what exactly is in a graph is getting problematic. A simple SQL query doesn’t cut it anymore and we’re forced to start debugging in order to see what an object actually contains. Knowing in advance how a complex JPA datamodel is populated through a JQPL query is a science on its own. So I thought, why not have the possibility to send the same JPQL query and have the result returned to us as JSON. The problem, I thought, would be those wretched circular dependencies. Luckily, the Jackson developers have since developed a solution to the problem: their JSON serializer now supports object references. And it’s usable out-of-the-box for JPA datamodels. Their JSON object reference requires an object to have a unique ID. Luckily, this is also the case for JPA entities. However, JSON id references need to be unique across the entire graph, whereas JPA id’s only need to be unique within the same entity. In our case, it wasn’t really an issue, as we use UUID’s for JPA id fields, which are unique throughout the entire database. So how do you serialize an object graph? Well, assume you have two entities with bidirectional relationships like this: @Entity public class ParentEntity { @Id private String id; private String description; @OneToMany(mappedBy = "parent") private List children; // getters and setters omitted for brevity } @Entity public class ChildEntity { @Id private String id; private String description; @ManyToOne private Parent parent; // getters and setters omitted for brevity } Adding Jackson JSON identities is very simple: @Entity @JsonIdentityInfo(generator=ObjectIdGenerators.PropertyGenerator.class, property="id") public class ParentEntity { ... } @Entity @JsonIdentityInfo(generator=ObjectIdGenerators.PropertyGenerator.class, property="id") public class ChildEntity { ... } And that’s it! If you would now serialize a parent object with 2 children, you’ll get something like this: { "id": "parent-id1", "description": "parent", "children": [ { "id": "child-id1", "description": "child1", "parent": "parent-id1" }, { "id": "child-id2", "description": "child2", "parent": "parent-id1" } ] }

MongoDB is an open source document-oriented NoSQL database system which stores data as JSON-like documents with dynamic schemas. As it doesn't store data in tables as is done in the usual relational database setup, it doesn't map well to the JPA way of storing data. Morphia is an open source lightweight type-safe library designed to bridge the gap between the MongoDB Java driver and domain objects. It can be an alternative to SpringData if you're not using the Spring Framework to interact with MongoDB. This post will cover the basics of persisting and querying entities along the lines of JPA by using Morphia and a MongoDB database instance. There are four POJOs this example will be using. First we have BaseEntity which is an abstract class containing the Id and Version fields: package com.city81.mongodb.morphia.entity; import org.bson.types.ObjectId; import com.google.code.morphia.annotations.Id; import com.google.code.morphia.annotations.Property; import com.google.code.morphia.annotations.Version; public abstract class BaseEntity { @Id @Property("id") protected ObjectId id; @Version @Property("version") private Long version; public BaseEntity() { super(); } public ObjectId getId() { return id; } public void setId(ObjectId id) { this.id = id; } public Long getVersion() { return version; } public void setVersion(Long version) { this.version = version; } } Whereas JPA would use @Column to rename the attribute, Morphia uses @Property. Another difference is that @Property needs to be on the variable whereas @Column can be on the variable or the get method. The main entity we want to persist is the Customer class: package com.city81.mongodb.morphia.entity; import java.util.List; import com.google.code.morphia.annotations.Embedded; import com.google.code.morphia.annotations.Entity; @Entity public class Customer extends BaseEntity { private String name; private List accounts; @Embedded private Address address; public String getName() { return name; } public void setName(String name) { this.name = name; } public List getAccounts() { return accounts; } public void setAccounts(List accounts) { this.accounts = accounts; } public Address getAddress() { return address; } public void setAddress(Address address) { this.address = address; } } As with JPA, the POJO is annotated with @Entity. The class also shows an example of @Embedded: The Address class is also annotated with @Embedded as shown below: package com.city81.mongodb.morphia.entity; import com.google.code.morphia.annotations.Embedded; @Embedded public class Address { private String number; private String street; private String town; private String postcode; public String getNumber() { return number; } public void setNumber(String number) { this.number = number; } public String getStreet() { return street; } public void setStreet(String street) { this.street = street; } public String getTown() { return town; } public void setTown(String town) { this.town = town; } public String getPostcode() { return postcode; } public void setPostcode(String postcode) { this.postcode = postcode; } } Finally, we have the Account class of which the customer class has a collection of: package com.city81.mongodb.morphia.entity; import com.google.code.morphia.annotations.Entity; @Entity public class Account extends BaseEntity { private String name; public String getName() { return name; } public void setName(String name) { this.name = name; } } The above show only a small subset of what annotations can be applied to domain classes. More can be found at http://code.google.com/p/morphia/wiki/AllAnnotations The Example class shown below goes through the steps involved in connecting to the MongoDB instance, populating the entities, persisting them and then retrieving them: package com.city81.mongodb.morphia; import java.net.UnknownHostException; import java.util.ArrayList; import java.util.List; import com.city81.mongodb.morphia.entity.Account; import com.city81.mongodb.morphia.entity.Address; import com.city81.mongodb.morphia.entity.Customer; import com.google.code.morphia.Datastore; import com.google.code.morphia.Key; import com.google.code.morphia.Morphia; import com.mongodb.Mongo; import com.mongodb.MongoException; /** * A MongoDB and Morphia Example * */ public class Example { public static void main( String[] args ) throws UnknownHostException, MongoException { String dbName = new String("bank"); Mongo mongo = new Mongo(); Morphia morphia = new Morphia(); Datastore datastore = morphia.createDatastore(mongo, dbName); morphia.mapPackage("com.city81.mongodb.morphia.entity"); Address address = new Address(); address.setNumber("81"); address.setStreet("Mongo Street"); address.setTown("City"); address.setPostcode("CT81 1DB"); Account account = new Account(); account.setName("Personal Account"); List accounts = new ArrayList(); accounts.add(account); Customer customer = new Customer(); customer.setAddress(address); customer.setName("Mr Bank Customer"); customer.setAccounts(accounts); Key savedCustomer = datastore.save(customer); System.out.println(savedCustomer.getId()); } Executing the first few lines will result in the creation of a Datastore. This interface will provide the ability to get, delete and save objects in the 'bank' MongoDB instance. The mapPackage method call on the morphia object determines what objects are mapped by that instance of Morphia. In this case all those in the package supplied. Other alternatives exist to map classes, including the method map which takes a single class (this method can be chained as the returning object is the morphia object), or passing a Set of classes to the Morphia constructor. After creating instances of the entities, they can be saved by calling save on the datastore instance and can be found using the primary key via the get method. The output from the Example class would look something like the below: 11-Jul-2012 13:20:06 com.google.code.morphia.logging.MorphiaLoggerFactory chooseLoggerFactory INFO: LoggerImplFactory set to com.google.code.morphia.logging.jdk.JDKLoggerFactory 4ffd6f7662109325c6eea24f Mr Bank Customer There are many other methods on the Datastore interface and they can be found along with the other Javadocs at http://morphia.googlecode.com/svn/site/morphia/apidocs/index.html An alternative to using the Datastore directly is to use the built in DAO support. This can be done by extending the BasicDAO class as shown below for the Customer entity: package com.city81.mongodb.morphia.dao; import com.city81.mongodb.morphia.entity.Customer; import com.google.code.morphia.Morphia; import com.google.code.morphia.dao.BasicDAO; import com.mongodb.Mongo; public class CustomerDAO extends BasicDAO { public CustomerDAO(Morphia morphia, Mongo mongo, String dbName) { super(mongo, morphia, dbName); } } To then make use of this, the Example class can be changed (and enhanced to show a query and a delete): ... CustomerDAO customerDAO = new CustomerDAO(morphia, mongo, dbName); customerDAO.save(customer); Query query = datastore.createQuery(Customer.class); query.and( query.criteria("accounts.name").equal("Personal Account"), query.criteria("address.number").equal("81"), query.criteria("name").contains("Bank") ); QueryResults retrievedCustomers = customerDAO.find(query); for (Customer retrievedCustomer : retrievedCustomers) { System.out.println(retrievedCustomer.getName()); System.out.println(retrievedCustomer.getAddress().getPostcode()); System.out.println(retrievedCustomer.getAccounts().get(0).getName()); customerDAO.delete(retrievedCustomer); } ... With the output from running the above shown below: 11-Jul-2012 13:30:46 com.google.code.morphia.logging.MorphiaLoggerFactory chooseLoggerFactory INFO: LoggerImplFactory set to com.google.code.morphia.logging.jdk.JDKLoggerFactory Mr Bank Customer CT81 1DB Personal Account This post only covers a few brief basics of Morphia but shows how it can help bridge the gap between JPA and NoSQL.

When you have a piece of code that often fails and must be retried, this Java 7/8 library provides rich and unobtrusive API with fast and scalable solution to this problem: ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); RetryExecutor executor = new AsyncRetryExecutor(scheduler). retryOn(SocketException.class). withExponentialBackoff(500, 2). //500ms times 2 after each retry withMaxDelay(10_000). //10 seconds withUniformJitter(). //add between +/- 100 ms randomly withMaxRetries(20); You can now run arbitrary block of code and the library will retry it for you in case it throws SocketException: final CompletableFuture future = executor.getWithRetry(() -> new Socket("localhost", 8080) ); future.thenAccept(socket -> System.out.println("Connected! " + socket) ); Please look carefully! getWithRetry() does not block. It returns CompletableFuture immediately and invokes given function asynchronously. You can listen for that Future or even for multiple futures at once and do other work in the meantime. So what this code does is: trying to connect to localhost:8080 and if it fails with SocketException it will retry after 500 milliseconds (with some random jitter), doubling delay after each retry, but not above 10 seconds. Equivalent but more concise syntax: executor. getWithRetry(() -> new Socket("localhost", 8080)). thenAccept(socket -> System.out.println("Connected! " + socket)); This is a sample output that you might expect: TRACE | Retry 0 failed after 3ms, scheduled next retry in 508ms (Sun Jul 21 21:01:12 CEST 2013) java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0-ea] //... TRACE | Retry 1 failed after 0ms, scheduled next retry in 934ms (Sun Jul 21 21:01:13 CEST 2013) java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0-ea] //... TRACE | Retry 2 failed after 0ms, scheduled next retry in 1919ms (Sun Jul 21 21:01:15 CEST 2013) java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0-ea] //... TRACE | Successful after 2 retries, took 0ms and returned: Socket[addr=localhost/127.0.0.1,port=8080,localport=46332] Connected! Socket[addr=localhost/127.0.0.1,port=8080,localport=46332] Imagine you connect to two different systems, one is slow, second unreliable and fails often: CompletableFuture stringFuture = executor.getWithRetry(ctx -> unreliable()); CompletableFuture intFuture = executor.getWithRetry(ctx -> slow()); stringFuture.thenAcceptBoth(intFuture, (String s, Integer i) -> { //both done after some retries }); thenAcceptBoth() callback is executed asynchronously when both slow and unreliable systems finally reply without any failure. Similarly (using CompletableFuture.acceptEither()) you can call two or more unreliable servers asynchronously at the same time and be notified when the first one succeeds after some number of retries. I can’t emphasize this enough - retries are executed asynchronously and effectively use thread pool, rather than sleeping blindly. Rationale Often we are forced to retry given piece of code because it failed and we must try again, typically with a small delay to spare CPU. This requirement is quite common and there are few ready-made generic implementations with retry support in Spring Batch through RetryTemplate class being best known. But there are few other, quite similar approaches ([1], [2]). All of these attempts (and I bet many of you implemented similar tool yourself!) suffer the same issue - they are blocking, thus wasting a lot of resources and not scaling well. This is not bad per se because it makes programming model much simpler - the library takes care of retrying and you simply have to wait for return value longer than usual. But not only it creates leaky abstraction (method that is typically very fast suddenly becomes slow due to retries and delay), but also wastes valuable threads since such facility will spend most of the time sleeping between retries. Therefore Async-Retry utility was created, targeting Java 8 (with Java 7 backport existing) and addressing issues above. The main abstraction is RetryExecutor that provides simple API: public interface RetryExecutor { CompletableFuture doWithRetry(RetryRunnable action); CompletableFuture getWithRetry(Callable task); CompletableFuture getWithRetry(RetryCallable task); CompletableFuture getFutureWithRetry(RetryCallable> task); } Don’t worry about RetryRunnable and RetryCallable - they allow checked exceptions for your convenience and most of the time we will use lambda expressions anyway. Please note that it returns CompletableFuture. We no longer pretend that calling faulty method is fast. If the library encounters an exception it will retry our block of code with preconfigured backoff delays. The invocation time will sky-rocket from milliseconds to several seconds. CompletableFuture clearly indicates that. Moreover it’s not a dumb java.util.concurrent.Future we all know - CompletableFuture in Java 8 is very powerful and most importantly - non-blocking by default. If you need blocking result after all, just call .get() on Future object. Basic API The API is very simple. You provide a block of code and the library will run it multiple times until it returns normally rather than throwing an exception. It may also put configurable delays between retries: RetryExecutor executor = //... executor.getWithRetry(() -> new Socket("localhost", 8080)); Returned CompletableFuture will be resolved once connecting to localhost:8080 succeeds. Optionally we can consume RetryContext to get extra context like which retry is currently being executed: executor. getWithRetry(ctx -> new Socket("localhost", 8080 + ctx.getRetryCount())). thenAccept(System.out::println); This code is more clever than it looks. During first execution ctx.getRetryCount() returns 0, therefore we try to connect to localhost:8080. If this fails, next retry will try localhost:8081 (8080 + 1) and so on. And if you realize that all of this happens asynchronously you can scan ports of several machines and be notified about first responding port on each host: Arrays.asList("host-one", "host-two", "host-three"). stream(). forEach(host -> executor. getWithRetry(ctx -> new Socket(host, 8080 + ctx.getRetryCount())). thenAccept(System.out::println) ); For each host RetryExecutor will attempt to connect to port 8080 and retry with higher ports. getFutureWithRetry() requires special attention. I you want to retry method that already returns CompletableFuture: e.g. result of asynchronous HTTP call: private CompletableFuture asyncHttp(URL url) { /*...*/} //... final CompletableFuture> response = executor.getWithRetry(ctx -> asyncHttp(new URL("http://example.com"))); Passing asyncHttp() to getWithRetry() will yield CompletableFuture>. Not only it’s awkward to work with, but also broken. The library will barely call asyncHttp() and retry only if it fails, but not if returned CompletableFuture fails. The solution is simple: final CompletableFuture response = executor.getFutureWithRetry(ctx -> asyncHttp(new URL("http://example.com"))); In this case RetryExecutor will understand that whatever was returned from asyncHttp() is the actually just a Future and will (asynchronously) wait for result or failure. This library is much more powerful, so let’s dive into: Configuration Options In general there are two important factors you can configure: RetryPolicy that controls whether next retry attempt should be made and Backoff - that optionally adds delay between subsequent retry attempts. By default RetryExecutor repeats user task infinitely on every Throwable and adds 1 second delay between retry attempts. Creating an Instance of RetryExecutor Default implementation of RetryExecutor is AsyncRetryExecutor which you can create directly: ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); RetryExecutor executor = new AsyncRetryExecutor(scheduler); //... scheduler.shutdownNow(); The only required dependency is standard ScheduledExecutorService from JDK. One thread is enough in many cases but if you want to concurrently handle retries of hundreds or more tasks, consider increasing the pool size. Notice that the AsyncRetryExecutor does not take care of shutting down the ScheduledExecutorService. This is a conscious design decision which will be explained later. AsyncRetryExecutor has few other constructors but most of the time altering the behaviour of this class is most convenient with calling chained with*() methods. You will see plenty of examples written this way. Later on we will simply use executor reference without defining it. Assume it’s of RetryExecutor type. Retrying Policy Exceptions By default every Throwable (except special AbortRetryException) thrown from user task causes retry. Obviously this is configurable. For example in JPA you may want to retry a transaction that failed due to OptimisticLockException - but every other exception should fail immediately: executor. retryOn(OptimisticLockException.class). withNoDelay(). getWithRetry(ctx -> dao.optimistic()); Where dao.optimistic() may throw OptimisticLockException. In that case you probably don’t want any delay between retries, more on that later. If you don’t like the default of retrying on every Throwable, just restrict that using retryOn(): executor.retryOn(Exception.class) Of course the opposite might also be desired - to abort retrying and fail immediately in case of certain exception being thrown rather than retrying. It’s that simple: executor. abortOn(NullPointerException.class). abortOn(IllegalArgumentException.class). getWithRetry(ctx -> dao.optimistic()); Clearly you don’t want to retry NullPointerException or IllegalArgumentException as they indicate programming bug rather than transient failure. And finally you can combine both retry and abort policies. User code will retry in case of any retryOn() exception (or subclass) unless it should abortOn() specified exception. For example we want to retry every IOException or SQLException but abort if FileNotFoundException or java.sql.DataTruncation is encountered (order is irrelevant): executor. retryOn(IOException.class). abortIf(FileNotFoundException.class). retryOn(SQLException.class). abortIf(DataTruncation.class). getWithRetry(ctx -> dao.load(42)); If this is not enough you can provide custom predicate that will be invoked on each failure: executor. abortIf(throwable -> throwable instanceof SQLException && throwable.getMessage().contains("ORA-00911") ); Max Number of Retries Another way of interrupting retrying “loop” (remember that this process is asynchronous, there is no blocking loop) is by specifying maximum number of retries: executor.withMaxRetries(5) In rare cases you may want to disable retries and barely take advantage from asynchronous execution. In that case try: executor.dontRetry() Delays Between Retries (backoff) Retrying immediately after failure is sometimes desired (see OptimisticLockException example) but in most cases it’s a bad idea. If you can’t connect to external system, waiting a little bit before next attempt sounds reasonably. You save CPU, bandwidth and other server’s resources. But there are quite a few options to consider: should we retry with constant intervals or increase delay after each failure? should there be a lower and upper limit on waiting time? should we add random “jitter” to delay times to spread retries of many tasks in time? This library answers all these questions. Fixed Interval Between Retries By default each retry is preceded by 1 second waiting time. So if initial attempt fails, first retry will be executed after 1 second. Of course we can change that default, e.g. to 200 milliseconds: executor.withFixedBackoff(200) If we are already here, by default backoff is applied after executing user task. If user task itself consumes some time, retries will be less frequent. For example with retry delay of 200ms and average time it takes before user task fails at about 50ms RetryExecutor will retry about 4 times per second (50ms + 200ms). However if you want to keep retry frequency at more predictable level you can use fixedRate flag: executor. withFixedBackoff(200). withFixedRate() This is similar to “fixed rate” vs. “fixed delay” approaches in ScheduledExecutorService. BTW don’t expect RetryExecutor to be very precise, it does it’s best but it heavily depends on aforementioned ScheduledExecutorService accuracy. Exponentially Growing Intervals Between Retries It’s probably an active research subject, but in general you may wish to expand retry delay over time, assuming that if the user task fails several times we should try less frequently. For example let’s say we start with 100ms delay until first retry attempt is made but if that one fails as well, we should wait two times more (200ms). And later 400ms, 800ms… You get the idea: executor.withExponentialBackoff(100, 2) This is an exponential function that can grow very fast. Thus it’s useful to set maximum backoff time at some reasonable level, e.g. 10 seconds: executor. withExponentialBackoff(100, 2). withMaxDelay(10_000) //10 seconds Random Jitter One phenomena often observed during major outages is that systems tend to synchronize. Imagine a busy system that suddenly stops responding. Hundreds or thousands of requests fail and are retried. It depends on your backoff but by default all these requests will retry exactly after one second producing huge wave of traffic at one point in time. Finally such failures are propagated to other systems that, in turn, synchronize as well. To avoid this problem it’s useful to spread retries over time, flattening the load. A simple solution is to add random jitter to delay time so that not all request are scheduled for retry at the exact same time. You have choice between uniform jitter (random value from -100ms to 100ms): executor.withUniformJitter(100) //ms …and proportional jitter, multiplying delay time by random factor, by default between 0.9 and 1.1 (10%): executor.withProportionalJitter(0.1) //10% You may also put hard lower limit on delay time to avoid to short retry times being scheduled: executor.withMinDelay(50) //ms Implementation Details This library was built with Java 8 in mind to take advantage of lambdas and new CompletableFuture abstraction (but Java 7 port with Guava dependency exists). It uses ScheduledExecutorService underneath to run tasks and schedule retries in the future - which allows best thread utilization. But what is really interesting is that the whole library is fully immutable, there is no single mutable field, at all. This might be counter-intuitive at first, take for example this trivial code sample: ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); AsyncRetryExecutor first = new AsyncRetryExecutor(scheduler). retryOn(Exception.class). withExponentialBackoff(500, 2); AsyncRetryExecutor second = first.abortOn(FileNotFoundException.class); AsyncRetryExecutor third = second.withMaxRetries(10); It might seem that all with*() methods or retryOn()/abortOn() mutate existing executor. But that’s not the case, each configuration change creates new instance, leaving the old one untouched. So for example while first executor will retry on FileNotFoundException, the second and third won’t. However they all share the same scheduler. This is the reason why AsyncRetryExecutor does not shut down ScheduledExecutorService (it doesn’t even have any close() method). Since we have no idea how many copies of AsyncRetryExecutor exist pointing to the same scheduler, we don’t even try to manage its lifecycle. However this is typically not a problem (see Spring integration below). You might be wondering, why such an awkward design decision? There are three reasons: when writing a concurrent code immutability can greatly reduce risk of multi-threading bugs. For example RetryContext holds number of retries. But instead of mutating it we simply create new instance (copy) with incremented but final counter. No race condition or visibility can ever occur. if you are given an existing RetryExecutor which is almost exactly what you want but you need one minor tweak, you simply call executor.with...() and get a fresh copy. You don’t have to worry about other places where the same executor was used (see: Spring integration for further examples) functional programming and immutable data structures are sexy these days ;-). N.B.: AsyncRetryExecutor is not marked final, does you can break immutability by subclassing it and adding mutable state. Please don’t do this, subclassing is only permitted to alter behaviour. Dependencies This library requires Java 8 and SLF4J for logging. Java 7 port additionally depends on Guava. Spring Integration If you are just about to use RetryExecutor in Spring - feel free, but the configuration API might not work for you. Spring promotes (or used to promote) the convention of mutable services with plenty of setters. In XML you define bean and invoke setters (via ) on it. This convention assumes the existence of mutating setters. But I found this approach error-prone and counter-intuitive under some circumstances. Let’s say we globally defined org.springframework.transaction.support.TransactionTemplate bean and injected it in multiple places. Great. Now there is this one single request that requires slightly different timeout: @Autowired private TransactionTemplate template; and later in the same class: final int oldTimeout = template.getTimeout(); template.setTimeout(10_000); //do the work template.setTimeout(oldTimeout); This code is wrong on so many levels! First of all if something fails we never restore oldTimeout. OK, finally to the rescue. But also notice how we changed global, shared TransactionTemplate instance. Who knows how many other beans and threads are just about to use it, unaware of changed configuration? And even if you do want to globally change the transaction timeout, fair enough, but it’s still wrong way to do this. private timeout field is not volatile and thus changes made to it may or may not be visible to other threads. What a mess! The same problem appears with many other classes like JmsTemplate. You see where I’m going? Just create one, immutable service class and safely adjust it by creating copies whenever you need it. And using such services is equally simple these days: @Configuration class Beans { @Bean public RetryExecutor retryExecutor() { return new AsyncRetryExecutor(scheduler()). retryOn(SocketException.class). withExponentialBackoff(500, 2); } @Bean(destroyMethod = "shutdownNow") public ScheduledExecutorService scheduler() { return Executors.newSingleThreadScheduledExecutor(); } } Hey! It’s 21st century, we don’t need XML in Spring any more. Bootstrap is simple as well: final ApplicationContext context = new AnnotationConfigApplicationContext(Beans.class); final RetryExecutor executor = context.getBean(RetryExecutor.class); //... context.close(); As you can see integrating modern, immutable services with Spring is just as simple. BTW if you are not prepared for such a big change when designing your own services, at least consider constructor injection. Maturity This library is covered with a strong battery of unit tests. However it wasn’t yet used in any production code and the API is subject to change. Of course you are encouraged to submit bugs, feature requests and pull requests. It was developed with Java 8 in mind but Java 7 backport exists with slightly more verbose API and mandatory Guava dependency (ListenableFuture instead of CompletableFuture from Java 8). Full source code on GitHub.

I’ve been playing around with Jersey over the past couple of days and one thing I wanted to do was create an entry point or index which listed all my resources, the available paths and the verbs they accepted. Guido Simone explained a neat way of finding the paths and verbs for a specific resource using Jersey’sIntrospectionModeller: AbstractResource resource = IntrospectionModeller.createResource(JacksonResource.class); System.out.println("Path is " + resource.getPath().getValue()); String uriPrefix = resource.getPath().getValue(); for (AbstractSubResourceMethod srm :resource.getSubResourceMethods()) { String uri = uriPrefix + "/" + srm.getPath().getValue(); System.out.println(srm.getHttpMethod() + " at the path " + uri + " return " + srm.getReturnType().getName()); } If we run that against j4-minimal‘s JacksonResource class we get the following output: Path is /jackson GET at the path /jackson/{who} return com.g414.j4.minimal.JacksonResource$Greeting GET at the path /jackson/awesome/{who} return javax.ws.rs.core.Response That’s pretty neat but I didn’t want to have to manually list all my resources since I’ve already done that using Guice . I needed a way to programatically get hold of them and I partially found the way to do this from this postwhich suggests using Application.getSingletons(). I actually ended up using Application.getClasses() and I ended up with ResourceListingResource: @Path("/") public class ResourceListingResource { @GET @Produces(MediaType.APPLICATION_JSON) public Response showAll( @Context Application application, @Context HttpServletRequest request) { String basePath = request.getRequestURL().toString(); ObjectNode root = JsonNodeFactory.instance.objectNode(); ArrayNode resources = JsonNodeFactory.instance.arrayNode(); root.put( "resources", resources ); for ( Class aClass : application.getClasses() ) { if ( isAnnotatedResourceClass( aClass ) ) { AbstractResource resource = IntrospectionModeller.createResource( aClass ); ObjectNode resourceNode = JsonNodeFactory.instance.objectNode(); String uriPrefix = resource.getPath().getValue(); for ( AbstractSubResourceMethod srm : resource.getSubResourceMethods() ) { String uri = uriPrefix + "/" + srm.getPath().getValue(); addTo( resourceNode, uri, srm, joinUri(basePath, uri) ); } for ( AbstractResourceMethod srm : resource.getResourceMethods() ) { addTo( resourceNode, uriPrefix, srm, joinUri( basePath, uriPrefix ) ); } resources.add( resourceNode ); } } return Response.ok().entity( root ).build(); } private void addTo( ObjectNode resourceNode, String uriPrefix, AbstractResourceMethod srm, String path ) { if ( resourceNode.get( uriPrefix ) == null ) { ObjectNode inner = JsonNodeFactory.instance.objectNode(); inner.put("path", path); inner.put("verbs", JsonNodeFactory.instance.arrayNode()); resourceNode.put( uriPrefix, inner ); } ((ArrayNode) resourceNode.get( uriPrefix ).get("verbs")).add( srm.getHttpMethod() ); } private boolean isAnnotatedResourceClass( Class rc ) { if ( rc.isAnnotationPresent( Path.class ) ) { return true; } for ( Class i : rc.getInterfaces() ) { if ( i.isAnnotationPresent( Path.class ) ) { return true; } } return false; } } The only change I’ve made from Guido Simone’s solution is that I also call resource.getResourceMethods()because resource.getSubResourceMethods() only returns methods which have a @Path annotation. Since we’ll sometimes define our path at the class level and then define different verbs that operate on that resource it misses some methods out. If we run a cURL command (piped through python to make it look nice) against the root we get the following output: $ curl http://localhost:8080/ -w "\n" 2>/dev/null | python -mjson.tool { "resources": [ { "/bench": { "path": "http://localhost:8080/bench", "verbs": [ "GET", "POST", "PUT", "DELETE" ] } }, { "/sample/{who}": { "path": "http://localhost:8080/sample/{who}", "verbs": [ "GET" ] } }, { "/jackson/awesome/{who}": { "path": "http://localhost:8080/jackson/awesome/{who}", "verbs": [ "GET" ] }, "/jackson/{who}": { "path": "http://localhost:8080/jackson/{who}", "verbs": [ "GET" ] } }, { "/": { "path": "http://localhost:8080/", "verbs": [ "GET" ] } } ] }