Curator's Note: The content of this article was originally written over at the MongoLab blog. So you’re checking in on your latest awesome application one day — it’s really getting traction! You’re proud of its uptime record, thanks in part to the MongoDB replica set underneath it. But now … something’s wrong. Users are complaining that some of their data has gone missing. Others are noticing stuff they deleted has suddenly reappeared. What’s going on?!? Don’t worry… we’ll get to the bottom of this! In doing so, we’ll examine a source of risk that’s easy to overlook in a MongoDB application: replication lag — what it means, why it happens, and what you can do about it. Here’s what we’re going to cover: What is replication lag? Why is lag problematic? What causes a secondary to fall behind? How do I measure lag? How do I monitor for lag? What can I do to minimize lag? Tip #1: Make sure your secondary has enough horsepower Tip #2: Consider adjusting your write concern Tip #3: Plan for index builds Tip #4: Take backups without blocking Tip #5: Be sure capped collections have an _id field & a unique index Tip #6: Check for replication errors Don’t let replication lag take you by surprise. Continuing this cautionary tale… Seriously, wtf?! You were doing everything right! Using MongoDB with a well-designed schema and lovingly-tuned indexes, your application back-end has been handling thousands of transactions per second without breaking a sweat. You’ve got multiple nodes arranged in a replica set with no single point of failure. Your application tier’s Mongo driver connections are aware of the replica set and can follow changes in the PRIMARY node during failover. All critical writes are “safe” writes. Your app has been up without interruption for almost six months now! How could this have happened? This unsettling situation has the hallmarks of an insidious foe in realm of high-availability data stewardship: unchecked replication lag. Closely monitoring a MongoDB replica set for replication lag is critical. What is replication lag? As you probably know, like many data stores MongoDB relies on replication — making redundant copies of data — to meet design goals around availability. In a perfect world, data replication would be instantaneous; but in reality, thanks to pesky laws of physics, some delay is inevitable — it’s a fact of life. We need to be able to reason about how it affects us so as to manage around the phenomenon appropriately. Let’s start with definitions… For a given secondary node, replication lag is the delay between the time an operation occurs on the primary and the time that same operation gets applied on the secondary. For the replica set as a whole, replication lag is (for most purposes) the smallest replication lag found among all its secondary nodes. In a smoothly running replica set, all secondaries closely follow changes on the primary, fetching each group of operations from its oplog and replaying them approximately as fast as they occur. That is, replication lag remains as close to zero as possible. Reads from any node are then reasonably consistent; and, should the current primary become unavailable, the secondary that assumes the PRIMARY role will be able to serve to clients a dataset that is almost identical to the original. For a variety of reasons, however, secondaries may fall behind. Sometimes elevated replication lag is transient and will remedy itself without intervention. Other times, replication lag remains high or continues to rise, indicating a systemic problem that needs to be addressed. In either case, the larger the replication lag grows and the longer it remains that way, the more exposure your database has to the associated risks. Why is lag problematic? Significant replication lag creates failure modes that can be problematic for a MongoDB database deployment that is meant to be highly available. Here’s why: If your replica set fails over to a secondary that is significantly behind the primary, a lot of un-replicated data may be on the original primary that will need to be manually reconciled. This will be painful or impossible if the original primary is unrecoverable. If the failed primary cannot be recovered quickly, you may be forced to run on a node whose data is not up-to-date, or forced to take down your database altogether until the primary can be recovered. If you have only one secondary, and it falls farther behind than the earliest history retained in the primary’s oplog, your secondary will require a full resynchronization from the primary. During the resync, your cluster will lack the redundancy of a valid secondary; the cluster will not return to high availability until the entire data set is copied. If you only take backups from your secondary (which we highly recommend), backups must be suspended for the duration of the resync. Replication lag makes it more likely that results of any read operations distributed across secondaries will be inconsistent. A “safe” write with ‘w’ > 1 — i.e., requiring multiple nodes acknowledge the write before it returns — will incur latency proportional to the current replication lag, and/or may time out. Strictly speaking, the problem of replication lag is distinct from the problem of data durability. But as the last point above regarding multi-node write concern illustrates, the two concepts are most certainly linked. Data that has not yet been replicated is not completely protected from single-node failure; and client writes specified to be safe from single-node failure must block until replication catches up to them. What causes a secondary to fall behind? In general, a secondary falls behind on replication any time it cannot keep up with the rate at which the primary is writing data. Some common causes: Secondary is weak To have the best chance of keeping up, a secondary host should match the primary host’s specs for CPU, disk IOPS, and network I/O. If it’s outmatched by the primary on any of these specs, a secondary may fall behind during periods of sustained write activity. Depending on load this will, at best, create brief excursions in replication lag and, at worst, cause the secondary to fall irretrievably behind. Bursty writes In the wake of a burst of write activity on the primary, a secondary may not be able to fetch and apply the ops quickly enough. If the secondary is underpowered, this effect can be quite dramatic. But even when the nodes have evenly matched specs, such a situation is possible. For example, a command like: db.coll.update({x: 7}, {$set: {y: 42}, {multi: true} can place an untold number of separate “update” ops in the primary’s oplog. To keep up, a secondary must fetch those ops (max 4MB at a time for each getMore command!), read into RAM any index and data pages necessary to satisfy each _id lookup (remember: each oplog entry references a single target document by _id; the original query about “x” is never directly reflected the oplog), and finally perform the update op, altering the document and placing the corresponding entry into its oplog; and it must do all this in the same amount of time that the primary does merely the last step. Multiplied by a large enough number of ops, that disparity can amount to a noticeable lag. Map/reduce output A specific type of the extreme write burst scenario might be a command like: db.coll.mapReduce( ... { out: other_coll ... }) Index build It may surprise you to learn that, even if you build an index in the background on the primary, it will be built in the foreground on each secondary. There is currently no way to build indexes in the background on secondary nodes (cf. SERVER-2771). Therefore, whenever a secondary builds an index, it will block all other operations, including replication, for the duration. If the index builds quickly, this may not be a problem; but long-running index builds can swiftly manifest as significant replication lag. Secondary is locked for backup One of the suggested methods for backing up data in a replica set involves explicitly locking a secondary against changes while the backup is taken. Assuming the primary is still conducting business as usual, of course replication lag will climb until the backup is complete and the lock is released. Secondary is offline Similarly, if the secondary is not running or cannot reach the primary for whatever reason, it cannot make progress against the replication backlog. When it rejoins the replica set, the replication lag will naturally reflect the time spent away. How do I measure lag? Run the db.printSlaveReplicationInfo() command To determine the current replication lag of your replica set, you can use the mongo shell and run the db.printSlaveReplicationInfo() command. rs-ds046297:PRIMARY db.printSlaveReplicationInfo() source: ds046297-a1.mongolab.com:46297 syncedTo: Tue Mar 05 2013 07:48:19 GMT-0800 (PST) = 7475 secs ago (2.08hrs) source: ds046297-a2.mongolab.com:46297 syncedTo: Tue Mar 05 2013 07:48:19 GMT-0800 (PST) = 7475 secs ago (2.08hrs) More than 2 hours — whoa, isn’t that a lot? Maybe! See, those “syncedTo” times don’t have much to do with the clock on the wall; they’re just the timestamp on the last operation that the replica has copied over from the PRIMARY. If the last write operation on the PRIMARY happened 5 minutes ago, then yes: 2 hours is a lot. On the other hand, if the last op was 2.08 hours ago, then this is golden! To fill in that missing piece of the story, we can use the db.printReplicationInfo() command. rs-ds046297:PRIMARY db.printReplicationInfo() configured oplog size: 1024MB log length start to end: 5589secs (1.55hrs) oplog first event time: Tue Mar 05 2013 06:15:19 GMT-0800 (PST) oplog last event time: Tue Mar 05 2013 07:48:19 GMT-0800 (PST) now: Tue Mar 05 2013 09:53:07 GMT-0800 (PST) Let’s see … PRIMARY’s “oplog last event time” – SECONDARY’s “syncedTo” = 0.0. Yay. As fun as that subtraction may be, it’s seldom called for. If there is a steady flow of write operations, the last op on the PRIMARY will usually have been quite recent. Thus, a figure like “2.08 hours” should probably raise eyebrows; you would expect to see a nice low number there instead — perhaps as high as a few seconds. And, having seen a low number, there would be no need to qualify its context with the second command. Examine the “repl lag” graph in MMS You can also view recent and historical replication lag using the MongoDB Monitoring Service (MMS) from 10gen. On the Status tab of each SECONDARY node, you’ll find the repl lag graph: How do I monitor for lag? It is critical that the replication lag of your replica set(s) be monitored continuously. Since you have to sleep occasionally, this is a job best done by robots. It is essential that these robots be reliable, and that they notify you promptly whenever a replica set is lagging too far behind. Here are a couple ways you can make sure this is taken care of: If MongoLab is hosting your replica set, relax! For any multi-node, highly-available replica set we host for you, you can monitor replication lag in our UI and by default you will receive automated alerts whenever the replication lag exceeds 10 minutes. You can also set up an alert using the MMS system. Its exciting new features allow you to configure a replication lag alert: What can I do to minimize lag? Out of courtesy (for them or for ourselves), we would like to make those lag-monitoring automata’s lives as boring as possible. Here are some tips: Tip #1: Make sure your secondary has enough horsepower It’s not uncommon for people to run under-powered secondaries to save money — this can be fine if the write load is light. But in scenarios where the write load is heavy, the secondary might not be able to keep up with the primary. To avoid this, you should beef up your secondary so that it’s as powerful as your primary. Specifically, a SECONDARY node should have enough network bandwidth that it can retrieve ops from the PRIMARY’s oplog at roughly the rate they’re created and also enough storage throughput that it can apply the ops — i.e., read any affected documents and their index entries into RAM, and commit the altered documents back to disk — at that same rate. CPU rarely becomes a bottleneck, but it may need to be considered if there are many index keys to compute and insert for the documents that are being added or changed. Tip #2: Consider adjusting your write concern Your secondary may be lagging simply because your primary’s oplog is filling up faster than it can be replicated. Even with an equally-brawny SECONDARY node, the PRIMARY will always be capable of depositing 4MB in its memory-mapped oplog in a fraction of the time those same 4MB will need to make it across a TCP/IP connection. One viable way to apply some back-pressure to the primary might be to adjust your write concern. If you are currently using a write concern that does not acknowledge writes (aka “fire-and-forget” mode), you can change your write concern to require an acknowledgement from the primary (w:1) and/or a write to the primary’s journal (j:true). Doing so will slow down the rate at which the concerned connection can generate new ops needing replication. Other times it may be appropriate to use a ‘w’ > 1 or a ‘w’ set to “majority” to ensure that each write to the cluster is replicated to more than one node before the command returns. Requiring confirmation that a write has replicated to secondaries will effectively guarantee that those secondaries have caught up (at least up to the timestamp of this write) before the next command on the same connection can produce more ops in the backlog. As previously alluded to, choosing the most appropriate write concern for the data durability requirements of your application — or for particular critical write operations within the application — is something you must give thought to irrespective of the replication lag issue we’re focusing on here. But you should be aware of the interrelationship: just as the durability guarantee of w>1 can be used as a means of forcing a periodic “checkpoint” on replication, excessive replication lag can show up as a surprisingly high latency (or timeout) for that very occasional critical write operation where you’ve used “w: majority” to make sure it’s truly committed. Adjust to taste Having servers acknowledge every write can be a big hit to system throughput. If it makes sense for your application, you can amortize that penalty by doing inserts in batches, requiring acknowledgement only at the end of each batch. The smaller the batch, the greater the back-pressure on PRIMARY data creation rate, and correspondingly greater potential adverse impact to overall throughput. Don’t overdo it Using a large value for ‘w’ can itself be problematic. It represents a demand that w nodes finish working through their existing backlog before the command returns. So, if replication lag is high (in the sense of there being a large volume of data waiting to copy over) when the write command is issued, the command execution time will suffer a proportionally high latency. Also, if enough nodes go offline such that ‘w’ cannot be satisfied, you have effectively locked up your database. This is basically the opposite of “high availability.” Tip #3: Plan for index builds As mentioned earlier, an index build on a secondary is a foreground, blocking operation. If you’re going to create an index that is sizeable, perhaps you can arrange to do it during a period of low write activity on the primary. Alternately, if you have more than one secondary, you can follow the steps here to minimize the impact of building large indexes. Tip #4: Take backups without blocking Earlier we discussed the technique of locking the secondary to do a backup. There are other alternatives to consider here, including filesystem snapshots and “point-in-time” backups using the “--oplog” option of mongodump without locking. These are preferable to locking the secondary during a period of active writes if there’s any chance you’ll use the secondary for anything other than backups. Tip #5: Be sure capped collections have an _id field & a unique index Reliable replication is not possible unless there is a unique index on the _id field. Before MongoDB version 2.2, capped collections did not have an _id field or index by default. If you have a collection like this, you should create an index on the _id field, specifying unique: true. Failing to do this can, in certain situations, cause replication to halt entirely. So … this should not be regarded as optional. Tip #6: Check for replication errors If you see that replication lag is only increasing (and never falling), your replica set could be experiencing replication errors. To check for errors, run rs.status() and look at the errmsg field in the result. Additionally, check the log file of your secondary and look for error messages there. One specific example: if you see “RS102 too stale to catch up” in the secondary’s mongodb.log or in the errmsg field when running rs.status(), it means that secondary has fallen so far behind that there is not enough history retained by the primary (its “oplog size”) to bring it up to date. In this case, your secondary will require a full resynchronization from the primary. In general, though, what you do in response to an error depends on the error. Sometimes you can simply restart the mongod process for your secondary; but the majority of the time you will need to understand the root cause of the error before you can fix the problem. Don’t let replication lag take you by surprise. At the end of the day, replication lag is just one more source of risk in any high-availability system that we need to understand and design around. Striking the right balance between performance and “safety” of write operations is an exercise in risk management — the “right” balance will be different in different situations. For an application on a tight budget with occasional spikes in write volume, for example, you might decide that a large replication lag in the wake of those spikes is acceptable given the goals of the application, and so an underpowered secondary makes sense. At the opposite extreme, for an application where every write is precious and sacred, the required “majority” write concern will mean you have essentially no tolerance for replication lag above the very minimum possible. The good news is that MongoDB makes this all very configurable, even on an operation by operation basis. We hope this article has given you some insight into the phenomenon of replication lag that will enable you to reason about the risk it poses for a high-availability MongoDB application, and armed you with some tools for managing it. As always, let us know if we can help!

let’s say you have a large piece of text and a dictionary of keywords. how do you quickly locate all the keywords? aho-corasick algorithm diagram well, there are many ways really, you could even iterate through the whole thing and compare words to keywords. but it turns out that’s going to be very slow. at least o(n_keywords * n_words) complexity. essentially you’re making as many passes over the text as your dictionary is big. in 1975 a couple of ibm researchers – alfred aho and margaret corasick – discovered an algorithm that can do this in a single pass. the aho-corasick string matching algorithm . i implemented it in haskell and it takes 0.005s to find 8 different keywords in oscar wilde’s the nightingale and the rose – a 12kb text. a quick naive keyword search implemented in python takes 0.023s . not a big difference practically speaking, but imagine a situation with megabytes of text and thousands of words in the dictionary. the authors mention printing out the result as a major bottleneck in their assessment of the algorithm. yep, printing . the aho-corasick algorithm at the core of this algorithm are three functions: the three functions of aho-corasick algorithm a parser based on a state machine, which maps (state, char) pairs to states and occasionally emits an output. this is called the goto function a failure function, which tells the goto function which state to jump into when the character it just read doesn’t match anything an output function, which maps states to outputs – potentially more than one per state the algorithm works in two stages. it will first construct the goto, failure and output functions. the complexity of this operation hinges solely on the size of our dictionary. then it iterates over the input text to produce all the matches. using state machines for parsing text is a well known trick – the real genius of this algorithm rests in that failure function if you ask me. it makes lateral transitions between states when the algorithm climbs itself into a wall. say you have she and hers in the dictionary. the goto machine eats your input string one character at the time. let’s say it’s already read s h . the next input is an e so it outputs she and reaches a final state. next it reads an r , but the state didn’t expect any more inputs, so the failure function puts us on the path towards hers . this is a bit tricky to explain in text, i suggest you look at the picture from the original article and look at what’s happening. my haskell implementation the first implementation i tried, relied on manully mapping inputs to outputs for the goto, failure and output functions by using pattern recognition. not very pretty, extremely hardcoded, but it worked and was easy to make. building the functions dynamically proved a bit trickier. type goto = map (int, char) int type failure = map int int type output = map int [string] first off, we build the goto function. -- builds the goto function build_goto::goto -> string -> (goto, string) build_goto m s = (add_one 0 m s, s) -- adds one string to goto function add_one::int -> goto -> [char] -> goto add_one _ m [] = m add_one state m (c:rest) | member key m = add_one (frommaybe 0 $ map.lookup key m) m rest | otherwise = add_one max (map.insert key max m) rest where key = (state, c) max = (size m)+1 essentially this builds a flattened prefix tree in a hashmap of (state, char) pairs mapping to the next state. it makes sure to avoid adding new edges to the three as much as possible. the reason it’s not simply a prefix tree are those lateral transitions; doing them in a tree would require backtracking and repeating of steps, so we haven’t achieved anything. once we have the goto function, building the output is trivial. -- builds the output function build_output::(?m::goto) => [string] -> output build_output [] = empty build_output (s:rest) = map.insert (fin 0 s) (list.filter (\x -> elem x dictionary) $ list.tails s) $ build_output rest -- returns the state in which an input string ends without using failures fin::(?m::goto) => int -> [char] -> int fin state [] = state fin state (c:rest) = fin next rest where next = frommaybe 0 $ map.lookup (state, c) ?m we are essentially going over the dictionary, finding the final state for each word and building a hash table mapping final states to their outputs. building the failure function was trickiest, because we need a way to iterate over the depths at which nodes are position in the goto state machine. but we threw that info away by using a hashmap. -- tells us which nodes in the goto state machine are at which traversal depth nodes_at_depths::(?m::goto) => [[int]] nodes_at_depths = list.map (\i -> list.filter (>0) $ list.map (\l -> if i < length l then l!!i else -1) paths) [0..(maximum $ list.map length paths)-1] where paths = list.map (path 0) dictionary we now have a list of lists, that tells us at which depth certain nodes are. -- builds the failure function build_fail::(?m::goto) => [[int]] -> int -> failure build_fail nodes 0 = fst $ mapaccuml (\f state -> (map.insert state 0 f, state)) empty (nodes!!0) build_fail nodes d = fst $ mapaccuml (\f state -> (map.insert state (decide_fail state lower) f, state)) lower (nodes!!d) where lower = build_fail nodes (d-1) -- inner step of building the failure function decide_fail::(?m::goto) => int -> failure -> int decide_fail state lower = findwithdefault 0 (s, c) ?m where (s', c) = key' state $ assocs ?m s = findwithdefault 0 s' lower -- gives us the key associated with a certain state (how to get there) key'::int -> [((int, char), int)] -> (int, char) key' _ [] = (-1, '_') -- this is ugly, being of maybe type would be better key' state ((k, v):rest) | state == v = k | otherwise = key' state rest here we are going over the list of nodes at depths and deciding what the failure should be for each depth based on the failures of depth-1. at depth zero, all failures go to the zeroth state. an important part of this process was inverting the goto hashmap so values point to keys, which is essentially what the key’ function does. finally, we can use the whole algorithm like this: main = do let ?m = fst $ mapaccuml build_goto empty dictionary let ?f = build_fail nodes_at_depths $ (length $ nodes_at_depths)-1 ?out = build_output dictionary print $ ahocorasick text a bit more involved than the usual example of haskell found online, it’s still pretty cool you can see the whole code on github here .

One of EclipseLink JAXB (MOXy)'s strengths is the ability to map an object model to both JSON and XML with a single set of metadata. The one weakness had been that you needed to compromise on the JSON key or XML element for collection properties. I'm happy to say that this issue has been solved in EclipseLink 2.5 (and EclipseLink 2.4.2), and I will demonstrate below with an example. You can try this out today by downloading an EclipseLink 2.5.0 (or EclipseLink 2.4.2) nightly build starting on March 15, 2013 from: http://www.eclipse.org/eclipselink/downloads/nightly.php Domain Model By default a JAXB (JSR-222) implementation will not output a grouping element around collection data. This can be done through the use of the @XmlElementWrapper annotation (see: JAXB & Collection Properties). This grouping element often has a plural name and is a better fit for the key of a JSON array than the repeating element defined by the @XmlElement annotation is. package blog.json.collections; import java.util.*; import javax.xml.bind.annotation.*; @XmlRootElement @XmlType(propOrder={"name", "emailAddresses"}) public class Customer { private String name; private List emailAddresses = new ArrayList(); public String getName() { return name; } public void setName(String name) { this.name = name; } @XmlElementWrapper(name="email-addresses") @XmlElement(name="email-address") public List getEmailAddresses() { return emailAddresses; } public void setEmailAddresses(List<'String> emailAddresses) { this.emailAddresses = emailAddresses; } } Demo We will specify the JSON_WRAPPER_AS_ARRAY_NAME property with a true value to tell MOXy that it should use the grouping element as the name for the JSON array value. Then we will use the same Marshaller to output the same object to both XML and JSON. package blog.json.collections; import java.util.*; import javax.xml.bind.*; import org.eclipse.persistence.jaxb.MarshallerProperties; public class Demo { public static void main(String[] args) throws Exception { Customer customer = new Customer(); customer.setName("Jane Doe"); customer.getEmailAddresses().add("[email protected]"); customer.getEmailAddresses().add("[email protected]"); Map properties = new HashMap(1); properties.put(MarshallerProperties.JSON_WRAPPER_AS_ARRAY_NAME, true); JAXBContext jc = JAXBContext.newInstance(new Class[] {Customer.class}, properties); Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); // Output XML marshaller.marshal(customer, System.out); // Output JSON marshaller.setProperty(MarshallerProperties.MEDIA_TYPE, "application/json"); marshaller.marshal(customer, System.out); } } XML Output Below is the XML output from running the demo code. We see that email-addresses is marshalled as the grouping element which contains an email-address element for each item in the collection. Jane Doe [email protected] [email protected] JSON Output The following JSON output is produced from the same metadata. The only difference is that we told MOXy to use the grouping element as the name for JSON array values. { "customer" : { "name" : "Jane Doe", "email-addresses" : [ "[email protected]", "[email protected]" ] } } JAX-RS You can easily use MOXy as your JSON-binding provider in a JAX-RS environment (see: MOXy as your JAX-RS JSON Provider - MOXyJsonProvider). You can specify that the grouping element should be used as the JSON array name with the wrapperAsArrayName property on MOXyJsonProvider. package blog.json.collections; import java.util.*; import javax.ws.rs.core.Application; import org.eclipse.persistence.jaxb.rs.MOXyJsonProvider; public class CustomerApplication extends Application { @Override public Set> getClasses() { HashSet> set = new HashSet>(1); set.add(CustomerService.class); return set; } @Override public Set

A couple of weeks ago I wrote part 2 of a short series of blogs on the Producer Consumer pattern. This blog focused upon the need to close down my Teletype’s worker thread, fixing a bug in the original code from part 1 of the series. The idea here is that the Teletype’s worker thread can be controlled by a command from the application’s main thread. This command tells the worker thread to shutdown thus allowing the app the gracefully shutdown as demonstrated by the code below: @Override public void run() { while (run) { try { Message message = queue.take(); printHead.print(message.toString()); messageCount++; } catch (InterruptedException e) { printHead.print("Teletype closing down..."); } } printHead.print("Teletype Off."); } public void destroy() { run = false; thread.interrupt(); } In this sample, the main thread calls the destroy() method, which sets the run variable to false and interrupts the worker’s blocking call to queue.take(). However, there’s a problem with this idea in certain circumstances. For example, will suddenly terminating the consumer’s worker thread cause problems in other parts of the system? Will there be data loss as important messages in the queue don’t get processed? If the answer to these questions is ‘yes’ then there’s another approach you can take: use a Poison Pill. Poison Pill is a rather melodramatic name for simply placing a certain, known, data item on the queue and when the consumer reads this item it closes down. Obviously, the poison pill has to be the last item placed on the queue or else the consumer will shut down prematurely. This idea is great in simple systems with only one producer and consumer as shown below: ...but takes a little more thought when there are multiple producers with a single consumer as in my football match updates scenario: ...and could fall apart completely in the case of multiple produces and consumers: ... as ensuring that each consumer receives a poison pill at the right time and all the data in the queue gets processed could be quite tricky. In this blog I’m updating my Teletype code to shut itself down once the two MatcherReporters have sent all their data. The first thing to do is to decide on the message that will act as a poison pill. In the snippet below you can see that I’ve inserted a message that contains the text “END OF FILE” at the end of the match update stream. 95:30 END OF FILE 95:00 Final Score Fulham 0 - 1 Man Utd 94:59 Full time The referee signals the end of the game. I’ve inserted one of these messages into each set of game data. The next thing to do is to modify the Teletype code adding a check for the poison pill message: public class Teletype implements Runnable { private static final String POISON_PILL_MESSAGE = "END OF FILE"; private final BlockingQueue queue; private final PrintHead printHead; private final int matchesPlayed; private volatile boolean run = true; private int pillsRecieved; public Teletype(PrintHead printHead, BlockingQueue queue, int matchesPlayed) { this.queue = queue; this.printHead = printHead; this.matchesPlayed = matchesPlayed; } public void start() { Thread thread = new Thread(this, "Studio Teletype"); thread.start(); printHead.print("Teletype Online."); } @Override public void run() { while (run) { try { Message message = queue.take(); handleMessage(message); } catch (InterruptedException e) { printHead.print("Teletype closing down..."); } } printHead.print("Teletype Off."); } private void handleMessage(Message message) { if (allGamesAreOver(message.getMessageText())) { run = false; } else { printHead.print(message.toString()); } } private boolean allGamesAreOver(String messageText) { if (POISON_PILL_MESSAGE.equals(messageText)) { pillsRecieved++; } return pillsRecieved == matchesPlayed ? true : false; } @VisibleForTesting boolean isRunning() { return run; } } One of the most significant changes here is the addition of the matchesPlayed instance variable. This variable tells the Teletype how many MatchReporters there are supplying it with data. Ultimately this breaks the Producer Consumer pattern in that the consumer now knows about the rest of the system; however, it’s necessary because we need to ensure that the Teletype shuts down at the end of all the data. In a single producer/consumer one to one system this isn’t necessary. The other big change in the Teletype code is to the run() loop. Once a message has been retrieved from the queue it’s passed to the new handleMessage(...) method. The handleMessage(...) method checks whether or not all the games it’s receiving data from are over by calling allGamesAreOver(...), which checks the message text against the poison pill string. If the message test is the poison pill string then the pillsRecieved counter is updated. If the pillsRecieved equals the matchesPlayed variable then all the all the games are over and allGamesAreOver(...) returns true. This sets the run instance variable to false and the worker thread’s run() method exits. So that’s about it, the melodramatic Poison Pill pattern in a nutshell, next time Murder in the Red Barn. The code for this sample is available on GitHub.

hello friends, recently i was asked by one of my friends what is the difference between factory method and abstract factory design patterns, but i didn't seem to convince him easily. of course i’d already read head first, gof, pattern hatching, refactoring to patterns, but not deeply all of them and it was not recent. so i decided to explore more and tried to simply the differences, and here ‘s my attempt. i assume that you are already familiar with both the patterns, so i’ll focus here on where the most people have confusion with the differences. let’s revisit the definitions and their structure first: factory method : define an interface for creating an object, but let subclasses decide which class to instantiate. factory method lets a class defer instantiation to subclasses. abstract factory : provide an interface for creating families of related or dependent objects without specifying their concrete classes. differences one product vs set of many this is perhaps the easiest one but important certainly. factory method is used to create one product only but abstract factory is about creating families of related or dependent products . inheritance vs composition this is perhaps the most confusing one (as both seems to be using inheritance). factory method depends on inheritance to decide which product to be created. in creator class (in structure diagram), it has other methods also (implemented, to manipulate the product) which use the only abstract method factorymethod() to create the product and it can only be implemented/changed by subclasses . here the creator class is also acting like client which depends only on a single method to create a product. we have to create a subclass of whole creator/client class to create a new different product. there’s no separate and dedicated class for creation of a product, had it been the case we could have used it with composition where we can pass an object of factory to client and client can use it without getting into inheritance hierarchy. on the other side, in abstract factory, there’s a separate class dedicated to create a family of related/dependent products and its (any concrete subclass factory) object can be passed to the client which uses it ( composition ). here the client gets a different object (concrete factory) to create the products, instead of creating itself (e.g. using factorymethod() and forcing inheritance), and thus uses composition . if we think of just a product creation facility and the client that uses it, it is clear that in factory method, we are restricted to use inheritance (class based) and in abstract factory we have the flexibility of composition (object based) to create specific products. //factory method class client { public void anoperation() { product p = factorymethod(); p.dosomething(); } protected product factorymethod() {//or it can be abstract as well return new defaultproduct(); } } class newclient extends client { protected product factorymethod() {//overriding return new specificproduct(); } } //abstract factory class client { private factory factory; public client (factory factory) { this.factory = factory; } public void anoperation() { producta p = factory.createproducta(); p.dosomething();//other products and operations as well } } interface factory { producta createproducta(); productb createproductb(); } //concrete factories also, implementing factory interface method vs full class (object) factory method is just a method while abstract factory is an object . the purpose of a class having factory method is not just create objects, it does other work also, only a method is responsible for creating object. in abstract factory, the whole purpose of the class is to create family of objects . level of abstraction abstract factory is one level higher in abstraction than factory method. factory method abstracts the way objects are created , while abstract factory also abstracts the way factories are created which in turn abstracts the way objects are created. one inside another as abstract factory is at higher level in abstraction, it often uses factory method to create the products in factories. i also believe that we should not be obsessed with design patterns, these are all built on good basic design principles, and often mixed while using in real world. i hope it helps somebody else also. let me also know if it can be improved. happy patterns, bye.

This Post explains Topics in Active MQ (Message Broker) with Subscribing and Publishing. For this we will write two java clients. As we did for wso2 Message Broker TopicSubscriber.java to Subcribe for messages TopicPublisher.java to to Publish the messages Let's Start. [1] Get Active MQ from http://activemq.apache.org/download.html [1.1] Start Active MQ from \bin\activemq.bat You can see the started server form http://localhost:8161/admin/ [2] Create Porject "Client" on IDE that you preferred [3] Add activemq-all-5.7.0.jar to lib Dir in the project (activemq-all-5.7.0.jar can be found in root folder) [4] Creat class "TopicSubscriber.java" to Subcribe for messages package simple; import java.util.Properties; import javax.jms.JMSException; import javax.jms.Message; import javax.jms.MessageListener; import javax.jms.Session; import javax.jms.TextMessage; import javax.jms.Topic; import javax.jms.TopicConnection; import javax.jms.TopicConnectionFactory; import javax.jms.TopicSession; import javax.naming.InitialContext; import javax.naming.NamingException; public class TopicSubscriber { private String topicName = "news.sport"; private String initialContextFactory = "" +"org.apache.activemq.jndi.ActiveMQInitialContextFactory"; private String connectionString = "tcp://" +"localhost:61616"; private boolean messageReceived = false; public static void main(String[] args) { TopicSubscriber subscriber = new TopicSubscriber(); subscriber.subscribeWithTopicLookup(); } public void subscribeWithTopicLookup() { Properties properties = new Properties(); TopicConnection topicConnection = null; properties.put("java.naming.factory.initial", initialContextFactory); properties.put("connectionfactory.QueueConnectionFactory", connectionString); properties.put("topic." + topicName, topicName); try { InitialContext ctx = new InitialContext(properties); TopicConnectionFactory topicConnectionFactory = (TopicConnectionFactory) ctx .lookup("QueueConnectionFactory"); topicConnection = topicConnectionFactory.createTopicConnection(); System.out .println("Create Topic Connection for Topic " + topicName); while (!messageReceived) { try { TopicSession topicSession = topicConnection .createTopicSession(false, Session.AUTO_ACKNOWLEDGE); Topic topic = (Topic) ctx.lookup(topicName); // start the connection topicConnection.start(); // create a topic subscriber javax.jms.TopicSubscriber topicSubscriber = topicSession .createSubscriber(topic); TestMessageListener messageListener = new TestMessageListener(); topicSubscriber.setMessageListener(messageListener); Thread.sleep(5000); topicSubscriber.close(); topicSession.close(); } catch (JMSException e) { e.printStackTrace(); } catch (NamingException e) { e.printStackTrace(); } catch (InterruptedException e) { e.printStackTrace(); } } } catch (NamingException e) { throw new RuntimeException("Error in initial context lookup", e); } catch (JMSException e) { throw new RuntimeException("Error in JMS operations", e); } finally { if (topicConnection != null) { try { topicConnection.close(); } catch (JMSException e) { throw new RuntimeException( "Error in closing topic connection", e); } } } } public class TestMessageListener implements MessageListener { public void onMessage(Message message) { try { System.out.println("Got the Message : " + ((TextMessage) message).getText()); messageReceived = true; } catch (JMSException e) { e.printStackTrace(); } } } } [5] Creat class "TopicPublisher.java" to to Publish the messages package simple; import javax.jms.*; import javax.naming.InitialContext; import javax.naming.NamingException; import java.util.Properties; public class TopicPublisher { private String topicName = "news.sport"; private String initialContextFactory = "org.apache.activemq" +".jndi.ActiveMQInitialContextFactory"; private String connectionString = "tcp://localhost:61616"; public static void main(String[] args) { TopicPublisher publisher = new TopicPublisher(); publisher.publishWithTopicLookup(); } public void publishWithTopicLookup() { Properties properties = new Properties(); TopicConnection topicConnection = null; properties.put("java.naming.factory.initial", initialContextFactory); properties.put("connectionfactory.QueueConnectionFactory", connectionString); properties.put("topic." + topicName, topicName); try { // initialize // the required connection factories InitialContext ctx = new InitialContext(properties); TopicConnectionFactory topicConnectionFactory = (TopicConnectionFactory) ctx .lookup("QueueConnectionFactory"); topicConnection = topicConnectionFactory.createTopicConnection(); try { TopicSession topicSession = topicConnection.createTopicSession( false, Session.AUTO_ACKNOWLEDGE); // create or use the topic System.out.println("Use the Topic " + topicName); Topic topic = (Topic) ctx.lookup(topicName); javax.jms.TopicPublisher topicPublisher = topicSession .createPublisher(topic); String msg = "Hi, I am Test Message"; TextMessage textMessage = topicSession.createTextMessage(msg); topicPublisher.publish(textMessage); System.out.println("Publishing message " +textMessage); topicPublisher.close(); topicSession.close(); Thread.sleep(20); } catch (InterruptedException e) { e.printStackTrace(); } } catch (JMSException e) { throw new RuntimeException("Error in JMS operations", e); } catch (NamingException e) { throw new RuntimeException("Error in initial context lookup", e); } } } [6] Firstly Run "TopicSubscriber.java" and then run "TopicPublisher.java" Here is out put from both TopicSubscriber:: Create Topic Connection for Topic news.sport Got the Message : Hi, I am Test Message TopicPublisher:: Use the Topic news.sport Publishing message ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId = ID:Madhuka-THINK-51683-1359787878456-1:1:1:1:1, originalDestination = null, originalTransactionId = null, producerId = null, destination = topic://news.sport, transactionId = null, expiration = 0, timestamp = 1359787878729, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false, text = Hi, I am Test Message} [More] Here is full message that we have send to TopicSubscriber. We can get that any parameter in above. Here is sample to get TimeStamp and ID from JMS message. public class TestMessageListener implements MessageListener { public void onMessage(Message message) { try { System.out.println("Got the Message TimeStamp: " + message.getJMSTimestamp()); System.out.println("Got the Message JMS ID : " + message.getJMSMessageID()); messageReceived = true; } catch (JMSException e) { e.printStackTrace(); } } } Now go to ActiveMQ server at http://localhost:8161/admin/ See that Topic and message count for that topic. Now you time to check more in 'Active MQ'

Let’s say you are writing an application in Ruby. You are probably talking to every API under the sun and are happily writing tests to make sure your code isn’t failing. Because you don’t want to rely on 3rd parties or an internet connection to make your tests pass or fail you mock everything with let’s say, Webmock. This also makes your tests much much faster. After all even the fastest internet is much slower than the processor talking to its memory. If you’re too lazy to mock out every API under the sun, you might use VCR to record requests and play them back later. The main advantage being, you don’t have to worry about meticulously reimplementing everything, and you can nuke the recordings at any time to make sure your code still works against the real API. Life is good. Enter Javascript, stage left Then Javascript becomes more and more prominent. Suddenly your application’s logic is shifting from backend to browser and before you know it, most of your tests are pretty irrelevant. You’re fine for a while with Capybara or Cucumber. Launch a headless browser, click around the site from the comfort of RSpec, make sure users see what they’re supposed to. Balance restored. Then you add a payment form. Or something. Suddenly your frontend is talking to an API. In case of Stripe or Balanced it’s even a feature. A great benefit for the user. jQuery(function($) { $('#payment-form').submit(function(event) { var $form = $(this); // Disable the submit button to prevent repeated clicks $form.find('button').prop('disabled', true); Stripe.createToken($form, stripeResponseHandler); // Prevent the form from submitting with the default action return false; }); }); Well that sucks, you’re suddenly back to square one. Your tests take minutes to execute. Your tests fail without an internet connection. Your tests rely on some 3rd party service being up. Your tests suck. Who wants to code when running ~5 tests takes 3 minutes? Nobody. Enter puffing-billy, stage right The problem is that neither Webmock nor VCR can handle requests originating in a browser because they happen in a different thread and they can’t mess around with those. Luckily, a year ago Olly Smith, created puffing-billy. The idea was great – spin up a web proxy, tell your headless browser to use it, when your code makes a request it will go through the proxy, which will try to use a Webmock to handle it, otherwise pass it on to the vast internet. But who wants to mock everything out manually? Over the past few weeks I set upon the task of fixing this problem and restoring sanity to my life. Good tests are transparent to the application and I’ll be damned if I use any of the suggested solutions on the internet like “Well you just put a switch in your code that knows if you’re in a test and then doesn’t talk to Stripe” Screw that. This morning I submitted a pull request to puffing-billy. I added the ability for puffing-billy to behave like it was VCR, but for your browser. When a request is made, it gets cached. The cache is then persisted between sessions, and requests are played back to the browser as needed. It’s not as sophisticated as VCR just yet, but it gets the job done and my test runtime has gone from 3 minutes to just under a minute. That’s a big deal in my book! The caching even understands that some URL’s are needlessly different on every request (social buttons, analytics etc.) so you can configure it to normalize those requests to a single recording that is played back every time. Your tests don’t really rely on gAnalytics working right? And the best thing is, you don’t even have to change your tests. You add something like this in your spec_helper.rb: Billy.configure do |c| c.cache = true c.ignore_params = ["http://www.google-analytics.com/__utm.gif", "http://b.siftscience.com/i.gif", "https://r.twimg.com/jot", "http://p.twitter.com/t.gif", "http://p.twitter.com/f.gif", "http://www.facebook.com/plugins/like.php", "https://www.facebook.com/dialog/oauth", "http://cdn.api.twitter.com/1/urls/count.json"] c.persist_cache = true c.cache_path = 'spec/req_cache/' end # need to call this because of a race condition between persist_cache # being set and the proxy being loaded for the first time Billy.proxy.restore_cache Capybara.javascript_driver = :poltergeist_billy A test for the payment form looks the same as usual: scenario "physical product" do product = start_buying build(:product, :physical, user: @seller, active: true) VCR.use_cassette('Balanced/purchase_with_cc') do within '#new_order' do fill_in 'order_email', with: Faker::Internet.safe_email fill_in_address fill_in_card click_on 'Buy Now' end page.should have_css('#receipt', :visible => true) end validate_receipt product, @seller end Puffing-billy will transparently cache every requests the browser makes and VCR records any requests made by your backend logic. It’s pretty sweet. What do you guys think? I only have 20 days of Ruby experience and the internet has told me it really wants something like this, but I couldn’t find anyone who’s already made it.

Apache Subversion’s ‘svn annotate’ command allows users to view a line-by-line breakdown of all the changes that have been applied to a human-readable file in your working copy. This information is printed alongside details on: The person responsible for each change The revision number where each change occurred Note that this line-by-line attribution is based on the file as it currently appears within the working copy. To run ‘svn annotate’ on a file, enter: svn annotate (working-copy-location)/file-being-examined In this example, we’re examining all the changes for the ‘Changelog’ file, which is located inside the trunk of our working copy. If you need a more comprehensive printout, the -verbose (-v) switch adds the full datestamp to each line. The –force Switch The ‘svn annotate’ command uses Multipurpose Internet Mail Extensions (MIME) types to automatically detect whether a file is human-readable. By default, ‘svn annotate’ will block any file that’s non human-readable. If you attempt to perform an ‘svn annotate’ on a file that Subversion judges not to be readable, you’ll get an error message. If you want to go ahead regardless, you can add the –force switch. Of course, this may result in a screen full of strange symbols if the file truly is non human-readable!

Is it ironic that it can be difficult to map the java.util.Map class in JAXB (JSR-222)? In this post I will cover some items that will make it much easier. Java Model Below is the Java model that we will use for this example. Customer The Customer class has a property of type Map. I chose this Map specifically since the key is a domain object and the value is a domain object. package blog.map; import java.util.*; import javax.xml.bind.annotation.*; @XmlRootElement public class Customer { private Map addressMap = new HashMap(); public Map getAddressMap() { return addressMap; } public void setAddressMap(Map addressMap) { this.addressMap = addressMap; } } Address The Address class is just a typical POJO. package blog.map; public class Address { private String street; public String getStreet() { return street; } public void setStreet(String street) { this.street = street; } } Demo Code In the demo code below we will create an instance of Customer and populate its Map property. Then we will marshal it to XML. package blog.map; import javax.xml.bind.*; public class Demo { public static void main(String[] args) throws Exception { JAXBContext jc = JAXBContext.newInstance(Customer.class); Address billingAddress = new Address(); billingAddress.setStreet("1 A Street"); Address shippingAddress = new Address(); shippingAddress.setStreet("2 B Road"); Customer customer = new Customer(); customer.getAddressMap().put("billing", billingAddress); customer.getAddressMap().put("shipping", shippingAddress); Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); marshaller.marshal(customer, System.out); } } Use Case #1 - Default Representation Below is a sample of XML corresponding to our domain model. We see that each item in the Map has key and value elements wrapped in an entry element. shipping 2 B Road billing 1 A Street Use Case #2 - Rename the Element The JAXB reference implementation uses the @XmlElementWrapper annotation to rename the element corresponding to a Map property (we've added this support to MOXy in EclipseLink 2.4.2 and 2.5.0). In previous versions of MOXy the @XmlElement annotation should be used. Customer We will use the @XmlElementWrapper annotation to rename the element corresponding to the addressMap property to be addresses. package blog.map; import java.util.*; import javax.xml.bind.annotation.*; @XmlRootElement public class Customer { private Map addressMap = new HashMap(); @XmlElementWrapper(name="addresses") public Map getAddressMap() { return addressMap; } public void setAddressMap(Map addressMap) { this.addressMap = addressMap; } } Output Now we see that the addressMap element has been renamed to addresses. shipping 2 B Road billing 1 A Street Use Case #3 - Add Namespace Qualification In this use case we will examine the impact of applying namespace qualification to a class that has a property of type java.util.Map. There was a MOXy bug related to the namespace qualification of Map properties that has been fixed in EclipseLink 2.4.2 and 2.5.0 (see: http://bugs.eclipse.org/399297). package-info We will use the package level @XmlSchema annotation to specify that all fields/properties belonging to classes in this package should be qualified with the http://www.example.com namespace (see: JAXB & Namespaces). @XmlSchema( namespace="http://www.example.com", elementFormDefault=XmlNsForm.QUALIFIED) package blog.map; import javax.xml.bind.annotation.*; Output We see that the elements corresponding to the Customer and Address classes are namespace qualified, but the elements corresponding to the Map class are not. This is because the Map class is from the java.util package and therefore the information we specified on the package level @XmlSchema annotation does not apply. shipping 2 B Road billing 1 A Street Use Case #4 - Fix Namespace Qualification with an XmlAdapter We can use an XmlAdapter to adjust the namespace qualification from the previous use case. XmlAdapter (MapAdapter) The XmlAdapter mechanism allows you to convert a class to another for the purpose of affecting the mapping (see: XmlAdapter - JAXB's Secret Weapon). To get the appropriate namespace qualification we will use an XmlAdapter to convert the Map to objects in the package for our domain model. package blog.map; import java.util.*; import javax.xml.bind.annotation.adapters.XmlAdapter; public class MapAdapter extends XmlAdapter> { public static class AdaptedMap { public List entry = new ArrayList(); } public static class Entry { public String key; public Address value; } @Override public Map unmarshal(AdaptedMap adaptedMap) throws Exception { Map map = new HashMap(); for(Entry entry : adaptedMap.entry) { map.put(entry.key, entry.value); } return map; } @Override public AdaptedMap marshal(Map map) throws Exception { AdaptedMap adaptedMap = new AdaptedMap(); for(Map.Entry mapEntry : map.entrySet()) { Entry entry = new Entry(); entry.key = mapEntry.getKey(); entry.value = mapEntry.getValue(); adaptedMap.entry.add(entry); } return adaptedMap; } } Customer The @XmlJavaTypeAdapter annotation is used to specify the XmlAdapter on the Map property. Note with an XmlAdaper applied we need to change the @XmlElementWrapper annotation to @XmlElement (evidence that @XmlElement should be used to annotate the element for a Map property). package blog.map; import java.util.*; import javax.xml.bind.annotation.*; import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter; @XmlRootElement public class Customer { private Map addressMap = new HashMap(); @XmlJavaTypeAdapter(MapAdapter.class) @XmlElement(name="addresses") public Map getAddressMap() { return addressMap; } public void setAddressMap(Map addressMap) { this.addressMap = addressMap; } } Output Now all the elements in the XML output are qualfied with the http://www.example.com namespace. shipping 2 B Road billing 1 A Street

With unit tests you can check that your code behaviours just as you expect it to. When writing your unit tests you shouldn't need to worry about if any other area of the application is working correctly. The benefits of unit testing are: Decouples your code Write more modular classes Functions are smaller and more focused Your functions are more defensive Quality of code becomes higher You will find it easier to reuse code. When writing unit tests you just need to test this one method of your application, if your method relies on another class/variable there should be a way you can inject this into the method. This is where dependency injection in your code comes in handy, it will allow you to inject objects into your classes to change the output of the class. There are a few things you need to do to make a method unit testable, methods will need an input from a parameter or a class variable and it will need a return or set a class variable in the method. If the method hasn't got these things then the method can not be unit testable. If there isn't a return of the method then there is no way in knowing how the method performs. Dependency Injection Dependency injection is when your object has a dependency on another object. The simplest form to understand what dependency injection is to think of a setter method. A setter method will take one parameter and set a class variable from this parameter. This is using code injection to pass in a parameter to be used as the class variable value. public function setValue( $val ) { $this->val = $val; } Without dependency injection this method will look like this. public function setValue() { $this->val = 10; } For unit testing you need to be aware of any classes that your class is dependent on. For example if you have a login class that will connect to a database. class login { private $db = false; public function __construct() { $this->db = new Database(); } public function loginUser( $user, $password ) { $this->db->checkLogin( $user, $password ); } } This login class has a dependency of the class Database in the constructor, which means that we can't unit test this correctly. If we want to unit test this then the database class has to be development and tested. If the database class is broken and we try to unit test the loginUser() method the test will always fail and we won't know that it's the database class which is broke or the loginUser() method that is broke. If the database class is finished development, tested and data is in the database then we can use this for the loginUser() function. But now our tests are dependent on data being correct in the database. If we pass in a username and password it must be in the database for our test to pass. Our code could be correct but if the data isn't there then our unit tests will fail. This isn't correct use of unit tests and is more suited to be an integration test. To fix this problem we can use dependency injection to pass in a database connector which will set the database class variable. There are 2 ways we can inject a variable into a class, it can either be in the constructor of the class or by using a setter method. I tend to use constructor for all required dependences and use the setter method if there is a default value for the class variable. class login { private $db = false; public function __construct( $db ) { $this->db = $db; } public function loginUser( $user, $password ) { $this->db->checkLogin( $user, $password ); } } Now this class isn't dependant on a certain database class we can pass in the database class by using the parameter on the login class constructor. We can unit test this loginUser() method by first setting the $this->db class variable. We don't want to rely on a real database as the data can change so we can either create a test harness database class or you can mock the database class. A test harness class will allow you to create your database class and hardcode any data that you need. In the example above we can create a method checkLogin(), in our test harness we can then hardcode a successful login username and password to make the loginUser() method pass. Or you can use a PHP mocking framework to mock a class/method/return value. Both methods have their benefits but mocking is normally quicker to code, but there are times when you want to hardcode certain variables in a class. Mocking Objects In TDD With PHP Mocking objects in test driven development allows you create objects to act as a certain class, if your test depends on another method to return a value, you can mock this method and make it return any value you want. In the example we used above you can mock the database class and choose what value we are expecting back from the checkLogin() method. When mocking a method you can choose what you want to return from this method, therefore we can write tests to see what will happen when checkLogin() returns TRUE and then we can write another test to see what happens when checkLogin() returns FALSE. Mocking objects means that you can run your unit tests without depending on another class returning the values you are expecting, ao you can test just your code in this one method. Here are some of the most popular PHP mocking frameworks: Mocking with PHPUnit - http://www.phpunit.de/manual/3.0/en/mock-objects.html Mocking with Phake - http://phake.digitalsandwich.com/docs/html/ Mocking with Mockery - https://github.com/padraic/mockery Mocking with Enchane PHP - https://github.com/Enhance-PHP/Enhance-PHP Mocking with FBMock - https://github.com/facebook/FBMock Dependency Injection With Interfaces If we are going to pass in a database connector in a constructor of the login class, then this database connector will always have to have a method of checkLogin(). This is why we should code our dependences by using interfaces to make sure that we are always passing in the correct type of class. class login { private $db = false; public function __construct( IDatabase $db ) { $this->db = $db; } } class database implements IDatabase { public function checkLogin( $username, $password ) { // check the login credentials } } interface IDatabase { public function checkLogin( $username, $password ); } This will make sure that the class we pass into the constructor is a type of IDatabase, so if our database class doesn't implement IDatabase then the code will fail and therefore our unit tests will fail. This means whatever we pass into the constructor we know that this class will be able to run the methods it needs for the unit tests to run.

When creating some Class Library you should pay attention to the visibility of its members and have a clear vision of what you’d like to expose to its users and what on the other side should be hidden. When writing unit tests against such assemblies however, you obviously want to test everything, from the internal members to the externally exposes parts. Assume you have a class library named Base containing the following classes and interfaces ICommandHandler - which is a publicly exposed interface CommandHandler - basically its concrete implementation IUndoRedoStack - an interface that is used only internally UndoRedoStack - its concrete implementation Note that I’m programming exclusively against interaces as that’s a major requirement for being able to create the necessary isolation for testing each component on its own. Testing Classes with Internal Visibility Lets take a closer look at the CommandHandler class class CommandHandler : ICommandHandler { //... } When creating a test for the CommandHandler class you would proceed as follows [TestClass] public class CommandHandlerTest { private CommandHandler commandHandler; [TestInitialize] public void Setup() { commandHandler = new CommandHandler(); } [TestCleanup] public void Teardown() { commandHandler = null; } [TestMethod] publiic void ShouldExecuteAGivenCommand() { //the test content } } When you execute such test, it won’t compile however. A best practice is to place the tests in a separate DLL (I usually name it like Base.UnitTests if the tested assembly is called Base) and as such, CommandHandler won’t be visible as it has been defined to only have internal visibility. In a previous blog post I already explained on how to overcome this issue, namely by specifying the InternalsVisibleTo attribute in the tested assembly. Check out that blog post for more details. Mocking Interfaces with Internal Visibility using Moq Now, CommandHandler has a dependency on IUndoRedoStack class CommandHandler : ICommandHandler { public CommandHandler(IUndoRedoStack undoRedoStack) { //... } } The CommandHandler has an Execute(command) method and suppose we’d like to test the fact that when calling it with a given ICommand object, that specific object gets added to the undoRedoStack. We would write [TestInitialize] public void Setup() { mockUndoRedo = new Mock

We are welcoming some new colleagues that come from a Java background in the Onebip team, both from the development and operations field. Here's a primer on learning PHP in this situation, that you may find useful when introducing similar people in your PHP-based projects. The absolute basics Before being able to discuss meaningful matters over PHP code, these pages from the manual should be sent to each interested developer. What is the basic syntax and the available data types of PHP. Primitive types are diffused like in Java, but there's no autoboxing as there is no object equivalent for them. Strings are immutable and one of the most important types. They can be defined with single or double quotes, and their manipulation functions follow the C api. Arrays are the glue of the language, working as lists, sets and maps while wrapped into objects. They can always be traversed with foreach(), and keep an eye on the available functions to avoid rewriting array_search() or sort() by accident. Operators work differently on primitive values and on objects: == is different from === in both these context, but in two ways. Other everyday operators are `.` and `+=`. PHP's object paradigm is borrowed from Java: it supports concrete and abstract classes, interfaces, and the private, protected and public scopes. Type hints (which are actually not hints but strong preconditions) are what is most similar to Java static type safety mechanisms. I personally strongly favors their usage. Namespaces are packages, use statements are imports. However, you don't always have to write them. The deployment model of PHP consists of an interpreter instantiated N times, running on many shared nothing processes. Don't care about: The installation process of Apache and PHP: if you work in a team, you will get good help on that, and you're going to do this only once per project probably. For example, we provide a virtual machine ready for development. The function syntax is also not very useful by itself in 2013, skip directly to objects and their methods. Take a look at anonymous functions, however. Exceptions work mostly as in Java, so don't bother reading about them before coding. Sessions are also evil in web services nowadays, so ignore anything that start with $_SESSION or session_*(): write shared nothing services (that may be restricted to our team and projects.) In general, ignore also low-level APIs like setcookie() or exec() if you already have an higher-level abstraction in the application you're working on, being this abstraction a library or your own code. It's important to know how cookies are transmitted according to HTTP, but coming from another language you already know the protocol. So you write objects that go into a graph At least that's how I see programming these days. However, you have to exit the process sometimes, and PHP provides several APIs for that. The database APIs such as the mongo and PDO extensions, working respectively with MongoDB and relational database. However, if you already use a database abstraction layer, learn just that as there is nothing interesting to see at this lower level. The DateTime object and its cousins, at least know how to call its constructor and the format() method. You should know these APIs exist in order to look them up on demand: json_*() functions, the SimpleXMLElement class, the mcrypt extension and hash_hmac(), mail(). They're just a click away at php.net/function_or_class_name, and I think these names are pretty self-explanatory to tell you when you should read about them. Watch out for: Some topics are always controversial, and get confusing for PHP beginners. Raise your alert level when you feel you're encountering some strange behavior. the difference between == and === comes out often, and it cannot be reduced to "just use === by default" since all primitive types coming from HTTP requests in the classic x-www-form-urlencoded Content-Type are usually strings. php.ini settings may change the output of your application and the actual flow of execution depending on error reporting levels and display options. One never ceases to learn, so when you encounter some problematic directive, take a quick read of the rest of the directives for that extension. Since some people believe web development is the concatenation of strings (it is not), it is tempting to reinvent the wheel; for example, writing functions for composing URLS from paths and query strings. http_build_query() solves that problem as other Composer-ready packages do, so take a look around before starting to implement commodity algorithms yourself. Stack Overflow and the PHP manual itself are good places to start if your problem has been already solved by someone else in the last 10 years. Perform a kata To test your level of proficiency with PHP, execute a small kata with TDD which incidentally will also check your usage of PHPUnit. For example, classic katas are: The Fizz Buzz, maybe with a web interface. The Roman numbers kata: write a function that transforms 42 in XLII. The Game of Life, although it is probably too complex for your first PHP project if you didn't already solve it in another language.

the most important part of building an integration with an api is actually writing the code that will connect with the web service and invoke its methods. i'll show you why using the façade pattern to decouple calls from your existing code is a good idea and help you identify what kind of problems you might be able to prevent. so, first things first, what is the façade pattern? a façade is an object that provides simple access to complex - or external - functionality. it might be used to group together several methods into a single one, to abstract a very complex method into several simple calls or, more generically, to decouple two pieces of code where there's a strong dependency of one over the other. what happens when you develop api calls inside your code and, suddenly, the api is upgraded and some of its methods or parameters change? you'll have to change your application code to handle those changes. also, by changing your internal application code, you might have to change the way some of your objects behave. it is easy to overlook every instance and can require you to doublecheck multiple lines of code. there's a better way to keep api calls up-to-date. by writing a façade with the single responsibility of interacting with the external web service, you can defend your code from external changes. now, whenever the api changes, all you have to do is update your façade. your internal application code will remain untouched. from a test driven development point-of-view, using a facade offers a big advantage. you're now able to write simple tests against the façade without affecting your internal code test results. by using this strategy, you'll be able to know immediately whenever an api is not working as you expected and make the necessary changes to the façade. this is the approach we follow at cloudwork when building integrations between any third-party apis. api façades act as tight compartments that protect the rest of the application from external changes and simplify the way we interact with different web services. this article is cross-posted at using facades to decouple api integrations .

There has been a lot of talking about NoSQL lately, and how this "new" databases can solve problems of Big Data. NoSQL is a movement that promotes alternatives to the relational model in order to provide higher scalability and availability using simple and lightweight systems of storage and retrieval. As part of the NoSQL movement, Graph Databases in particular are especially suited for applications that need to analyze relationships in a network of data assuring high scalability. They are also suited to deal with ad-hoc data or including different type of data as they are less rigid with the schema. Among the most well-known databases there are solutions specially for .Net developers, for instance DEX graph database offers full support for their native C# API with direct and fast access to the graph core, popular Neo4j offers third-party .NET client via REST API, Microsoft research Trinity also supports C# language (although it is more a key-value solution) and CloudGraph seems to focus also on .Net with a beta-release. So, for all .Net developers that are interested into using graphs for their applications, this article will give them a first hint of how to construct a first graph in C# using Visual Studio. For the example we are going to use DEX's native .Net API for a simple first approach. Download your API (in this case DEX API can be downloaded from here ) Create the C# console application Replace the Program.cs with the sample below. This is a typical HelloWorld example where the developer will create a graph database with two nodes (of the same type) and an edge between them. Nodes and edges are the main units of a graph (graph theory ), nodes can be seen as a row of a relational table whereas edges could be seen as the foreign key relating two rows from different tables. The example also includes a neighbors operation, in order to see how information is retrieved from a graph database. Following the relational simile, this retrieval from the graph would be like a select operation from the rows of the two tables joined by the foreign key. using System; using System.Collections.Generic; using System.Linq; using System.Text; using com.sparsity.dex.gdb; namespace HelloWorld { class Program { static void Main(string[] args) { // // Create a sample database // DexConfig cfg = new DexConfig(); Dex dex = new Dex(cfg); Database db = dex.Create("HelloWorld.dex", "HelloWorld"); Session sess = db.NewSession(); Graph g = sess.GetGraph(); // Add a node type with two attributes int nodeType = g.NewNodeType("TheNodeType"); int idAttribute = g.NewAttribute(nodeType, "id", DataType.Long, AttributeKind.Unique); int nameAttribute = g.NewAttribute(nodeType, "name", DataType.String, AttributeKind.Indexed); // Add a directed edge type with an attribute int edgeType = g.NewEdgeType("TheEdgeType", true, false); // Add a node long hellow = g.NewNode(nodeType); Value value = new Value(); g.SetAttribute(hellow, idAttribute, value.SetLong(1)); g.SetAttribute(hellow, nameAttribute, value.SetString("Hellow")); // Add another node long world = g.NewNode(nodeType); g.SetAttribute(world, idAttribute, value.SetLong(2)); g.SetAttribute(world, nameAttribute, value.SetString("World")); // Add an edge long theEdge = g.NewEdge(edgeType, hellow, world); // Get the neighbors of the first node using the edges of "TheEdgeType" type Objects neighbors = g.Neighbors(hellow, edgeType, EdgesDirection.Outgoing); // Say hello to the neighbors ObjectsIterator it = neighbors.Iterator(); while (it.HasNext()) { long neighborOid = it.Next(); g.GetAttribute(neighborOid, edgeType, value); System.Console.WriteLine("Hello " + value.GetString()); } // The ObjectsIterator must be closed it.Close(); // The Objects must be closed neighbors.Close(); // Close the database sess.Close(); db.Close(); dex.Close(); } } } Add a reference in the project (Menu Project -> AddReference) to the library "dexnet.dll". Put the other dlls in a directory where the application can found it. The easier way is to put them where the application exe is created. Or set the working directory to the directory where you have the dlls. But you could also put them with the windows dlls. Here it is your first .Net graph! What next? populate your graph database with more nodes and edges (play with different types of nodes and edges!) and also do not forget to use popular graph algorithms (take a look at this dzone article here or this more advanced here )

Introduction The IT buzzword of 2012 is without a doubt Big Data. It’s new and here to stay, and for a good reason. Big data is data that exceeds the processing capacity of conventional database systems. Great examples are CERN with the Large Hadron Collider, whose experiments generate 25 petabytes of data annually, or Walmart, which handles more than one million customer transaction every hour. Problems These vast amounts of data leave us with two problems. Problem 1: To gain value from this data, one must choose an alternative way to process it. The value of big data to an organization falls into two categories: analytical use, and enabling new products. Big data analytics can reveal insights hidden previously by data too costly to process, such as peer influence among customers, revealed by analyzing shoppers’ transactions, social and geographical data. Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data, in contrast to the somewhat static nature of running predetermined reports. Problem 2: The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. Remember the CERN case where the LHC produces over 25 Petabytes of data annually? No “classic” database architecture or setup is capable of holding these amounts of data. Solutions Fortunately, both problems can be solved by implementing the correct infrastructure and rethinking data storage. There are two critical factors in Big Data environments: size and speed. We already discussed the vast amounts of data and desire to be able to access and process the data fast. The latter is the main differentiator from more traditional data warehouses. Just imagine what you can do when you can access all your data real-time. Enter big data. A common Big Data implementation is an in-memory data grid that lives in a distributed cluster, ensuring both speed, by storing data in-memory, and capacity by using scalability features provided by a cluster. As a bonus, availability is ensured by using a distributed cluster. As for the data storage, there are typically two kinds: in-memory databases and in-memory data grids. But first some background. It is not a new attempt to use main memory as a storage area instead of a disk. In our daily lives there are numerous examples of main memory databases (MMDB), as they perform much faster than disk-based databases. An every day example is a mobile phone. When you SMS or call someone most mobile service providers use MMDB to get the information on your contact as soon as possible. The same applies to your phone. When someone calls you, the caller details are looked up in the contacts application, usually providing a name and sometimes a picture. In memory data grids In Memory Data Grid (IMDG) is the same as MMDB in that it stores data in main memory, but it has a totally different architecture. The features of IMDG can be summarized as follows: Data is distributed and stored on multiple servers. Each server operates in the active mode. A data model is usually object-oriented (serialized) and non-relational. According to the necessity, you often need to add or reduce servers. No traditional database features such as tables. In other words, IMDG is designed to store data in main memory, ensure scalability and store an object itself. These days, there are many IMDG products, both commercial and open source. Some of the most commonly used products are: Hazelcast (http://www.hazelcast.com) JBoss Infinispan (http://www.jboss.org/infinispan) GridGain DataGrid (http://www.gridgain.com/features/in-memory-data-grid/) VMware Gemfire (http://www.vmware.com/nl/products/application-platform/vfabric-gemfire/overview.html) Oracle Coherence (http://www.oracle.com/technetwork/middleware/coherence/overview/index.html) Gigaspaces XAP (http://www.gigaspaces.com/datagrid) Terracotta Enterprise Suite (http://terracotta.org/products/enterprise-suite) Why Memory? The main reasons for using main memory for data storage are once again the two main themes of Big Data: speed and capacity. The processing performance of main memory is 800 times faster than an HDD and up to 40 times faster than an. Moreover, the latest x86 server supports main memory of hundreds of GB per server. It is said that the limit of a traditional processing database’s (OLTP) data capacity is approximately 1 TB and that the OLTP processing data capacity would not increment well. If servers using main memory of 1 TB or larger become more commonly used, you will be able to conduct operations with the entire data placed in main memory, at least in the field of OLTP. IMDG Architecture To use main memory as a storage area, two weak points should be overcome: Limited capacity: involves data that exceeds the maximum capacity of the main memory of the server Reliability: involves data loss in case of a (system) failure. IMDG overcomes the limit of capacity by ensuring horizontal scalability using a distributed architecture, and resolves the issue of reliability through a replication system as part of the grid (or a distributed cluster). Now let’s discuss how an IMDG actually works. First of all, it is important to understand that an IMDG is not the same as an in-memory database, also referred to as MMDB (main memory databases). Typical examples of MMDBs are Oracle TimesTen or Sap Hana. MMDBs are full database products that simply reside in memory. As a result of being a full-blown database, they also carry the weight and overhead of database management features. IMDG is different. No tables, indexes, triggers, stored procedures, process managers etc. Just plain storage. The data model used in IMDG is key-value pairs. A key-value pair is a list with only two parts: a key and a value. The key can be used for storing and retrieving the values in the list. A key can be compared to the index or primary key of a table in a database. Note that IMDG are closely tied to development environments such as Java as the key-value pairs are represented by the structures provided by such a programming environment. Most IMDGs are written in Java, and can only be used within other Java applications. Therefore, the values of key-value pairs can be anything supported by Java, ranging from simple data types such as a string or number, to complex objects. This overcomes the two important hurdles: as you can store complex Java objects as value, there’s no need to translate these objects into a relational datamodel (which is the case in more traditional applications using a database for storage). Furthermore, the seeming limitation of being able to store only one value per key, is actually no limitation at all. Large memory sizes Most of the products introduced above use Java as an implementation language. Java reserves and uses a part of the RAM (internal memory) for dynamic memory allocation. This reserved memory space is called the Java heap. All runtime objects created by a Java application are stored in heap. Using large amounts of data causes two problems. Size limitation: By default, the heap size is 128 MB, but for current business applications, this limit is reached easily. Once the heap is “full”, no new objects can be created and the Java application will show some nasty errors. Performance: It is possible to increase the size of the heap, but this introduces some new problems. When a heap reaches a size of more than 4 gigabytes, Java will have serious issues with memory managements, causing your application to slow down or even freeze. Java has a feature called Garbage Collector, which periodically scans the heap and checks each object if it is still valid and being used. If not, the garbage collector removes the object and defragments the newly available space. The problem is, the larger the heap size, the more work to do for the garbage collector, resulting in performance degradation. Imagine a large bank has a Java application that manages customers, accounts and transactions. We have seen that an IMDG allows the application to store and access all data very quickly by caching it in memory, instead of storing the data in relatively slow databases. Let’s assume the combined data has a size of 40 gigabytes. Storing it in heap is simply not possible, considering the performance penalties of Java’s memory management capabilities. The graph below illustrates the garbage collection pause time when placing cached data in heap: Terracotta’s BigMemory product has a method to overcome these limitations. The method is to use an off-heap memory (direct buffer). Data will not be stored in Java’s heap, but directly in the available internal memory (RAM). Since, this is not subject to Java’s garbage collector, there are no performance penalties. The differences on performance are significant, as can be seen in the graph below: Using off-heap storage has some major benefits: You can use all the available memory on your machine, not just the memory that is allocated to the heap (usually less that 512 Mb). This allows you to store more data in a in-memory data grid, greatly speeding up your application. The heap can be relieved by storing data in native memory, speeding up Java applications as less heap space has to be garbage collected. Clustering, fail over and high availability So far, we have seen IMDG features that are applicable to a single server. However, the real power of IMDG lies in it’s networking and clustering capabilities, providing features as data replication, data synchronization between clients, fail over and high availability. To achieve this, a cluster of servers (or server array) acts a backbone of the infrastructure. Applications (that still can have their own IMDG or off-heap cache) that are connected to the cluster can share, replicate and backup their data with either the cluster or other applications. The graph below depicts a typical setup using Terracotta's BigMemory: The caches on the application servers are usually referred to as “level 1” cache, while the data cache on the server array is referred to as “level 2” cache. There are many different scenarios possible for storing, clustering, synchronizing and replicating data. Covering all these topics goes far beyond the scope of this article. For more information, consult the technical documentation of the product of your choice. Conclusion Big Data brings us some new challenges. First of all, storing and accessing vast amounts of data makes us rethink traditional methods and technologies. Next, there’s the question what to do with all the available data. The potential value for marketing, financial and other businesses is huge. In order to facilitate Big Data, in-memory data grids are considered the best option. IMDGs with off-heap storage are even more powerful, allowing data centric enterprise application to overcome certain limits of the Java platform, such as memory and performance constraints. As the amount of data that (large) companies produce and store, grows exponentially, databases will hit a limit. Accessing your data without a performance penalty simply will not be possible. The answer to this is using an IMDG.

In this part, I will be briefing about different types of Database Normalizations using a sample data model. What is Database Normalization? Normalization is the process of efficiently organizing data in the database. Primary Goal of Normalization? Eliminating redundant data & ensuring meaningful data dependencies. Types of Normalization The following are the three most common normal forms in the database normalization process First Normal Form (1NF) Second Normal Form (2NF) Third Normal Form (3NF) Sample Data Model for Demonstration The following data model will be used to demonstrate all the three normal forms First Normal Form (1NF) First Normal Form (1NF) sets the very basic rules for an organized database: Create separate set of tables for each group of related data and identify each row with a unique columns [primary key] or set of columns [composite key] Eliminate duplicate columns from the table The following data model depicts the tables after 1NF rules are applied - Second Normal Form (2NF) Second Normal Form (2NF) further addresses the concept of removing duplicate data: Meet all the requirements of the first normal form Remove subsets of data that apply to multiple rows of a table and place them in separate tables Create relationships between these new tables and their predecessors through the use of foreign keys So basically the objective of the Second Normal Form is to take that is only partly dependent on the primary key and enter that data into another table. The following data model depicts the tables after 2NF rules are applied. Data from EMPLOYEE_TABLE is split into 2 tables – EMPLOYEE_TABLE and EMPLOYEE_HR_TABLE. Similarly data from CUSTOMER_TABLE is moved to CUSTOMER_TABLE and CUSTOMER_ORDER table Third Normal Form (3NF) Third normal form (3NF) goes one large step further: Meet all the requirements of the second normal form. Remove columns that are not dependent upon the primary key. The following data model depicts the tables after 3NF rules are applied. Further state and country details are moved to their own tables because they are not dependent on the primary key. Advantages of Normalizing the Database There are several advantages of normalization - Data can be stored as small atomic pieces Saves space Increases speed Reduces data anomalies Easy maintenance Other parts of this series include: Part 1 – ACID Properties Part 2 – Keys Part 4 – Database Transactions [coming soon] Part 5 – Indexes [coming soon]

Need help dealing with Maven's non-resolvable parent problem? Check out this post to learn how.

Spring articles are becoming a trend on this blog, I should probably apply for a SpringSource position Colleagues of mine sometimes curse me for my stubbornness in using XML configuration for Spring. Yes, it seems so 2000′s but XML has definite advantages: Configuration is centralized, it’s not scattered among all different components so you can have a nice overview of beans and their wirings in a single place If you need to split your files, no problem, Spring let you do that. It then reassembles them at runtime through internal tags or external context files aggregation Only XML configuration allows for explicit wiring – as opposed to autowiring. Sometimes, the latter is a bit too magical for my own taste. Its apparent simplicity hides real complexity: not only do we need to switch between by-type and by-name autowiring, but more importantly, the strategy for choosing the relevant bean among all eligible ones escapes but the more seasoned Spring developers. Profiles seem to make this easier, but is relatively new and is known to few Last but not least, XML is completely orthogonal to the Java file: there’s no coupling between the 2 so that the class can be used in more than one context with different configurations The sole problem with XML is that you have to wait until runtime to discover typos in a bean or some other stupid boo-boo. On the other side, using Spring IDE plugin (or the integrated Spring Tools Suite) definitely can help you there. An interesting alternative to both XML and direct annotations on bean classes is JavaConfig, a former separate project embedded into Spring itself since v3.0. It merges XML decoupling advantage with Java compile-time checks. JavaConfig can be seen as the XML file equivalent, only written in Java. The whole documentation is of course available online, but this article will just let you kickstart using JavaConfig. As an example, let us migrate from the following XML file to a JavaConfig The equivalent file is the following: import java.net.MalformedURLException; import java.net.URL; import javax.swing.Icon; import javax.swing.ImageIcon; import javax.swing.JButton; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class MigratedConfiguration { @Bean public JButton button() { return new JButton("Hello World"); } @Bean public JButton anotherButton() { return new JButton(icon()); } @Bean public Icon icon() throws MalformedURLException { URL url = new URL("http://morevaadin.com/assets/images/learning_vaadin_cover.png"); return new ImageIcon(url); } } Usage is simpler than simple: annotate the main class with @Configuration and individual producer methods with @Bean. The only drawback, IMHO, is that it uses autowiring. Apart from that, It just works. Note that in a Web environment, the web deployment descriptor should be updated with the following lines: contextClass org.springframework.web.context.support.AnnotationConfigWebApplicationContext contextConfigLocation com.packtpub.learnvaadin.springintegration.SpringIntegrationConfiguration Sources for this article are available in Maven/Eclipse format here. To go further: Java-based container configuration documentation AnnotationConfigWebApplicationContextJavaDoc @ContextConfigurationJavaDoc (to configure Spring Test to use JavaConfig)

n this post i’m going to cover how to do job chaining in quartz versus obsidian scheduler . both are java job schedulers, but they have different approaches so i thought i’d highlight them here and give some guidance to users using both options. it’s very common when using a job scheduler to need to chain one job to another. chaining in this case refers to executing a specific job after a certain job completes (or maybe even fails). often we want to do this conditionally, or pass on data to the target job so it can receive it as input from the original job. we’ll start with demonstrating how to do this in quartz, which will take a fair bit of work. obsidian will come after since it’s so simple. chaining in quartz quartz is the most popular job scheduler out there, but unfortunately it doesn’t provide any way to give you chaining without you writing some code. quartz is a low-level library at heart, and it doesn’t try to solve these types of problems for you, which in my mind is unfortunate since it puts the onus on developers. but despite this, many teams still end up using quartz, so hopefully this is useful to some of you. i’m going to outline probably the most basic way to perform chaining. it will allow a job to chain to another, passing on its jobdatamap (for state). this is simpler than using listeners, which would require extra configuration, but if you want to take a look, check out this listener for a starting point. sample code this will rely on an abstract class that will provided basic flow and chaining functionality to any subclasses. it acts as a very simple template class. first, let’s create the abstract class that gives us chaining behaviour: import static org.quartz.jobbuilder.newjob; import static org.quartz.triggerbuilder.newtrigger; import org.quartz.*; import org.quartz.impl.*; public abstract class chainablejob implements job { private static final string chain_job_class = "chainedjobclass"; private static final string chain_job_name = "chainedjobname"; private static final string chain_job_group = "chainedjobgroup"; @override public void execute(jobexecutioncontext context) throws jobexecutionexception { // execute actual job code doexecute(context); // if chainjob() was called, chain the target job, passing on the jobdatamap if (context.getjobdetail().getjobdatamap().get(chain_job_class) != null) { try { chain(context); } catch (schedulerexception e) { e.printstacktrace(); } } } // actually schedule the chained job to run now private void chain(jobexecutioncontext context) throws schedulerexception { jobdatamap map = context.getjobdetail().getjobdatamap(); @suppresswarnings("unchecked") class jobclass = (class) map.remove(chain_job_class); string jobname = (string) map.remove(chain_job_name); string jobgroup = (string) map.remove(chain_job_group); jobdetail jobdetail = newjob(jobclass) .withidentity(jobname, jobgroup) .usingjobdata(map) .build(); trigger trigger = newtrigger() .withidentity(jobname + "trigger", jobgroup + "trigger") .startnow() .build(); system.out.println("chaining " + jobname); stdschedulerfactory.getdefaultscheduler().schedulejob(jobdetail, trigger); } protected abstract void doexecute(jobexecutioncontext context) throws jobexecutionexception; // trigger job chain (invocation waits for job completion) protected void chainjob(jobexecutioncontext context, class jobclass, string jobname, string jobgroup) { jobdatamap map = context.getjobdetail().getjobdatamap(); map.put(chain_job_class, jobclass); map.put(chain_job_name, jobname); map.put(chain_job_group, jobgroup); } } there’s a fair bit of code here, but it’s nothing too complicated. we create the basic flow for job chaining by creating an abstract class which calls a doexecute() method in the child class, then chains the job if it was requested by calling chainjob() . so how do we use it? check out the job below. it actually chains to itself to demonstrate that you can chain any job and that it can be conditional. in this case, we will chain the job to another instance of the same class if it hasn’t already been chained, and we get a true value from new random().nextboolean() . import java.util.*; import org.quartz.*; public class testjob extends chainablejob { @override protected void doexecute(jobexecutioncontext context) throws jobexecutionexception { jobdatamap map = context.getjobdetail().getjobdatamap(); system.out.println("executing " + context.getjobdetail().getkey().getname() + " with " + new linkedhashmap(map)); boolean alreadychained = map.get("jobvalue") != null; if (!alreadychained) { map.put("jobtime", new date().tostring()); map.put("jobvalue", new random().nextlong()); } if (!alreadychained && new random().nextboolean()) { chainjob(context, testjob.class, "secondjob", "secondjobgroup"); } } } the call to chainjob() at the end will result in the automatic job chaining behaviour in the parent class. note that this isn’t called immediately, but only executes after the job completes its doexecute() method. here’s a simple harness that demonstrates everything together: import org.quartz.*; import org.quartz.impl.*; public class test { public static void main(string[] args) throws exception { // start up scheduler stdschedulerfactory.getdefaultscheduler().start(); jobdetail job = jobbuilder.newjob(testjob.class) .withidentity("firstjob", "firstjobgroup").build(); // trigger our source job to triggers another trigger trigger = triggerbuilder.newtrigger() .withidentity("firstjobtrigger", "firstjobbtriggergroup") .startnow() .withschedule( simpleschedulebuilder.simpleschedule().withintervalinseconds(1) .repeatforever()).build(); stdschedulerfactory.getdefaultscheduler().schedulejob(job, trigger); thread.sleep(5000); // let job run a few times stdschedulerfactory.getdefaultscheduler().shutdown(); } } sample output executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=5420204983304142728, jobtime=sat mar 02 15:19:29 pst 2013} executing firstjob with {} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=-2361712834083016932, jobtime=sat mar 02 15:19:31 pst 2013} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=7080718769449337795, jobtime=sat mar 02 15:19:32 pst 2013} executing firstjob with {} chaining secondjob executing secondjob with {jobvalue=7235143258790440677, jobtime=sat mar 02 15:19:33 pst 2013} executing firstjob with {} deficiencies well, we’re up and chaining, but there are some problems with this approach: it doesn’t integrate with a container like spring to use configured jobs. more code would be required. it forces you to know up front which jobs you want to chain, and write code for it. configuration is fixed, unless, once again, you write more code. no real-time changes (unless you write more code). a fair bit of code to maintain , and high likelihood you will have to expand it for more functionality. the theme here is that it’s doable, but it’s up to you to do the work to make it happen. obsidian avoids these problems by making chaining configurable, instead of it being a feature of the job itself. read on to find out how. chaining in obsidian in contrast to quartz, chaining in obsidian requires no code and no up-front knowledge of which jobs will chain or how you might want to chain them later. chaining is a form of configuration, and like all job-related configuration in obsidian, you can make live changes at any time without a build or any code at all. job configuration can use a native rest api or the web ui that’s included with obsidian. the following chaining features are available for free: no code and no redeploy to add or remove chains. you can chain specific configurations of job classes. you can chain only on certain states, including failure. chain conditionally based on source job saved state (equivalent to quartz’s jobdatamap), including multiple conditions. regexp/equals/greater than, etc. chain only when matching a schedule. check out the feature and ui documentation to find out more. now that we know what’s possible, let’s see an example. once you have your jobs configured , just create a new chain using the ui. rest api support will be here shortly but as of 1.5.1 chaining isn’t included in the api. if you need to script this right now, we can provide pointers . in the ui, it looks like the following: easy, huh? all configuration is stored in a database, so it’s easy to replicate it in various environments or to automate it via scripting. as a bonus, obsidian tracks and shows you all chaining state including what job triggered a chained job. it will even tell you why a job chain didn’t fire, whether it’s because the job status didn’t match, or one of your conditions didn’t. conclusion that summarizes how you can go about chaining in quartz and obsidian. quartz definitely has a minimalist approach, but that leaves developers with a lot of work to do. meanwhile, obsidian provides rich functionality out of the box to keep developers working on their own rich functionality, instead of the plumbing that so often seems to consume their time. if you have any suggestions or feature requests for obsidian, drop us a note by leaving a comment or by contacting us .

The term 'datastore’ is one that many of you who work with VMware are familiar, but which doesn’t really translate to the world of Microsoft’s Hyper-V. “Since Hyper-V does not require a different formatting of the underlying physical disk structure like VMFS(VMware’s proprietary disk format) we are able to browse the ‘datastore’ with File Explorer(In Windows 8/Server 2012…formerly known as Windows Explorer).” “Who said that?” That quote was from my friend Tommy Patterson, who writes about datastores and how they compare to the file system structures used in Hyper-V in Part 4 of our “20+ Days of Server Virtualization” series. In his article he describes the locations of the various components that define and make up a Hyper-V virtual machine, and even provides a script to help you quickly locate filesystem locations for your virtual machine bits. READ HIS ARTICLE HERE