This article introduces GapList, an implementation which strives for combining the strengths of both ArrayList and LinkedList.

Hey, let’s be careful out there. --Sergeant Esterhaus, daily briefing to the force of Hill Street Blues When developers run into an unexpected bug and can’t fix it, they’ll “add some defensive code” to make the code safer and to make it easier to find the problem. Sometimes just doing this will make the problem go away. They’ll tighten up data validation – making sure to check input and output fields and return values. Review and improve error handling – maybe add some checking around “impossible” conditions. Add some helpful logging and diagnostics. In other words, the kind of code that should have been there in the first place. Expect the Unexpected The whole point of defensive programming is guarding against errors you don’t expect. ---Steve McConnell, Code Complete The few basic rules of defensive programming are explained in a short chapter in Steve McConnell’s classic book on programming, Code Complete: Protect your code from invalid data coming from “outside”, wherever you decide “outside” is. Data from an external system or the user or a file, or any data from outside of the module/component. Establish “barricades” or “safe zones” or “trust boundaries” – everything outside of the boundary is dangerous, everything inside of the boundary is safe. In the barricade code, validate all input data: check all input parameters for the correct type, length, and range of values. Double check for limits and bounds. After you have checked for bad data, decide how to handle it. Defensive Programming is NOT about swallowing errors or hiding bugs. It’s about deciding on the trade-off between robustness (keep running if there is a problem you can deal with) and correctness (never return inaccurate results). Choose a strategy to deal with bad data: return an error and stop right away (fast fail), return a neutral value, substitute data values, … Make sure that the strategy is clear and consistent. Don’t assume that a function call or method call outside of your code will work as advertised. Make sure that you understand and test error handling around external APIs and libraries. Use assertions to document assumptions and to highlight “impossible” conditions, at least in development and testing. This is especially important in large systems that have been maintained by different people over time, or in high-reliability code. Add diagnostic code, logging and tracing intelligently to help explain what’s going on at run-time, especially if you run into a problem. Standardize error handling. Decide how to handle “normal errors” or “expected errors” and warnings, and do all of this consistently. Use exception handling only when you need to, and make sure that you understand the language’s exception handler inside out. Programs that use exceptions as part of their normal processing suffer from all the readability and maintainability problems of classic spaghetti code. --The Pragmatic Programmer I would add a couple of other rules. From Michael Nygard’s Release It! n Never ever wait forever on an external call, especially a remote call. Forever can be a long time when something goes wrong. Use time-out/retry logic and his Circuit Breaker stability pattern to deal with remote failures. And for languages like C and C++, defensive programming also includes using safe function calls to avoid buffer overflows and common coding mistakes. Different Kinds of Paranoia The Pragmatic Programmer describes defensive programming as “Pragmatic Paranoia”. Protect your code from other people’s mistakes, and your own mistakes. If in doubt, validate. Check for data consistency and integrity. You can’t test for every error, so use assertions and exception handlers for things that “can’t happen”. Learn from failures in test and production – if this failed, look for what else can fail. Focus on critical sections of code – the core, the code that runs the business. Healthy Paranoid Programming is the right kind of programming. But paranoia can be taken too far. In the Error Handling chapter of Clean Code, Michael Feathers cautions that “many code bases are dominated by error handling” --Michael Feathers, Clean Code Too much error handling code not only obscures the main path of the code (what the code is actually trying to do), but it also obscures the error handling logic itself – so that it is harder to get it right, harder to review and test, and harder to change without making mistakes. Instead of making the code more resilient and safer, it can actually make the code more error-prone and brittle. There’s healthy paranoia, then there’s over-the-top-error-checking, and then there’s bat shit crazy crippling paranoia – where defensive programming takes over and turns in on itself. The first real world system I worked on was a “Store and Forward” network control system for servers (they were called minicomputers back then) across the US and Canada. It shared data between distributed systems, scheduled jobs, and coordinated reporting across the network. It was designed to be resilient to network problems and automatically recover and restart from operational failures. This was ground breaking stuff at the time, and a hell of a technical challenge. The original programmer on this system didn’t trust the network, didn’t trust the O/S, didn’t trust Operations, didn’t trust other people’s code, and didn’t trust his own code – for good reason. He was a chemical engineer turned self-taught system programmer who drank a lot while coding late at night and wrote thousands of lines of unstructured FORTRAN and Assembler under the influence. The code was full of error checking and self diagnostics and error-correcting code, the files and data packets had their own checksums and file-level passwords and hidden control labels, and there was lots of code to handle sequence accounting exceptions and timing-related problems – code that mostly worked most of the time. If something went wrong that it couldn’t recover from, programs would crash and report a “label of exit” and dump the contents of variables – like today’s stack traces. You could theoretically use this information to walk back through the code to figure out what the hell happened. None of this looked anything like anything that I learned about in school. Reading and working with this code was like programming your way out of Arkham Asylum. If the programmer ran into bugs and couldn’t fix them, that wouldn’t stop him. He would find a way to work around the bugs and make the system keep running. Then later after he left the company, I would find and fix a bug and congratulate myself until it broke some “error-correcting” code somewhere else in the network that now depended on the bug being there. So after I finally figured out what was going on, I took out as much of this “protection” as I could safely remove, and cleaned up the error handling so that I could actually maintain the system without losing what was left of my mind. I setup trust boundaries for the code – although I didn’t know that’s what it was called then – deciding what data couldn’t be trusted and what could. Once this was done I was able to simplify the defensive code so that I could make changes without the system falling over itself, and still protect the core code from bad data, mistakes in the rest of the code, and operational problems. Making code safer is simple The point of defensive coding is to make the code safer and to help whoever is going to maintain and support the code – not make their job harder. Defensive code is code – all code has bugs, and, because defensive code is dealing with exceptions, it is especially hard to test and to be sure that it will work when it has to. Understanding what conditions to check for and how much defensive coding is needed takes experience, working with code in production and seeing what can go wrong in the real world. A lot of the work involved in designing and building secure, resilient systems is technically difficult or expensive. Defensive programming is neither – like defensive driving, it’s something that everyone can understand and do. It requires discipline and awareness and attention to detail, but it’s something that we all need to do if we want to make the world safe.

The Map Reduce Framework works in two main phases to process the data, which are the "map" phase and the "reduce" phase.

One of the things that all Spring developers like to do is to shoehorn Spring into any application they work on - it’s one of my guilty pleasures in life: you see some code, think it’s rubbish because it contains several well known anti-patterns and then think how cool it would be if this app was a Spring app. When working with legacy code, you can’t convert it into a fully fledged Spring app over night, that takes time. What you need to do is to add Spring code a little at a time: piece by piece and there’s one good way of doing that. In the following scenario, you’re working on some legacy code and you’ve written a Spring bean called: MySpringBean and it needs to use the legacy class: LegacyAppClass The legacy class looks like this: public class LegacyAppClass { // some old code goes here public void legacyDoSomethingMethod() { System.out.println("This is so old it doesn't use a logger...."); } } ...whilst your new SpringBean looks like this: public class MySpringBean { private LegacyAppClass injectedBean; @Override public String toString() { return "The toString()"; } public LegacyAppClass getInjectedBean() { return injectedBean; } public void setInjectedBean(LegacyAppClass injectedBean) { this.injectedBean = injectedBean; } public void myDoSomethingMethod() { injectedBean.legacyDoSomethingMethod(); } } ...as you can see, the myDoSomethingMethod() method needs to call the legacy legacyDoSomethingMethod() method. Given that any legacy application will have its own way of creating various objects, and that your new Spring code will need to use those objects to get its job done, then you need a way of combining the legacy objects with your shiny new ones. This will usually involve adding the legacy objects into your Spring Context and injecting them into your objects and to do this you need Spring’s StaticApplicationContext. @Test public void loadExternalClassTest2() { LegacyAppClass myInstance = new LegacyAppClass(); GenericApplicationContext parentContext = new StaticApplicationContext(); parentContext.getBeanFactory().registerSingleton("injectedBean", myInstance); parentContext.refresh(); // seems to be required sometimes ApplicationContext context = new ClassPathXmlApplicationContext( new String[] { "SpringIntegrationExample.xml" }, parentContext); MySpringBean mySpringBean = context.getBean(MySpringBean.class); assertNotNull(mySpringBean); mySpringBean.myDoSomethingMethod(); System.out.println(mySpringBean.toString()); } In the test code above the first point to note is that I create an instance of LegacyAppClass for use by the test, but in a real world app this will have already been created somewhere in your legacy code base. The next three lines is where the magic happens... GenericApplicationContext parentContext = new StaticApplicationContext(); parentContext.getBeanFactory().registerSingleton("injectedBean", myInstance); parentContext.refresh(); // seems to be required sometimes ...in the snippet above, you can see that I’m creating a StaticApplicationContext and then pragmatically adding my legacy class instance to it. ApplicationContext context = new ClassPathXmlApplicationContext( new String[] { "SpringIntegrationExample.xml" }, parentContext); The final task, as shown above, is to then create a new Spring application context using whatever method is suitable for your project. In this case, I’ve used the proverbial ClassPathXmlApplicationContext but other types of app context work just as well. You may say that this is a simple Micky-Mouse example, but from experience it does scale very well. It’s currently being used by a couple of full scale old style JSP Front Strategy MVC applications, (covered in detail in my blog from last October called Everybody Knows About MVC), as part of an implementation of Martin Fowler’s Strangler Pattern. Finally, in the interests of completeness, below is the XML config for this example:

Marker Interfaces in Java have special significance because of the fact that they have no methods declared in them which means that the classes implementing these interfaces don't have to override any of the methods. A few of the marker interfaces already exist in the JDK like Serializable and Cloneable. One can also create their own custom interfaces which doesn't have any method. The purpose of these interfaces is to force some kind of functionality in the classes by providing some functionality to a class if it implements the marker interface. A common question asked very frequently is about Runnable interface being marker or not. Runnable interface is not marker because Runnable interface has the public void run() method declared inside it. A very good example of marker interface is Serializable where the class implements can be used with ObjectOutputStream and ObjectInputStream classes. The Java language specification doesn't itself define the term marker interface and the term has been coined by authors, developers and designers. One common question asked is if we can create a marker interface or not and the answer is yes because of following reason: We can't create marker interface similar to Serializable or Cloneable but we can simulate the functionality by writing extra code around the custom marker interface.

One question that came up a couple times this week is how to set gradle up to deploy jars locally. For the most part I was satisfied with just having people push snapshot releases to our Artifactory server but some people did express a real desire to be able to publish a jar to the local resolution cache to test changes out locally. I’m still a fan of deploying snapshots from feature branches but luckily you can do a local publish and resolve with gradle. First off, ask yourself if the dependency is coupled enough to warrant being a submodule. Also, could just linking the project in your IDE be enough to get what you want done? If the answer to both questions are no then your next recourse is to use gradle’s excellent maven compatibility (don’t run!). For the project you want to publish locally you simply need to apply the maven plugin and make sure you have version and group set for the project (usually I put group and version in gradle.properties). apply plugin: 'java' apply plugin: 'maven' version = '0.5.1-SNAPSHOT' group = 'org.jamescarr.examples' That’s all you need to install it locally, just run gradle install from the project root to install it to the local m2 cache. Now let’s update your project that will depend on it. ... repositories { mavenCentral() mavenLocal() } dependencies { compile('org.jamecarr.examples:example-api:0.5.1-SNAPSHOT'){ changing=true } } The magic sauce here is using mavenLocal() as one of your resolution repositories. This will resolve against the local m2 cache. mavenCentral() can be replaced by whatever repositories you might use, it is only included since it’s the most often used. That’s it! I know some people dislike this approach due to ingrained disdain for maven but the beauty of it is that maven is silently at work and you really don’t get bothered by it.

The one major difference between IDEA and Eclipse is that IDEA "feels context", which effectively makes IDEA intelligent.

In database programming, it happens a lot that you need to bind a picture box to a field with type of photo or image. For example, if you want to show an Employee’s picture from Northwind.mdb database, you might want to try the following code: picEmployees.DataBindings.Add(“Image”, bsEmployees, “Photo”, true); This code works if the images are stored in the database with no OLE header or the images stored as a raw image file formats. As the pictures stored in the Northwind database in are not stored in raw image file formats and they are stored as an OLE image documents, then you have to strip off the OLE header to work with the image properly. Binding imageBinding = new Binding("Image", bsEmployees, "ImageBlob.ImageBlob", true); imageBinding.Format += new ConvertEventHandler(this.PictureFormat); private void PictureFormat(object sender, ConvertEventArgs e) { Byte[] img = (Byte[])e.Value; MemoryStream ms = new MemoryStream(); int offset = 78; ms.Write(img, offset, img.Length - offset); Bitmap bmp = new Bitmap(ms); ms.Close(); // Writes the new value back e.Value = bmp; } Fortunately, there are some overload methods in .NET Framework to take care of this mechanism, but it cannot be guaranteed whether you need to strip off the OLE object by yourself or not. For example, you can use the following technique to access the images of the Northwind.mdb that ships with Microsoft Access and they will be rendered properly. picEmployees.DataBindings.Add(“Image”, bsEmployees, “Photo”, true, DataSourceUpdateMode.Never, new Bitmap(typeof(Button), “Button.bmp”)); Unfortunately, there are some scenarios that you need a better solution. For example, the Xtreme.mdb database that ships with Crystal Reports has a photo filed that cannot be handled by the preceding methods. For these complex scenarios, you can download the OLEStripper classes from here and re-write the PictureFormat method as it is shown below: private void PictureFormat(object sender, ConvertEventArgs e) { // photoIndex is same as Employee ID int photoIndex = Convert.ToInt32(e.Value); // Read the original OLE object ReadOLE olePhoto = new ReadOLE(); string PhotoPath = olePhoto.GetOLEPhoto(photoIndex); // Strip the original OLE object StripOLE stripPhoto = new StripOLE(); string StripPhotoPath = stripPhoto.GetStripOLE(PhotoPath); FileStream PhotoStream = new FileStream(StripPhotoPath , FileMode.Open); Image EmployeePhoto = Image.FromStream(PhotoStream); e.Value = EmployeePhoto; PhotoStream.Close(); }

One of my favorite subjects is real-time communication. A chat component is one of the most basic forms of real-time communication. In the past, I've blogged about how to create a JavaFX 1.3 based chat application using a Comet Server. Meanwhile, we updated the internal Chat Application we are using at LodgON and it is now using JavaFX 2 and RedFX 2. The most basic JavaFX Chat Application is described at http://redfx.org/samples/chat/index.html and the required binaries can also be downloaded from that site. The example shown on the RedFX samples page is very very basic and of course not very useful in real-world cases. However, the basics about sending and receiving messages, processing them and visualizing them, are very similar. We use the exact same principles for a more complex Chat Application for focus groups that we are currently migrating from JavaFX 1.3 to JavaFX 2.1. A Chat application requires a client component and a server component. If you look at the client code that can be downloaded from the RedFX.org download section, you'll probably agree that JavaFX is the perfect candidate platform for writing chat applications. Very few lines of code are needed, and any Java developer can easily extend this application. The server component, which can be downloaded here contains a Java EE 6 Web Archive. It includes the RedFX server components, a few configuration files, and it runs out of the box on Glassfish 3.1.2 (if you enable Comet and/or Web sockets -- those are disabled by default unfortunately). A couple of weeks ago, we made the beta-release of RedFX available in a binary form. It is our intention to make available all source code that is required to run the basic samples. However, this takes time. A couple of months ago, we open-sourced the DaliCore platform, and that takes time. Making a project open-source involves much much more than putting the code in a zip and make it available online. It is my goal to make the RedFX client part of the open-source JFXtras.org project. We still have to figure out how we will deal with dependencies, and how/where we can host the RedFX server components. We will do this as fast as we can, but there are only 25 hours in a day...

storing massive amounts of data in a nosql data store is just one side of the big data equation. being able to visualize your data in such a way that you can easily gain deeper insights , is where things really start to get interesting. lately, i've been exploring various options for visualizing (directed) graphs, including circos . circos is an amazing software package that visualizes your data through a circular layout . although it's originally designed for displaying genomic data , it allows to create good-looking figures from data in any field. just transform your data set into a tabular format and you are ready to go. the figure below illustrates the core concept behind circos. the table's columns and rows are represented by segments around the circle. individual cells are shown as ribbons , which connect the corresponding row and column segments. the ribbons themselves are proportional in width to the value in the cell. when visualizing a directed graph , nodes are displayed as segments on the circle and the size of the ribbons is proportional to the value of some property of the relationships. the proportional size of the segments and ribbons with respect to the full data set allows you to easily identify the key data points within your table. in my case, i want to better understand the flow of visitors to and within the datablend site and blog; where do visitors come from (direct, referral, search, ...) and how do they navigate between pages. the rest of this article details how to 1) retrieve the raw visit information through the google analytics api, 2) persist this information as a graph in neo4j and 3) query and preprocess this data for visualization through circos. as always, the complete source code can be found on the datablend public github repository . 1. retrieving your google analytics data let's start by retrieving the raw google analytics data . the google analytics data api provides access to all dimensions and metrics that can be queried through the web application. in my case, i'm interested in retrieving the previous page path property for each page view. if a visitor enters through a page outside of the datablend website, the previous page path is marked as (entrance) . otherwise, it contains the internal path . we will use google's java data api to connect and retrieve this information. we are particularly interested in the pagepath , pagetitle , previouspagepath and medium dimensions, while our metric of choice is the number of pageviews . after setting the date range, the feed of entries that satisfy this criteria can be retrieved. for ease of use, we transform this data to a domain entity and filter/clean the data accordingly. if a visit originates from outside the datablend website, we store the specific medium (direct, referral, search, ...) as previous path. // authenticate analyticsservice = new analyticsservice(configuration.service); analyticsservice.setusercredentials(configuration.client_username, configuration.client_pass); // create query dataquery query = new dataquery(new url(configuration.data_url)); query.setids(configuration.table_id); query.setdimensions("ga:medium,ga:previouspagepath,ga:pagepath,ga:pagetitle"); query.setmetrics("ga:pageviews"); query.setstartdate(datestring); query.setenddate(datestring); // execute datafeed feed = analyticsservice.getfeed(createqueryurl(date), datafeed.class); // iterate and clean for (dataentry entry : feed.getentries()) { string pagepath = entry.stringvalueof("ga:pagepath"); string pagetitle = entry.stringvalueof("ga:pagetitle"); string previouspagepath = entry.stringvalueof("ga:previouspagepath"); string medium = entry.stringvalueof("ga:medium"); long views = entry.longvalueof("ga:pageviews"); // filter the data if (filter(pagepath) && filter(previouspagepath) && (!clean(previouspagepath).equals(clean(pagepath)))) { // check criteria are satisfied navigation navigation = new navigation(clean(previouspagepath), clean(pagepath), pagetitle, date, views); if (navigation.getsource().equals("(entrance)")) { // in case of an entrace, save its medium instead navigation.setsource(medium); } navigations.add(navigation); } } 2. storing navigational data as a directed graph in neo4j the set of site navigations can easily be stored as a directed graph in the neo4j graph database . nodes are site paths (or mediums), while relationships are the navigations themselves. we start by retrieving the navigations for a particular date range and retrieve (or lazily create) the nodes representing the source and target paths (or mediums). next we de-normalize the pageviews metric (for instance, 6 individual relationships will be created for 6 page-views). although this de-normalization step is not really required, i did so to make sure that the degree of my nodes is correct if i would perform other types of calculations. for each individual navigation relationship, we also store the date of visit . // retrieve navigations for a particular date list navigations = retrieval.getnavigations(date); // save them in the graph database transaction tx = graphdb.begintx(); // iterate and create for (navigation nav : navigations) { node source = getpath(nav.getsource()); node target = getpath(nav.gettarget()); if (!target.hasproperty("title")) { target.setproperty("title", nav.gettargettitle()); } for (long i = 0; i < nav.getamount(); i++) { // duplicate relationships relationship transition = source.createrelationshipto(target, relationships.navigation); transition.setproperty("date", date.gettime()); // save time as long } } // commit tx.success(); tx.finish(); 3. creating the circos tabular data format the circos tabular data format is quite easy to construct. it's basically a tab-delimited file with row and column headers. a cell is interpreted as a value that flows from the row entity to the column entity . we will use the neo4j cypher query language to retrieve the data of interest, namely all navigations that occurred within a certain time period . doing so allows us to create historical visualizations of our navigations and observe how visit flow behaviors are changing over time. // access the graph database graphdb = new embeddedgraphdatabase("var/analytics"); engine = new executionengine(graphdb); // execute the data range cypher query map params = new hashmap(); params.put("fromdate", from.gettime()); params.put("todate", to.gettime()); // execute the query executionresult result = engine.execute("start sourcepath=node:index(\"path:*\") " + "match sourcepath-[r]->targetpath " + "where r.date >= {fromdate} and r.date <= {todate} " + "return sourcepath,targetpath", params); next, we create the tab delimited file itself. we iterate through all entries (i.e. navigations) that match our cypher query and store them in a temporary list. afterwards, we start building the two-dimensional array by normalizing (i.e. summing) the number of navigations between the source and target paths. at the end, we filter this occurrence matrix on the minimal number of required navigations. this ensures that we will only create segments for paths that are relevant in the total population. as a final step, we print the occurrences matrix as a tab-delimited file. for each path, we will use a shorthand as the circos renderer seems to have problem with long string identifiers. // retrieve the results iterator> it = result.javaiterator(); list navigations = new arraylist(); map titles = new hashmap(); set paths = new hashset(); // iterate the results while (it.hasnext()) { map record = it.next(); string source = (string)((node) record.get("sourcepath")).getproperty("path"); string target = (string) ((node) record.get("targetpath")).getproperty("path"); string targettitle = (string) ((node) record.get("targetpath")).getproperty("title"); // reuse the navigation object as temorary holder navigations.add(new navigation(source, target, targettitle, new date(), 1)); paths.add(source); paths.add(target); if (!titles.containskey(target)) { titles.put(target, targettitle); } } // retrieve the various paths list pathids = arrays.aslist(paths.toarray(new string[]{})); // create the matrix that holds the info int[][] occurences = new int[pathids.size()][pathids.size()]; // iterate through all the navigations and update accordingly for (navigation navigation : navigations) { int sourceindex = pathids.indexof(navigation.getsource()); int targetindex = pathids.indexof(navigation.gettarget()); occurences[sourceindex][targetindex] = occurences[sourceindex][targetindex] + 1; } // matrix build, filter on threshold for (int i = 0; i < occurences.length; i++) { for (int j = 0; j < occurences.length; j++) { if (occurences[i][j] < threshold) { occurences[i][j] = 0; } } // print printcircosdata(pathids, titles, occurences); the text below is a sample of the output generated by the printcircosdata method. it first prints the legend (matching shorthands with actual paths). next it prints the tab-delimited circos table. link0 - /?p=411/wp-admin - storing and querying rdf data in neo4j through sail - datablend link1 - /?p=1146 - visualizing rdf schema inferencing through neo4j, tinkerpop, sail and gephi - datablend link2 - /?p=164 - big data / concise articles - datablend link3 - referral - null link4 - /?p=1400 - the joy of algorithms and nosql revisited: the mongodb aggregation framework - datablend ... datal0l1l2l3l4... l000000 l100000 l200000 l3059400197 l400000 4. use the circos power although circos can be installed on your local computer, we will use its online version to create the visualization of our data. upload your tab-delimited file and just wait a few seconds before enjoying the beautiful rendering of your site's navigation information. with just a glimpse of an eye we can already see that the l3-segment (i.e. the referrals) is significantly larger (almost 6000 navigations) compared to the others segments. the outer 3 rings visualize the total amounts of navigations that are leaving and entering this particular path. in case of referrals, no navigations have this path as target (indicated by the empty middle ring). its total segment count (inner ring) is entirely build up out of navigations that have a referral as source. the l6-segment seems to be the path that attracts the most traffic (around 2500 navigations). this segment visualizes the navigation data related to my "the joy of algorithms and nosql: a mongodb example" -article. most of its traffic is received through referrals, while a decent amount is also generated through direct (l17-segment) and search (l27-segment) traffic. the l15-segment (my blog's main page) is the only path that receives an almost equal amount of incoming and outgoing traffic. with just a few tweaks to the circos input data, we can easily focus on particular types of navigation data. in the figure below, i made sure that referral and search navigations are visualized more prominently through the use of 2 separate colors. 5. conclusions in the era of big data, visualizations are becoming crucial as they enable us to mine our large data sets for certain patterns of interest. circos specializes in a very specific type of visualization, but does its job extremely well. i would be delighted to hear about other types of visualizations for directed graphs.

JMS providers like ActiveMQ are based on the concept of passing one-directional messages between nodes and brokers asynchronously. A thorough knowledge of the type of messages that can be sent through a JMS middleware can simplify a lot your work in mapping the communication patterns to real code. The basic Message interface Some object members are shared by all messages: header fields, used to identify univocally a message and to route it to the right brokers and consumers. A dynamic map of properties which can be read programmatically by JMS brokers in order to filter or to route messages. A body, which is differentiated in the various implementations we'll see. Header fields The set of getJMS*() methods on the Message interface defines the available headers. Two of them are oriented to message identification: getJMSMessageID() contains a generated ID for identifying a message, unique at least for the current broker. All generated IDs start with the prefix 'ID:', but you can override it with the corresponding setter. getJMSCorrelationID() (and getJMSCorrelationID() as bytes) can link a message with another, usually one that has been sent previously. For example, a reply can carry the ID of the original message when put in another queue. Two to sender and recipient identification: getJMSDestination() returns a Destination object (a Topic or a Queue, or their temporary version) describing where the message was directed. getJMSReplyTo() is a Destination object where replies should be sent; it can be null of course. Three tune the delivery mechanism: getJMSDeliveryMode() can be DeliveryMode.NON_PERSISTENT or DeliveryMode.PERSISTENT; only persistent messages guarantee delivery in case of a crash of the brokers that transport it. getJMSExpiration() returns a timestamp indicating the expiration time of the message; it can be 0 on a message without a defined expiration. getJMSPriority() returns a 0-9 integer value (higher is better) defining the priority for delivery. It is only a best-effort value. While the remaining ones contain metadata: getJMSRedelivered() returns a boolean indicating if the message is being delivered again after a delivery which was not acknowledge. getJMSTimestamp() returns a long indicating the time of sending. getJMSType() defines a field for provider-specific or application-specific message types. Of these headers, only JMSCorrelationID, JMSReplyTo and JMSType have to be set when needed. The others are generated or managed by the send() and publish() methods if not specified (there are setters available for each of these headers.) Properties Generic properties with a String name can be added to messages and read with getBooleanProperty(), getStringProperty() and similar methods. The corresponding setters setBooleanProperty(), setStringProperty(), ... can be used for their addition. The reason for keeping some properties out of the content of the message (which a MapMessage can contain) is so they could be read before reaching the destination, for example in a JMS broker. The use case for this access to message properties is routing and filtering: downstream brokers and consumers may define a filter such as "I am interested only on messages on this Topic that have property X = 'value' and Y = 'value2'". Bodies All the subinterfaces of javax.jms.Message defined by the API provide different types of message bodies (while actual classes are defined by the providers and are not part of the API). Actual instantiation is then handled by the Session, which implements an Abstract Factory pattern. On the receival side, a cast is necessary for any message type (at least in the Java JMS Api), since only a generic Message is read. BytesMessage is the most basic type: it contains a sequence of uninterpreted bytes. Hence, it can in theory contain anything, but the generation and interpretation of the content is the client's job. BytesMessage m = session.createBytesMessage(); m.writeByte(65); m.writeBytes(new byte[] { 66, 68, 70 }); // on receival (cast shown only here) BytesMessage m = (BytesMessage) genericMessage; byte[] content = new byte[4]; m.readBytes(content); MapMessage defines a message containing an (unordered) set of key/value pairs, also called a map or dictionary or hash. However, the keys are String objects, while the values are primitives or Strings; since they are primitives, they shouldn't be null. MapMessage = session.createMapMessage(); m.setString('key', 'value'); // or m.setObject('key', 'value') to avoid specifying a type // on receival m.getString('key'); // or m.getObject('key') ObjectMessage wraps a generic Object for transmission. The Object should be Serializable. ObjectMessage m = session.createObjectMessage(); m.setObject(new ValueObject('field1', 42)); // on receival ValueObject vo = (ValueObject) m.getObject(); StreamMessage wraps a stream of primitive values of indefinite length. StreamMessage m = session.createStreamMessage(); m.writeBoolean(true); m.writeBoolean(false); m.writeBoolean(true); // receival System.out.println(m.readBoolean()); System.out.println(m.readBoolean()); System.out.println(m.readBoolean()); // prints true, false, true TextMessage wraps a String of any length. TextMessage m = session.createTextMessage("Contents"); // or use m.setText() afterwards // receival String text = m.getText(); Usually all messages are in a read-only phase after receival, so only getters can be called on them.

i have been reading up on join implementations available for hadoop for past few days. in this post i recap some techniques i learnt during the process. the joins can be done at both map side and join side according to the nature of data sets of to be joined. reduce side join let’s take the following tables containing employee and department data. let’s see how join query below can be achieved using reduce side join. select employees.name, employees.age, department.name from employees inner join department on employees.dept_id=department.dept_id map side is responsible for emitting the join predicate values along with the corresponding record from each table so that records having same department id in both tables will end up at on same reducer which would then do the joining of records having same department id. however it is also required to tag the each record to indicate from which table the record originated so that joining happens between records of two tables. following diagram illustrates the reduce side join process. here is the pseudo code for map function for this scenario. map (k table, v rec) { dept_id = rec.dept_id tagged_rec.tag = table tagged_rec.rec = rec emit(dept_id, tagged_rec) } at reduce side join happens within records having different tags. reduce (k dept_id, list tagged_recs) { for (tagged_rec : tagged_recs) { for (tagged_rec1 : taagged_recs) { if (tagged_rec.tag != tagged_rec1.tag) { joined_rec = join(tagged_rec, tagged_rec1) } emit (tagged_rec.rec.dept_id, joined_rec) } } map side join (replicated join) using distributed cache on smaller table for this implementation to work one relation has to fit in to memory. the smaller table is replicated to each node and loaded to the memory. the join happens at map side without reducer involvement which significantly speeds up the process since this avoids shuffling all data across the network even-though most of the records not matching are later dropped. smaller table can be populated to a hash-table so look-up by dept_id can be done. the pseudo code is outlined below. map (k table, v rec) { list recs = lookup(rec.dept_id) // get smaller table records having this dept_id for (small_table_rec : recs) { joined_rec = join (small_table_rec, rec) } emit (rec.dept_id, joined_rec) } using distributed cache on filtered table if the smaller table doesn’t fit the memory it may be possible to prune the contents of it if filtering expression has been specified in the query. consider following query. select employees.name, employees.age, department.name from employees inner join department on employees.dept_id=department.dept_id where department.name="eng" here a smaller data set can be derived from department table by filtering out records having department names other than “eng”. now it may be possible to do replicated map side join with this smaller data set. replicated semi-join reduce side join with map side filtering even of the filtered data of small table doesn’t fit in to the memory it may be possible to include just the dept_id s of filtered records in the replicated data set. then at map side this cache can be used to filter out records which would be sent over to reduce side thus reducing the amount of data moved between the mappers and reducers. the map side logic would look as follows. map (k table, v rec) { // check if this record needs to be sent to reducer boolean sendtoreducer = check_cache(rec.dept_id) if (sendtoreducer) { dept_id = rec.dept_id tagged_rec.tag = table tagged_rec.rec = rec emit(dept_id, tagged_rec) } } reducer side logic would be same as the reduce side join case. using a bloom filter a bloom filter is a construct which can be used to test the containment of a given element in a set. a smaller representation of filtered dept_ids can be derived if dept_id values can be augmented in to a bloom filter. then this bloom filter can be replicated to each node. at the map side for each record fetched from the smaller table the bloom filter can be used to check whether the dept_id in the record is present in the bloom filter and only if so to emit that particular record to reduce side. since a bloom filter is guaranteed not to provide false negatives the result would be accurate. references [1] hadoop in action [2] hadoop : the definitive guide

My prefered solution was to use the Maven ‘update snapshots’ command line argument.

I solved a little problem last night when playing with named pipes. I created a named pipe that writes all output to a file. Named pipes are opened for all users on a single machine. In this post I will show you a simple class that works as a pipe server. In .NET-based languages we can use the System.IO.Pipes namespace classes to work with named pipes. Here is my simple pipe server that writes all client output to file. public class MyPipeServer { public void Run() { var sid = new SecurityIdentifier(WellKnownSidType.WorldSid, null); var rule = new PipeAccessRule(sid, PipeAccessRights.ReadWrite, AccessControlType.Allow); var sec = new PipeSecurity(); sec.AddAccessRule(rule); using (NamedPipeServerStream pipeServer = new NamedPipeServerStream ("testpipe",PipeDirection.InOut, 100, PipeTransmissionMode.Byte, PipeOptions.None, 0, 0, sec)) { pipeServer.WaitForConnection(); var read = 0; var bytes = new byte[4096]; using(var file=File.Open(@"c:\tmp\myfile.dat", FileMode.Create)) while ((read = pipeServer.Read(bytes, 0, bytes.Length)) > 0) { file.Write(bytes, 0, read); file.Flush(); } } } } Real-life pipe scenarios are usually more complex but this simple class is good to get things running like they should be.

Use short enough and long enough variable names in each scope of code. Generally length may be 1 char for loop counters, 1 word for condition/loop variables, 1-2 words for methods, 2-3 words for classes, 3-4 words for globals. Use specific names for variables, for example "value", "equals", "data", ... are not valid names for any case. Use meaningful names for variables. Variable name must define the exact explanation of its content. Don't start variables with o_, obj_, m_ etc. A variable does not need tags which states it is a variable. Obey company naming standards and write variable names consistently in application: e.g. txtUserName, lblUserName, cmbSchoolType, ... Otherwise readability will reduce and find/replace tools will be unusable. Obey programming language standards and don't use lowercase/uppercase characters inconsistently: e.g. userName, UserName, USER_NAME, m_userName, username, ... use Camel Case (aka Upper Camel Case) for classes: VelocityResponseWriter use Lower Case for packages: com.company.project.ui use Mixed Case (aka Lower Camel Case) for variables: studentName use Upper Case for constants : MAX_PARAMETER_COUNT = 100 use Camel Case for enum class names and Upper Case for enum values. don't use '_' anywhere except constants and enum values (which are constants). For example for Java, Don't reuse same variable name in the same class in different contexts: e.g. in method, constructor, class. So you can provide more simplicity for understandability and maintainability. Don't use same variable for different purposes in a method, conditional etc. Create a new and different named variable instead. This is also important for maintainability and readability. Don't use non-ASCII chars in variable names. Those may run on your platform but may not on others. Don't use too long variable names (e.g. 50 chars). Long names will bring ugly and hard-to-read code, also may not run on some compilers because of character limit. Decide and use one natural language for naming, e.g. using mixed English and German names will be inconsistent and unreadable. Use meaningful names for methods. The name must specify the exact action of the method and for most cases must start with a verb. (e.g. createPasswordHash) Obey company naming standards and write method names consistently in application: e.g. getTxtUserName(), getLblUserName(), isStudentApproved(), ... Otherwise readability will reduce and find/replace tools will be unusable. Obey programming language standards and don't use lowercase/uppercase characters inconsistently: e.g. getUserName, GetUserName, getusername, ... For example for Java, use Mixed Case for method names: getStudentSchoolType use Mixed Case for method parameters: setSchoolName(String schoolName) Use meaningful names for method parameters, so it can documentate itself in case of no documentation.

Microsoft Downloads - Microsoft Robotics Developer Studio 4 Microsoft® Robotics Developer Studio 4 is a freely available .NET-based programming environment for building robotics applications. It can be used by both professional and non-professional developers as well as hobbyists. Version: 4.0.261.0 Date Published: 3/6/2012 Language:English Microsoft Robotics Developer Studio 4.exe, 289.6 MB Microsoft® Robotics Developer Studio 4 enables hobbyists and professional or non-professional developers to create robotics applications targeting a wide range of scenarios. This release is an update that has the functionality of the previous RDS 2008 R3 Standard Edition with the addition of support for the Kinect sensor and a defined Reference Platform, also referred to as MARK (Mobile Autonomous Robot using Kinect). Microsoft Robotics Developer Studio can support a broad set of robotics platforms by either running directly on the platform (if it has an embedded PC running Windows) or controlling the robot from a Windows PC through a communication channel such as Wi-Fi or Bluetooth®. In addition to providing support for Microsoft Visual Studio 2010, Microsoft Robotics Developer Studio 4 provides a Visual Programming Language (VPL) which allows developers to create applications simply by dragging and dropping components onto a canvas and wiring them together. The powerful Visual Simulation Environment (VSE) provides a high-fidelity simulation environment powered by NVIDIA™ PhysX™ engine for running game-quality 3D simulations with real-world physics interactions. To help developers getting started, Robotics Developer Studio contains extensive Documentationand a large set of samples and tutorials that illustrate how to write applications ranging from simple "Hello Robot" to complex applications that simultaneously run on multiple robots. ..." Microsoft Robotics Developer Studio "excitement with Microsoft Kinect within the robotics space signaling the potential opportunities that exist in transforming robots to low-cost mainstream consumer devices. RDS 4, with support from the Kinect sensor, aims to make it easier for developers to build applications, including those directed at personal robotics and consumer scenarios, both in hardware and in simulation. With RDS 4 simulation tools, you don’t need physical hardware to develop a Kinect-based robot! For hardware design, Microsoft has published the Robotics Developer Studio: Reference Platform Design specification for a standardized Kinect-based robot, to work with RDS 4.Parallax, Inc has manufactured a hardware kit based on the reference platform design specification which is available now. RDS 4 also adds support for Microsoft .NET Framework 4.0, XNA 4.0, and Silverlight 4.0. Developers can use the tool to build commercial products based on the RDS 4 platform. Here is an example of what can be done with RDS 4: ..." Robot's! Kinect! Free Development Environment! Yeah, I know I've most of that before, but come-on! Robots! Kinect!... :)

Django handles database connections transparently in almost all cases. It will start a new connection when your request starts up, and commit it at the end of the request lifetime. Other times you need to dive in further and do your own granular transaction management. But for the most part, it's fully automatic. However, sometimes your use case may require that you close the current database connection and open a new one. While this is possible in Django, it's not well documented. Why would you want to do this? I my case, I was writing an automation test framework. Some of the automation tests make database calls through the Django ORM to setup records, clean up after the test, etc. Each test is executed in the same process space, via a thread pool. We found that if one of the early tests threw an unrecoverable database error, such as an IntegrityError due to violating a unique constraint, the database connection would be aborted. Subsequent tests that tried to use the database would raise a DatabaseError: Traceback (most recent call last): File /home/user/project/app/test.py, line 73, in tearDown MyModel.objects.all() File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 444, in delete collector.collect(del_query) File /usr/local/lib/python2.6/dist-packages/django/db/models/deletion.py, line 146, in collect reverse_dependency=reverse_dependency) File /usr/local/lib/python2.6/dist-packages/django/db/models/deletion.py, line 91, in add if not objs: File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 113, in __nonzero__ iter(self).next() File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 107, in _result_iter self._fill_cache() File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 772, in _fill_cache self._result_cache.append(self._iter.next()) File /usr/local/lib/python2.6/dist-packages/django/db/models/query.py, line 273, in iterator for row in compiler.results_iter(): File /usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py, line 680, in results_iter for rows in self.execute_sql(MULTI): File /usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py, line 735, in execute_sql cursor.execute(sql, params) File /usr/local/lib/python2.6/dist-packages/django/db/backends/postgresql_psycopg2/base.py, line 44, in execute return self.cursor.execute(query, args) DatabaseError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. It turns out that it's relatively easy to reset the database connection. We just called the following function at the start of every test. Django is smart enough to re-initialize the connection the next time it's used, assuming that it's disconnected properly. def reset_database_connection(): from django import db db.close_connection()

This is the seventh of a series of articles about setting up a secure RESTful Web Service using Spring 3.1 and Spring Security 3.1 with Java based configuration. This article will focus on the implementation of pagination in a RESTful web service. The REST with Spring series: Part 1 – Bootstrapping a web application with Spring 3.1 and Java based Configuration Part 2 – Building a RESTful Web Service with Spring 3.1 and Java based Configuration Part 3 – Securing a RESTful Web Service with Spring Security 3.1 Part 4 – RESTful Web Service Discoverability Part 5 – REST Service Discoverability with Spring Part 6 – Basic and Digest authentication for a RESTful Service with Spring Security 3.1 You can check out the entire REST with Spring Series here. Page as resource vs Page as representation The first question when designing pagination in the context of a RESTful architecture is whether to consider the page an actual resource or just a representation of resources. Treating the page itself as a resource introduces a host of problems such as no longer being able to uniquely identify resources between calls. This, coupled with the fact that outside the RESTful context, the page cannot be considered a proper entity, but a holder that is constructed when needed makes the choice straightforward: the page is part of the representation. The next question in the pagination design in the context of REST is where to include the paging information: in the URI path: /foo/page/1 the URI query: /foo?page=1 Keeping in mind that a page is not a resource, encoding the page information in the URI is no longer an option. Page information in the URI query Encoding paging information in the URI query is the standard way to solve this issue in a RESTful service. This approach does however have one downside – it cuts into the query space for actual queries: /foo?page=1&size=10 The Controller Now, for the implementation – the Spring MVC Controller for pagination is straightforward: @RequestMapping( value = "admin/foo",params = { "page", "size" },method = GET ) @ResponseBody public List< Foo > findPaginated( @RequestParam( "page" ) int page, @RequestParam( "size" ) int size, UriComponentsBuilder uriBuilder, HttpServletResponse response ){ Page< Foo > resultPage = service.findPaginated( page, size ); if( page > resultPage.getTotalPages() ){ throw new ResourceNotFoundException(); } eventPublisher.publishEvent( new PaginatedResultsRetrievedEvent< Foo > ( Foo.class, uriBuilder, response, page, resultPage.getTotalPages(), size ) ); return resultPage.getContent(); } The two query parameters are defined in the request mapping and injected into the controller method via @RequestParam; the HTTP response and the Spring UriComponentsBuilder are injected in the Controller method to be included in the event, as both will be needed to implement discoverability. Discoverability for REST pagination Withing the scope of pagination, satisfying the HATEOAS constraint of REST means enabling the client of the API to discover the next and previous pages based on the current page in the navigation. For this purpose, the Link HTTP header will be used, coupled with the official “next“, “prev“, “first” and “last” link relation types. In REST, Discoverability is a cross cutting concern, applicable not only to specific operations but to types of operations. For example, each time a Resource is created, the URI of that resource should be discoverable by the client. Since this requirement is relevant for the creation of ANY Resource, it should be dealt with separately and decoupled from the main Controller flow. With Spring, this decoupling is achieved with events, as was thoroughly discussed in the previous article focusing on Discoverability of a RESTful service. In the case of pagination, the event – PaginatedResultsRetrievedEvent – was fired in the Controller, and discoverability is achieved in a listener for this event: void addLinkHeaderOnPagedResourceRetrieval( UriComponentsBuilder uriBuilder, HttpServletResponse response, Class clazz, int page, int totalPages, int size ){ String resourceName = clazz.getSimpleName().toString().toLowerCase(); uriBuilder.path( "/admin/" + resourceName ); StringBuilder linkHeader = new StringBuilder(); if( hasNextPage( page, totalPages ) ){ String uriNextPage = constructNextPageUri( uriBuilder, page, size ); linkHeader.append( createLinkHeader( uriForNextPage, REL_NEXT ) ); } if( hasPreviousPage( page ) ){ String uriPrevPage = constructPrevPageUri( uriBuilder, page, size ); appendCommaIfNecessary( linkHeader ); linkHeader.append( createLinkHeader( uriForPrevPage, REL_PREV ) ); } if( hasFirstPage( page ) ){ String uriFirstPage = constructFirstPageUri( uriBuilder, size ); appendCommaIfNecessary( linkHeader ); linkHeader.append( createLinkHeader( uriForFirstPage, REL_FIRST ) ); } if( hasLastPage( page, totalPages ) ){ String uriLastPage = constructLastPageUri( uriBuilder, totalPages, size ); appendCommaIfNecessary( linkHeader ); linkHeader.append( createLinkHeader( uriForLastPage, REL_LAST ) ); } response.addHeader( HttpConstants.LINK_HEADER, linkHeader.toString() ); } In short, the listener logic checks if the navigation allows for a next, previous, first and last pages and, if it does, adds the relevant URIs to the Link HTTP Header. It also makes sure that the link relation type is the correct one – “next”, “prev”, “first” and “last”. This is the single responsibility of the listener (the full code here). Test Driving Pagination Both the main logic of pagination and discoverability should be extensively covered by small, focused integration tests; as in the previous article, the rest-assured library is used to consume the REST service and to verify the results. These are a few example of pagination integration tests; for a full test suite, check out the github project (link at the end of the article): @Test public void whenResourcesAreRetrievedPaged_then200IsReceived(){ Response response = givenAuth().get( paths.getFooURL() + "?page=1&size=10" ); assertThat( response.getStatusCode(), is( 200 ) ); } @Test public void whenPageOfResourcesAreRetrievedOutOfBounds_then404IsReceived(){ Response response = givenAuth().get( paths.getFooURL() + "?page=" + randomNumeric( 5 ) + "&size=10" ); assertThat( response.getStatusCode(), is( 404 ) ); } @Test public void givenResourcesExist_whenFirstPageIsRetrieved_thenPageContainsResources(){ restTemplate.createResource(); Response response = givenAuth().get( paths.getFooURL() + "?page=1&size=10" ); assertFalse( response.body().as( List.class ).isEmpty() ); } Test Driving Pagination Discoverability Testing Discoverability of Pagination is relatively straightforward, although there is a lot of ground to cover. The tests are focused on the position of the current page in navigation and the different URIs that should be discoverable from each position: @Test public void whenFirstPageOfResourcesAreRetrieved_thenSecondPageIsNext(){ Response response = givenAuth().get( paths.getFooURL()+"?page=0&size=10" ); String uriToNextPage = extractURIByRel( response.getHeader( LINK ), REL_NEXT ); assertEquals( paths.getFooURL()+"?page=1&size=10", uriToNextPage ); } @Test public void whenFirstPageOfResourcesAreRetrieved_thenNoPreviousPage(){ Response response = givenAuth().get( paths.getFooURL()+"?page=0&size=10" ); String uriToPrevPage = extractURIByRel( response.getHeader( LINK ), REL_PREV ); assertNull( uriToPrevPage ); } @Test public void whenSecondPageOfResourcesAreRetrieved_thenFirstPageIsPrevious(){ Response response = givenAuth().get( paths.getFooURL()+"?page=1&size=10" ); String uriToPrevPage = extractURIByRel( response.getHeader( LINK ), REL_PREV ); assertEquals( paths.getFooURL()+"?page=0&size=10", uriToPrevPage ); } @Test public void whenLastPageOfResourcesIsRetrieved_thenNoNextPageIsDiscoverable(){ Response first = givenAuth().get( paths.getFooURL()+"?page=0&size=10" ); String uriToLastPage = extractURIByRel( first.getHeader( LINK ), REL_LAST ); Response response = givenAuth().get( uriToLastPage ); String uriToNextPage = extractURIByRel( response.getHeader( LINK ), REL_NEXT ); assertNull( uriToNextPage ); } These are just a few examples of integration tests consuming the RESTful service. Getting All Resources On the same topic of pagination and discoverability, the choice must be made if a client is allowed to retrieve all the Resources in the system at once, or if the client MUST ask for them paginated. If the choice is made that the client cannot retrieve all Resources with a single request, and pagination is not optional but required, then several options are available for the response to a get all request. One option is to return a 404 (Not Found) and use the Link header to make the first page discoverable: Link=; rel=”first“, ; rel=”last“ Another option is to return redirect – 303 (See Other) – to the first page of the pagination. A third option is to return a 405 (Method Not Allowed) for the GET request. REST Paginag with Range HTTP headers A relatively different way of doing pagination is to work with the HTTP Range headers – Range, Content-Range, If-Range, Accept-Ranges – and HTTP status codes – 206 (Partial Content), 413 (Request Entity Too Large), 416 (Requested Range Not Satisfiable). One view on this approach is that the HTTP Range extensions were not intended for pagination, and that they should be managed by the Server, not by the Application. Implementing pagination based on the HTTP Range header extensions is nevertheless technically possible, although not nearly as common as the implementation discussed in this article. Conclusion This article covered the implementation of Pagination in a RESTful service with Spring, discussing how to implement and test Discoverability. For a full implementation of pagination, check out the github project. From the original REST Pagination in Spring of the REST with Spring series

Over the last couple of months I have read up on volumes of Technical Analysis (“TA”) information, I have back tested probably hundreds of automated trading strategies against massive amounts of data, both exchange intraday- and tick data, as well as other sources. Some of these strategies have been massively profitable in back testing, others not so much. Some of the TA patterns, I’ve discarded before they even left the book, because they did not stand up to any sort of scientific scrutiny because they lacked a clear predictive thesis, where riddled with forward-looking bias (“Head and Shoulders patterns”), and in some cases where just plain bulls**t (“Elliott Wave Principle” comes to mind). The outcomes of my testing has made me think about the implications of large scale data analysis in general: it is very easy to get fooled by randomness. In many cases in my testing results have been amazing, but I cannot come up with a plausible causal explanation as to why, and when I gently nudge the parameters just ever so slightly, outcomes can look entirely different. Taking a step back from the data, looking at it in a larger perspective, I’m inclined to conclude that if data across multiple parameter variations looks like a random walk and lacks a plausible causal explanation, then it is a random walk. If I cannot say “X is caused by A and B”, I’m inclined to believe that the actual reason is “X is the result because A and B fit the historical data D, but may not do so in the future”. And herein lies the crux of the matter: how many data scientists are inclined to take a step back, rather than just assume that there is a pattern there? How many are prepared to do so if their livelihood is largely based on them finding patterns, rather than discarding them because they do not hold up to deeper scrutiny? I’d say very few. My conclusion to this is that the age of Big Data will see a radical increase of pseudo-scientific “discoveries”, driven out of an interest in announcing new great “patterns”. This pseudo-science will pervade both academia, public sector and private sector, God knows I’ve seen a fair number of academic research papers already that simply do not hold if you investigate their thesis in a deeper manner. I suspect we will arrive at a point much like with any new technology whereby people will tire of the claims made by “Big Data Scientists”, because at least half of what they say will have been proven to be hokey and pseudo-science in the pursuit of being able to make even more outlandish claims in a game of one-upping the competition. Some of this will be driven by malice and self-interest, but I suspect in equal parts it will be driven by ignorance and perverted incentives putting blinders on people in the business.

When you’re just starting out with Django, it can be overwhelming to see there’s no standard approach to deal with settings. However, there are a few simple best practices that work when you start needing more than the basic settings file. What’s the problem? Django stores all the settings in a project-wide settings.py file. All is fine until the moment you need different settings for different environments(such as a production environment and a development environment to start with). Of the necessity to have different settings files Your development settings should be different from your production settings. Why? you should protect sensitive things like database passwords, api secrets, private keys in a separate file the behavior of production code is not suited for development : for example you don’t want to send an email when you’re developing new features Here are a few things that may change from one environment to another: database details: database name, user name and password api keys your own private keys for encrypting data debug variables (DEBUG and TEMPLATE_DEBUG) path to different tools needed by your Django applications and any other flags needed to make your application work differently in development and in production We’re going to compare three different ways to organize your settings when having one setting file doesn’t cut it anymore. First solution: local settings This solution relies on having one settings.py file with common settings and a local_settings file where you define environment-specific settings. Let’s see: ### settings.py file ### settings that are not environment dependent try: from local_settings import * except ImportError: pass ### local_settings.py ### environment-specific settings ### example with a development environment DEBUG = True DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'django', 'USER': 'django', 'PASSWORD': '1234', 'HOST': '', 'PORT': '', } Advantages: simple if you only need a development and a production environment (no staging) – the local_settings.py should stay out of source control and you need to have a separate one for development and production. Disadvantages: it limits what you can do with settings such as modify common settings in the local_settings for example. It can work for the most simple case though. Second solution: environment-based settings This is one from Ches Martin. It relies on having an environment variable pointing to the right python module. It has the advantage of being explicit, so you can name your specific settings files explicity (production file being production.py for example). [myapp]$ls settings __init__.py defaults.py dev.py staging.py production.py What happens when we do import settings? Settings is in this case a package (a package in Python being a directory with an __init__.py file inside), so when the interpreter loads the package it executes __init__.py. By default, let’s make it work in development environment. ### __init.py__ from dev import * ### default.py__ ### sensible choices for default settings ### dev.py from defaults import * DEBUG = True ### other development-specific stuff ### production.py from defaults import * DEBUG = False ### other production-specific stuff How to use it in production and staging environments? The trick is that the settings module location can be overriden by settings an environment Django variable. So we need to override that variable before starting our webserver. For example if you use Apache as your Django web server, modify your Apache configuration file with: SetEnv DJANGO_SETTINGS_MODULE myapp.settings.production Third solution: system-wide settings ### settings.py import os ENVIRONMENT_SETTING_FILE = '/etc/django.myproject.settings' ### this will load all environment file settings in here execfile(ENVIRONMENT_SETTING_FILE) ### all common settings ### ... We can see one problem of doing it this way is that we cannot modify variables defined in the common settings.py file in the environment-specific files. On the other hand, it simplifies the management of environment-specific settings files. Create a file for development, staging and production, secure it with some tight permissions, and forget about it. To conclude There are other ways to manage your settings, so you can experiment a little bit. Check the list of resources to know more about managing your settings. And drop a comment here if you want to share the way you’re doing it! Resources: Splitting up the settings file, very comprehensive resource from the official Django website Django settings, again from the official Django website Extending Django settings for the real world from Yipit Django settings at Disqus: for a more complex and modular settings configutation. Stack Overflow: How to modularize Django settings.py? How do you configure Django for simple development and deployment? Django Snippets Keep settings.py in version control safely