How do you create a range from 1 to 10 in SQL? Have you ever thought about it? This is such an easy problem to solve in any imperative language, it’s ridiculous. Take Java (or C, whatever) for instance: for (int i = 1; i <= 10; i++) System.out.println(i); This was easy, right? Things even look more lean when using functional programming. Take Scala, for instance: (1 to 10) foreach { t => println(t) } We could fill about 25 pages about various ways to do the above in Scala, agreeing on how awesome Scala is (or what hipsters we are). But how to create a range in SQL? … And we’ll exclude using stored procedures, because that would be no fun. In SQL, the data source we’re operating on are tables. If we want a range from 1 to 10, we’d probably need a table containing exactly those ten values. Here are a couple of good, bad, and ugly options of doing precisely that in SQL. OK, they’re mostly bad and ugly. By creating a table The dumbest way to do this would be to create an actual temporary table just for that purpose: CREATE TABLE "1 to 10" AS SELECT 1 value FROM DUAL UNION ALL SELECT 2 FROM DUAL UNION ALL SELECT 3 FROM DUAL UNION ALL SELECT 4 FROM DUAL UNION ALL SELECT 5 FROM DUAL UNION ALL SELECT 6 FROM DUAL UNION ALL SELECT 7 FROM DUAL UNION ALL SELECT 8 FROM DUAL UNION ALL SELECT 9 FROM DUAL UNION ALL SELECT 10 FROM DUAL See also this SQLFiddle This table can then be used in any type of select. Now that’s pretty dumb but straightforward, right? I mean, how many actual records are you going to put in there? By using a VALUES() table constructor This solution isn’t that much better. You can create a derived table and manually add the values from 1 to 10 to that derived table using the VALUES() table constructor. In SQL Server, you could write: SELECT V FROM ( VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10) ) [1 to 10](V) See also this SQLFiddle By creating enough self-joins of a sufficent number of values Another “dumb”, yet a bit more generic solution would be to create only a certain amount of constant values in a table, view or CTE (e.g. two) and then self join that table enough times to reach the desired range length (e.g. four times). The following example will produce values from 1 to 10, “easily”: WITH T(V) AS ( SELECT 0 FROM DUAL UNION ALL SELECT 1 FROM DUAL ) SELECT V FROM ( SELECT 1 + T1.V + 2 * T2.V + 4 * T3.V + 8 * T4.V V FROM T T1, T T2, T T3, T T4 ) WHERE V <= 10 ORDER BY V See also this SQLFiddle By using grouping sets Another way to generate large tables is by using grouping sets, or more specifically by using the CUBE() function. This works much in a similar way as the previous example when self-joining a table with two records: SELECT ROWNUM FROM ( SELECT 1 FROM DUAL GROUP BY CUBE(1, 2, 3, 4) ) WHERE ROWNUM <= 10 See also this SQLFiddle By just taking random records from a “large enough” table In Oracle, you could probably use ALL_OBJECTs. If you’re only counting to 10, you’ll certainly get enough results from that table: SELECT ROWNUM FROM ALL_OBJECTS WHERE ROWNUM <= 10 See also this SQLFiddle What’s so “awesome” about this solution is that you can cross join that table several times to be sure to get enough values: SELECT ROWNUM FROM ALL_OBJECTS, ALL_OBJECTS, ALL_OBJECTS, ALL_OBJECTS WHERE ROWNUM <= 10 OK. Just kidding. Don’t actually do that. Or if you do, don’t blame me if your productive system runs low on memory. By using the awesome PostgreSQL GENERATE_SERIES() function Incredibly, this isn’t part of the SQL standard. Neither is it available in most databases but PostgreSQL, which has the GENERATE_SERIES() function. This is much like Scala’s range notation: (1 to 10) SELECT * FROM GENERATE_SERIES(1, 10) See also this SQLFiddle By using CONNECT BY If you’re using Oracle, then there’s a really easy way to create such a table using the CONNECT BY clause, which is almost as convenient as PostgreSQL’s GENERATE_SERIES() function: SELECT LEVEL FROM DUAL CONNECT BY LEVEL < 10 See also this SQLFiddle By using a recursive CTE Recursive common table expressions are cool, yet utterly unreadable. the equivalent of the above Oracle CONNECT BY clause when written using a recursive CTE would look like this: WITH "1 to 10"(V) AS ( SELECT 1 FROM DUAL UNION ALL SELECT V + 1 FROM "1 to 10" WHERE V < 10 ) SELECT * FROM "1 to 10" See also this SQLFiddle By using Oracle’s MODEL clause A decent “best of” comparison of how to do things in SQL wouldn’t be complete without at least one example using Oracle’s MODEL clause (see this awesome use-case for Oracle’s spreadsheet feature). Use this clause only to make your co workers really angry when maintaining your SQL code. Bow before this beauty! SELECT V FROM ( SELECT 1 V FROM DUAL ) T MODEL DIMENSION BY (ROWNUM R) MEASURES (V) RULES ITERATE (10) ( V[ITERATION_NUMBER] = CV(R) + 1 ) ORDER BY 1 See also this SQLFiddle Conclusion There aren’t actually many nice solutions to do such a simple thing in SQL. Clearly, PostgreSQL’s GENERATE_SERIES() table function is the most beautiful solution. Oracle’s CONNECT BY clause comes close. For all other databases, some trickery has to be applied in one way or another. Unfortunately.

Unfortunately, I have seen it misused several times when a better design would be the solution.

At the Graph Database meet up in Antwerp, we discussed how you would model a hyper edge in a property graph like Neo4j, and I realized that I’d done this in my football graph without realizing. A hyper edge is defined as follows: A hyperedge is a connection between two or more vertices, or nodes, of a hypergraph. A hypergraph is a graph in which generalized edges (called hyperedges) may connect more than two nodes with discrete properties. In Neo4j, an edge (or relationship) can only be between itself or another node; there’s no way of creating a relationship between more than 2 nodes. I had problems when trying to model the relationship between a player and a football match because I wanted to say that a player participated in a match and represented a specific team in that match. I started out with the following model: Unfortunately, creating a direct relationship from the player to the match means that there’s no way to work out which team they played for. This information is useful because sometimes players transfer teams in the middle of a season and we want to analyze how they performed for each team. In a property graph, we need to introduce an extra node which links the match, player and team together: Although we are forced to adopt this design it actually helps us realize an extra entity in our domain which wasn’t visible before – a player’s performance in a match. If we want to capture information about a players’ performance in a match we can store it on this node. We can also easily aggregate players stats by following the played relationship without needing to worry about the matches they played in. The Neo4j manual has a few more examples of domain models containing hyper edges which are worth having a look at if you want to learn more.

The lack of Base64 encoding API in Java is, in my opinion, by far one of the most annoying holes in the libraries. Finally Java 8 includes a decent API for it in the java.util package. Here is a short introduction of this new API (apparently it has a little more than the regular encode/decode API). The java.util.Base64 Utility Class The entry point to Java 8's Base64 support is the java.util.Base64 class. It provides a set of static factory methods used to obtain one of the three Base64 encodes and decoders: Basic URL MIME Basic Encoding The standard encoding we all think about when we deal with Base64: no line feeds are added to the output and the output is mapped to characters in the Base64 Alphabet: A-Za-z0-9+/ (we see in a minute why is it important). Here is a sample usage: // Encode String asB64 = Base64.getEncoder().encodeToString("some string".getBytes("utf-8")); System.out.println(asB64); // Output will be: c29tZSBzdHJpbmc= // Decode byte[] asBytes = Base64.getDecoder().decode("c29tZSBzdHJpbmc="); System.out.println(new String(asBytes, "utf-8")); // And the output is: some string It can't get simpler than that and, unlike in earlier versions of Java, there is any need for external dependencies (commons-codec), Sun classes (sun.misc.BASE64Decoder) or JAXB's DatatypeConverter (Java 6 and above). However it gets even better. URL Encoding Most of us are used to get annoyed when we have to encode something to be later included in a URL or as a filename - the problem is that the Base64 Alphabet contains meaningful characters in both URIs and filesystems (most specifically the forward slash (/)). The second type of encoder uses the alternative "URL and Filename safe" Alphabet which includes -_ (minus and underline) instead of +/. See the following example: String basicEncoded = Base64.getEncoder().encodeToString("subjects?abcd".getBytes("utf-8")); System.out.println("Using Basic Alphabet: " + basicEncoded); String urlEncoded = Base64.getUrlEncoder().encodeToString("subjects?abcd".getBytes("utf-8")); System.out.println("Using URL Alphabet: " + urlEncoded); The output will be: Using Basic Alphabet: c3ViamVjdHM/YWJjZA== Using URL Alphabet: c3ViamVjdHM_YWJjZA== The sample above illustrates some content which if encoded using a basic encoder will result in a string containing a forward slash while when using a URL safe encoder the output will include an underscore instead (URL Safe encoding is described in clause 5 of the RFC) MIME Encoding The MIME encoder generates a Base64 encoded output using the Basic Alphabet but in a MIME friendly format: each line of the output is no longer than 76 characters and ends with a carriage return followed by a linefeed (\r\n). The following example generates a block of text (this is needed just to make sure we have enough 'body' to encode into more than 76 characters) and encodes it using the MIME encoder: StringBuilder sb = new StringBuilder(); for (int t = 0; t < 10; ++t) { sb.append(UUID.randomUUID().toString()); } byte[] toEncode = sb.toString().getBytes("utf-8"); String mimeEncoded = Base64.getMimeEncoder().encodeToString(toEncode); System.out.println(mimeEncoded); The output: NDU5ZTFkNDEtMDVlNy00MDFiLTk3YjgtMWRlMmRkMWEzMzc5YTJkZmEzY2YtM2Y2My00Y2Q4LTk5 ZmYtMTU1NzY0MWM5Zjk4ODA5ZjVjOGUtOGMxNi00ZmVjLTgyZjctNmVjYTU5MTAxZWUyNjQ1MjJj NDMtYzA0MC00MjExLTk0NWMtYmFiZGRlNDk5OTZhMDMxZGE5ZTYtZWVhYS00OGFmLTlhMjgtMDM1 ZjAyY2QxNDUyOWZiMjI3NDctNmI3OC00YjgyLThiZGQtM2MyY2E3ZGNjYmIxOTQ1MDVkOGQtMzIz Yi00MDg0LWE0ZmItYzkwMGEzNDUxZTIwOTllZTJiYjctMWI3MS00YmQzLTgyYjUtZGRmYmYxNDA4 Mjg3YTMxZjMxZmMtYTdmYy00YzMyLTkyNzktZTc2ZDc5ZWU4N2M5ZDU1NmQ4NWYtMDkwOC00YjIy LWIwYWItMzJiYmZmM2M0OTBm Wrapping All encoders and decoders created by the java.util.Base64 class support streams wrapping - this is a very elegant construct - both coding and efficiency wise (since no redundant buffering are needed) - to stream in and out buffers via encoders and decoders. The sample bellow illustrates how a FileOutputStream can be wrapped with an encoder and a FileInputStream is wrapped with a decoder - in both cases there is no need to buffer the content: public void wrapping() throws IOException { String src = "This is the content of any resource read from somewhere" + " into a stream. This can be text, image, video or any other stream."; // An encoder wraps an OutputStream. The content of /tmp/buff-base64.txt will be the // Base64 encoded form of src. try (OutputStream os = Base64.getEncoder().wrap(new FileOutputStream("/tmp/buff-base64.txt"))) { os.write(src.getBytes("utf-8")); } // The section bellow illustrates a wrapping of an InputStream and decoding it as the stream // is being consumed. There is no need to buffer the content of the file just for decoding it. try (InputStream is = Base64.getDecoder().wrap(new FileInputStream("/tmp/buff-base64.txt"))) { int len; byte[] bytes = new byte[100]; while ((len = is.read(bytes)) != -1) { System.out.print(new String(bytes, 0, len, "utf-8")); } } }

In the last post I wrote about Nathan and my attempts at the Kaggle Titanic Problem, I mentioned that our next step was to try out scikit-learn, so I thought I should summarize where we’ve got up to. We needed to write a classification algorithm to work out whether a person onboard the Titanic survived, and luckily, scikit-learn has extensive documentation on each of the algorithms. Unfortunately almost all those examples use numpy data structures and we’d loaded the data using pandas and didn’t particularly want to switch back! Luckily it was really easy to get the data into numpy format by calling ‘values’ on the pandas data structure, something we learnt from a reply on Stack Overflow. For example if we were to wire up an ExtraTreesClassifier which worked out survival rate based on the ‘Fare’ and ‘Pclass’ attributes we could write the following code: import pandas as pd from sklearn.ensemble import ExtraTreesClassifier from sklearn.cross_validation import cross_val_score train_df = pd.read_csv("train.csv") et = ExtraTreesClassifier(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0) columns = ["Fare", "Pclass"] labels = train_df["Survived"].values features = train_df[list(columns)].values et_score = cross_val_score(et, features, labels, n_jobs=-1).mean() print("{0} -> ET: {1})".format(columns, et_score)) To start with with read in the CSV file which looks like this: $ head -n5 train.csv PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 1,0,3,"Braund, Mr. Owen Harris",male,22,1,0,A/5 21171,7.25,,S 2,1,1,"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",female,38,1,0,PC 17599,71.2833,C85,C 3,1,3,"Heikkinen, Miss. Laina",female,26,0,0,STON/O2. 3101282,7.925,,S 4,1,1,"Futrelle, Mrs. Jacques Heath (Lily May Peel)",female,35,1,0,113803,53.1,C123,S Next we create our classifier which “fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting.“ i.e. a better version of a random forest. On the next line we describe the features we want the classifier to use, then we convert the labels and features into numpy format so we can pass the to the classifier. Finally we call the cross_val_score function which splits our training data set into training and test components and trains the classifier against the former and checks its accuracy using the latter. If we run this code we’ll get roughly the following output: $ python et.py ['Fare', 'Pclass'] -> ET: 0.687991021324) This is actually a worse accuracy than we’d get by saying that females survived and males didn’t. We can introduce ‘Sex’ into the classifier by adding it to the list of columns: columns = ["Fare", "Pclass", "Sex"] If we re-run the code we’ll get the following error: $ python et.py An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (514, 0)) ... Traceback (most recent call last): File "et.py", line 14, in et_score = cross_val_score(et, features, labels, n_jobs=-1).mean() File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/cross_validation.py", line 1152, in cross_val_score for train, test in cv) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/parallel.py", line 519, in __call__ self.retrieve() File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/parallel.py", line 450, in retrieve raise exception_type(report) sklearn.externals.joblib.my_exceptions.JoblibValueError/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/my_exceptions.py:26: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message, : JoblibValueError ___________________________________________________________________________ Multiprocessing exception: ... ValueError: could not convert string to float: male ___________________________________________________________________________ This is a slightly verbose way of telling us that we can’t pass non numeric features to the classifier – in this case ‘Sex’ has the values ‘female’ and ‘male’. We’ll need to write a function to replace those values with numeric equivalents. train_df["Sex"] = train_df["Sex"].apply(lambda sex: 0 if sex == "male" else 1) Now if we re-run the classifier we’ll get a slightly more accurate prediction: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) The next step is to use the classifier against the test data set, so let’s load the data and run the prediction: test_df = pd.read_csv("test.csv") et.fit(features, labels) et.predict(test_df[columns].values) Now if we run that: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) Traceback (most recent call last): File "et.py", line 22, in et.predict(test_df[columns].values) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 444, in predict proba = self.predict_proba(X) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 479, in predict_proba X = array2d(X, dtype=DTYPE) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 91, in array2d X_2d = np.asarray(np.atleast_2d(X), dtype=dtype, order=order) File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/numeric.py", line 235, in asarray return array(a, dtype, copy=False, order=order) ValueError: could not convert string to float: male which is the same problem we had earlier! We need to replace the ‘male’ and ‘female’ values in the test set too so we’ll pull out a function to do that now. def replace_non_numeric(df): df["Sex"] = df["Sex"].apply(lambda sex: 0 if sex == "male" else 1) return df Now we’ll call that function with our training and test data frames: train_df = replace_non_numeric(pd.read_csv("train.csv")) ... test_df = replace_non_numeric(pd.read_csv("test.csv")) If we run the program again: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) Traceback (most recent call last): File "et.py", line 26, in et.predict(test_df[columns].values) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 444, in predict proba = self.predict_proba(X) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 479, in predict_proba X = array2d(X, dtype=DTYPE) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 93, in array2d _assert_all_finite(X_2d) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 27, in _assert_all_finite raise ValueError("Array contains NaN or infinity.") ValueError: Array contains NaN or infinity. There are missing values in the test set so we’ll replace those with average values from our training set using an Imputer: from sklearn.preprocessing import Imputer imp = Imputer(missing_values='NaN', strategy='mean', axis=0) imp.fit(features) test_df = replace_non_numeric(pd.read_csv("test.csv")) et.fit(features, labels) print et.predict(imp.transform(test_df[columns].values)) If we run that it completes successfully: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) [0 1 0 0 1 0 0 1 1 0 0 0 1 0 1 1 0 0 1 1 0 0 1 0 1 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 1 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 0 1 0 1 0 0 1 0 1 1 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 1 1 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 0 1 1 1 1 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 1 0 1 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 0] The final step is to add these values to our test data frame and then write that to a file so we can submit it to Kaggle. The type of those values is ‘numpy.ndarray’ which we can convert to a pandas Series quite easily: predictions = et.predict(imp.transform(test_df[columns].values)) test_df["Survived"] = pd.Series(predictions) We can then write the ‘PassengerId’ and ‘Survived’ columns to a file: test_df.to_csv("foo.csv", cols=['PassengerId', 'Survived'], index=False) Then output file looks like this: $ head -n5 foo.csv PassengerId,Survived 892,0 893,1 894,0 The code we’ve written is on github in case it’s useful to anyone.

Introduction I had an interesting conversation the other day about custom domain-specific languages and we happened to talk about a feature of Spring that I’ve used before but doesn’t seem to be widely known: the static application context. This post illustrates a basic example I wrote that introduces the static application context and shows how it might be useful. It’s also an interesting topic as it shows some of the well-architected internals of the Spring framework. Most uses of Spring start with XML or annotations and wind up with an application context instance. Behind the scenes, Spring has been working hard to instantiate objects, inject properties, invoke context aware listeners, and so forth. There are a set of classes internal to Spring to help this process along, as Spring needs to hold all of the configuration data about beans before any beans are instantiated. (This is because the beans may be defined in any order, and Spring doesn’t have the exhaustive set of dependencies until all beans are defined.) Spring Static Application Context Spring offers a class called StaticApplicationContext that gives programmatic access from Java to this whole configuration and registration process. This means we can define an entire application context from pure Java code, without using XML or Java annotations or any other tricks. The Javadoc for StaticApplicationContext is here, but an example is coming. Why might we use this? As the Javadoc says, it’s mainly useful for testing. Spring uses it for its own testing, but I’ve found it useful for testing applications that use Spring or other dependency management frameworks. Often, for unit testing, we want to inject different objects into a class from those used in production (e.g. mock objects, or objects that simulate remote invocation, database, or messaging). Of course, we can just keep a separate Spring XML configuration file for testing, but it’s very nice to have our whole configuration right there in the Java unit test class as it makes it easier to maintain. Example I’ve added an example to my intro-to-java repository on GitHub. I created aStaticContext class that provides a very basic Java domain-specific language (DSL) for Spring beans. This is just to make it easier to use from the unit test. The DSL only includes the most basic Spring capabilities: register a bean, set properties, and wire dependencies. package org.anvard.introtojava.spring; import org.springframework.beans.MutablePropertyValues; import org.springframework.beans.factory.config.ConstructorArgumentValues; import org.springframework.beans.factory.config.RuntimeBeanReference; import org.springframework.beans.factory.support.RootBeanDefinition; import org.springframework.context.ApplicationContext; import org.springframework.context.support.StaticApplicationContext; public class StaticContext { public class BeanContext { private String name; private Class beanClass; private ConstructorArgumentValues args; private MutablePropertyValues props; private BeanContext(String name, Class beanClass) { this.name = name; this.beanClass = beanClass; this.args = new ConstructorArgumentValues(); this.props = new MutablePropertyValues(); } public BeanContext arg(Object arg) { args.addGenericArgumentValue(arg); return this; } public BeanContext arg(int index, Object arg) { args.addIndexedArgumentValue(index, arg); return this; } public BeanContext prop(String name, Object value) { props.add(name, value); return this; } public BeanContext ref(String name, String beanRef) { props.add(name, new RuntimeBeanReference(beanRef)); return this; } public void build() { RootBeanDefinition def = new RootBeanDefinition(beanClass, args, props); ctx.registerBeanDefinition(name, def); } } private StaticApplicationContext ctx; private StaticContext() { this.ctx = new StaticApplicationContext(); } public static StaticContext create() { return new StaticContext(); } public ApplicationContext build() { ctx.refresh(); return ctx; } public BeanContext bean(String name, Class beanClass) { return new BeanContext(name, beanClass); } } This class uses several classes that are normally internal to Spring: StaticApplicationContext: Holds bean definitions and provides regular Java methods for registering beans. ConstructorArgumentValues: A smart list for a bean’s constructor arguments. Can hold both wire-by-type and indexed constructor arguments. MutablePropertyValues: A smart list for a bean’s properties. Can hold regular objects and references to other Spring beans. RuntimeBeanReference: A reference by name to a bean in the context. Used for wiring beans together because it allows Spring to delay resolution of a dependency until it’s been instantiated. The StaticContext class uses the builder pattern and provides for method chaining. This makes for cleaner use from our unit test code. Here’s the simplest example: @Test public void basicBean() { StaticContext sc = create(); sc.bean("basic", InnerBean.class).prop("prop1", "abc"). prop("prop2", "def").build(); ApplicationContext ctx = sc.build(); assertNotNull(ctx); InnerBean bean = (InnerBean) ctx.getBean("basic"); assertNotNull(bean); assertEquals("abc", bean.getProp1()); assertEquals("def", bean.getProp2()); } A slightly more realistic example that includes wiring beans together is not much more complicated: @Test public void innerBean() { StaticContext sc = create(); sc.bean("outer", OuterBean.class).prop("prop1", "xyz"). ref("inner", "inner").build(); sc.bean("inner", InnerBean.class).prop("prop1", "ghi"). prop("prop2", "jkl").build(); ApplicationContext ctx = sc.build(); assertNotNull(ctx); InnerBean inner = (InnerBean) ctx.getBean("inner"); assertNotNull(inner); assertEquals("ghi", inner.getProp1()); assertEquals("jkl", inner.getProp2()); OuterBean outer = (OuterBean) ctx.getBean(OuterBean.class); assertNotNull(outer); assertEquals("xyz", outer.getProp1()); assertEquals(inner, outer.getInner()); } Note that once we build the context, we can use it like any other Spring application context, including fetching beans by name or type. Also note that the two contexts we created here are completely separate, which is important for unit testing. Conclusion Much like my post on custom Spring XML, the static application context is a specialty feature that isn’t intended for everyday users of Spring. But I’ve found it convenient when unit testing and it provides an interesting peek into how Spring works.

It is a very rare usecase, but sometimes you need it. How to embed Maven in your application, so that you can programatically run goals? Short answer is: it's tricky. I dabbled into the matter for my java webapp automatic syncing project, and at some point I decided not to embed it. Ultimately, I used a library that does what I needed, but anyway, here are the steps and tools that might be helpful. What you usually need the embedded maven for, is to execute some goals on a maven project. There are two scenarios. The first one is, if you are running inside the maven container, i.e. you are writing a mojo/plugin. Then it's fairly easy, because you have everything managed by the already-initialized plexus container. In that case you can use the mojo-executor. Easy to use, but expects a "project", "pluginManager" and "session", which you can't easily obtain. The second scenario is completely embedded maven. There is a library that does what I needed it to do (thanks to MariuszS for pointing it out) - it's Maven Embedder. Its usage is described in this SO question. Use both the first and the second answer. Before finding that library, I tried two more libraries: the jenkins maven embedded and the Maven Invoker. The problem in both libraries is: they need a maven home. That is, the path to where a maven installation resides. Which is kind of contrary to the idea of "embedded" maven. If the Maven Embedder suits you, you can stop reading. However, there might be cases where the Maven Embedder might not be what you are looking for. In that case, you should use one of the two aforementioned libraries. So, how to find and set a maven home? Ask the user to specify it. Not too much of a hassle, probably Use M2_HOME. One of the libraries uses that by default, but the problem is it might not be set. I don't usually set it, for example. If it is not, you can fallback to the previous approach Scan the entire file system for a maven installation - sounds ok, and it can be done only once, and then stored in some entry. The problem is - there might not be a maven installation. Even if it's a developer's machine - IDEs (Eclipse, at least) have an "embedded" maven. And while it probably stores it somewhere internally in the same format a manual installation would, it may change it's path or structure depending on the version. You can, of course, re-scan the file tree every once in a while to find such an installation Download Maven programatically yourself. Then you can be sure where it is located and that it will always be located there in the same format. The problem here is version mismatch - the user might be using another version of maven. Making the version configurable is an option. All of these work in some cases, and don't work in others. So, in order of preference: 1. make sure you really need to embed maven 2. use the Maven Embedder 3. use another option with its considerations

we all know junit test-classes can be parameterized , which means that for a given set of test-elements, the test class is instantiated a few times, but using constructors for that isn’t always what you want. i’ve taken the stringsorttest from this blog as an example. @runwith(parameterized.class) public class stringsorttest { @parameters public static collection data() { return arrays.aslist(new object[][] { { "abc", "abc" }, { "cba", "abc" }, }); } private final string input; private final string expected; public stringsorttest(final string input, final string expected) { this.input = input; this.expected = expected; } @test public void testsort() { assertequals(expected, mysortmethod(input)); } } this is pretty darn obnoxious sometimes if you have multiple sets of data for various tests, which all go through the constructor, which would force you to write multiple test classes. testng solves this better by allowing you to provide separate data sets to individual test methods using the @dataprovider annotation . but don’t worry, now you can achieve the same with the junit-dataprovider , available on github. pull in the dependency with e.g. maven. com.tngtech.java junit-dataprovider 1.5.0 test the above example now could be rewritten as: @runwith(dataproviderrunner.class) public class stringsorttest { @dataprovider public static object[][] data() { return new object[][] { { "abc", "abc" }, { "cba", "abc" }, }; } @test @usedataprovider("data") public void testsort(final string input, final string expected) { assertequals(expected, mysortmethod(input)); } } you’ll see: no constructor. in this example it doesn’t have many benefits, except maybe for less boiler-plate, but now you can create as many @dataprovider-annotated methods which are fed directly to your @usedataprovider-annotated testmethod(s). sidenote for eclipse: if you’re using that ide, the junit plugin is unable to map the names of the passed/failed testmethods to the ones in the testclass correctly. if a method fails, you’ll have to find it back manually in the testclass (or vote for the patch andreas smidt created to get this fixed in eclipse). in the mean time, if you’re stuck with junit and you’d love to use this feature you’re so accustomed to using with testng, go ahead and try the junit-dataprovider now.

I came across a debate about whether naming a class [Something]Manager is proper. Mainly, I came across this article, but also found other references here and here. I already had my own thoughts on this topic and decided I would share them with you. I started thinking on this a while ago when working in a Swing-powered desktop app. We decided we would make use of the Listeners model that Swing establishes for its components. To register a listener in a Swing component you say: // Some-Type = {Action, Key, Mouse, Focus, ...} component.add[Some-Type]Listener(listener); That's not a problem until you need to dynamically remove those listeners, or override them (remove and add a new listener that listens to the same event but fires a different action), or do any other 'unnatural' thing with them. This turns out to be a little difficult if you use the interface Swing provides out-of-the-box. For instance, to remove a listener, you need to pass the whole instance you want to remove; you would have to hold the reference to the listener if you intend to remove it some time later. So I decided I would take a step further and try to abstract the mess by creating a Manager class. What you could do with this class was something like this: EventsListenersManager.registerListener( "some unique name", component, listener); EventsListenersManager.overrideListener("some unique name", newListener); EventsListenerManager.disableListener("some unique name"); EventsListenerManager.enableListener("some unique name"); EventsListenerManager.unregisterListener("some unique name"); What we gained here??? We got rid of the explicit specification of [Some-Type] and have a single function for every type of listener; and we defined a unique name for the binding, which will be used as an id to remove, enable/disable, and override listeners; a.k.a. no need to hold references. Obviously, it is now convenient to always use the manager’s interface, since it reduces the need to trick our code and gives us a simple and more readable interface to achieve many things with listeners. But that’s not the only advantage of this manager class. You see, manager classes in general have one big advantage: they create an abstraction of a service and work as a centralized point to add logic of any nature, like security logic or performance logic. In the case of our EventsListenersManager, we put some extra logic to enchance the way we can use listeners by providing an interface that simplifies their use: it is easier to make a switch between two listeners now than it was before. Another thing we could have done was to restrict the number of listeners registered in one component for performance sake (Hint: listeners execution don’t run in parallel). So manager classes seem to be a good thing, but we can’t make a manager for every object we want to control in an application. This would consume time and would make us start thinking in a manager class even when we don’t need it. But we can discover a pattern for the need of defining a manager. I’ve seen them used when the resources they control are expensive, like database connections, media resources, hardware resources, etc; and as I see it, expensiveness can come in many flavors: Listeners are expensive because they may be difficult to play with and can be a bottleneck in performance (we must keep them short and fast). Connections are expensive to construct. Media resources are expensive to load. Hardware resources are expensive because they are scarce. So we create different managers for them: We create a manager for listeners to ease and control their use. We create a connections pool in order to limit and control the number of connections to a database. Games developers create a resources manager to have a single point to access resources and reduce the possibility to load them multiple times. We all know there is a driver for every piece of hardware. That’s their manager. Here we can see multiple variations of manager classes implementations, but they all attempt to solve a single thing: reducing the cost we may incur in when using expensive resources directly. I think people should have this line of thought: make a manager class for every expensive resource you detect in your application. You will be more than thankful when you want to add extra logic and you only have to go to one place. Also, enforce the use of manager objects instead of accessing the resources directly. The next time you are planning to make a manager class, ask yourself: is the 'thing' I want to control expensive in any way???

This blog is more of a tutorial where we describe the development of a simple data access module, more for fun and learning than anything else. All code can be found here for those who don’t want to type along: https://github.com/ricston-git/tododb As a heads-up, we will be covering the following: Using Groovy in a Maven project within Eclipse Using Groovy to interact with our database Testing our code using the Spock framework We include Spring in our tests with ContextConfiguration A good place to start is to write a pom file as shown here. The only dependencies we want packaged with this artifact are groovy-all and commons-lang. The others are either going to be provided by Tomcat or are only used during testing (hence the scope tags in the pom). For example, we would put the jar with PostgreSQL driver in Tomcat’s lib, and tomcat-jdbc and tomcat-dbcp are already there. (Note: regarding the postgre jar, we would also have to do some minor configuration in Tomcat to define a DataSource which we can get in our app through JNDI – but that’s beyond the scope of this blog. See here for more info). Testing-wise, I’m depending on spring-test, spock-core, and spock-spring (the latter is to get spock to work with spring-test). Another significant addition in the pom is the maven-compiler-plugin. I have tried to get gmaven to work with Groovy in Eclipse, but I have found the maven-compiler-plugin to be a lot easier to work with. With your pom in an empty directory, go ahead and mkdir -p src/main/groovy src/main/java src/test/groovy src/test/java src/main/resources src/test/resources. This gives us a directory structure according to the Maven convention. Now you can go ahead and import the project as a Maven project in Eclipse (install the m2e plugin if you don’t already have it). It is important that you do not mvn eclipse:eclipse in your project. The .classpath it generates will conflict with your m2e plugin and (at least in my case), when you update your pom.xml the plugin will not update your dependencies inside Eclipse. So just import as a maven project once you have your pom.xml and directory structure set up. Okay, so our tests are going to be integration tests, actually using a PostgreSQL database. Since that’s the case, lets set up our database with some data. First go ahead and create a tododbtest database which will only be used for testing purposes. Next, put the following files in your src/test/resources: Note, fill in your username/password: DROP TABLE IF EXISTS todouser CASCADE; CREATE TABLE todouser ( id SERIAL, email varchar(80) UNIQUE NOT NULL, password varchar(80), registered boolean DEFAULT FALSE, confirmationCode varchar(280), CONSTRAINT todouser_pkey PRIMARY KEY (id) ); insert into todouser (email, password, registered, confirmationCode) values ('abc.j123@gmail.com', 'abc123', FALSE, 'abcdefg') insert into todouser (email, password, registered, confirmationCode) values ('def.123@gmail.com', 'pass1516', FALSE, '123456') insert into todouser (email, password, registered, confirmationCode) values ('anon@gmail.com', 'anon', FALSE, 'codeA') insert into todouser (email, password, registered, confirmationCode) values ('anon2@gmail.com', 'anon2', FALSE, 'codeB') Basically, testContext.xml is what we’ll be configuring our test’s context with. The sub-division into datasource.xml and initdb.xml may be a little too much for this example… but changes are usually easier that way. The gist is that we configure our data source in datasource.xml (this is what we will be injecting in our tests), and the initdb.xml will run the schema.sql and test-data.sql to create our table and populate it with data. So lets create our test, or should I say, our specification. Spock is specification framework that allows us to write more descriptive tests. In general, it makes our tests easier to read and understand, and since we’ll be using Groovy, we might as well make use of the extra readability Spock gives us. package com.ricston.blog.sample.model.spec; import javax.sql.DataSource import org.springframework.beans.factory.annotation.Autowired import org.springframework.test.annotation.DirtiesContext import org.springframework.test.annotation.DirtiesContext.ClassMode import org.springframework.test.context.ContextConfiguration import spock.lang.Specification import com.ricston.blog.sample.model.data.TodoUser import com.ricston.blog.sample.model.dao.postgre.PostgreTodoUserDAO // because it supplies a new application context after each test, the initialize-database in initdb.xml is // executed for each test/specification @DirtiesContext(classMode=ClassMode.AFTER_EACH_TEST_METHOD) @ContextConfiguration('classpath:testContext.xml') class PostgreTodoUserDAOSpec extends Specification { @Autowired DataSource dataSource PostgreTodoUserDAO postgreTodoUserDAO def setup() { postgreTodoUserDAO = new PostgreTodoUserDAO(dataSource) } def "findTodoUserByEmail when user exists in db"() { given: "a db populated with a TodoUser with email anon@gmail.com and the password given below" String email = 'anon@gmail.com' String password = 'anon' when: "searching for a TodoUser with that email" TodoUser user = postgreTodoUserDAO.findTodoUserByEmail email then: "the row is found such that the user returned by findTodoUserByEmail has the correct password" user.password == password } } One specification is enough for now, just to make sure that all the moving parts are working nicely together. The specification itself is easy enough to understand. We’re just exercising the findTodoUserByEmail method of PostgreTodoUserDAO – which we will be writing soon. Using the ContextConfiguration from Spring Test we are able to inject beans defined in our context (the dataSource in our case) through the use of annotations. This keeps our tests short and makes them easier to modify later on. Additionally, note the use of DirtiesContext. Basically, after each specification is executed, we cannot rely on the state of the database remaining intact. I am using DirtiesContext to get a new Spring context for each specification run. That way, the table creation and test data insertions happen all over again for each specification we run. Before we can run our specification, we need to create at least the following two classes used in the spec: TodoUser and PostgreTodoUserDAO package com.sample.data import org.apache.commons.lang.builder.ToStringBuilder class TodoUser { long id; String email; String password; String confirmationCode; boolean registered; @Override public String toString() { ToStringBuilder.reflectionToString(this); } } package com.ricston.blog.sample.model.dao.postgre import groovy.sql.Sql import javax.sql.DataSource import com.ricston.blog.sample.model.dao.TodoUserDAO import com.ricston.blog.sample.model.data.TodoUser class PostgreTodoUserDAO implements TodoUserDAO { private Sql sql public PostgreTodoUserDAO(DataSource dataSource) { sql = new Sql(dataSource) } /** * * @param email * @return the TodoUser with the given email */ public TodoUser findTodoUserByEmail(String email) { sql.firstRow """SELECT * FROM todouser WHERE email = $email""" } } package com.ricston.blog.sample.model.dao; import com.ricston.blog.sample.model.data.TodoUser; public interface TodoUserDAO { /** * * @param email * @return the TodoUser with the given email */ public TodoUser findTodoUserByEmail(String email); } We’re just creating a POGO in TodoUser, implementing its toString using common’s ToStringBuilder. In PostgreTodoUserDAO we’re using Groovy’s SQL to access the database, for now, only implementing the findTodoUserByEmail method. PostgreTodoUserDAO implements TodoUserDAO, an interface which specifies the required methods a TodoUserDAO must have. Okay, so now we have all we need to run our specification. Go ahead and run it as a JUnit test from Eclipse. You should get back the following error message: org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object '{id=3, email=anon@gmail.com, password=anon, registered=false, confirmationcode=codeA}' with class 'groovy.sql.GroovyRowResult' to class 'com.ricston.blog.sample.model.data.TodoUser' due to: org.codehaus.groovy.runtime.metaclass.MissingPropertyExceptionNoStack: No such property: confirmationcode for class: com.ricston.blog.sample.model.data.TodoUser Possible solutions: confirmationCode at com.ricston.blog.sample.model.dao.postgre.PostgreTodoUserDAO.findTodoUserByEmail(PostgreTodoUserDAO.groovy:23) at com.ricston.blog.sample.model.spec.PostgreTodoUserDAOSpec.findTodoUserByEmail when user exists in db(PostgreTodoUserDAOSpec.groovy:37) Go ahead and connect to your tododbtest database and select * from todouser; As you can see, our confirmationCode varchar(280), ended up as the column confirmationcode with a lower case ‘c’. In PostgreTodoUserDAO’s findTodoUserByEmail, we are getting back GroovyRowResult from our firstRow invocation. GroovyRowResult implements Map and Groovy is able to create a POGO (in our case TodoUser) from a Map. However, in order for Groovy to be able to automatically coerce the GroovyRowResult into a TodoUser, the keys in the Map (or GroovyRowResult) must match the property names in our POGO. We are using confirmationCode in our TodoUser, and we would like to stick to the camel case convention. What can we do to get around this? Well, first of all, lets change our schema to use confirmation_code. That’s a little more readable. Of course, we still have the same problem as before since confirmation_code will not map to confirmationCode by itself. (Note: remember to change the insert statements in test-data.sql too). One way to get around this is to use Groovy’s propertyMissing methods as show below: def propertyMissing(String name, value) { if(isConfirmationCode(name)) { this.confirmationCode = value } else { unknownProperty(name) } } def propertyMissing(String name) { if(isConfirmationCode(name)) { return confirmationCode } else { unknownProperty(name) } } private boolean isConfirmationCode(String name) { 'confirmation_code'.equals(name) } def unknownProperty(String name) { throw new MissingPropertyException(name, this.class) } By adding this to our TodoUser.groovy we are effectively tapping in on how Groovy resolves property access. When we do something like user.confirmationCode, Groovy automatically calls getConfirmationCode(), a method which we got for free when declared the property confirmationCode in our TodoUser. Now, when user.confirmation_code is invoked, Groovy doesn’t find any getters to invoke since we never declared the property confirmation_code, however, since we have now implemented the propertyMissing methods, before throwing any exceptions it will use those methods as a last resort when resolving properties. In our case we are effectively checking whether a get or set on confirmation_code is being made and mapping the respective operations to our confirmationCode property. It’s as simple as that. Now we can keep the auto coercion in our data access object and the property name we choose to have in our TodoUser. Assuming you’ve made the changes to the schema and test-data.sql to use confirmation_code, go ahead and run the spec file and this time it should pass. That’s it for this tutorial. In conclusion, I would like to discuss some finer points which someone who’s never used Groovy’s SQL before might not know. As you can see in PostgreTodoUserDAO.groovy, our database interaction is pretty much a one-liner. What about resource handling (e.g. properly closing the connection when we’re done), error logging, and prepared statements? Resource handling and error logging are done automatically, you just have to worry about writing your SQL. When you do write your SQL, try to stick to using triple quotes as used in the PostgreTodoUserDAO.groovy example. This produces prepared statements, therefore protecting against SQL injection and avoids us having to put ‘?’ all over the place and properly lining up the arguments to pass in to the SQL statement. Note that transaction management is something which the code using our artifact will have to take care of. Finally, note that a bunch of other operations (apart from findTodoUserByEmail) are implemented in the project on GitHub: https://github.com/ricston-git/tododb. Additionally, there is also a specification test for TodoUser, making sure that the property mapping works correctly. Also, in the pom.xml, there is some maven-surefire-plugin configuration in order to get the surefire-plugin to pick up our Spock specifications as well as any JUnit tests which we might have in our project. This allows us to run our specifications when we, for example, mvn clean package. After implementing all the operations you require in PostgreTodoUserDAO.groovy, you can go ahead and compile the jar or include in a Maven multi-module project to get a data access module you can use in other applications.

transitioning from the relational world to the beautiful world of graphs requires a shift in thinking about data. although graphs are often much more intuitive than tables, there are certain mistakes people tend to make when modelling their data as a graph for the first time. in this article, we look at one common source of confusion: bidirectional relationships. directed relationships relationships in neo4j must have a type, giving the relationship a semantic meaning, and a direction. frequently, the direction becomes part of the relationship's meaning. in other words, the relationship would be ambiguous without it. for example, the following graph shows that the czech republic defeated sweden in ice hockey. had the direction of the relationship been reversed, the swedes would be much happier. with no direction at all, the relationship would be ambiguous, since it would not be clear who the winner was. note that the existence of this relationship implies a relationship of a different type going in the opposite direction, as the next graph illustrates. this is often the case. to give another example, the fact that pulp fiction was directed_by quentin tarantino implies that quentin tarantino is_director_of pulp fiction. you could come up with a huge number of such relationship pairs. one common mistake people often make when modelling their domain in neo4j is creating both types of relationships. since one relationship implies the other, this is wasteful, both in terms of space and traversal time. neo4j can traverse relationships in both directions. more importantly, thanks to the way neo4j organizes its data, the speed of traversal does not depend on the direction of the relationships being traversed. bidirectional relationships some relationships, on the other hand, are naturally bidirectional. a classic example is facebook or real-life friendship. this relationship is mutual - when someone is your friend, you are (hopefully) his friend, too. depending on how we look at the model, we could also say such relationship is undirected. graphaware and neo technology are partner companies. since this is a mutual relationship, we could model it as bidirectional or undirected relationship, respectively. but since none of this is directly possible in neo4j, beginners often resort to the following model, which suffers from the exact same problem as the incorrect ice hockey model: an extra unnecessary relationship. neo4j apis allow developers to completely ignore relationship direction when querying the graph, if they so desire. for example, in neo4j's own query language, cypher, the key part of a query finding all partner companies of neo technology would look something like match (neo)-[:partner]-(partner) the result would be the same as executing and merging the results of the following two different queries: match (neo)-[:partner]->(partner) and match (neo)<-[:partner]-(partner) therefore, the correct (or at least most efficient) way of modelling the partner relationships is using a single partner relationship with an arbitrary direction . conclusion relationships in neo4j can be traversed in both directions with the same speed. moreover, direction can be completely ignored. therefore, there is no need to create two different relationships between nodes, if one implies the other.

A quick overview on how to see your heaps in Eclipse.

We'll use a custom CXF interceptor to add these headers.

What is Functional Programming? Functional programming is a specific way to look at problems and model their solutions. Pragmatically, functional programming is a coding style that exhibits the following characteristics: Power and flexibility. We can solve many general real-world problems using functional constructs Simplicity. Most functional programs exhibit a small set of keywords and concise syntax for expressing concepts Suitable for parallel processing. Via immutable values and operators, functional programs lend themselves to asynchronous and parallel processing. In functional programming, programs are executed by evaluating expressions, in contrast with imperative programming where programs are composed of statements which change global state when executed. Functional programming typically avoids using mutable state. Everything is a mathematical function. Functional programming languages can have objects, but generally those objects are immutable -- either arguments or return values to functions. There are no for/next loops, as those imply state changes. Instead, that type of looping is performed with recursion and by passing functions as arguments. Functional Programming vs. Imperative Programming: We can think of imperative programming as writing code that describes in exacting detail the steps the software must take to execute a given computation. These steps are generally expressed as a combination of statement executions, state changes, loops and conditional branches. Many programming languages support programming in both functional and imperative style but the syntax and facilities of a language are typically optimised for only one of these styles, and social factors like coding conventions and libraries often force the programmer towards one of the styles. Therefore, programming languages may be categorized into functional and imperative ones. Why Functional Programming? While you can develop concurrent, scalable and asynchronous software without embracing functional programming, it's simpler, safer, and easier to use the right tool for the job. Functional programming enables you to take advantage of multi-core systems, develop robust concurrent algorithms, parallelize compute-intensive algorithms, and to readily leverage the growing number of cloud computing platforms. Imagine you've implemented a large program in a purely functional way. All the data is properly threaded in and out of functions, and there are no truly destructive updates to speak of. Now pick the two lowest-level and most isolated functions in the entire codebase. They're used all over the place, but are never called from the same modules. Now make these dependent on each other: function A behaves differently depending on the number of times function B has been called and vice-versa. In addition, if you've not already explored non-imperative programming, it's a great way to expand your problem solving skills and your horizons. The new concepts that you'll learn will help you to look at many problems from a different perspective and will help you to become a better and more insightful OO programmer. I encourage everyone who wants to be a better programmer: consider learning a functional language. Haskell and OCaml are both great choices, and F# and Erlang are pretty good as well. It won’t be easy, but that is probably a good sign. Try and identify the difficult concepts you encounter and see if other people are leveraging them; frequently you can break through a mental roadblock by finding out what the intent of an unfamiliar abstraction really is. While you’re learning, do be careful not to take it too seriously. Like anything that requires time and effort, there is a danger of becoming over-invested in FP. Falling into this cognitive trap will ruin your investment. It’s easy to forget how many models of computation there are out there, and even easier to forget how much beautiful software has been written with any of them. It’s a narrow path to walk down, but on the other side, you emerge with more core concepts and models to leverage in your everyday programming. You will almost certainly become more comfortable with denser code, and will certainly gain new insights into how to be a better software engineer.

Android 4.4 has made a big change in the OS’ internals for HTML5 development: it has replaced its original WebKit-based WebView with modern Chromium. The new Android Browser is also powered by Chromium, but it’s not clear yet its future. Besides the good news, not everything looks exciting in these changes: let’s see why. Every web developer that has played with native webapps, PhoneGap and the Android’s WebView knows how terrible it was in terms of performance and HTML5 compatibility. These are the same problems that most web developers suffer right now with the Android Browser, which is reported to be 32% of the mobile web browsing market share, compared with just 5% of the modern Chrome for Android according to Akamai. I’ve been talking about this problem in a recent post this year: Android Browser: the eternal mobile browser. Therefore I’m the first one to celebrate the beginning of the end for this dying web platform and the Chrome team now in charge of Android’s web runtimes. Chroming Android From Android 4.4, Chromium 30 is the web engine for the WebView native widget, including the V8 JavaScript engine. Let’s start with good news: Support for remote debugging Support for new HTML5 features Better performance Now why we should take this change with moderated excitement: We will still deal with the old WebView for a couple of years. It won’t be upgraded without an OS upgrade There might be some compatibility issues Where is My Browser? Everybody at Android and Chrome team is talking about the new WebView but nobody is even mentioning what will happen to the browser. We all want Chrome as the default browser, but it seems it’s not there yet (licenses issues, I guess). I’ve even seen a couple of members of the Chrome team saying that the stock Android Browser didn’t exist in the latest previous versions, which is not true. From Google’s perspective, Android Browser sounds much like IE6 and nobody wants to talk about it. They give us the idea that Chrome has been powering web browsing in Android for a while, but that is only true for some particular Android devices - Nexuses and devices from top manufactures. However, as I’ve mentioned before, the relationship between users browsing with Android Browser and Chrome is still 7 to 1. Besides what some people believe, the previous version of Android, 4.3, included minor upgrades to the Browser, so it is there for sure. The question is: what will happen on 4.4 with the stock browser? We know that the Nexus 5 has Google Chrome by default; the question here is what will happen with other devices having in mind that average users don’t download browsers from the store and use what the devices offers for browsing. Based on the emulator, the Android Browser is still there on the emulator and it’s using the classic browser UI with the Chromium 30 engine (it can coexist with Chrome but they will be radically different) Unfortunately, there is no mention of this on docs and blogs on Android 4.4. I hope we can get a real answer from the Android team soon about the future of the browser itself. The Good News Remote debugging Finally we have the ability to debug remotely Android native webviews, including PhoneGap apps, and the Android Browser works smoothly both from real devices and from the emulator. When we have an Android app opened with a web view or the Android Browser, the Chrome remote debugger tools will recognize it as a “Chrome 30” session and we have the full package of excellent tools for debug, profile and test our webapps. HTML5 new features Compared with the classic web view and the Android Browser until 4.3, we now have support for: Server Sent events Web Sockets Web Workers Advanced form input selectors, such as date and time FileSystem API IndexedDB MediaCapture Stream ??? test Animation Timing API Page Visibility API Canvas Blend modes CSS3 Flexbox (latest version) CSS3 Filters Even matching Chrome 30 for Android, the Web View (and potentially the Android Browser) will not have support (no reasons given) for: WebGL WebRTC WebAudio FullScreen Form validation Compared with the classic Web View, the new one doesn’t have Network Information API Performance difference Having V8 as the JavaScript engine for the new web view, the JavaScript performance if much better, besides general performance on CSS thanks to hardware acceleration. The Not so Good News The Classic Web View is still alive Don’t get so excited. We will deal with the old Web View (known as “classic”) for a couple of years. In fact, some devices such as Galaxy Nexus that are today on 4.3 will not get the update. And remember that still today 30% of Android users are on 2.x after 2 years of being replaced by 4.0, so it’s fair to guess that at the beginning of 2016 we will still have around a third of the users on the “classic” WebView that we hate today. The migration on the market will be slow based on Android’s fragmentation. WebView upgrade The KitKat WebView is based on Chromium 30 and it won’t be updated. That means you are stuck with it unless to get an upgrade in the future of the whole OS to next version. Even Google has announced OS delta updates without vendors’ intervention, but it seems the WebView will not get that deal yet. Therefore and based on Chrome's release cycle, in one year we will have Chrome 40 and the WebView will still be in 30. In a couple of years we might be complaining about an “old and outdated” webview again Compatibility issues Because there are changes between the old WebKit-based rendering engine and the modern Chromium engine, you should test your native webapp on KitKat to make sure it’s still working great. To reduce problems, if our app was packaged before KitKat the WebView will enter a “quirks mode” (any similarity with IE6 is pure coincidence) that will reduce the risk of incompatibilities while still getting the new APIs. In fact, this compatibility mode will get in action if the configuration file of your app has a target SDK lower than 19 (the API number for KitKat). To get more detailed information on migration and compatibility issues you can try the new Guides at Android and Chrome websites: http://developer.android.com/guide/webapps/migrating.html http://developers.google.com/chrome/mobile/docs/webview Looking Forward I’m really looking forward to remove the old WebKit and Android Browser from the market. The Chrome team is doing a great job empowering the mobile web (just remember homescreen webapps from Chrome 31), but sometimes the Android ecosystem is slowing down HTML5 penetration and helping promoting companies to avoid using web technologies. I hope this is the beginning of a change.

In this post I’m going to explain you about Linq AddRange method. This method is quite useful when you want to add multiple elements to a end of list. Following is a method signature for this. public void AddRange( IEnumerable collection ) To Understand let’s take simple example like following. using System; using System.Collections.Generic; namespace Linq { class Program { static void Main(string[] args) { List names=new List {"Jalpesh"}; string[] newnames=new string[]{"Vishal","Tushar","Vikas","Himanshu"}; foreach (var newname in newnames) { names.Add(newname); } foreach (var n in names) { Console.WriteLine(n); } } } } Here in the above code I am adding content of array to a already created list via foreach loop. You can use AddRange method instead of for loop like following.It will same output as above. using System; using System.Collections.Generic; namespace Linq { class Program { static void Main(string[] args) { List names=new List {"Jalpesh"}; string[] newnames=new string[]{"Vishal","Tushar","Vikas","Himanshu"}; names.AddRange(newnames); foreach (var n in names) { Console.WriteLine(n); } } } } Now when you run that example output is like following. Add Range in more complex scenario: You can also use add range to more complex scenarios also like following.You can use other operator with add range as following. using System; using System.Collections.Generic; using System.Linq; namespace Linq { class Program { static void Main(string[] args) { List names=new List {"Jalpesh"}; string[] newnames=new string[]{"Vishal","Tushar","Vikas","Himanshu"}; names.AddRange(newnames.Where(nn=>nn.StartsWith("Vi"))); foreach (var n in names) { Console.WriteLine(n); } } } } Here in the above code I have created array with string and filter it with where operator while adding it to an existing list. Following is output as expected. That’s it. Hope you like it. Stay tuned for more..

Since git hooks can be any executable script with an appropriate #! line, Python is more than suitable for writing your git hooks. Simply stated, git hooks are scripts which are called at different points of time in the life cycle of working with your git repository. Let’s start by creating a new git repository: ~/work> git init git-hooks-exp Initialized empty Git repository in /home/gene/work/git-hooks-exp/.git/ ~/work> cd git-hooks-exp/ ~/work/git-hooks-exp (master)> tree -al .git/ .git/ ├── branches ├── config ├── description ├── HEAD ├── hooks │ ├── applypatch-msg.sample │ ├── commit-msg.sample │ ├── post-update.sample │ ├── pre-applypatch.sample │ ├── pre-commit.sample │ ├── prepare-commit-msg.sample │ ├── pre-rebase.sample │ └── update.sample ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags 9 directories, 12 files Inside the .git are a number of directories and files, one of them being hooks/ which is where the hooks live. By default, you will have a number of hooks with the file names ending in .sample. They may be useful as starting points for your own scripts. However, since they all have an extension .sample, none of the hooks are actually activated. For a hook to be activated, it must have the right file name and it should be executable. Let’s see how we can write a hook using Python. We will write a post-commit hook. This hook is called immediately after you have made a commit. We are going to do something fairly useless, but quite interesting in this hook. We will take the commit SHA1 of this commit, and print how it may look like in a more human form. I do the latter using the humanhash module. You will need to have it installed. Here is how the hook looks like: #!/usr/bin/python import subprocess import humanhash # get the last commit SHA and print it after humanizing it # https://github.com/zacharyvoase/humanhash print humanhash.humanize( subprocess.check_output( ['git','rev-parse','HEAD'])) I use the subprocess.check_output() function to execute the command git rev-parse HEAD so that I can get the commit SHA1 and then call the humanhash.humanize() function with it. Save the hook as a file, post-commit in your hooks/ directory and make it executable using chmod +x .git/hooks/post-commit. Let’s see the hook in action: ~/work/git-hooks-exp (master)> touch file ~/work/git-hooks-exp (master)> git add file ~/work/git-hooks-exp (master)> git commit -m "Added a file" carbon-network-connecticut-equal [master (root-commit) 2d7880b] Added a file 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 file The commit SHA1 for the commit turned out to be 2d7880be746a1c1e75844fc1aa161e2b8d955427. Let’s check it with the humanize function and check if we get the same message as above: >>> humanhash.humanize('2d7880be746a1c1e75844fc1aa161e2b8d955427') 'carbon-network-connecticut-equal' And you can see the same message above as well. For some of the hooks, you will see that they are called with some parameters. In Python you can access them using the sys.argv attribute from the sys module, with the first member being the name of the hook of course and the others will be the parameters that the hook is called with.

One piece to Docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it. One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker! Generate The SSL Certificates We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx. # Create the CA Key and Certificate for signing Client Certs openssl genrsa -des3 -out ca.key 4096 openssl rsa -in ca.key -out ca.key # remove password! openssl req -new -x509 -days 365 -key ca.key -out ca.crt # Create the Server Key, CSR, and Certificate openssl genrsa -des3 -out server.key 1024 openssl rsa -in server.key -out server.key # remove password! openssl req -new -key server.key -out server.csr # We're self signing our own server cert here. This is a no-no in production. openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt # Create the Client Key and CSR openssl genrsa -des3 -out client.key 1024 openssl rsa -in client.key -out client.key # no password! openssl req -new -key client.key -out client.csr # Sign the client certificate with our CA cert. Unlike signing our own server cert, this is what we want to do. openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security. We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs. Setup Nginx The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default. upstream docker { server unix:/var/run/docker.sock fail_timeout=0; } server { listen 4242; server localhost; ssl on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; ssl_client_certificate /etc/nginx/certs/ca.crt; ssl_verify_client on; access_log on; error_log /dev/null; location / { proxy_pass http://docker; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 120; proxy_read_timeout 120; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else! Hack It Up! With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one. # Is normal http traffic denied? curl -v http://localhost:4242/info # How about https, sans client cert and key? curl -v -s -k https://localhost:4242/info # And the final good request! curl -v -s -k --key client.key --cert client.crt https://localhost:4242/info For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot! But wait there’s more… let’s build a container that can call the service to launch other containers! For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting). First, the node.js application that will connect and display information: https = require 'https' fs = require 'fs' options = host: 172.42.1.62 port: 4242 method: 'GET' path: '/containers/json' key: fs.readFileSync('ssl/client.key') cert: fs.readFileSync('ssl/client.crt') headers: { 'Accept': 'application/json'} # not required, but being semantic here! req = https.request options, (res) -> console.log res req.end() And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it! FROM shykes/nodejs MAINTAINER James R. Carr ADD ssl/client* /srv/app/ssl ADD package.json /srv/app/package.json ADD app.coffee /srv/app/app.coffee RUN cd /srv/app && npm install . CMD cd /srv/app && npm start That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!

I just spent the better part of an hour trying to effectively install a version of python that doesn’t affect the rest of my system. This is usually pretty trivial effort according to most online articles. There are a few subtleties you might want to be aware of before you go and and well you know. I am writing this mostly for myself so pardon the casual nature, and/or grammatical errors. So you have a clean install of Mavericks, the world is beautiful. You have a new beautiful new background and you have sudden inspiration to get some work done. So many guides will tell you to do simply use system python which comes with easy_install and install pip with sudo. Don’t do this, For many reasons. Homebrew always has the most recent version (currently 2.7.5). Apple has made significant changes to its bundled Python, potentially resulting in hidden bugs. Homebrew’s Python includes the latest Python package management tools: pip and Setuptools First thing you do is install homebrew. Then do a little something like this. brew install python Once that is done, you will want to update your PATH variable to include the following directories in PATH /usr/local: /usr/local/bin: You want to put local on PATH because it contains other important directories like lib and sbin. What we are doing here is modifying the PATH variable which your system looks for commands, it will traverse the paths until it finds a match and use that. We want it to use copy of python in /usr/local You can put these modifications in your .zshrc or .bashrc whichever floats your boat. Now to test to see if you are using the correct python on your system (you have multiple copies now after the brew install) you want to you use: which python If you get /usr/bin/python you either didn’t set the PATH properly or didn’t reload either .zshrc or .bashrc (here is a hint: restart your terminal) If you get a usr/local/bin/python you are winning. You will also want to make sure you are using the correct copy of pip on your system as well, use the same steps as above. Never install anything with sudo. Don’t listen to random READMEs online. You make the decision, my son. Now you install things you want system wide without having to worry about breaking your system’s copy of Python. Swaggin'.

This blog post demonstrates how easy it is to use Apache Camel and its new json-path component along with the camel-sqs component to produce and consume messages on Amazon SQS. Amazon Web Services SQS is a message queuing “software as a service” (SaaS) in the cloud. To be able to use it, you need to sign up for AWS. It’s primary access mechanism is XML over HTTP through various AWS SDK clients provided by Amazon. Please check out the SQS documentation for more. And as “luck” would have it, one of the users in the Apache Camel community created a component to be able to integrate with SQS. This makes it trivial to add a producer or consumer to an SQS queue and plugs in nicely with the Camel DSL. SQS, however, is not a “one-size fits all” queueing service; you must be aware of your use case and make sure it fits (current requirements as well as somewhat into the future…). There are limitations that, if not studied and accounted for ahead of time, could come back to sink your project. An example of a viable alternative, and one that more closely fits the profile of a high performance and full featured message queue is Apache ActiveMQ. For example, one limitation to keep in mind is that unlike traditional JMS consumers, you cannot create a subscription to a queue that filters messages based on some predicate (at least not using the AWS-SQS API — you’d have to build that into your solution). Some other things to keep in mind when using SQS: The queue does not preserve FIFO messaging That is, message order is not preserved. They can arrive out of order from when they were sent. Apache Camel can help with its resequencer pattern. Bilgin Ibryam, now a colleague of mine at Red Hat, has written a great blog post about how to restore message order using the resequencer pattern. Message size is limited to 256K This is probably sufficient, but if your message sizes are variable, or contain more data that 256K, you will have to chunk them and send in smaller chunks. No selector or selective consumption If you’re familiar with JMS, you know that you can specify consumers to use a “selector” or a predicate expression that is evaluated on the broker side to determine whether or not a specific message should be dispatched to a specific consumer. For example, Durability constraints Some use cases call for the message broker to store messages until consumers return. SQS allows a limit of up to 14 days. This is most likely sufficient, but something to keep in mind. Binary payloads not allowed SQS only allows text-based messages, e.g., XML, JSON, fixed format text, etc. Binary such as Avro, Protocol Buffers, or Thrift are not allowed. For some of these limitations, you can work around them by building out the functionality yourself. I would always recommend taking a look at how an integration library like Apache Camel can help — which has out-of-the-box support for doing some of these things. Doing JMS-style selectors So the basic problem is we want to subscribe to a SQS queue, but we want to filter which messages we process. For those messages that we do not process, those should be left in the queue. To do this, we will make use of Apache Camel’s Filter EIP as well as the visibility timeouts available on the SQS queue. By default, SQS will dispatch all messages in its queue when it’s queried. We cannot change this, and thus not avoid the message being dispatched to us — we’ll have to do the filtering on our side (this is different than how a full-featured broker like ActiveMQ does it, i.e., filtering is done on the broker side so the consumer doesn’t even see the message it does not want to see). Once SQS dispatches a message, it does not remove it from the queue unless the consumer has acknowledged that it has it and is finished with it. The consumer does this by sending a DeleteMessage command. Until the DeleteMessage command is sent, the message is always in the queue, however visibility comes in to play here. When a message is dispatched to a consumer, there is a period of time which it will not be visible to other consumers. So if you browsed the queue, you would not see it (it should appear in the stats as “in-flight”). However, there is a configurable period of time you can specify for how long this “visibility timeout” should be active. So if you set the visibility to a lower time period (default is 30 seconds), you can more quickly get messages re-dispatched to consumers that would be able to handle the message. Take a look at the following Camel route which does just that: @Override public void configure() throws Exception { // every two seconds, send a message to the "demo" queue in SQS from("timer:kickoff?period=5000") .setBody().method(this, "generateJsonString") .to("aws-sqs://demo?amazonSQSClient=#sqsClient&defaultVisibilityTimeout=2"); } In the above Camel Route, we create a new message every 5 seconds and send it to an SQS queue named demo — note we set the defaultVisibilityTimeout to 2 seconds. This means that after a message gets dispatched to a consumer, SQS will wait about 2 seconds before considering it eligible to be dispatched to another consumer if it has not been deleted. On the consumer side, we take advantage of a couple Apache Camel conveniences Using JSON Path + Filter EIP Camel has an excellent new component named JSON-Path. Claus Ibsen tweeted about it when he hacked it up. This allows you to do Content-Based Routing on a JSON payload very easily by using XPath-style expressions to pick out and evaluate attributes in a JSON encoded object. So in the following example, we can test an attribute named ‘type’ to be equal to ‘LOGIN’ and use Camel’s Filter EIP to allow only those messages that match to go through and continue processing: public class ConsumerRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { from("aws-sqs://demo?amazonSQSClient=#sqsClient&deleteIfFiltered=false") .setHeader("identity").jsonpath("$['type']") .filter(simple("${header.identity} == 'login'")) .log("We have a message! ${body}") .to("file:target/output?fileName=login-message-${date:now:MMDDyy-HHmmss}.json"); } } To complete the functionality, we have to pay attention to a new configuration option added for the Camel-SQS component: deleteIfFiltered — Whether or not to send the DeleteMessage to the SQS queue if an exchange fails to get through a filter. If ‘false’ and exchange does not make it through a Camel filter upstream in the route, then don’t send DeleteMessage. By default, Camel will send the “DeleteMessage” command to SQS after a route has completed successfully (without an exception). However, in this case, we are specifying to not send the DeleteMessage command if the message had been previously filtered by Camel. This example demonstrates how easy it is to use Apache Camel and its new json-path component along with the camel-sqs component to produce and consume messages on Amazon SQS. Please take a look at the source code on my github repo to play with the live code and try it out yourself.