This blog is more of a tutorial where we describe the development of a simple data access module, more for fun and learning than anything else. All code can be found here for those who don’t want to type along: https://github.com/ricston-git/tododb As a heads-up, we will be covering the following: Using Groovy in a Maven project within Eclipse Using Groovy to interact with our database Testing our code using the Spock framework We include Spring in our tests with ContextConfiguration A good place to start is to write a pom file as shown here. The only dependencies we want packaged with this artifact are groovy-all and commons-lang. The others are either going to be provided by Tomcat or are only used during testing (hence the scope tags in the pom). For example, we would put the jar with PostgreSQL driver in Tomcat’s lib, and tomcat-jdbc and tomcat-dbcp are already there. (Note: regarding the postgre jar, we would also have to do some minor configuration in Tomcat to define a DataSource which we can get in our app through JNDI – but that’s beyond the scope of this blog. See here for more info). Testing-wise, I’m depending on spring-test, spock-core, and spock-spring (the latter is to get spock to work with spring-test). Another significant addition in the pom is the maven-compiler-plugin. I have tried to get gmaven to work with Groovy in Eclipse, but I have found the maven-compiler-plugin to be a lot easier to work with. With your pom in an empty directory, go ahead and mkdir -p src/main/groovy src/main/java src/test/groovy src/test/java src/main/resources src/test/resources. This gives us a directory structure according to the Maven convention. Now you can go ahead and import the project as a Maven project in Eclipse (install the m2e plugin if you don’t already have it). It is important that you do not mvn eclipse:eclipse in your project. The .classpath it generates will conflict with your m2e plugin and (at least in my case), when you update your pom.xml the plugin will not update your dependencies inside Eclipse. So just import as a maven project once you have your pom.xml and directory structure set up. Okay, so our tests are going to be integration tests, actually using a PostgreSQL database. Since that’s the case, lets set up our database with some data. First go ahead and create a tododbtest database which will only be used for testing purposes. Next, put the following files in your src/test/resources: Note, fill in your username/password: DROP TABLE IF EXISTS todouser CASCADE; CREATE TABLE todouser ( id SERIAL, email varchar(80) UNIQUE NOT NULL, password varchar(80), registered boolean DEFAULT FALSE, confirmationCode varchar(280), CONSTRAINT todouser_pkey PRIMARY KEY (id) ); insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'abc123', FALSE, 'abcdefg') insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'pass1516', FALSE, '123456') insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'anon', FALSE, 'codeA') insert into todouser (email, password, registered, confirmationCode) values ('[email protected]', 'anon2', FALSE, 'codeB') Basically, testContext.xml is what we’ll be configuring our test’s context with. The sub-division into datasource.xml and initdb.xml may be a little too much for this example… but changes are usually easier that way. The gist is that we configure our data source in datasource.xml (this is what we will be injecting in our tests), and the initdb.xml will run the schema.sql and test-data.sql to create our table and populate it with data. So lets create our test, or should I say, our specification. Spock is specification framework that allows us to write more descriptive tests. In general, it makes our tests easier to read and understand, and since we’ll be using Groovy, we might as well make use of the extra readability Spock gives us. package com.ricston.blog.sample.model.spec; import javax.sql.DataSource import org.springframework.beans.factory.annotation.Autowired import org.springframework.test.annotation.DirtiesContext import org.springframework.test.annotation.DirtiesContext.ClassMode import org.springframework.test.context.ContextConfiguration import spock.lang.Specification import com.ricston.blog.sample.model.data.TodoUser import com.ricston.blog.sample.model.dao.postgre.PostgreTodoUserDAO // because it supplies a new application context after each test, the initialize-database in initdb.xml is // executed for each test/specification @DirtiesContext(classMode=ClassMode.AFTER_EACH_TEST_METHOD) @ContextConfiguration('classpath:testContext.xml') class PostgreTodoUserDAOSpec extends Specification { @Autowired DataSource dataSource PostgreTodoUserDAO postgreTodoUserDAO def setup() { postgreTodoUserDAO = new PostgreTodoUserDAO(dataSource) } def "findTodoUserByEmail when user exists in db"() { given: "a db populated with a TodoUser with email [email protected] and the password given below" String email = '[email protected]' String password = 'anon' when: "searching for a TodoUser with that email" TodoUser user = postgreTodoUserDAO.findTodoUserByEmail email then: "the row is found such that the user returned by findTodoUserByEmail has the correct password" user.password == password } } One specification is enough for now, just to make sure that all the moving parts are working nicely together. The specification itself is easy enough to understand. We’re just exercising the findTodoUserByEmail method of PostgreTodoUserDAO – which we will be writing soon. Using the ContextConfiguration from Spring Test we are able to inject beans defined in our context (the dataSource in our case) through the use of annotations. This keeps our tests short and makes them easier to modify later on. Additionally, note the use of DirtiesContext. Basically, after each specification is executed, we cannot rely on the state of the database remaining intact. I am using DirtiesContext to get a new Spring context for each specification run. That way, the table creation and test data insertions happen all over again for each specification we run. Before we can run our specification, we need to create at least the following two classes used in the spec: TodoUser and PostgreTodoUserDAO package com.sample.data import org.apache.commons.lang.builder.ToStringBuilder class TodoUser { long id; String email; String password; String confirmationCode; boolean registered; @Override public String toString() { ToStringBuilder.reflectionToString(this); } } package com.ricston.blog.sample.model.dao.postgre import groovy.sql.Sql import javax.sql.DataSource import com.ricston.blog.sample.model.dao.TodoUserDAO import com.ricston.blog.sample.model.data.TodoUser class PostgreTodoUserDAO implements TodoUserDAO { private Sql sql public PostgreTodoUserDAO(DataSource dataSource) { sql = new Sql(dataSource) } /** * * @param email * @return the TodoUser with the given email */ public TodoUser findTodoUserByEmail(String email) { sql.firstRow """SELECT * FROM todouser WHERE email = $email""" } } package com.ricston.blog.sample.model.dao; import com.ricston.blog.sample.model.data.TodoUser; public interface TodoUserDAO { /** * * @param email * @return the TodoUser with the given email */ public TodoUser findTodoUserByEmail(String email); } We’re just creating a POGO in TodoUser, implementing its toString using common’s ToStringBuilder. In PostgreTodoUserDAO we’re using Groovy’s SQL to access the database, for now, only implementing the findTodoUserByEmail method. PostgreTodoUserDAO implements TodoUserDAO, an interface which specifies the required methods a TodoUserDAO must have. Okay, so now we have all we need to run our specification. Go ahead and run it as a JUnit test from Eclipse. You should get back the following error message: org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object '{id=3, [email protected], password=anon, registered=false, confirmationcode=codeA}' with class 'groovy.sql.GroovyRowResult' to class 'com.ricston.blog.sample.model.data.TodoUser' due to: org.codehaus.groovy.runtime.metaclass.MissingPropertyExceptionNoStack: No such property: confirmationcode for class: com.ricston.blog.sample.model.data.TodoUser Possible solutions: confirmationCode at com.ricston.blog.sample.model.dao.postgre.PostgreTodoUserDAO.findTodoUserByEmail(PostgreTodoUserDAO.groovy:23) at com.ricston.blog.sample.model.spec.PostgreTodoUserDAOSpec.findTodoUserByEmail when user exists in db(PostgreTodoUserDAOSpec.groovy:37) Go ahead and connect to your tododbtest database and select * from todouser; As you can see, our confirmationCode varchar(280), ended up as the column confirmationcode with a lower case ‘c’. In PostgreTodoUserDAO’s findTodoUserByEmail, we are getting back GroovyRowResult from our firstRow invocation. GroovyRowResult implements Map and Groovy is able to create a POGO (in our case TodoUser) from a Map. However, in order for Groovy to be able to automatically coerce the GroovyRowResult into a TodoUser, the keys in the Map (or GroovyRowResult) must match the property names in our POGO. We are using confirmationCode in our TodoUser, and we would like to stick to the camel case convention. What can we do to get around this? Well, first of all, lets change our schema to use confirmation_code. That’s a little more readable. Of course, we still have the same problem as before since confirmation_code will not map to confirmationCode by itself. (Note: remember to change the insert statements in test-data.sql too). One way to get around this is to use Groovy’s propertyMissing methods as show below: def propertyMissing(String name, value) { if(isConfirmationCode(name)) { this.confirmationCode = value } else { unknownProperty(name) } } def propertyMissing(String name) { if(isConfirmationCode(name)) { return confirmationCode } else { unknownProperty(name) } } private boolean isConfirmationCode(String name) { 'confirmation_code'.equals(name) } def unknownProperty(String name) { throw new MissingPropertyException(name, this.class) } By adding this to our TodoUser.groovy we are effectively tapping in on how Groovy resolves property access. When we do something like user.confirmationCode, Groovy automatically calls getConfirmationCode(), a method which we got for free when declared the property confirmationCode in our TodoUser. Now, when user.confirmation_code is invoked, Groovy doesn’t find any getters to invoke since we never declared the property confirmation_code, however, since we have now implemented the propertyMissing methods, before throwing any exceptions it will use those methods as a last resort when resolving properties. In our case we are effectively checking whether a get or set on confirmation_code is being made and mapping the respective operations to our confirmationCode property. It’s as simple as that. Now we can keep the auto coercion in our data access object and the property name we choose to have in our TodoUser. Assuming you’ve made the changes to the schema and test-data.sql to use confirmation_code, go ahead and run the spec file and this time it should pass. That’s it for this tutorial. In conclusion, I would like to discuss some finer points which someone who’s never used Groovy’s SQL before might not know. As you can see in PostgreTodoUserDAO.groovy, our database interaction is pretty much a one-liner. What about resource handling (e.g. properly closing the connection when we’re done), error logging, and prepared statements? Resource handling and error logging are done automatically, you just have to worry about writing your SQL. When you do write your SQL, try to stick to using triple quotes as used in the PostgreTodoUserDAO.groovy example. This produces prepared statements, therefore protecting against SQL injection and avoids us having to put ‘?’ all over the place and properly lining up the arguments to pass in to the SQL statement. Note that transaction management is something which the code using our artifact will have to take care of. Finally, note that a bunch of other operations (apart from findTodoUserByEmail) are implemented in the project on GitHub: https://github.com/ricston-git/tododb. Additionally, there is also a specification test for TodoUser, making sure that the property mapping works correctly. Also, in the pom.xml, there is some maven-surefire-plugin configuration in order to get the surefire-plugin to pick up our Spock specifications as well as any JUnit tests which we might have in our project. This allows us to run our specifications when we, for example, mvn clean package. After implementing all the operations you require in PostgreTodoUserDAO.groovy, you can go ahead and compile the jar or include in a Maven multi-module project to get a data access module you can use in other applications.

transitioning from the relational world to the beautiful world of graphs requires a shift in thinking about data. although graphs are often much more intuitive than tables, there are certain mistakes people tend to make when modelling their data as a graph for the first time. in this article, we look at one common source of confusion: bidirectional relationships. directed relationships relationships in neo4j must have a type, giving the relationship a semantic meaning, and a direction. frequently, the direction becomes part of the relationship's meaning. in other words, the relationship would be ambiguous without it. for example, the following graph shows that the czech republic defeated sweden in ice hockey. had the direction of the relationship been reversed, the swedes would be much happier. with no direction at all, the relationship would be ambiguous, since it would not be clear who the winner was. note that the existence of this relationship implies a relationship of a different type going in the opposite direction, as the next graph illustrates. this is often the case. to give another example, the fact that pulp fiction was directed_by quentin tarantino implies that quentin tarantino is_director_of pulp fiction. you could come up with a huge number of such relationship pairs. one common mistake people often make when modelling their domain in neo4j is creating both types of relationships. since one relationship implies the other, this is wasteful, both in terms of space and traversal time. neo4j can traverse relationships in both directions. more importantly, thanks to the way neo4j organizes its data, the speed of traversal does not depend on the direction of the relationships being traversed. bidirectional relationships some relationships, on the other hand, are naturally bidirectional. a classic example is facebook or real-life friendship. this relationship is mutual - when someone is your friend, you are (hopefully) his friend, too. depending on how we look at the model, we could also say such relationship is undirected. graphaware and neo technology are partner companies. since this is a mutual relationship, we could model it as bidirectional or undirected relationship, respectively. but since none of this is directly possible in neo4j, beginners often resort to the following model, which suffers from the exact same problem as the incorrect ice hockey model: an extra unnecessary relationship. neo4j apis allow developers to completely ignore relationship direction when querying the graph, if they so desire. for example, in neo4j's own query language, cypher, the key part of a query finding all partner companies of neo technology would look something like match (neo)-[:partner]-(partner) the result would be the same as executing and merging the results of the following two different queries: match (neo)-[:partner]->(partner) and match (neo)<-[:partner]-(partner) therefore, the correct (or at least most efficient) way of modelling the partner relationships is using a single partner relationship with an arbitrary direction . conclusion relationships in neo4j can be traversed in both directions with the same speed. moreover, direction can be completely ignored. therefore, there is no need to create two different relationships between nodes, if one implies the other.

(NOTE: A version of this article posted previously contained incorrect information. The below version corrects those errors. The author apologises for any inconvenience.) I’m not a programmer. I’m an art collection manager with a fierce DIY streak that has helped me to develop a database application, and build and manage a website that incorporates it. Ever since I accidentally lobotomised my first Windows 3.1 computer, I’ve taught myself how to seek out, find and apply the information I need, sometimes through long hours of trial and error, and I owe almost all of it to the Internet. If it weren’t for people’s willingness to share information for free on innumerable forums and websites like this one, I would not have been able even to scratch the surface of completing the sorts of tasks that are now behind me. In that spirit of sharing, I thought I might humbly offer the solution I’ve cobbled together from various sources, and tweaked to automate tasks related to collecting the IP addresses of visitors to my website. I’m sure it’s nothing earth-shattering to an experienced coder, but it works well for me, and I’ve never seen a complete solution like it presented anywhere on the Internet before. It is a Windows-centric solution, since that’s the platform I’ve always used. The first step is to gather and record the IP addresses of website visitors. I’ve chosen to do this using php. Insert this code into the html of each page for which you’d like to capture IP addresses, just before the closing tag: If you want to record the IP addresses for each webpage to a different file, use a different name each time for filename.txt in the above example. Next, create the blank filename.txt file(s) in the same directory of your web server in which these html files reside. Now each time a visitor loads these pages, their IP address will be written to the text file(s) you’ve indicated. Next, you’ll need a way to download the text file(s) from the server to your local machine. If you’re writing to multiple files on the server, I’ve found it’s helpful to download them separately, then combine them into one list. Also, I like to sort the list and remove duplicate entries (you’ll see why a little later). Following is a Visual Basic script to do all of that. Let’s designate it C:\Folder\Subfolder\DloadCmbnDdupe.vbs. In the script below, substitute YourWebsite.com with the domain name of your website, C:\Folder\Subfolder with the actual location on your local computer and filename*.txt with the file name(s) on the web server to which you’ve chosen to write. Option Explicit On Error Resume Next Download "http://www.YourWebsite.com/filename.txt", _ " C:\Folder\Subfolder\filename.txt " Download "http://www.YourWebsite.com/filename2.txt", _ " C:\Folder\Subfolder\filename2.txt " Download "http://www.YourWebsite.com/filename3.txt", _ " C:\Folder\Subfolder\filename3.txt " CmbnDdupe() If Err <> 0 Then Wscript.echo "Error Type = " & Err.Description End If WScript.Quit '----------------------------------------------------------------------------------------- Function Download(strURL, strPath) Dim i, objFile, objFSO, objHTTP, strFile, strMsg Const ForReading = 1, ForWriting = 2, ForAppending = 8 Set objFSO = CreateObject("Scripting.FileSystemObject") If objFSO.FolderExists(strPath) Then strFile = objFSO.BuildPath(strPath, Mid(strURL, InStrRev(strURL, "/") + 1)) ElseIf objFSO.FolderExists(Left(strPath, InStrRev(strPath, "\") - 1)) Then strFile = strPath Else WScript.Echo "ERROR: Target folder not found." Exit Function End If Set objFile = objFSO.OpenTextFile(strFile, ForWriting, True) Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1") objHTTP.Open "GET", strURL, False objHTTP.Send For i = 1 To LenB(objHTTP.ResponseBody) objFile.Write Chr(AscB(MidB(objHTTP.ResponseBody, i, 1))) Next objFile.Close() End Function '----------------------------------------------------------------------------------------- Function CmbnDdupe() Dim shell Set shell=createobject("wscript.shell") shell.run "CmbnDdupe.bat" Set shell=nothing End Function You’ll notice that Function CmbnDdupe calls a batch file, CmbnDdupe.bat, in the same directory (C:\Folder\Subfolder). Here it is, below. Again, substitute filename*.txt with the file name(s) you used at the beginning. @echo off for %%x in (filename.txt) do type %%x>>templist for %%x in (filename2.txt) do type %%x>>templist for %%x in (filename3.txt) do type %%x>>templist ren templist IPlist.txt setlocal disableDelayedExpansion set file=IPlist.txt set "sorted=%file%.sorted" set "deduped=%file%.deduped" ::Define a variable containing a linefeed character set LF=^ ::The 2 blank lines above are critical, do not remove sort "%file%" >"%sorted%" >"%deduped%" ( set "prev=" for /f usebackq^ eol^=^%LF%%LF%^ delims^= %%A in ("%sorted%") do ( set "ln=%%A" setlocal enableDelayedExpansion if /i "!ln!" neq "!prev!" ( endlocal (echo %%A) set "prev=%%A" ) else endlocal ) ) >nul move /y "%deduped%" "%file%" del "%sorted%" exit This routine combines the downloaded files into one (templist), then renames it to IPlist.txt. It then sorts the IP addresses into ascending order, saving to IPlist.txt.sorted, and then removes any duplicates, saving to IPlist.txt.deduped. Finally, it moves (overwrites and deletes) IPlist.txt.deduped to IPlist.txt and deletes IPlist.txt.sorted, leaving behind IPlist.txt (the sorted and de-dupe-ified list). Now, we have a list of one IP address per visitor to the pages from which we’re collecting. At this point I like to ping each IP address to collect whatever information is available about it. This is why I remove the duplicate entries, which are caused by a visitor viewing more than one of the collecting pages, or by a visitor returning to the pages. I don’t need to waste time and bandwidth pinging the same IP address more than once. If I want to see which IPs visited which pages multiple times, I can always just look at filename.txt, filename2.txt and filename3.txt. I’ve named the ping routine PingList.vbs, and put it in C:\Folder\Subfolder. Here it is: Option Explicit On Error Resume Next Dim srcFile srcFile = "IPlist.txt" PingList(srcFile) If Err <> 0 Then Wscript.echo "Error Type = " & Err.Description End If WScript.Quit '----------------------------------------------------------------------------------------- Function PingList(srcFile) Dim objFSO Dim objShell Dim strCommand Dim opnFile Dim strText Dim logFile Set objFSO = CreateObject("Scripting.FileSystemObject") Set objShell = Wscript.CreateObject("Wscript.Shell") logFile = "Log.txt" If objFSO.FileExists(srcFile) Then Set opnFile = objFSO.OpenTextFile(srcFile, 1) Do While Not opnFile.AtEndOfStream strText = opnFile.ReadLine If Trim(strText) <> "" Then strCommand = strText objShell.run "%comspec% /c ping -a -n 1 " & strText & " >> " & logFile, , True End If Loop opnFile.Close Else WScript.Echo "File '" & srcFile & "' was not found." End If End Function This script pings each IP address in IPlist.txt, resolves the hostname if possible and writes the results to Log.txt in the same directory. Go ahead and create a blank Log.txt now in C:\Folder\Subfolder. The next thing you’ll want to do is clear the contents of the text files on the server, so that they will retain only the IP addresses from new visits. Create the following file in C:\Folder\Subfolder. I’ve named it Upload.cmd. It's critical that it has the .cmd file type. @echo off echo user YourUsername> ftpcmd.dat echo YourPassword>> ftpcmd.dat echo bin>> ftpcmd.dat echo cd /YourWebDirectory/>> ftpcmd.dat echo prompt>> ftpcmd.dat echo mput %1 %2 %3>> ftpcmd.dat echo rename filename_.txt filename.txt>> ftpcmd.dat echo rename filename 2_.txt filename2.txt>> ftpcmd.dat echo rename filename 3_.txt filename3.txt>> ftpcmd.dat echo quit>> ftpcmd.dat ftp -n -s:ftpcmd.dat ftp.YourWebsite.com del ftpcmd.dat exit Substitute YourUsername and YourPassword with the username and password with which you access your website files, and YourWebDirectory with the location of your website files on the server. In C:\Folder\Subfolder, create the blank text file(s) that will overwrite the ones on the server. Give them a different name (for instance, add an underscore), as you’ll want to distinguish them from the files you downloaded at the beginning of this exercise. Hence, the blank filename_.txt will be copied to the server as filename.txt, overwriting the existing file. The number of per-cent-sign-plus-integer combinations (variables) needs to correspond with the number of files you upload and overwrite; in this case, three (%1 %2 %3 = filename_.txt, filename2_.txt and filename3_.txt). Substitute YourWebsite.com for the domain name of your website. Before moving to the final step, create a blank text file named ErrorLog.txt in C:\Folder\Subfolder. This is where we’ll record any errors encountered during the execution of the combined routines. Now to put it all together and automate it. Create the following batch file (I’ve name it DLPingUL.bat) and put it in C:\. @echo off title Download Ping Upload - Scheduled task, please wait cd "C:\Folder\Subfolder" start "" /wait CScript DloadCmbnDdupe.vbs 2>> ErrorLog.txt start "" /wait Upload.cmd filename_.txt filename2_.txt filename3_.txt 2>> ErrorLog.txt copy /d /y /a IPlist.txt NewIPlist.txt /a 2>> ErrorLog.txt start "" /wait CScript PingList.vbs 2>> ErrorLog.txt del IPlist.txt exit The reason we’ve put this in C:\ is so that Windows Task Scheduler will have no problem with permissions when running it. This routine moves to the directory in which you’ve stored all of the relevant files (C:\Folder\Subfolder, substitute with the actual location); executes the Visual Basic script that creates the list; executes the upload, passing the file names of the blank replacement files to the echo mput %1 %2 %3 command; copies IPlist.txt to NewIPlist.txt, overwriting the latter if it exists (this is so you have a list to which to refer if you want); and executes the VB Script to ping and record the results. Finally, it deletes IPlist.txt, as it needs to be created programmatically each time. Any errors are recorded in C:\Folder\Subfolder\ErrorLog.txt. The final step is to create a new task in Task Scheduler that runs C:\DLPingUL.bat at a time of your choosing. I run it every Saturday at 3:00 am, so that when I wake up I have C:\Folder\Subfolder\Log.txt waiting for me with all of its pinged IP address information. Having the hostname can be especially helpful; it can show you which bots and spiders crawled your site, or from which corporation the visit originated. The only manual task I do is any further research on those IP addresses, like running them through Whois or whatismyipaddress.com/ip-lookup . These sites help me to determine, for example, if an IP address is static or dynamic, or if it is associated with hackers or spammers, among other useful bits of information. When I’m finished, I clear the contents of Log.txt so that it is ready for next time. If you’ve read all the way down to here then you’ve been very patient with me, and for that I thank you kindly. Addendum: For best results, when copying the above code, click on "View Source" in the upper right corner of the code box, and copy and paste the source. This will ensure consistency. _____________________________________________ Phillip Schubert is the founder of Schubert & Associates www.schubertassociates.com.au

A quick overview on how to see your heaps in Eclipse.

We'll use a custom CXF interceptor to add these headers.

What is Functional Programming? Functional programming is a specific way to look at problems and model their solutions. Pragmatically, functional programming is a coding style that exhibits the following characteristics: Power and flexibility. We can solve many general real-world problems using functional constructs Simplicity. Most functional programs exhibit a small set of keywords and concise syntax for expressing concepts Suitable for parallel processing. Via immutable values and operators, functional programs lend themselves to asynchronous and parallel processing. In functional programming, programs are executed by evaluating expressions, in contrast with imperative programming where programs are composed of statements which change global state when executed. Functional programming typically avoids using mutable state. Everything is a mathematical function. Functional programming languages can have objects, but generally those objects are immutable -- either arguments or return values to functions. There are no for/next loops, as those imply state changes. Instead, that type of looping is performed with recursion and by passing functions as arguments. Functional Programming vs. Imperative Programming: We can think of imperative programming as writing code that describes in exacting detail the steps the software must take to execute a given computation. These steps are generally expressed as a combination of statement executions, state changes, loops and conditional branches. Many programming languages support programming in both functional and imperative style but the syntax and facilities of a language are typically optimised for only one of these styles, and social factors like coding conventions and libraries often force the programmer towards one of the styles. Therefore, programming languages may be categorized into functional and imperative ones. Why Functional Programming? While you can develop concurrent, scalable and asynchronous software without embracing functional programming, it's simpler, safer, and easier to use the right tool for the job. Functional programming enables you to take advantage of multi-core systems, develop robust concurrent algorithms, parallelize compute-intensive algorithms, and to readily leverage the growing number of cloud computing platforms. Imagine you've implemented a large program in a purely functional way. All the data is properly threaded in and out of functions, and there are no truly destructive updates to speak of. Now pick the two lowest-level and most isolated functions in the entire codebase. They're used all over the place, but are never called from the same modules. Now make these dependent on each other: function A behaves differently depending on the number of times function B has been called and vice-versa. In addition, if you've not already explored non-imperative programming, it's a great way to expand your problem solving skills and your horizons. The new concepts that you'll learn will help you to look at many problems from a different perspective and will help you to become a better and more insightful OO programmer. I encourage everyone who wants to be a better programmer: consider learning a functional language. Haskell and OCaml are both great choices, and F# and Erlang are pretty good as well. It won’t be easy, but that is probably a good sign. Try and identify the difficult concepts you encounter and see if other people are leveraging them; frequently you can break through a mental roadblock by finding out what the intent of an unfamiliar abstraction really is. While you’re learning, do be careful not to take it too seriously. Like anything that requires time and effort, there is a danger of becoming over-invested in FP. Falling into this cognitive trap will ruin your investment. It’s easy to forget how many models of computation there are out there, and even easier to forget how much beautiful software has been written with any of them. It’s a narrow path to walk down, but on the other side, you emerge with more core concepts and models to leverage in your everyday programming. You will almost certainly become more comfortable with denser code, and will certainly gain new insights into how to be a better software engineer.

Android 4.4 has made a big change in the OS’ internals for HTML5 development: it has replaced its original WebKit-based WebView with modern Chromium. The new Android Browser is also powered by Chromium, but it’s not clear yet its future. Besides the good news, not everything looks exciting in these changes: let’s see why. Every web developer that has played with native webapps, PhoneGap and the Android’s WebView knows how terrible it was in terms of performance and HTML5 compatibility. These are the same problems that most web developers suffer right now with the Android Browser, which is reported to be 32% of the mobile web browsing market share, compared with just 5% of the modern Chrome for Android according to Akamai. I’ve been talking about this problem in a recent post this year: Android Browser: the eternal mobile browser. Therefore I’m the first one to celebrate the beginning of the end for this dying web platform and the Chrome team now in charge of Android’s web runtimes. Chroming Android From Android 4.4, Chromium 30 is the web engine for the WebView native widget, including the V8 JavaScript engine. Let’s start with good news: Support for remote debugging Support for new HTML5 features Better performance Now why we should take this change with moderated excitement: We will still deal with the old WebView for a couple of years. It won’t be upgraded without an OS upgrade There might be some compatibility issues Where is My Browser? Everybody at Android and Chrome team is talking about the new WebView but nobody is even mentioning what will happen to the browser. We all want Chrome as the default browser, but it seems it’s not there yet (licenses issues, I guess). I’ve even seen a couple of members of the Chrome team saying that the stock Android Browser didn’t exist in the latest previous versions, which is not true. From Google’s perspective, Android Browser sounds much like IE6 and nobody wants to talk about it. They give us the idea that Chrome has been powering web browsing in Android for a while, but that is only true for some particular Android devices - Nexuses and devices from top manufactures. However, as I’ve mentioned before, the relationship between users browsing with Android Browser and Chrome is still 7 to 1. Besides what some people believe, the previous version of Android, 4.3, included minor upgrades to the Browser, so it is there for sure. The question is: what will happen on 4.4 with the stock browser? We know that the Nexus 5 has Google Chrome by default; the question here is what will happen with other devices having in mind that average users don’t download browsers from the store and use what the devices offers for browsing. Based on the emulator, the Android Browser is still there on the emulator and it’s using the classic browser UI with the Chromium 30 engine (it can coexist with Chrome but they will be radically different) Unfortunately, there is no mention of this on docs and blogs on Android 4.4. I hope we can get a real answer from the Android team soon about the future of the browser itself. The Good News Remote debugging Finally we have the ability to debug remotely Android native webviews, including PhoneGap apps, and the Android Browser works smoothly both from real devices and from the emulator. When we have an Android app opened with a web view or the Android Browser, the Chrome remote debugger tools will recognize it as a “Chrome 30” session and we have the full package of excellent tools for debug, profile and test our webapps. HTML5 new features Compared with the classic web view and the Android Browser until 4.3, we now have support for: Server Sent events Web Sockets Web Workers Advanced form input selectors, such as date and time FileSystem API IndexedDB MediaCapture Stream ??? test Animation Timing API Page Visibility API Canvas Blend modes CSS3 Flexbox (latest version) CSS3 Filters Even matching Chrome 30 for Android, the Web View (and potentially the Android Browser) will not have support (no reasons given) for: WebGL WebRTC WebAudio FullScreen Form validation Compared with the classic Web View, the new one doesn’t have Network Information API Performance difference Having V8 as the JavaScript engine for the new web view, the JavaScript performance if much better, besides general performance on CSS thanks to hardware acceleration. The Not so Good News The Classic Web View is still alive Don’t get so excited. We will deal with the old Web View (known as “classic”) for a couple of years. In fact, some devices such as Galaxy Nexus that are today on 4.3 will not get the update. And remember that still today 30% of Android users are on 2.x after 2 years of being replaced by 4.0, so it’s fair to guess that at the beginning of 2016 we will still have around a third of the users on the “classic” WebView that we hate today. The migration on the market will be slow based on Android’s fragmentation. WebView upgrade The KitKat WebView is based on Chromium 30 and it won’t be updated. That means you are stuck with it unless to get an upgrade in the future of the whole OS to next version. Even Google has announced OS delta updates without vendors’ intervention, but it seems the WebView will not get that deal yet. Therefore and based on Chrome's release cycle, in one year we will have Chrome 40 and the WebView will still be in 30. In a couple of years we might be complaining about an “old and outdated” webview again Compatibility issues Because there are changes between the old WebKit-based rendering engine and the modern Chromium engine, you should test your native webapp on KitKat to make sure it’s still working great. To reduce problems, if our app was packaged before KitKat the WebView will enter a “quirks mode” (any similarity with IE6 is pure coincidence) that will reduce the risk of incompatibilities while still getting the new APIs. In fact, this compatibility mode will get in action if the configuration file of your app has a target SDK lower than 19 (the API number for KitKat). To get more detailed information on migration and compatibility issues you can try the new Guides at Android and Chrome websites: http://developer.android.com/guide/webapps/migrating.html http://developers.google.com/chrome/mobile/docs/webview Looking Forward I’m really looking forward to remove the old WebKit and Android Browser from the market. The Chrome team is doing a great job empowering the mobile web (just remember homescreen webapps from Chrome 31), but sometimes the Android ecosystem is slowing down HTML5 penetration and helping promoting companies to avoid using web technologies. I hope this is the beginning of a change.

My intensive work on EasyNetQ (our super simple .NET API for RabbitMQ) continues. I’ve been taking lessons learned from nearly two years of production and the fantastic feedback from EasyNetQ’s users, mashing this together, and making lots changes to both the internals and the API. I know that API changes cause problems for users; they break your application and force you to revisit your code. But the longer term benefits should outweigh the immediate costs as EasyNetQ slowly morphs into a solid, reliable library that really does make working with RabbitMQ as easy as possible. Changes in version 0.17 are all around the request-response pattern. The initial implementation was very rough with lots of nasty ways that resource use could run away when things went wrong. The lack of timeouts also meant that your application could wait forever when messages got lost. Lastly the API was quite clunky, with call-backs where Tasks are a far better choice. All these problems have been corrected in this version. API changes There is now a synchronous Request method. Of course messaging is by nature a asynchronous operation, but sometimes you just want the simplest possible thing and you don’t care about blocking your thread while you wait for a response. Here’s what it looks like: var response = bus.Request(request); The old call-back Request method has been removed. There was no need for it when the RequestAsync that returned a Task was always a better choice: var task = bus.RequestAsync(request) task.ContinueWith(response => { Console.WriteLine("Got response: '{0}'", response.Result.Text); }); Timeouts Timeouts are an essential ingredient of any distributed system. This probably deserves a blog post of its own, but no matter how resilient you make your architecture, if an important piece simply goes away (like the network for example), you need a circuit breaker. EasyNetQ now has a global timeout that you can configure via the connection string: var bus = RabbitHutch.CreateBus("host=localhost;timeout=60"); Here we’ve configured the timeout as 60 seconds. The default is 10 seconds. If you make a request, but no response is received within the timeout period, a System.Timeout exception will be thrown. If the connection goes away while a request is in-flight, EasyNetQ doesn’t wait for the timeout to fire, but immediately throws an EasyNetQException with a message saying that the connection has been lost. Your application should catch both Timeout and EasyNetQ exceptions and react appropriately. Internal Changes My last blog post was a discussion of the implementation options of request-response with RabbitMQ. As I said there, I now believe that a single exclusive queue for all responses to a client is the best option. Version 0.17 implements this. When you call bus.Request(…) you will see a queue created named easynetq.response.. This will last for the lifetime of the current connection. Happy requesting!

The second blog post in the Exploring the landscape of large scale agile frameworks - series “Wet agile”, “the agile waterfall” and “the Agile-Waterfall Hybrid” … this controversial, mixed-method baby has as many names as formats. Some have received a lot of dedicated thought, are fit-for-purpose and manage to preserve the main benefits of the more pure methods. Other hybrid models have resulted by accident; either as a consequence of… … a half-way-stopped agile transitioning program that dropped dead somewhere between waterfall and scrum in an unsuccessful change initiative … a compromise between method-promoters of different ideologies (often management being on the more waterfall friendly side vs. developers generally promoting more agility) The result of the latter two cases can be horrible to witness and worse to experience for everyone involved regardless if they sit within the development organization or are on the receiving end waiting for the product. However, I do not think the frequent occurrence of poor agile-waterfall hybrids we can witness in the industry is a reason to consistently argue against mixing the two arts as many agilists do. Rather, these cases highlight the need to bring the successful and proven versions of agile-waterfall hybrids into the light and reserve a place for them among their purer cousins in the landscape for large scale agile frameworks. Founders of more successful hybrid versions often come from product development companies that deal with both software and hardware. The driver behind the method mixing in these cases is, if we simplify to the extreme, the realization that many aspects of hardware development benefit from waterfall processes, whereas software development has much to gain from an agile approach. The Agile-Waterfall Hybrid described by Erick Bergmann and Andy Hamilton from Schneider Electric is a great example of a model that merges Agile and Project Management Process/waterfall in a good way without compromising the methods’ core principles to much. Erick Bergmann and Andy Hamilton presented the Agile-Waterfall Hybrid at Agile 2013 and the key concepts of the model can be well understood from the presenting material; https://submissions.agilealliance.org/system/sessions/attachments/000/000/760/original/Agile-Waterfall_Hybrid_01AUG2013.pdf The model allows software teams to adapt agile practices while hardware development and overall product management is handled through a traditional PMP/waterfall approach The overall PMP process has a well-defined interface to the agile software development which is continuous right from the start of the concept/feasibility study up until validation and production. The interface has the format of close collaboration for all activities ranging from definition of requirements and scoping to continuous software deliveries and feedback between the agile side and the waterfallish PMP side The model accommodates development where the same/similar software is used in several products with separate POs. Eg., the software teams need to deliver features to multiple stakeholders continuously in a similar fashion as component teams typically need to act towards feature teams. The very challenging backlog management situation resulting for teams in environments like this is addressed in a good way by the model. In short, it describes how you can achieve swift product releases by putting effort into the feature releases planning on the software side There is no perfect alternative. Just like any other hybrid model, this Agile-Waterfall Hybrid compromises with some of the principles of the non-hybrid methods. The waterfall side of development is forced to live with the flexibility surrounding the software requirements and the agile teams must commit to time-fixed delivery dates, cost forecasts and risk assessment. Hybrid development models face many challenges. It is not easy to combine the dependency tracking and clarity of waterfall with the flexibility and openness to change of Agile development without diluting the benefits and create complex work processes. However, examples like the one referred to above prove that it is possible and as the industry need for these types of models is evident I hope to see more of these being published in the future.

During a day off I stumbled upon a kata that had a simple definition: the Harry Potter books kata. It is old enough that it considers 5 as the total number of Hary Potter books instead of 7, but it has aged well. Unlike for the bowling kata, there is nothing specific to Harry Potter in the problem. The domain is that of e-commerce and the problem consists in calculating the discount a given set of books is eligible for; for example, considering every copy has a list price of 8 pounds, how much a purchasing 2 Philosopher Stone books and 1 Chamber of Secrets would cost. I don't want to talk much about the problem itself here to avoid spoilers for you; suffice to say that it took me 8 hours to get to a solution that I could be proud of (the first couple of which to get to a potentially shippable solution, still not being confident of it being bugs-free or of its extensibility.) I want instead to enumerate some of the traits of the problem that make it interesting with respect to the more popular katas. Some of these traits could even invalidate the problem as a kata; still it's a challenging exercise. Difficulty First of all, the Harry Potter kata does not have a trivial solution. Some katas focus on learning a new technique like TDD or how to follow the Open/Closed Principle; they keep the related problem simple enough not to change two different variables of the experiment at the same time. Still, most programmers that take the time to perform katas can see the solutions to these problems once they have read it. They can in principle write a long procedure that will solve the problem, satisfying functional requirements and probably not the non-functional ones such as maintainability or extensibility. Following a discipline even when you see an easy solution is probably a useful skill; however I feel that too simple problems prime the head of a programmer with the first design he can think of, so that applying TDD becomes a way of putting the design into practice instead of let it emerge bit by bit while adding new specifications. Instead, consider a problem which is just out of your reach. You can's solve it in a single shot and when you take a look at the problem you see some statements that can make you panic: "I don't know how to do that... yet". This problem is a much better simulation of real world conditions. It forces you to add assumptions to simplify the problem and follow an incremental path towards the solution instead of adding now components that you will only use later, if only to keep the cognitive load manageable. Incrementality The Harry Potter kata divides nicely into almost-independent test scenarios. The input and output state are simple to express and the same API used for the first test can be used without much expansion for the last ones. For example, the first test comprehends 0 or 1 book. Intermediate ones get to 4 or 5 books some of which are duplicated copies with the same title. Large acceptance tests get to more than 20 books (there's a reason for that.) Breaking assumptions I can't say too much about the steps I took to solve the problem without ruining your fun. However here are some beliefs I think are challenged by the experience: One new test case does not result in wild modifications to the production code. You can follow a monotonic path to the solution: you don't ever have to abandon functionality that you have already written when refactoring. Code organization and documentation are really a secondary aspect as code can be self-documenting. Once you have added a few corner cases you don't have to worry about the code working even if the solution is difficult to understand. Worry about efficiency later. First make it work. Conclusions The Harry Potter kata is a nice and challenging exercise, in the category that keeps you glued to the keyboard for a long time. It's definitely more time-consuming than smaller katas, like for the application katas, so you have to allocate a sensible amount of time to solve it. However, the benefits I feel from this kind of training are different from the ones of toy problems: problem-solving is a different art than training a particular practice.

Apache Shiro can be used in a wide range of applications as part of the Java Security Framework.

there are a lot of design patterns, architectures and design concepts that can be applied to technical aspects of implementing a product. for exapmle, we have mvc architecture that isolates the view, controller and model of the application. we have a factory pattern that defines how to create objects and so on. yet, these all limit themselves to the technical aspects of the product, while there exists no definite pre-defined architecture or patterns to design and build "business functions". the plug and play architecture i want to describe in this article is a "functional architecture", that defines a pattern to design the business functionality to get the most out of it. modular architecture is a design technique where functionality of a program is separated into independent, interchangeable modules, such that each module contains everything necessary to execute one aspect of the desired functionality. typically, in modular architecture, separation is done based on technical aspects. for eg., a module is created for database interaction, another is created for logging and so on. the advantage of a modular architecture is that you can easily replace or add components without affecting the rest of the application. the modules clearly define the interfaces that are used to interact with the module and these are tied at compile time to other modules. what if we could extend this architecture to "business functionality" rather than just limiting it to technical aspects of a product. we already design "business functionality" in a modular manner. but, we do not consciously look at it as modules with clear-cut interfaces for inputs and outputs. defining the plug and play architecture the plug and play architecture extends the techniques of modular architecture to business functions. for eg., a module can be created to encompass all functions related to order, while another can be created to encompass all functions related to quote. i will call these independent modules implementing business functions as flows. an application, in this architecture, is defined as a "collection of flows working together". the same advantage of modular architecture applies here, i.e., you can replace, add or remove flows without affecting the rest of the application. an added advantage to extending modules to business functions, is that the flows can be combined in different paths to get different business functionality. further, extending this definition to be able to add, remove or replace modules at "runtime" rather than tying them together at compile time gives us the flexibility to customize the application at runtime without "in-built flags and if clauses". the beauty of this extension of the architecture is that "it gives us the flexibility to create a set of loosely coupled flows that can be bound together at runtime as opposed to coding one tightly integrated application that is rigid and not malleable". to further see how this architecture works, let us consider a set of loosely coupled flows coded as per the diagram: there are 5 flows coded and deployed. each flow exposes an output and accepts a set of inputs. for eg., the "product listing" flow encapsulates the following business functions create products manually by admin import products by admin provide services to view, search and list products expose an output, "product" data that can be linked to other flows another eg., is the cart flow. this encapsulates the following business functions create a new cart add items to the cart check out the cart expose an output, "cart" data that can be linked to other flows accept an input, "cart item" that can be linked to the output from other flows we can connect the output "product" data from "product listing" flow to the "cart item" input of the cart flow. this is not tightly coupled at compile time and is left as an open gate or an input for the cart flow. at runtime the gate is closed or left open as required by the customer. from the diagram we see that we have 5 gates that can be closed. leading us to create different applications at runtime by closing the correct gates. for eg., we can just enable "product listing" for a customer which provides just the basic features of product listing. gates 4 and 6 can be closed to create an application with the features create, list products add products to a cart send an enquiry for the products added to the cart in another variation instead of gates 4 and 6, gate 7 can be closed. the application now has the features create, list products send an enquiry for single products by plugging the "inputs" of flows with different "outputs" from other flows, varied applications can be created. this is called a plug and play architecture. smart is implemented in this architecture to provide a highly flexible container to build saas products. need for plug and play architecture when a product is exposed as a multi-tenanted saas product, it cannot be created with "one size fits all" tenet. to break this tenet we need a flexible, customizable runtime environment where features can be varied based on a customer requirement without affecting other customers serviced by the same application. the plug and play architecture provides this flexibility to the saas.

In this post I’m going to explain you about Linq AddRange method. This method is quite useful when you want to add multiple elements to a end of list. Following is a method signature for this. public void AddRange( IEnumerable collection ) To Understand let’s take simple example like following. using System; using System.Collections.Generic; namespace Linq { class Program { static void Main(string[] args) { List names=new List {"Jalpesh"}; string[] newnames=new string[]{"Vishal","Tushar","Vikas","Himanshu"}; foreach (var newname in newnames) { names.Add(newname); } foreach (var n in names) { Console.WriteLine(n); } } } } Here in the above code I am adding content of array to a already created list via foreach loop. You can use AddRange method instead of for loop like following.It will same output as above. using System; using System.Collections.Generic; namespace Linq { class Program { static void Main(string[] args) { List names=new List {"Jalpesh"}; string[] newnames=new string[]{"Vishal","Tushar","Vikas","Himanshu"}; names.AddRange(newnames); foreach (var n in names) { Console.WriteLine(n); } } } } Now when you run that example output is like following. Add Range in more complex scenario: You can also use add range to more complex scenarios also like following.You can use other operator with add range as following. using System; using System.Collections.Generic; using System.Linq; namespace Linq { class Program { static void Main(string[] args) { List names=new List {"Jalpesh"}; string[] newnames=new string[]{"Vishal","Tushar","Vikas","Himanshu"}; names.AddRange(newnames.Where(nn=>nn.StartsWith("Vi"))); foreach (var n in names) { Console.WriteLine(n); } } } } Here in the above code I have created array with string and filter it with where operator while adding it to an existing list. Following is output as expected. That’s it. Hope you like it. Stay tuned for more..

When you think of a Doctrine 2 DBAL Type you think of an atomic thing, but how can you work programmatically on this type without defining an event? A DBAL Type doesn't allow access to the Symfony 2 service container, you must use a hack. But before this let me explain the classic way (using events), why you should use this hack and why you shouldn't. The classic way is defined in the Symfony 2 Cookbook: How to Register Event Listeners and Subscribers Doctrine 2 events unlike Symfony 2 events aren't defined by the developer, the developer can only attach listeners on them. Why? Because Doctrine 2 isn't a framework that you can use for everything, persistence is its only job. When should you use this hack? When your stored object isn't a 1:1 representation of the PHP object and its elaboration can be memoizable or really fast. I use this hack for browscaps: with the BrowscapBundle I can convert from an user agent string to a stdClass object (like the get_browser function). Our object is container = $container; } public function prePersist(LifecycleEventArgs $args) { $this->doObjectToString($args); } public function postPersist(LifecycleEventArgs $args) { $this->doStringToObject($args); } public function preUpdate(LifecycleEventArgs $args) { $this->doObjectToString($args); } public function postUpdate(LifecycleEventArgs $args) { $this->doStringToObject($args); } public function postLoad(LifecycleEventArgs $args) { $this->doStringToObject($args); } private function doStringToObject($args) { $entity = $args->getEntity(); if ($entity instanceof Agent && !is_object($entity->getHeader())) { $browscap = $this->container->get('browscap'); $browser = $browscap->getBrowser($entity->getHeader()); $entity->setHeader($browser); } } private function doObjectToString($args) { $entity = $args->getEntity(); if ($entity instanceof Agent && is_object($entity->getHeader())) { $user_agent = $entity->getHeader()->browser_name; $entity->setHeader($user_agent); } } } With this code, everytime you will persist, update or extract a Agent entity from/to related storage system it'll be converted from string to object. The problem is that these callbacks will be invoked everytime and numerous events aren't recommended for your application. But with this hack I can write: services: acme.demo_bundle.event_listener.container_listener: arguments: - "@service_container" class: "Acme\DemoBundle\EventListener\ContainerListener" tags: - { name: doctrine.event_listener, event: getContainer } Doctrine ignores this event but it exists and results attached! container = $container; } public function getContainer() { return $this->container; } } This listener seems useless, but it's the only way for this hack because Doctrine 2 DBAL Type doesn't allow direct access to the service container but allows access to events listeners. getVarcharTypeDeclarationSQL($fieldDeclaration); } public function convertToPHPValue($value, AbstractPlatform $platform) { if (is_null($value)) { return null; } $listeners = $platform->getEventManager()->getListeners('getContainer'); $listener = array_shift($listeners); $container = $listener->getContainer(); return $container->get('browscap')->getBrowser($value); } public function convertToDatabaseValue($value, AbstractPlatform $platform) { if ($value instanceof Browscap) { return $value->getBrowser()->browser_name; } elseif ($value instanceof stdClass) { return $value->browser_name; } return $value; } public function getName() { return 'browscap'; } public function requiresSQLCommentHint(AbstractPlatform $platform) { return true; } } I use this hack to define only the events related to application flow (less events is better). Now that you know when you can use this, you must read why you shouldn't use it. Let me explain the reason with one simple example: imagine that one day PHP will allow external hooks in native classes constructor, how can you work without knowing what you're doing while initializing a new stdClass? The same reason here: everytime you extract a value from the database you want extract it fast (hopefully you'll extract more than one records), but how can you be sure that extraction is fast if every attribute of a single record depends on external libraries and logics? Quoting Ocramius, member of the Doctrine 2 development team: DBAL types are not designed for Dependency Injection. We explicitly avoided using DI for DBAL types because they have to stay simple. We’ve been asked many many times to change this behaviour, but doctrine believes that complex data manipulation should NOT happen within the very core of the persistence layer itself. That should be handled in your service layer.

Transaction Script (TS) is the simplest domain logic pattern we can find. It needs less work to implement than other domain logic patterns and therefore it’s perfect fit for smaller applications that doesn’t need big architecture behind them. This posting focuses on TS pattern by illustrating and analysing it. Also some good hints for implementation are provided. Definition Transaction Script organizes business logic by procedures where each procedure handles a single request from the presentation. pg. 110, Patterns of Enterprise Applications Architecture by Martin Fowler Example Imagine application that provides some functionalities: load orders to deliver, generate invoices, calculate bonuses for customers. These functionalities can be handled as two transactions: generate invoices, calculate bonuses. We cań create classes to handle invoice generating and bonus calculation but we can also go with one class that provides both methods. It’s up to you to decide which is better for you. class InvoiceGenerator { public void GenerateInvoices() { load orders to deliver iterate through orders generate invoice mark them ready to deliver } } This example uses one class for all operations. If you have many operations then use multiple classes to organize code better. Where I use TS? I have used TS mainly in small applications where I really don’t want to apply any bigger architecture. TS will save you a lot of time when applied in correct context. Some applications where I used TS are: Estonian ID-card based door lock control – Windows service and simple Windows Forms applications that allow to manage users, give permissions for opening doors and see who and when opened the door. It may sound like big application but the actual complexity lies in hardware and the software I wrote is actually very simple. Jira importer – my company uses small service that imports work logs and tasks from Jira to our main database where programming work, sales and billing meet. Getting data from Jira can be tricky but still this service is simple enough to avoid more complex patterns. On both cases I got simple classes that provide some functionalities needed by applications and it took less time than implementing full domain model by example. Tips TS works well for small applications that doesn’t implement any complex logic, When used in bigger applications TS has problems like code and functionalities duplication between procedures. If you see application growing then switch to some other domain logic pattern that is better fit for your application. It’s not easy usually to refactor TS to other domain logic patterns as other patterns deal more or less with classes that represent some business entity. In TS you can use different data source patterns to access and manipulate data. In my door lock control system I used ADO.NET objects. My Jira importer uses simple Entity Framework classes. When building your TS classes you can also apply design patterns to organize code better. Don’t let your TS classes grow to big bad spaghetti code. Don’t make TS classes dependent on presentation layer classes because dependencies like these makes it harder to share those classes between applications and also testing of TS classes will be very hard this way. Wrapping up TS is simple and lightweight domain logic pattern that is perfect fit for smaller applications that do nothing complex. It is possible to gather all operations together to one class but if you have many operations then it’s better to use more than one class for operations. If you see your application growing bigger and more complex then switch quickly to some other domain logic pattern that fits your needs better. Related Posts ASP.NET MVC: Moving code from controller action to service layer Patterns of Enterprise Application Architecture Analysis Patterns: Reusable Object Models Some points about services layer Writing cache based repositories for web application prototyping The post Transaction Script Pattern appeared first on Gunnar Peipman - Programming Blog.

Publisher confirms are a RabbitMQ addition to AMQP to guarantee message delivery. You can read all about them here and here. In short they provide a asynchronous confirmation that a publish has successfully reached all the queues that it was routed to. To turn on publisher confirms with EasyNetQ set the publisherConfirms connection string parameter like this: var bus = RabbitHutch.CreateBus("host=localhost;publisherConfirms=true"); When you set this flag, EasyNetQ will wait for the confirmation, or a timeout, before returning from the Publish method: bus.Publish(new MyMessage { Text = "Hello World!" }); // here the publish has been confirmed. Nice and easy. There’s a problem though. If I run the above code in a while loop without publisher confirms, I can publish around 4000 messages per second, but with publisher confirms switched on that drops to around 140 per second. Not so good. With EasyNetQ 0.15 we introduced a new PublishAsync method that returns a Task. The Task completes when the publish is confirmed: bus.PublishAsync(message).ContinueWith(task => { if (task.IsCompleted) { Console.WriteLine("Publish completed fine."); } if (task.IsFaulted) { Console.WriteLine(task.Exception); } }); Using this code in a while loop gets us back to 4000 messages per second with publisher confirms on. Happy confirms!

One piece to Docker that is interesting AMAZING is the Remote API that can be used to programatically interact with docker. I recently had a situation where I wanted to run many containers on a host with a single container managing the other containers through the API. But the problem I soon discovered is that at the moment when you turn networking on it is an all or nothing type of thing… you can’t turn networking off selectively on a container by container basis. You can disable IPv4 forwarding, but you can still reach the docker remote API on the machine if you can guess the IP address of it. One solution I came up with for this is to use nginx to expose the unix socket for docker over HTTPS and utilize client-side ssl certificates to only allow trusted containers to have access. I liked this setup a lot so I thought I would share how it’s done. Disclaimer: assumes some knowledge of docker! Generate The SSL Certificates We’ll use openssl to generate and self-sign the certs. Since this is for an internal service we’ll just sign it ourselves. We also remove the password from the keys so that we aren’t prompted for it each time we start nginx. # Create the CA Key and Certificate for signing Client Certs openssl genrsa -des3 -out ca.key 4096 openssl rsa -in ca.key -out ca.key # remove password! openssl req -new -x509 -days 365 -key ca.key -out ca.crt # Create the Server Key, CSR, and Certificate openssl genrsa -des3 -out server.key 1024 openssl rsa -in server.key -out server.key # remove password! openssl req -new -key server.key -out server.csr # We're self signing our own server cert here. This is a no-no in production. openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt # Create the Client Key and CSR openssl genrsa -des3 -out client.key 1024 openssl rsa -in client.key -out client.key # no password! openssl req -new -key client.key -out client.csr # Sign the client certificate with our CA cert. Unlike signing our own server cert, this is what we want to do. openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt Another option may be to leave the passphrase in and provide it as an environment variable when running a docker container or through some other means as an extra layer of security. We’ll move ca.crt, server.key and server.crt to /etc/nginx/certs. Setup Nginx The nginx setup for this is pretty straightforward. We just listen for traffic on localhost on port 4242. We require client-side ssl certificate validation and reference the certificates we generated in the previous step. And most important of all, set up an upstream proxy to the docker unix socket. I simply overwrote what was already in /etc/nginx/sites-enabled/default. upstream docker { server unix:/var/run/docker.sock fail_timeout=0; } server { listen 4242; server localhost; ssl on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; ssl_client_certificate /etc/nginx/certs/ca.crt; ssl_verify_client on; access_log on; error_log /dev/null; location / { proxy_pass http://docker; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 120; proxy_read_timeout 120; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; } } One important piece to make this work is you should add the user nginx runs as to the docker group so that it can read from the socket. This could be www-data, nginx, or something else! Hack It Up! With this setup and nginx restarted, let’s first run a curl command to make sure that this setup correctly. First we’ll make a call without the client cert to double check that we get denied access then a proper one. # Is normal http traffic denied? curl -v http://localhost:4242/info # How about https, sans client cert and key? curl -v -s -k https://localhost:4242/info # And the final good request! curl -v -s -k --key client.key --cert client.crt https://localhost:4242/info For the first two we should get some run of the mill 400 http response codes before we get a proper JSON response from the final command! Woot! But wait there’s more… let’s build a container that can call the service to launch other containers! For this example we’ll simply build two containers: one that has the client certificate and key and one that doesn’t. The code for these examples are pretty straightforward and to save space I’ll leave the untrusted container out. You can view the untrusted container on github (although it is nothing exciting). First, the node.js application that will connect and display information: https = require 'https' fs = require 'fs' options = host: 172.42.1.62 port: 4242 method: 'GET' path: '/containers/json' key: fs.readFileSync('ssl/client.key') cert: fs.readFileSync('ssl/client.crt') headers: { 'Accept': 'application/json'} # not required, but being semantic here! req = https.request options, (res) -> console.log res req.end() And the Dockerfile used to build the container. Notice we add the client.crt and client.key as part of building it! FROM shykes/nodejs MAINTAINER James R. Carr ADD ssl/client* /srv/app/ssl ADD package.json /srv/app/package.json ADD app.coffee /srv/app/app.coffee RUN cd /srv/app && npm install . CMD cd /srv/app && npm start That’s about it. Run docker build . and docker run -n >IMAGE ID< and we should see a json dump to the console of the actively running containers. Doing the same in the untrusted directory should present us with some 400 error about not providing a client ssl certificate. I’ve shared a project with all this code plus a vagrant file on github for your own prusual. Enjoy!

Since git hooks can be any executable script with an appropriate #! line, Python is more than suitable for writing your git hooks. Simply stated, git hooks are scripts which are called at different points of time in the life cycle of working with your git repository. Let’s start by creating a new git repository: ~/work> git init git-hooks-exp Initialized empty Git repository in /home/gene/work/git-hooks-exp/.git/ ~/work> cd git-hooks-exp/ ~/work/git-hooks-exp (master)> tree -al .git/ .git/ ├── branches ├── config ├── description ├── HEAD ├── hooks │ ├── applypatch-msg.sample │ ├── commit-msg.sample │ ├── post-update.sample │ ├── pre-applypatch.sample │ ├── pre-commit.sample │ ├── prepare-commit-msg.sample │ ├── pre-rebase.sample │ └── update.sample ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags 9 directories, 12 files Inside the .git are a number of directories and files, one of them being hooks/ which is where the hooks live. By default, you will have a number of hooks with the file names ending in .sample. They may be useful as starting points for your own scripts. However, since they all have an extension .sample, none of the hooks are actually activated. For a hook to be activated, it must have the right file name and it should be executable. Let’s see how we can write a hook using Python. We will write a post-commit hook. This hook is called immediately after you have made a commit. We are going to do something fairly useless, but quite interesting in this hook. We will take the commit SHA1 of this commit, and print how it may look like in a more human form. I do the latter using the humanhash module. You will need to have it installed. Here is how the hook looks like: #!/usr/bin/python import subprocess import humanhash # get the last commit SHA and print it after humanizing it # https://github.com/zacharyvoase/humanhash print humanhash.humanize( subprocess.check_output( ['git','rev-parse','HEAD'])) I use the subprocess.check_output() function to execute the command git rev-parse HEAD so that I can get the commit SHA1 and then call the humanhash.humanize() function with it. Save the hook as a file, post-commit in your hooks/ directory and make it executable using chmod +x .git/hooks/post-commit. Let’s see the hook in action: ~/work/git-hooks-exp (master)> touch file ~/work/git-hooks-exp (master)> git add file ~/work/git-hooks-exp (master)> git commit -m "Added a file" carbon-network-connecticut-equal [master (root-commit) 2d7880b] Added a file 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 file The commit SHA1 for the commit turned out to be 2d7880be746a1c1e75844fc1aa161e2b8d955427. Let’s check it with the humanize function and check if we get the same message as above: >>> humanhash.humanize('2d7880be746a1c1e75844fc1aa161e2b8d955427') 'carbon-network-connecticut-equal' And you can see the same message above as well. For some of the hooks, you will see that they are called with some parameters. In Python you can access them using the sys.argv attribute from the sys module, with the first member being the name of the hook of course and the others will be the parameters that the hook is called with.

Because Agile development teams work from a backlog of stories, one way to inject application security into software development is by writing up application security risks and activities as stories, making them explicit and adding them to the backlog so that application security work can be managed, estimated, prioritized and done like everything else that the team has to do. Security Stories SAFECode has tried to do this by writing a set of common, non-functional Security Stories following the well-known “As a [type of user] I want {something} so that {reason}” template. These stories are not customer- or user-focused: not the kind that a Product Owner would understand or care about. Instead, they are meant for the development team (architects, developers and testers). Example: As a(n) architect/developer, I want to ensure AND as QA, I want to verify that sensitive data is kept restricted to actors authorized to access it. There are stories to prevent/check for the common security vulnerabilities in applications: XSS, path traversal, remote execution, CSRF, OS command injection, SQL injection, password brute forcing. Checks for information exposure through error messages, proper use of encryption, authentication and session management, transport layer security, restricted uploads and URL redirection to un-trusted sites; and basic code quality issues: NULL pointer checking, boundary checking, numeric conversion, initialization, thread/process synchronization, exception handling, use of unsafe/restricted functions. SAFECode also includes a list of secure development practices (operational tasks) for the team that includes making sure that you’re using the latest compiler, patching the run-time and libraries, static analysis, vulnerability scanning, code reviews of high-risk code, tracking and fixing security bugs; and more advanced practices that require help from security experts like fuzzing, threat modeling, pen tests, environmental hardening. Altogether this is a good list of problems that need to be watched out for and things that should be done on most projects. But although SAFECode’s stories look like stories, they can’t be used as stories by the team. These Security Stories are non-functional requirements (NFRs) and technical constraints that (like requirements for scalability and maintainability and supportability) need to be considered in the design of the system, and may need to be included as part of the definition of done and conditions of acceptance for every user story that the team works on. Security Stories can’t be pulled from the backlog and delivered like other stories and removed from the backlog when they are done, because they are never “done”. The team has to keep worrying about them throughout the life of the project and of the system. As Rohit Sethi points out, asking developers to juggle long lists of technical constraints like this is not practical: If you start adding in other NFR constraints, such as accessibility, the list of constraints can quickly grow overwhelming to developers. Once the list grows unwieldy, our experience is that developers tend to ignore the list entirely. They instead rely on their own memories to apply NFR constraints. Since the number of NFRs continues to grow in increasingly specialized domains such as application security, the cognitive burden on developers’ memories is substantial. OWASP Evil User Stories – Hacking the Backlog Someone at OWASP has suggested an alternative, much smaller set of non-functional Evil User Stories that can be "hacked" into the backlog: A way for a security guy to get security on the agenda of the development team is by “hacking the backlog”. The way to do this is by crafting Evil User Stories, a few general negative cases that the team needs to consider when they implement other stories. Example #1. "As a hacker, I can send bad data in URLs, so I can access data and functions for which I'm not authorized." Example #2. "As a hacker, I can send bad data in the content of requests, so I can access data and functions for which I'm not authorized." Example #3. "As a hacker, I can send bad data in HTTP headers, so I can access data and functions for which I'm not authorized." Example #4. "As a hacker, I can read and even modify all data that is input and output by your application." Thinking like a Bad Guy – Abuse Cases and Abuser Stories Another way to beef up security in software development is to get the team to carefully look at the system they are building from the bad guy's perspective. In “Misuse and Abuse Cases: Getting Past the Positive”, Dr. Gary McGraw at Cigital talks about the importance of anticipating things going wrong, and thinking about behaviour that the system needs to prevent. Assume that the customer/user is not going to behave, or is actively out to attack the application. Question all of the assumptions in the design (the can’ts and won’ts), especially trust conditions – what if the bad guy can be anywhere along the path of an action (for example, using an attack proxy between the client and the server)? Abuse Cases are created by security experts working with the team as part of a critical review – either of the design or of an existing application. The goal of a review like this is to understand how the system behaves under attack/failure conditions, and document any weaknesses or gaps that need to be addressed. At Agile 2013 Judy Neher presented a hands-on workshop on how to write Abuser Stories, a lighter-weight, Agile practice which makes “thinking like a bad guy” part of the team’s job of defining and refining user requirements. Take a story, and as part of elaborating the story and listing the scenarios, step back and look at the story through a security lens. Don’t just think of what the user wants to do and can do - think about what they don’t want to do and can’t do. Get the same people who are working on the story to “put their black hats on” and think evil for a little while, brainstorm to come up with negative cases. As {some kind of bad guy} I want to {do some bad thing}… The {bad guy} doesn’t have to be a hacker. They could be an insider with a grudge or a selfish customer who is willing to take advantage of other users, or an admin user who needs to be protected from making expensive mistakes, or an external system that may not always function correctly. Ask questions like: How do I know who the user is and that I can trust them? Who is allowed to do what, and where are the authorization checks applied? Look for holes in multi-step workflows – what happens if somebody bypasses a check or tries to skip a step or do something out of sequence? What happens if an action or a check times-out or blocks or fails – what access should be allowed, what kind of information should be shown, what kind shouldn’t be? Are we interacting with children? Are we dealing with money? With dangerous command-and-control/admin functions? With confidential or pirvate data? Look closer at the data. Where is it coming from? Can I trust it? Is the source authenticated? Where is it validated – do I have to check it myself? Where is it stored (does it have to be stored)? If it has to be stored, should it be encrypted or masked (including in log files)? Who should be able to see it? Who shouldn’t be able to see it? Who can change it, and to the changes need to be audited? Do we need to make sure the data hasn't been tampered with (checksum, HMAC, digital signature)? Use this exercise to come up with refutation criteria (user can do this, but can’t do that; they can see this but they can’t see that), instead of, or as part of the conditions of acceptance for the story. Prioritize these cases based on risk, add the cases that you agree need to be taken care of as scenarios to the current story, or as new stories to the backlog if they are big enough. “Thinking like a bad guy” as you are working on a story seems more useful and practical than other story-based approaches. It doesn’t take a lot of time, and it’s not expensive. You don’t need to write Abuser Stories for every user Story and the more Abuser Stories that you do, the easier it will get – you'll get better at it, and you’ll keep running into the same kinds of problems that can be solved with the same patterns. You end up with something concrete and functional and actionable, work that has to be done and can be tested. Concrete, actionable cases like this are easier for the team to understand and appreciate – including the Product Owner, which is critical in Scrum, because the Product Owner decides what is important and what gets done. And because Abuser Stories are done in phase, by the people who are working on the stories already (rather than a separate activity that needs to be setup and scheduled) they are more likely to get done. Simple, quick, informal threat modeling like this isn’t enough to make a system secure – the team won’t be able to find and plug all of the security holes in the system this way, even if the developers are well-trained in secure software development and take their work seriously. Abuser Stories are good for identifying business logic vulnerabilities, reviewing security features (authentication, access control, auditing, password management, licensing) improving error handling and basic validation, and keeping onside of privacy regulations. Effective software security involves a lot more work than this: choosing a good framework and using it properly, watching out for changes to the system's attack surface, carefully reviewing high-risk code for design and coding errors, writing good defensive code as much as possible, using static analysis to catch common coding mistakes, and regular security testing (pen testing and dynamic analysis). But getting developers and testers to think like a bad guy as they build a system should go a long way to improving the security and robustness of your app.

If you are programming against a web service, the natural pattern is request-response. It’s always initiated by the client, which then waits for a response from the server. It’s great if the client wants to send some information to a server, or request some information based on some criteria. It’s not so useful if the server wants to initiate the send of some information to the client. There we have to rely on somewhat extended HTTP tricks like long-polling or web-hooks. With messaging systems, the natural pattern is send-receive. A producer node publishes a message which is then passed to a consuming node. There is no real concept of client or server; a node can be a producer, a consumer, or both. This works very well when one node wants to send some information to another or vice-versa, but isn’t so useful if one node wants to request information from another based on some criteria. All is not lost though. We can model request-response by having the client node create a reply queue for the response to a query message it sends to the server. The client can set the request message properties’ reply_to field with the reply queue name. The server inspects the reply_to field and publishes the reply to the reply queue via the default exchange, which is then consumed by the client. The implementation is simple on the request side; it looks just like a standard send-receive. But on the reply side, we have some choices to make. If you Google for ‘RabbitMQ RPC’, or ‘RabbitMQ request response’, you will find several different opinions concerning the nature of the reply queue. Should there be a reply queue per request, or should the client maintain a single reply queue for multiple requests? Should the reply queue be exclusive, only available to this channel, or not? Note that an exclusive queue will be deleted when the channel is closed, either intentionally, or if there is a network or broker failure that causes the connection to be lost. Let’s have a look at the pros and cons of these choices. Exclusive Reply Queue Per Request. Here each request creates a reply queue. The benefits are that it is simple to implement. There is no problem with correlating the response with the request, since each request has its own response consumer. If the connection between the client and the broker fails before a response is received, the broker will dispose of any remaining reply queues and the response message will be lost. The main implementation issue is that we need to clean up any replies queues in the event that a problem with the server means that it never publishes the response. This pattern has a performance cost because a new queue and consumer has to be created for each request. Exclusive Reply Queue Per Client Here each client connection maintains a reply queue which many requests can share. This avoids the performance cost of creating a queue and consumer per request, but adds the overhead that the client needs to keep track of the reply queue and match up responses with their respective requests. The standard way of doing this is with a correlation id that is copied by the server from the request to the response. Once again, there is no problem with deleting the reply queue when the client disconnects because the broker will do this automatically. It does mean that any responses that are in-flight at the time of a disconnection will be lost. Durable Reply Queue Both the options above have the problem that the response message can be lost if the connection between the client and broker goes down while the response is in flight. This is because they use exclusive queues that are deleted by the broker when the connection that owns them is closed. The natural answer to this is to use a non-exclusive reply queue. However this creates some management overhead. You need some way to name the reply queue and associate it with a particular client. The problem is that it’s difficult for the client to know if any one reply queue belongs to itself, or to another instance. It’s easy to naively create a situation where responses are being delivered to the wrong instance of the client. You will probably wind up manually creating and naming response queues, which removes one of the main benefits of choosing broker based messaging in the first place. EasyNetQ For a high-level re-useable library like EasyNetQ the durable reply queue option is out of the question. There is no sensible way of knowing whether a particular instance of the library belongs to a single logical instance of a client application. By ‘logical instance’ I mean an instance that might have been stopped and started, as opposed to two separate instances of the same client. Instead we have to use exclusive queues and accept the occasional loss of response messages. It is essential to implement a timeout, so that an exception can be raised to the client application in the event of response loss. Ideally the client will catch the exception and re-try the message if appropriate. Currently EasyNetQ implements the ‘reply queue per request’ pattern, but I’m planning to change it to a ‘reply queue per client’. The overhead of matching up responses to requests is not too onerous, and it is both more efficient and easier to manage. I’d be very interested in hearing other people’s experiences in implementing request-response with RabbitMQ.