scalability has become one of those core concept slash buzzwords of big data. it’s all about scaling out, web scale, and so on. in principle, the idea is to be able to take one piece of code and then throw any number of computers at it to make it fast. the terms “scalable” and “large scale” have been used in machine learning circles long before there was big data. there had always been certain problems which lead to a large amount of data, for example in bioinformatics, or when dealing with large number of text documents. so finding learning algorithms, or more generally data analysis algorithms which can deal with a very large set of data was always a relevant question. interestingly, this issue of scalability were seldom solved using actual scaling in in machine learning, at least not in the big data kind of sense. part of the reason is certainly that multicore processors didn’t yet exist at the scale they do today and that the idea of “just scaling out” wasn’t as pervasive as it is today. instead, “scalable” machine learning is almost always based on finding more efficient algorithms, and most often, approximations to the original algorithm which can be computed much more efficiently. to illustrate this, let’s search for nips papers (the annual advances in neural information processing systems, short nips, conference is one of the big ml community meetings) for papers which have the term “scalable” in the title. here are some examples: scalable inference for logistic-normal topic models … this paper presents a partially collapsed gibbs sampling algorithm that approaches the provably correct distribution by exploring the ideas of data augmentation … partially collapsed gibbs sampling is a kind of estimation algorithm for certain graphical models. a scalable approach to probabilistic latent space inference of large-scale networks … with […] an efficient stochastic variational inference algorithm, we are able to analyze real networks with over a million vertices […] on a single machine in a matter of hours … stochastic variational inference algorithm is both an approximation and an estimation algorithm. scalable kernels for graphs with continuous attributes … in this paper, we present a class of path kernels with computational complexity $o(n^2(m + \delta^2 ))$ … and this algorithm has squared runtime in the number of data points, so wouldn’t even scale out well even if you could. usually, even if there is potential for scalability, it usually something that is “embarassingly parallel” (yep, that’s a technical term), meaning that it’s something like a summation which can be parallelized very easily. still, the actual “scalability” comes from the algorithmic side. so how do scalable ml algorithms look like? a typical example are the stochastic gradient descent (sgd) class of algorithms. these algorithms can be used, for example, to train classifiers like linear svms or logistic regression. one data point is considered at each iteration. the prediction error on that point is computed and then the gradient is taken with respect to the model parameters, giving information about how to adapt these parameters slightly to make the error smaller. vowpal wabbit is one program based on this approach and it has a nice definition of what it considers to mean scalable in machine learning: there are two ways to have a fast learning algorithm: (a) start with a slow algorithm and speed it up, or (b) build an intrinsically fast learning algorithm. this project is about approach (b), and it’s reached a state where it may be useful to others as a platform for research and experimentation. so “scalable” means having a learning algorithm which can deal with any amount of data, without consuming ever growing amounts of resources like memory. for sgd type algorithms this is the case, because all you need to store are the model parameters, usually a few ten to hundred thousand double precision floating point value, so maybe a few megabytes in total. the main problem to speed this kind of computation up is how to stream the data by fast enough. to put it differently, not only does this kind of scalability not rely on scaling out, it’s actually not even necessary or possible to scale the computation out because the main state of the computation easily fits into main memory and computations on it cannot be distributed easily. i know that gradient descent is often taken as an example for map reduce and other approaches like in this paper on the architecture of spark , but that paper discusses a version of gradient descent where you are not taking one point at a time, but aggregate the gradient information for the whole data set before making the update to the model parameters. while this can be easily parallelized, it does not perform well in practice because the gradient information tends to average out when computed over the whole data set. if you want to know more, this large scale learning challenge sören sonnneburg organized in 2008 still has valuable information on how to deal with massive data sets. of course, there are things which can be easily scaled well using hadoop or spark, in particular any kind of data preprocessing or feature extraction where you need to apply the same operation to each data point in your data set. another area where parallelization is easy and useful is when you are using cross validation to do model selection where you usually have to train a large number of models for different parameter sets to find the combination which performs best. again, even here there is more potential for even speeding up such computations using better algorithms like in this paper of mine . i’ve just scratched the surface of this, but i hope you got the idea that scalability can mean quite different things. in big data (meaning the infrastructure side of it) what you want to compute is pretty well defined, for example some kind of aggregate over your data set, so you’re left with the question of how to parallelize that computation well. in machine learning, you have much more freedom because data is noisy and there’s always some freedom in how you model your data, so you can often get away with computing some variation of what you originally wanted to do and still perform well. often, this allows you to speed up your computations significantly by decoupling computations. parallelization is important, too, but alone it won’t get you very far. luckily, there are projects like spark and stratosphere/flink which work on providing more useful abstractions beyond map and reduce to make the last part easier for data scientists, but you won’t get rid of the algorithmic design part any time soon.

Some languages including C, C++ support pointers. Other languages including C++, Java, Python, Ruby, Perl and PHP all support references. On the surface both references and pointers are very similar, both are used to have one variable provide access to another. With both providing a lot of the same capabilities, it’s often unclear what is different between these different mechanisms. In this article I will illustrate the difference between pointers and references. Why Does This Matter Pointers are at the very core of effective Go. Most programmers are learning Go with a foundation in one of the languages mentioned above. Consequently understanding the difference between pointers and references is critical to understanding Go. Even if you are coming from a language that uses pointers, Go’s implementation of pointers differs from C and C++ in that it retains some of the nice properties of references while retaining the power of pointers. The remainder of this article is written with the intent of speaking broadly about the concept of references rather than about a specific implementation. We will be using Go as the reference implementation for pointers. What Is The Difference? A pointer is a variable which stores the address of another variable. A reference is a variable which refers to another variable. To illustrate our point, use the following example in C++ which supports both pointers and references. int i = 3; int *ptr = &i; int &ref = i; The first line simply defines a variable. The second defines a pointer to that variable’s memory address. The third defines a reference to the first variable. Not only are the operators different, but you use the differently as well. With pointers must use the * operator to dereference it. With a reference no operator is required. It is understood that you are intending to work with the referred variable. Continuing with our example, the following two lines will both change the value of i to 13. *ptr = 13; ref = 13; You may be asking, what happens if I try to access the ptr directly without dereferencing first. This takes us to our second critical difference between pointers and references. Pointers can be reassigned while references cannot. In other words, a pointer can be assigned to a different address. Consider the following example in Go: package main import "fmt" var ap *int func main() { a := 1 // define int b := 2 // define int ap = &a // set ap to address of a (&a) // ap address: 0x2101f1018 // ap value : 1 *ap = 3 // change the value at address &a to 3 // ap address: 0x2101f1018 // ap value : 3 a = 4 // change the value of a to 4 // ap address: 0x2101f1018 // ap value : 4 ap = &b // set ap to the address of b (&b) // ap address: 0x2101f1020 // ap value : 2 } So far you could do all of the above in a reasonably similar manner using references, and often with a simpler syntax. Stay with me, the following example will illustrate why pointers are more powerful than references. Extending the function above: ... ap2 := ap // set ap2 to the address in ap // ap address: 0x2101f1020 // ap value : 2 // ap2 address: 0x2101f1020 // ap2 value : 2 *ap = 5 // change the value at the address &b to 5 // ap address: 0x2101f1020 // ap value : 5 // ap2 address: 0x2101f1020 // ap2 value : 5 // If this was a reference ap & ap2 would now // have different values ap = &a // change ap to address of a (&a) // ap address: 0x2101f1018 // ap value : 4 // ap2 address: 0x2101f1020 // ap2 value : 5 // Since we've changed the address of ap, it now // has a different value then ap2 } You can experiment and play yourself at go play:http://play.golang.org/p/XJtdLxFoeO The key to understanding the difference is in the second example. If we were working with references we would not be able to change the value of b through *ap and have that reflected in *ap2. This is because once you make a copy of a reference they are now independent. While they may be referring to the same variable, when you manipulate the reference it will change what it refers to, rather than the referring value. The final example demonstrates the behavior when you change the assignment of one of the pointers to point to a new address. Due to the limitations of references this is the only operation available. Stay tuned… Next post will feature another property exclusively available to pointers, the pointer pointer. For more information on pointers I’ve found the following resources helpful understanding pointers and memory allocation Pointers in Go Pointers

In this article I would like to highlight the importance of governance in software development life cycle. What does 'Governance' means; in Oxford dictionary the meaning it has is: the action or manner of governing a state, organization, etc. rule, control In software development life cycle we can think of governance as monitor, measure and management of software. Now what aspect to governance in software depends on the stage at which software development is currently? In the following table I am trying to show the various software development stages, what entities need governance and tools/techniques used for governance: Development Stage What needs governance How do we (or tools/techniques used for) govern? Business Problem The source of problem statement (real business case vs. developer’s wish) Project roadmap, timeline and resource constraints Process exists to ensure right stakeholders are engaged, agreed upon project roadmap is defined, and resource pool, delivery timeline and risks are identified Requirement Gathering Use Case and Requirements Effort level and Risks Requirement document reviewed and signed off from all major stakeholders (business, development and testing teams) Architecture Alignment with corporate IT Non FRs like Performance, Security Architecture Review Design Design guidelines Design review Implementation Code Quality Feature implementation Code Review Code Coverage Feature Demo Testing Test Plan and Strategy Test Results and Coverage Score card on Non-FRs Review of Test Plan, Strategy, Coverage, and Results Review of Non-FRs score card or compliance report Documentation Documents Documentation Review Deployment Deployment Instructions Deployment document review Deployment team’s feedback on ease of deployment and 1st few weeks report from production Service / Maintenance Ease of troubleshooting Number of Customer found defects New feature request Tracking production defects and ensuring root cause analysis done on the same Assess and feed feature requests to next release’s project requirements Process to have regular feedback from service team’s In summary what is required for effective governance is a right process along with checklist, handoff templates, check points, retrospection meeting, open feedback culture and guidelines/standards. The software life cycle and architecture management frameworks like ITIL, TOGAF do help to implement an effective governance process in in software development life cycle.

In an earlier blog entry I had touched on a fictitious rube goldberg flow for capitalizing a string through a complicated series of steps, the premise of the article was to introduce Spring Integration Java DSL as an alternative to defining integration flows through xml configuration files. I learned a few new things after writing that blog entry, thanks to Artem Bilan and wanted to document those learnings here: So, first my original sample, here I have the following flow(the one's in bold): Take in a message of this type - "hello from spring integ" Split it up into individual words(hello, from, spring, integ) Send each word to a ActiveMQ queue Pick up the word fragments from the queue and capitalize each word Place the response back into a response queue Pick up the message, re-sequence based on the original sequence of the words Aggregate back into a sentence("HELLO FROM SPRING INTEG") and Return the sentence back to the calling application. EchoFlowOutbound.java: @Bean public DirectChannel sequenceChannel() { return new DirectChannel(); } @Bean public DirectChannel requestChannel() { return new DirectChannel(); } @Bean public IntegrationFlow toOutboundQueueFlow() { return IntegrationFlows.from(requestChannel()) .split(s -> s.applySequence(true).get().getT2().setDelimiters("\\s")) .handle(jmsOutboundGateway()) .get(); } @Bean public IntegrationFlow flowOnReturnOfMessage() { return IntegrationFlows.from(sequenceChannel()) .resequence() .aggregate(aggregate -> aggregate.outputProcessor(g -> Joiner.on(" ").join(g.getMessages() .stream() .map(m -> (String) m.getPayload()).collect(toList()))) , null) .get(); } @Bean public JmsOutboundGateway jmsOutboundGateway() { JmsOutboundGateway jmsOutboundGateway = new JmsOutboundGateway(); jmsOutboundGateway.setConnectionFactory(this.connectionFactory); jmsOutboundGateway.setRequestDestinationName("amq.outbound"); jmsOutboundGateway.setReplyChannel(sequenceChannel()); return jmsOutboundGateway; } It turns out, based on Artem Bilan's feedback, that a few things can be optimized here. First notice how I have explicitly defined two direct channels, "requestChannel" for starting the flow that takes in the string message and the "sequenceChannel" to handle the message once it returns back from the jms message queue, these can actually be totally removed and the flow made a little more concise this way: @Bean public IntegrationFlow toOutboundQueueFlow() { return IntegrationFlows.from("requestChannel") .split(s -> s.applySequence(true).get().getT2().setDelimiters("\\s")) .handle(jmsOutboundGateway()) .resequence() .aggregate(aggregate -> aggregate.outputProcessor(g -> Joiner.on(" ").join(g.getMessages() .stream() .map(m -> (String) m.getPayload()).collect(toList()))) , null) .get(); } @Bean public JmsOutboundGateway jmsOutboundGateway() { JmsOutboundGateway jmsOutboundGateway = new JmsOutboundGateway(); jmsOutboundGateway.setConnectionFactory(this.connectionFactory); jmsOutboundGateway.setRequestDestinationName("amq.outbound"); return jmsOutboundGateway; } "requestChannel" is now being implicitly created just by declaring a name for it. The sequence channel is more interesting, quoting Artem Bilan - do not specify outputChannel for AbstractReplyProducingMessageHandler and rely on DSL , what it means is that here jmsOutboundGateway is a AbstractReplyProducingMessageHandler and its reply channel is implicitly derived by the DSL. Further, two methods which were earlier handling the flows for sending out the message to the queue and then continuing once the message is back, is collapsed into one. And IMHO it does read a little better because of this change. The second good change and the topic of this article is the introduction of the Jms namespace factories, when I had written the previous blog article, DSL had support for defining the AMQ inbound/outbound adapter/gateway, now there is support for Jms based inbound/adapter adapter/gateways also, this simplifies the flow even further, the flow now looks like this: @Bean public IntegrationFlow toOutboundQueueFlow() { return IntegrationFlows.from("requestChannel") .split(s -> s.applySequence(true).get().getT2().setDelimiters("\\s")) .handle(Jms.outboundGateway(connectionFactory) .requestDestination("amq.outbound")) .resequence() .aggregate(aggregate -> aggregate.outputProcessor(g -> Joiner.on(" ").join(g.getMessages() .stream() .map(m -> (String) m.getPayload()).collect(toList()))) , null) .get(); } The inbound Jms part of the flow also simplifies to the following: @Bean public IntegrationFlow inboundFlow() { return IntegrationFlows.from(Jms.inboundGateway(connectionFactory) .destination("amq.outbound")) .transform((String s) -> s.toUpperCase()) .get(); } Thus, to conclude, Spring Integration Java DSL is an exciting new way to concisely configure Spring Integration flows. It is already very impressive in how it simplifies the readability of flows, the introduction of the Jms namespace factories takes it even further for JMS based flows.

Closely approaching the one year mark of when I first joined MongoLab (and the MongoDB community), I had the pleasure of attending the inaugural MongoDB World conference put together by the incredible MongoDB team. Second only to the excitement around major MongoDB feature announcements was the collective disbelief that this was MongoDB’s first multi-day conference ever. A big congratulations to all those that worked hard to put on such a massive (did you see the Intrepid!?) event. All this planning would have been for naught if MongoDB leaders and engineers failed to deliver announcements and features that would meet and exceed expectations. From major public cloud announcements to the reveal of document-level locking in version 2.8, developers and conference goers had plenty to be excited about. There was a lot to digest from the conference… we’ll cover the major highlights in case you missed them. Big announcements in public cloud Our time at the MongoLab booth yielded many high-quality conversations, predominantly those about offloading previously internal processes and workloads to the public cloud. It was remarkable to see and hear so many enterprise teams with the exact same message: the public cloud is the future, and the future is now. It’s no surprise then that MongoDB, Inc. released not one, but two press releases around MongoDB solutions for the public cloud. Fully-managed MongoDB on the Microsoft Azure Store Nearly one year ago, MongoDB, Inc. chose to partner with the MongoLab team to build a production-ready MongoDB solution for developers on Microsoft Azure. On the first day of World, MongoDB, Inc. announced the product of our collaboration – a fully-managed highly available MongoDB-as-a-Service Add-On offering on the Microsoft Azure Store. This new service runs MongoDB Enterprise and offers replication, monitoring and support from MongoDB, Inc. It’s also backed by MongoDB Management Service (MMS), allowing for point-in-time recovery of MongoDB deployments. Now, teams without the expertise or resources to manage their MongoDB deployment(s) can outsource all the database operations (monitoring and alerting, backups, performance tuning, etc.) to both MongoLab and MongoDB’s expert support teams. You can check out the MongoDB add-on in the Azure Store: https://azure.microsoft.com/en-us/gallery/store/mongodb/mongodb-inc/ MongoDB solutions on Google Cloud Platform MongoDB, Inc. also announced the arrival of new resources to help Google Cloud Platform customers deploy MongoDB on Google Compute Engine. These resources include a “Click to Deploy” feature and a MongoDB on Google Compute Engine Solutions paper covering MongoDB best practices. If you are looking for a fully-managed solution, with automated provisioning, backups, integrated monitoring and alerting, along with expert support, MongoLab recently announced the arrival of production-ready replica sets on Google. Product Roadmap – MongoDB version 2.8 On the second day of MongoDB World, Eliot Horowitz, MongoDB, Inc. CTO & Co-founder, took center stage and announced two huge changes to the MongoDB core project: document-level locking and pluggable storage engines. These features not only reflect improvements to the core project, but also signal to the community that the MongoDB team is listening to its users and is capable of delivering the software needed to power the workloads of tomorrow. Document-level locking The slides above from Eliot’s keynote point to a current obstacle (database-level locking) in MongoDB that limits overall scalability. With database-level locking, any write operation to the database holds the write lock and prevents subsequent writes from executing on the database until the original operation holding the write lock completes. Eliot’s announcement of document-level locking moves the write lock contention from the database level to the document (MongoDB equivalent to SQL “records”) level. This change will allow users to achieve much higher write throughput (we saw a 10x performance improvement in the live demo) across their MongoDB deployments, improving write scalability. If you’d like to try out document-level locking, the MongoDB team has already pushed the feature to the master branch on GitHub. This should only be used for experimentation, not to be run in production. Pluggable storage engine As MongoDB matures, feature releases like document level locking will continue to allow developers to build robust systems on top of MongoDB. But as the number of use cases grows, different tooling tailored to specific use cases may prove to be extremely beneficial. For example, if Company X decides that they want to use MongoDB to warehouse some of their data, they would likely want to optimize their database for slow-moving data and storage efficiency (compression). With the introduction of pluggable storage engines, many new possibilities are open to the community. Teams can now write their own storage engine for a particular use case, configure replica set nodes with different storage engines for specific situations, or collaborate with the open-source community to architect innovative solutions. This feature not only allows for more granular control of the database, but also encourages the MongoDB community to work together. Takeaways: A maturing and thriving ecosystem Roughly a year ago, MongoLab CTO Todd Dampier recapped MongoSF 2013 and spoke to the health of the MongoDB ecosystem. How far we’ve come! After attending the inaugural MongoDB World and chatting with MongoDB Masters, interns, hackathon winners, power users and those new to the community, the enthusiasm is still surging and as positive as ever. This enthusiasm is well placed. Developers and hackers use MongoDB because so much rich data on the web is shared as JSON (think Facebook, Twitter, Google, etc.). As a result, MongoDB is the de-facto database for hackathons and bootstrapped projects. Just learn the API for the site you want to mine, throw the JSON in MongoDB and query your data with the rich query language- it’s that easy. The MongoDB ecosystem is maturing as well. Take a look at the Customer Success Stories and you’ll get a feel for the extent in which enterprises leverage the solution and use it in production. To further drive enterprise adoption, MongoDB, Inc.’s public cloud solutions and product roadmap features aim to help teams run MongoDB in production and give teams the confidence that MongoDB will continue to improve scalability and meet their growing project requirements. Congratulations again to the MongoDB team on their big announcements and for creating such a fantastic forum at which to learn and meet fellow MongoDB users. Our team at MongoLab had a great time making new friends and talking shop; we look forward to meeting more MongoDB users soon (at a MongoDB Days near you)! -Chris@MongoLab

some diagrams for the spring petclinic application following on from my previous post ( software architecture as code ) where i demonstrated how to create a software architecture model as code, i decided to throw together a quick implementation of a spring component finder that could be used to (mostly) automatically create a model of a spring mvc web application. spring has a bunch of annotations (e.g. @controller, @component, @service and @repository) and these are often/can be used to signify the major building blocks of a web application. to illustrate this, i took the spring petclinic application and produced some diagrams for it. first is a context diagram. next up are the containers, which in this case are just a web server (e.g. apache tomcat) and a database (hsqldb by default). and finally we have a diagram showing the components that make up the web application. these, and their dependencies, were found by scanning the compiled version of the application (i cloned the project from github and ran the maven build). here is the code that i used to generate the model behind the diagrams. package com.structurizr.example; import com.fasterxml.jackson.databind.objectmapper; import com.fasterxml.jackson.databind.serializationfeature; import com.structurizr.componentfinder.componentfinder; import com.structurizr.componentfinder.springcomponentfinderstrategy; import com.structurizr.model.*; import com.structurizr.view.componentview; import com.structurizr.view.containerview; import com.structurizr.view.contextview; /** * this is a c4 representation of the spring petclinic sample app (https://github.com/spring-projects/spring-petclinic/). */ public class springpetclinic { public static void main(string[] args) throws exception { model model = new model(); // create the basic model (the stuff we can't get from the code) softwaresystem springpetclinic = model.addsoftwaresystem(location.internal, "spring petclinic", ""); person user = model.addperson(location.external, "user", ""); user.uses(springpetclinic, "uses"); container webapplication = springpetclinic.addcontainer("web application", "", "apache tomcat 7.x"); container relationaldatabase = springpetclinic.addcontainer("relational database", "", "hsqldb"); user.uses(webapplication, "uses"); webapplication.uses(relationaldatabase, "reads from and writes to"); // and now automatically find all spring @controller, @component, @service and @repository components componentfinder componentfinder = new componentfinder(webapplication, "org.springframework.samples.petclinic", new springcomponentfinderstrategy()); componentfinder.findcomponents(); // finally create some views contextview contextview = model.createcontextview(springpetclinic); contextview.addallsoftwaresystems(); contextview.addallpeople(); containerview containerview = model.createcontainerview(springpetclinic); containerview.addallpeople(); containerview.addallsoftwaresystems(); containerview.addallcontainers(); componentview componentview = model.createcomponentview(springpetclinic, webapplication); componentview.addallcomponents(); objectmapper objectmapper = new objectmapper(); objectmapper.enable(serializationfeature.indent_output); string modelasjson = objectmapper.writevalueasstring(model); system.out.println(modelasjson); } } the resulting json representing the model was then copy-pasted across into my simple (and very much in progress) diagramming tool . admittedly the diagrams are lacking on some details (i.e. component responsibilities and arrow annotations, although those can be fixed), but this approach proves you can expend very little effort to get something that is relatively useful. as i've said before, it's all about getting the abstractions right.

Earlier this month I wrote about vmstat iowait cpu numbers, and some of the comments I got were advertising the use of util% as reported by the iostat tool instead. I find this number even more useless for MySQL performance tuning and capacity planning. Now let me start by saying this is a really tricky and deceptive number. Many DBAs who report instances of their systems having a very busy IO subsystem said the util% in vmstat was above 99% and therefore they believe this number is a good indicator of an overloaded IO subsystem. Indeed – when your IO subsystem is busy, up to its full capacity, the utilization should be very close to 100%. However, it is perfectly possible for the IO subsystem and MySQL with it to have plenty more capacity than when utilization is showing 100% – as I will show in an example. Before that though lets see what the iostat manual page has to say on this topic – from this main page we can read: %util Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits. Which says right here that the number is useless for pretty much any production database server that is likely to be running RAID, Flash/SSD, SAN or cloud storage (such as EBS) capable of handling multiple requests in parallel. Let’s look at the following illustration. I will run sysbench on a system with a rather slow storage data size larger than buffer pool and uniform distribution to put pressure on the IO subsystem. I will use a read-only benchmark here as it keeps things more simple… sysbench –num-threads=1 –max-requests=0 –max-time=6000000 –report-interval=10 –test=oltp –oltp-read-only=on –db-driver=mysql –oltp-table-size=100000000 –oltp-dist-type=uniform –init-rng=on –mysql-user=root –mysql-password= run I’m seeing some 9 transactions per second, while disk utilization from iostat is at nearly 96%: [ 80s] threads: 1, tps: 9.30, reads/s: 130.20, writes/s: 0.00 response time: 171.82ms (95%) [ 90s] threads: 1, tps: 9.20, reads/s: 128.80, writes/s: 0.00 response time: 157.72ms (95%) [ 100s] threads: 1, tps: 9.00, reads/s: 126.00, writes/s: 0.00 response time: 215.38ms (95%) [ 110s] threads: 1, tps: 9.30, reads/s: 130.20, writes/s: 0.00 response time: 141.39ms (95%) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-0 0.00 0.00 127.90 0.70 4070.40 28.00 31.87 1.01 7.83 7.52 96.68 This makes a lot of sense – with read single thread read workload the drive should be only used getting data needed by the query, which will not be 100% as there is some extra time needed to process the query on the MySQL side as well as passing the result set back to sysbench. So 96% utilization; 9 transactions per second, this is a close to full-system capacity with less than 5% of device time to spare, right? Let’s run a benchmark with more concurrency – 4 threads at the time; we’ll see… [ 110s] threads: 4, tps: 21.10, reads/s: 295.40, writes/s: 0.00 response time: 312.09ms (95%) [ 120s] threads: 4, tps: 22.00, reads/s: 308.00, writes/s: 0.00 response time: 297.05ms (95%) [ 130s] threads: 4, tps: 22.40, reads/s: 313.60, writes/s: 0.00 response time: 335.34ms (95%) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-0 0.00 0.00 295.40 0.90 9372.80 35.20 31.75 4.06 13.69 3.38 100.01 So we’re seeing 100% utilization now, but what is interesting – we’re able to reclaim much more than less than 5% which was left if we look at utilization – throughput of the system increased about 2.5x Finally let’s do the test with 64 threads – this is more concurrency than exists at storage level which is conventional hard drives in RAID on this system… [ 70s] threads: 64, tps: 42.90, reads/s: 600.60, writes/s: 0.00 response time: 2516.43ms (95%) [ 80s] threads: 64, tps: 42.40, reads/s: 593.60, writes/s: 0.00 response time: 2613.15ms (95%) [ 90s] threads: 64, tps: 44.80, reads/s: 627.20, writes/s: 0.00 response time: 2404.49ms (95%) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util dm-0 0.00 0.00 601.20 0.80 19065.60 33.60 31.73 65.98 108.72 1.66 100.00 In this case we’re getting 4.5x of throughput compared to single thread and 100% utilization. We’re also getting almost double throughput of the run with 4 thread where 100% utilization was reported. This makes sense – there are 4 drives which can work in parallel and with many outstanding requests they are able to optimize their seeks better hence giving a bit more than 4x. So what have we so ? The system which was 96% capacity but which could have driven still to provide 4.5x throughput – so it had plenty of extra IO capacity. More powerful storage might have significantly more ability to run requests in parallel so it is quite possible to see 10x or more room after utilization% starts to be reported close to 100% So if utilization% is not very helpful what can we use to understand our database IO capacity better ? First lets look at the performance reported from those sysbench runs. If we look at 95% response time you can see 1 thread and 4 threads had relatively close 95% time growing just from 150ms to 250-300ms. This is the number I really like to look at- if system is able to respond to the queries with response time not significantly higher than it has with concurrency of 1 it is not overloaded. I like using 3x as multiplier – ie when 95% spikes to be more than 3x of the single concurrency the system might be getting to the overload. With 64 threads the 95% response time is 15-20x of the one we see with single thread so it is surely overloaded. Do we have anything reported by iostat which we can use in a similar way? It turns out we do! Check out the “await” column which tells us how much the requester had to wait for the IO request to be serviced. With single concurrency it is 7.8ms which is what this drives can do for random IO and is as good as it gets. With 4 threads it is 13.7ms – less than double of best possible, so also good enough… with concurrency of 64 it is however 108ms which is over 10x of what this device could produce with no waiting and which is surely sign of overload. A couple words of caution. First, do not look at svctm which is not designed with parallel processing in mind. You can see in our case it actually gets better with high concurrency while really database had to wait a lot longer for requests submitted. Second, iostat mixes together reads and writes in single statistics which specifically for databases and especially on RAID with BBU might be like mixing apples and oranges together – writes should go to writeback cache and be acknowledged essentially instantly while reads only complete when actual data can be delivered. The tool pt-diskstats from Percona Tookit breaks them apart and so can be much more for storage tuning for database workload. Some of the recent operating systems also ship with sysstat/iostat which breaks out await to r_await and w_await which is much more useful. Final note – I used a read-only workload on purpose – when it comes to writes things can be even more complicated – MySQL buffer pool can be more efficient with more intensive writes plus group commit might be able to commit a lot of transactions with single disk write. Still, the same base logic will apply. Summary: The take away is simple – util% only shows if a device has at least one operation to deal with or is completely busy, which does not reflect actual utilization for a majority of modern IO subsystems. So you may have a lot of storage IO capacity left even when utilization is close to 100%.

So you already know what Hadoop is? Why it is used? What problems you can solve with it?

Every SQL developer uses JOINs and aggregate functions, but some may not be aware that the two interacting can return incorrect results if not handled carefully. Brett Sauve at Looker has covered the issue in this recent post - specifically, he focuses on the issue of "fanouts" - and it's an easy problem to overlook. A fanout occurs when the primary table in a query contains fewer rows than the combined table resulting from a JOIN. Because the new table contains more rows, aggregate functions like COUNT or SUM may include duplicates, throwing off the results. According to Sauve, the answer is careful ordering of JOIN tables: Herein lies a key point: to help avoid fanouts, begin your joins with the most granular table. Sauve also provides some tips to help SQL developers understand how to avoid fanouts and ensure the validity of their data. He points to three types of JOINs: 1-to-1 Many-to-1 1-to-Many Knowing which JOIN you are using can rule out the possibility of a fanout, for example, because it is not possible in a 1-to-1 or Many-to-1 JOIN. If you are using a 1-to-Many JOIN, though, Sauve has some tips there, too. For example, a messy JOIN can be cleaned up by grouping data to create a 1-to-1 relationship. Check out the full article for the rest of Sauve's tips to make sure your aggregate functions are working correctly, and if you're looking for a few more tips to keep your SQL skills sharp, you might try one of these: Yet Another 10 Common Mistakes Java Developers Make When Writing SQL 6 Simple Performance Tips for SQL SELECT Statements

The expected behaviour for volatile fields is that they should behave in a multi-threaded application the same as they do in a single threaded application. They are not forbidden to behave the same way, but they are not guaranteed to behave the same way. The solution in Java 5.0+ is to use AtomicXxxx classes however these are relatively inefficient in terms of memory (they add a header and padding), performance (they add a references and little control over their relative positions), and syntactically they are not as clear to use. IMHO A simple solution if for volatile fields to act as they might be expected to do, the way JVM must support in AtomicFields which is not forbidden in the current JMM (Java- Memory Model) but not guaranteed. Why make fields volatile? The benefit of volatile fields is that they are visible across threads and some optimisations which avoid re-reading them are disabled so you always check again the current value even if you didn't change them. e.g. without volatile Thread 2: int a = 5; Thread 1: a = 6; (later) Thread 2: System.out.println(a); // prints 5 With volatile Thread 2: volatile int a = 5; Thread 1: a = 6; (later) Thread 2: System.out.println(a); // prints 6 Why not use volatile all the time? Volatile read and write access is substantially slower. When you write to a volatile field it stalls the entire CPU pipeline to ensure the data has been written to cache. Without this, there is a risk the next read of the value sees an old value, even in the same thread (See AtomicLong.lazySet() which avoids stalling the pipeline) The penalty can be in the order of 10x slower which you don't want to be doing on every access. What are the limitations of volatile? A significant limitation is that operations on the field is not atomic, even when you might think it is. Even worse than that is that usually, there is no difference. I.e. it can appear to work for a long time even years and suddenly/randomly break due to an incidental change such as the version of Java used, or even where the object is loaded into memory. e.g. which programs you loaded before running the program. e.g. updating a value Thread 2: volatile int a = 5; Thread 1: a += 1; Thread 2: a += 2; (later) Thread 2: System.out.println(a); // prints 6, 7 or 8 This is an issue because the read of a and the write of a are done separately and you can get a race condition. 99%+ of the time it will behave as expect, but sometimes it won't/ What can you do about it? You need to use AtomicXxxx classes. These wrap volatile fields with operations which behave as expected. Thread 2: AtomicInteger a = new AtomicInteger(5); Thread 1: a.incrementAndGet(); Thread 2: a.addAndGet(2); (later) Thread 2: System.out.println(a); // prints 8 What do I propose? The JVM has a means to behave as expected, the only surprising thing is you need to use a special class to do what the JMM won't guarantee for you. What I propose is that the JMM be changed to support the behaviour currently provided by the concurrency AtomicClasses. In each case the single threaded behaviour is unchanged. A multi-threaded program which does not see a race condition will behave the same. The difference is that a multi-threaded program does not have to see a race condition but changing the underlying behaviour current method suggested syntax notes x.getAndIncrement() x++ or x += 1 x.incrementAndGet() ++x x.getAndDecrment() x-- or x -= 1 x.decrementAndGet() --x x.addAndGet(y) (x += y) x.getAndAdd(y) ((x += y)-y) x.compareAndSet(e, y) (x == e ? x = y, true : false) Need to add the comma syntax used in other languages. These operations could be supported for all the primitive types such as boolean, byte, short, int, long, float and double. Additional assignment operators could be supported such as current method suggested syntax notes Atomic multiplication x *= 2; Atomic subtraction x -= y; Atomic division x /= y; Atomic modulus x %= y; Atomic shift x <<= y; Atomic shift x >>= z; Atomic shift x >>>= w; Atomic and x &= ~y; clears bits Atomic or x |= z; sets bits Atomic xor x ^= w; flips bits What is the risk? This could break code which relies on these operations occasionally failing due to race conditions. It might not be possible to support more complex expressions in a thread safe manner. This could lead to surprising bugs as the code can look like the works, but it doesn't. Never the less it will be no worse than the current state. JEP 193 - Enhanced Volatiles There is a JEP 193 to add this functionality to Java. An example is; class Usage { volatile int count; int incrementCount() { return count.volatile.incrementAndGet(); } } IMHO there is a few limitations in this approach. The syntax is fairly significant change. Changing the JMM might not require many changes the the Java syntax and possibly no changes to the compiler. It is a less general solution. It can be useful to support operations like volume += quantity; where these are double types. It places more burden on the developer to understand why he/she should use this instead of x++; Conclusion Much of the syntactic and performance overhead of using AtomicInteger and AtomicLong could be removed if the JMM guaranteed the equivalent single threaded operations behaved as expected for multi-threaded code. This feature could be added to earlier versions of Java by using byte code instrumentation.

We had a lengthy discussion recently during code review whether scala.Option.fold() is idiomatic and clever or maybe unreadable and tricky? Let's first describe what the problem is. Option.fold does two things: maps a function f overOption's value (if any) or returns an alternative alt if it's absent. Using simple pattern matching we can implement it as follows: val option: Option[T] = //... def alt: R = //... def f(in: T): R = //... val x: R = option match { case Some(v) => f(v) case None => alt } If you prefer one-liner, fold is actually a combination of map and getOrElse val x: R = option map f getOrElse alt Or, if you are a C programmer that still wants to write in C, but using Scala compiler: val x: R = if (option.isDefined) f(option.get) else alt Interestingly this is similar to how fold() is actually implemented, but that's an implementation detail. OK, all of the above can be replaced with single Option.fold(): val x: R = option.fold(alt)(f) Technically you can even use /: and \: operators (alt /: option) - but that would be simply masochistic. I have three problems with option.fold() idiom. First of all - it's anything but readable. We are folding (reducing) over Option - which doesn't really make much sense. Secondly it reverses the ordinary positive-then-negative-case flow by starting with failure (absence, alt) condition followed by presence block (f function; see also: Refactoring map-getOrElse to fold). Interestingly this method would work great for me if it was named mapOrElse: ** * Hypothetical in Option */ def mapOrElse[B](f: A => B, alt: => B): B = this map f getOrElse alt Actually there is already such method in Scalaz, called OptionW.cata. cata. Here is whatMartin Odersky has to say about it: "I personally find methods like cata that take two closures as arguments are often overdoing it. Do you really gain in readability over map + getOrElse? Think of a newcomer to your code[...]"While cata has some theoretical background, Option.fold just sounds like a random name collision that doesn't bring anything to the table, apart from confusion. I know what you'll say, that TraversableOnce has fold and we are sort-of doing the same thing. Why it's a random collision rather than extending the contract described inTraversableOnce? fold() method in Scala collections typically just delegates to one offoldLeft()/foldRight() (the one that works better for given data structure), thus it doesn't guarantee order and folding function has to be associative. But inOption.fold() the contract is different: folding function takes just one parameter rather than two. If you read my previous article about folds you know that reducing function always takes two parameters: current element and accumulated value (initial value during first iteration). But Option.fold() takes just one parameter: current Option value! This breaks the consistency, especially when realizing Option.foldLeft() andOption.foldRight() have correct contract (but it doesn't mean they are more readable). The only way to understand folding over option is to imagine Option as a sequence with0 or 1 elements. Then it sort of makes sense, right? No. def double(x: Int) = x * 2 Some(21).fold(-1)(double) //OK: 42 None.fold(-1)(double) //OK: -1 but: Some(21).toList.fold(-1)(double) : error: type mismatch; found : Int => Int required: (Int, Int) => Int Some(21).toList.fold(-1)(double) ^ If we treat Option[T] as a List[T], awkward Option.fold() breaks because it has different type than TraversableOnce.fold(). This is my biggest concern. I can't understand why folding wasn't defined in terms of the type system (trait?) and implemented strictly. As an example take a look at: Data.Foldable in Haskell (advanced) Data.Foldable typeclass describes various flavours of folding in Haskell. There are familiar foldl/foldr/foldl1/foldr1, in Scala namedfoldLeft/foldRight/reduceLeft/reduceRight accordingly. They have the same type as Scala and behave unsurprisingly with all types that you can fold over, including Maybe, lists, arrays, etc. There is also a function named fold, but it has a completely different meaning: class Foldable t where fold :: Monoid m => t m -> m While other folds are quite complex, this one barely takes a foldable container of ms (which have to be Monoids) and returns the same Monoid type. A quick recap: a type can be aMonoid if there exists a neutral value of that type and an operation that takes two values and produces just one. Applying that function with one of the arguments being neutral value yields the other argument. String ([Char]) is a good example with empty string being neutral value (mempty) and string concatenation being such operation (mappend). Notice that there are two different ways you can construct monoids for numbers: under addition with neutral value being 0 (x + 0 == 0 + x == x for any x) and under multiplication with neutral 1 (x * 1 == 1 * x == x for any x). Let's stick to strings. If I fold empty list of strings, I'll get an empty string. But when a list contains many elements, they are being concatenated: > fold ([] :: [String]) "" > fold [] :: String "" > fold ["foo", "bar"] "foobar" In the first example we have to explicitly say what is the type of empty list []. Otherwise Haskell compiler can't figure out what is the type of elements in a list, thus which monoid instance to choose. In second example we declare that whatever is returned from fold [], it should be a String. From that the compiler infers that [] actually must have a type of [String]. Last fold is the simplest: the program folds over elements in list and concatenates them because concatenation is the operation defined in Monoid Stringtypeclass instance. Back to options (or more precisely Maybe). Folding over Maybe monad having type parameter being Monoid (I can't believe I just said it) has an interesting interpretation: it either returns value inside Maybe or a default Monoid value: > fold (Just "abc") "abc" > fold Nothing :: String "" Just "abc" is same as Some("abc") in Scala. You can see here that if Maybe Stringis Nothing, neutral String monoid value is returned, that is an empty string. Summary Haskell shows that folding (also over Maybe) can be at least consistent. In ScalaOption.fold is unrelated to List.fold, confusing and unreadable. I advise avoiding it and staying with slightly more verbose map/getOrElse transformations or pattern matching. PS: Did I mention there is also Either.fold() (with even different contract) but noTry.fold()?

Fact: Batch Jobs are tricky to handle when exceptions raise. The problem is the huge amounts of data that these jobs are designed to take. If you’re processing 1 million records you simply can’t log everything. Logs would become huge and unreadable. Not to mention the performance toll it would take. On the other hand, if you log too little then it’s impossible to know what went wrong, and if 30 thousand records failed, not knowing what’s wrong with them can be a royal pain. This is a trade-off not simple to overcome. We took the feedback we got from the early releases around this issue, and now that Mule 3.5.0 and the Batch Module finally went GA with the May 2014 release, small but significant improvements took place to help you with these errors. Let’s take a look at some simple scenarios. Before we start This post assumes you’re familiar with the new Batch module included in Mule 3.5. If that’s not the case, here’re some links you might want to read before continuing: When ETL met the ESB: Introducing the Batch Module Batch Module Reloaded Batch Module documentation Records Failing in a Step So, let’s take a simple scenario. I have a simple step and records tend to fail there. This is the simplest manifestation of the trade-off we talked about at the beginning of this post. Logging a whole lot of unmanageable text or log so little that is barely useful. The strategy we took here is to not log exceptions but to log stack traces. Yeap, you read correctly. When we found an exception in a step we perform the following algorithm: Get the exception’s full stack trace Strip that stack trace from any messages. Even if all records always throw the same exception, those messages probably contain record specific information. E.g: ‘user Walter White is not eligible because he’s wanted by the DEA’. The user Jesse Pinkman would throw a slightly different message, but bottom line is that the exception is actually the same. Verify if this stack trace was already logged in the current step. If this is the first time that stack trace is found on the current step, then we log it adding a nice message saying that we will not log it again for the current step. Otherwise, if we already saw that stack trace in the given step, then we simply ignore it but keep count of how many times that exception was found Let’s see an example of how the WantedByDEAException would be logged: This is the first record to show this exception on this step for this job instance. Subsequent records with the same failureswill not be logged for performance and log readability reasons: ******************************************************************************** Message : null (org.mule.blog.WantedByDEAException). Message payload is of type: String Code : MULE_ERROR--2 -------------------------------------------------------------------------------- Exception stack is: 1. null (org.mule.blog.WantedByDEAException) org.mule.blog.FraudValidator:11 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/blog/WantedByDEAException.html) 2. null (org.mule.blog.WantedByDEAException). Message payload is of type: String (org.mule.api.MessagingException) org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor:32 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html) -------------------------------------------------------------------------------- Root Exception stack trace: org.mule.blog.WantedByDEAException at org.mule.blog.FraudValidator.process(FraudValidator.java:11) at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24) at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:58) + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything) ******************************************************************************** GOTCHA: This is done on a by step basis. If another step also throws the same exception, it WILL be logged once more for that step. Exception Summary The above is good. We get to see the exception without clogging the logs. However, because I only see the exception once, I don’t really know how many times it happened in total. For that we added an exception summary which is logged when the on-complete phase is reached. This summary shows for every exception type, how many times it happened in each step. Suppose that now we have a job with two steps, and both are capable of throwing BatchException. Then the exception summary would look like this: ************************************************************************************************** * - - + Exception Type + - - * - - + Step + - - * - - + Count + - - * ************************************************************************************************** * com.mulesoft.module.batch.exception.BatchException * Batch_Step * 10 * * com.mulesoft.module.batch.exception.BatchException * Second_Batch_Step * 9 * ************************************************************************************************** Here we can see that the first step failed 10 times and the second failed 9. Getting more out of the logs As this post starts saying, the motivation for all of this is the trade-off between large chunks of data and logging efficiency, that’s why all the items described above work with Mule’s default configuration of only logging INFO level messages (if you’re not familiar with logging levels, take a look at this) . There’re special cases like debugging or when you’re motivation to use batch is not the ability to handle large datasets but recoverability, resilience, management, etc (see this post for more examples). For those cases, you can get a whole lot of extra and extremely verbose information by setting the batch category to DEBUG level in your app’s log4j configuration like this: log4j.logger.com.mulesoft.module.batch=DEBUG CAUTION: Use the DEBUG level with care. Definitively not advised for production environments unless you’re absolutely sure that your jobs will not handle significantly large data sets. Error Handling MEL functions The early access release of the Batch module already shipped with some error handling MEL functions that can be used in the context of a batch step. Let’s refresh that real quick: #[isSuccessfulRecord()]: Boolean functions which returns true if the current record has not thrown exceptions in any prior step. #[isFailedRecord()]: Boolean functions which returns true if the current record has thrown exceptions in any prior step. #[failureExceptionForStep(String)]: This function receives the name of a step as a String argument. If the current record threw exception on that step, then it returns the actual Exception object. Otherwise it returns null Again, the above is just fine. However, we quickly learn that those felt short in some cases. Let’s consider the following scenario: A batch job that polls files containing contact information. Those contacts are transformer using DataMapper and fed into the job The first step transforms them again and inserts them into Salesforce The second step transforms again and inserts them into Google Contacts So far so good. The challenging new requirement is to be able to write into a JMS dead-letter queue per each record that has failed. To keep it simple, let’s just say that the message will be the exception itself. In case you haven’t noticed, there’s a gotcha in this requirement: each record could have failed in both steps, which means that the same record would translate into two JMS messages. To support this and other similar scenarios, we added these additional functions which also require to be executed in the context of a batch step: #[getStepExceptions()]: Returns a java Map in which the keys are the name of a batch step in which the current record has failed and the value is the exception itself. If the record hasn’t failed in any step, this Map will be empty but will never be null. Also, the Map contains no entries for steps in which the record hasn’t failed. #[getFirstException()]: Returns the Exception for the very first step in which the current record has failed. If the record hasn’t failed in any step, then it returns null. #[getLastException()]: Returns the Exception for the last step in which the current record has failed. If the record hasn’t failed in any step, then it returns null. So let’s take a look at how this application would look like: As you see, a pretty standard Batch Job. Because this is an error handling post, I won’t go into detail of the whole app and I’ll just focus on the last step. Let’s take a closer look: Since the goal of this step is to gather failures, it makes sense to skip the ones who haven’t got any, thus we set the accept policy to failures only. The first thing we do is to use the set-payload processor and the getStepExceptions() MEL function together so that the message payload is set to the exceptions map. Remember how we said before that a given record could have many exceptions? That’s the nice thing about this map, it gives you access to the exceptions found in all steps. Since our goal is to send the exceptions through JMS and we don’t care about the steps, we can simply use a foreach to iterate over the map’s values (the exceptions) and send them through a JMS outbound endpoint: And we’re done! We have happily aggregated exceptions! Summary The above features were introduced in the GA release of Mule 3.5.0. We tried our best to help you find equilibrium between useful and meaningful messages without generating undesired verbosity. Have better ideas or suggestions? Please reach out!

We have published the results of the Eclipse Community Survey 2014. Thank you to everyone who participated in the survey this year. The complete results and data are available for anyone to download [xls] [ods]. As in other years, I think the results provide an interesting perspective on what tools software developers are using and the type of applications they are building. Here are some key highlights from the results this year: 1) Git #1 Code Management Tool. Git has finally surpassed Subversion to be the top code management tool used by software developers. A third of developers (33.3%) report they use Git as their primary code management tool compared to 30.7% using Subversion. Subversion continues to show a downward trend from previous years when it was used by more than half the developers. Of note, 9.6% claim GitHub is their primary code management tool so the prevalence of overall Git usage is becoming dominate. 2) Maven and Jenkins Key Tools. For Build and Release tools, Maven and Jenkins continue to be key tools used by developers. Of interest is the growth of Gradle from 2013 (4.5%) to 2014 (11%). 3) Top 3 Application Servers. Tomcat (32.6%), JBoss (11.8%) and Jetty (7.2%) continue to be the top 3 application servers. 4) Java 8 Adoption. Java 8 was released in March 2014 and already 9.2% of Java developers have migrated to Java 8 as their primary version of Java. 59.2% are using Java 7 but close to a quarter are using Java 6 or before. 5) Majority of Developers Use JavaScript. More and more software developers use multiple languages to develop software. Due to the Eclipse biased of the survey, Java is not surprisingly the top primary language. However, when asked what other languages developers might use, JavaScript stands out to be a popular language with a the majority of developers (56.2%) claiming it as a secondary language. 6) Developers Experimenting With Open Hardware. The Internet of Things (IoT) and Open Hardware have become important industry trends in the last couple of years. Over a third (35.7%) of software developers are spending their own personal time learning about devices like the BeagleBone, Arduino and Raspberry Pi. Thanks again to everyone who participated in the survey. I hope everyone finds the results of interest.

NOTE: Always Agile Consulting now offers an Introduction To Trunk Based Development training course! Trunk Based Development supports Optimistic and Pessimistic Release Branching Trunk Based Development is a style of software development in which all developers commit their changes to a single shared trunk in source control, and every commit yields a production-ready build. It is a prerequisite for Continuous Delivery as it ensures that all code is continuously integrated into a single workstream, that developers always work against the latest code, and that merge/integration pain is minimised. It is important to note that Trunk Based Development does not prohibit branching. Trunk Based Development is compatible with a Release Branching strategy of short-lived release branches that are used for post-development defect fixes. That Release Branching strategy might be optimistic and defer branch creation until a defect occurs, or be pessimistic and immediately incur branch creation. For example, consider an application developed using Trunk Based Development. The most recent commits to trunk were source revisions a and b which yielded application versions 610 and 611 respectively, and version 610 is intended to be the next production release. With Optimistic Branching, the release of version 610 is immediate as there is no upfront branching. If a defect is subsequently found then a decision must be made where to commit the fix, as trunk has progressed since 610 from a to b. If the risk of pulling forward from a to b is acceptable then the simple solution is to commit the fix to trunk as c, and consequently release version 612. However, if the risk of pulling forward from a to b is unacceptable then a 610.x release branch is created from a, with the fix committed to the branch as c and released as version 610.1. That fix is then merged back into trunk as d to produce the next release candidate 612, and the 610.x branch is earmarked for termination. With Pessimistic Branching, the release of version 610 is accompanied by the upfront creation of a 610.x release branch in anticipation of defect(s). If a defect is found in version 610 then as with Optimistic Branching a decision must be made as to where the defect fix should be committed. If the risk of pulling forward from a to b is deemed insignificant then trunk can be pulled forward from a to b and the fix committed to trunk as c for release as version 612. The 610.x branch is therefore terminated without ever being used. If on the other hand the risk is deemed significant then the fix is committed to the 610.x branch as c and released as version 610.1. The fix is merged back into trunk as d and version 612, which will also receive its own branch upon release. The choice between Optimistic Branching and Pessimistic Branching for Trunk Based Development is dependent upon product quality and lead times. If product quality is poor and lead times are long, then the upfront cost of Pessimistic Branching may be justifiable. Alternatively, if post-development defects are rare and production releases are frequent then Optimistic Branching may be preferable.

if you've been following the blog, you will have seen a couple of posts recently about the alignment of software architecture and code. software architecture vs code talks about the typical gap between how we think about the software architecture vs the code that we write, while an architecturally-evident coding style shows an example of how to ensure that the code does reflect those architectural concepts. the basic summary of the story so far is that things get much easier to understand if your architectural ideas map simply and explicitly into the code. regular readers will also know that i'm a big fan of using diagrams to visualise and communicate the architecture of a software system, and this "big picture" view of the world is often hard to see from the thousands of lines of code that make up our software systems. one of the things that i teach people during my sketching workshops is how to sketch out a software system using a small number of simple diagrams, each at very separate levels of abstraction. this is based upon my c4 model , which you can find an introduction to at simple sketches for diagramming your software architecture . the feedback from people using this model has been great, and many have a follow-up question of "what tooling would you recommend?". my answer has typically been "visio or omnigraffle", but it's obvious that there's an opportunity here. representing the software architecture model in code i've had a lot of different ideas over the past few months for how to create, what is essentially, a lightweight modelling tool and for some reason, all of these ideas came together last week while i was at the goto amsterdam conference. i'm not sure why, but i had a number of conversations that inspired me in different ways, so i skipped one of the talks to throw some code together and test out some ideas. this is basically what i came up with... model model = new model(); softwaresystem techtribes = model.addsoftwaresystem(location.internal, "techtribes.je", "techtribes.je is the only way to keep up to date with the it, tech and digital sector in jersey and guernsey, channel islands"); person anonymoususer = model.addperson(location.external, "anonymous user", "anybody on the web."); person aggregateduser = model.addperson(location.external, "aggregated user", "a user or business with content that is aggregated into the website."); person adminuser = model.addperson(location.external, "administration user", "a system administration user."); anonymoususer.uses(techtribes, "view people, tribes (businesses, communities and interest groups), content, events, jobs, etc from the local tech, digital and it sector."); aggregateduser.uses(techtribes, "manage user profile and tribe membership."); adminuser.uses(techtribes, "add people, add tribes and manage tribe membership."); softwaresystem twitter = model.addsoftwaresystem(location.external, "twitter", "twitter.com"); techtribes.uses(twitter, "gets profile information and tweets from."); softwaresystem github = model.addsoftwaresystem(location.external, "github", "github.com"); techtribes.uses(github, "gets information about public code repositories from."); softwaresystem blogs = model.addsoftwaresystem(location.external, "blogs", "rss and atom feeds"); techtribes.uses(blogs, "gets content using rss and atom feeds from."); container webapplication = techtribes.addcontainer("web application", "allows users to view people, tribes, content, events, jobs, etc from the local tech, digital and it sector.", "apache tomcat 7.x"); container contentupdater = techtribes.addcontainer("content updater", "updates profiles, tweets, github repos and content on a scheduled basis.", "standalone java 7 application"); container relationaldatabase = techtribes.addcontainer("relational database", "stores people, tribes, tribe membership, talks, events, jobs, badges, github repos, etc.", "mysql 5.5.x"); container nosqlstore = techtribes.addcontainer("nosql data store", "stores content from rss/atom feeds (blog posts) and tweets.", "mongodb 2.2.x"); container filesystem = techtribes.addcontainer("file system", "stores search indexes.", null); anonymoususer.uses(webapplication, "view people, tribes (businesses, communities and interest groups), content, events, jobs, etc from the local tech, digital and it sector."); authenticateduser.uses(webapplication, "manage user profile and tribe membership."); adminuser.uses(webapplication, "add people, add tribes and manage tribe membership."); webapplication.uses(relationaldatabase, "reads from and writes data to"); webapplication.uses(nosqlstore, "reads from"); webapplication.uses(filesystem, "reads from"); contentupdater.uses(relationaldatabase, "reads from and writes data to"); contentupdater.uses(nosqlstore, "reads from and writes data to"); contentupdater.uses(filesystem, "writes to"); contentupdater.uses(twitter, "gets profile information and tweets from."); contentupdater.uses(github, "gets information about public code repositories from."); contentupdater.uses(blogs, "gets content using rss and atom feeds from."); it's a description of the context and container levels of my c4 model for the techtribes.je system. hopefully it doesn't need too much explanation if you're familiar with the model, although there are some ways in which the code can be made simpler and more fluent. since this is code though, we can easily constrain the model and version it. this approach works well for the high-level architectural concepts because there are very few of them, plus it's hard to extract this information from the code. but i don't want to start crafting up a large amount of code to describe the components that reside in each container, particularly as there are potentially lots of them and i'm unsure of the exact relationships between them. scanning the codebase for components if your code does reflect your architecture (i.e. you're using an architecturally-evident coding style), the obvious solution is to just scan the codebase for those components, and use those to automatically populate the model. how do we signify what a "component" is? in java, we can use annotations... package je.techtribes.component.tweet; import com.structurizr.annotation.component; ... @component(description = "provides access to tweets.") public interface tweetcomponent { /** * gets the most recent tweets by page number. */ list getrecenttweets(int page, int pagesize); ... } identifying those components is then a matter of scanning the source or the compiled bytecode. i've played around with this idea on and off for a few months, using a combination of java annotations along with annotation processors and libraries including scannotation, javassist and jdepend. the reflections library on google code makes this easy to do, and now i have simple java program that looks for my component annotation on classes in the classpath and automatically adds those to the model. as for the dependencies between components, again this is fairly straightforward to do with reflections. i have a bunch of other annotations too, for example to represent dependencies between a component and a container or software system, but the principle is still the same - the architecturally significant elements and their dependencies can mostly be embedded in the code. creating some views the model itself is useful, but ideally i want to look at that model from different angles, much like the diagrams that i teach people to draw when they attend my sketching workshop. after a little thought about what this means and what each view is constrained to show, i created a simple domain model to represent the context, container and component views... model model = ... softwaresystem techtribes = model.getsoftwaresystemwithname("techtribes.je"); container contentupdater = techtribes.getcontainerwithname("content updater"); // context view contextview contextview = model.createcontextview(techtribes); contextview.addallsoftwaresystems(); contextview.addallpeople(); // container view containerview containerview = model.createcontainerview(techtribes); containerview.addallsoftwaresystems(); containerview.addallpeople(); containerview.addallcontainers(); // component view for the content updater container componentview componentview = model.createcomponentview(techtribes, contentupdater); componentview.addallsoftwaresystems(); componentview.addallcontainers(); componentview.addallcomponents(); // let's exclude the logging component as it's used by everything componentview.remove(contentupdater.getcomponentwithname("loggingcomponent")); componentview.removeelementswithnorelationships(); again, this is all in code so it's quick to create, versionable and very customisable. exporting the model now that i have a model of my software system and a number of views that i'd like to see, i could do with drawing some pictures. i could create a diagramming tool in java that reads the model directly, but perhaps a better approach is to serialize the object model out to an external format so that other tools can use it. and that's what i did, courtesy of the jackson library . the resulting json file is over 600 lines long ( you can see it here ), but don't forget most of this has been generated automatically by java code scanning for components and their dependencies. visualising the views the last question is how to visualise the information contained in the model and there are a number of ways to do this. i'd really like somebody to build a google maps or prezi-style diagramming tool where you can pinch-zoom in and out to see different views of the model, but my ui skills leave something to be desired in that area. for the meantime, i've thrown together a simple diagramming tool using html 5, css and javascript that takes a json string and visualises the views contained within it. my vision here is to create a lightweight model visualisation tool rather than a visio clone where you have to draw everything yourself. i've deployed this app on pivotal web services and you can try it for yourself . you'll have to drag the boxes around to lay out the elements and it's not very pretty, but the concept works. the screenshot that follows shows the techtribes.je context diagram. thoughts? all of the c4 model java code is open source and sitting on github . this is only a few hours of work so far and there are no tests, so think of this as a prototype more than anything else at the moment. i really like the simplicity of capturing a software architecture model in code, and using an architecturally-evident coding style allows you to create large chunks of that model automatically. this also opens up the door to some other opportunities such automated build plugins, lightweight documentation tooling, etc. caveats apply with the applicability of this to all software systems, but i'm excited at the possibilities. thoughts?

while writing my spring-hibernate integration post i realize that we (as java developers) have so many frameworks available which can make life easy by rapidly developing so many common things using frameworks. this ease of development is not available in java particularly when you considering ui development, we still have to struggle with old, bad looking jsp with combination of css, jquery. thank god we now have so many javascript toolkits, framework that make developers' lives easy by offering so many ready-to-use widgets. extjs is the most advanced among those client side ui frameworks. today i am going to demonstrate to you how you can leverage extjs 4 to create crud application. in the next post i will try to use the same js code with spring mvc as a backend . create a html page which include extjs library and aur books.js file. here is the extjs crud code, i have combined it in a single file. you can follow the extjs 4 mvc folder structure. ext.onready(function () { ext.define('techzoo.model.book', { extend: 'ext.data.model', fields: [ {name: 'title', type: 'string'}, {name: 'author', type: 'string'}, {name: 'price', type: 'int'}, {name: 'qty', type: 'int'} ] }); ext.define('techzoo.store.books', { extend : 'ext.data.store', model : 'techzoo.model.book', fields : ['title', 'author','price', 'qty'], data : [ { title: 'jdbc, servlet and jsp', author: 'santosh kumar', price: 300, qty : 12000 }, { title: 'head first java', author: 'kathy sierra', price: 550, qty : 2500 }, { title: 'java scjp certification', author: 'khalid mughal', price: 650, qty : 5500 }, { title: 'spring and hinernate', author: 'santosh kumar', price: 350, qty : 2500 }, { title: 'mastering c++', author: 'k. r. venugopal', price: 400, qty : 1200 } ] }); ext.define('techzoo.view.bookslist', { extend: 'ext.grid.panel', alias: 'widget.bookslist', title: 'books list - (extjs 4 crud example - @ tousif khan)', store: 'books', initcomponent: function () { this.tbar = [{ text : 'add book', action : 'add', iconcls : 'book-add' }]; this.columns = [ { header: 'title', dataindex: 'title', flex: 1 }, { header: 'author', dataindex: 'author' }, { header: 'price', dataindex: 'price' , width: 60 }, { header: 'quantity', dataindex: 'qty', width: 80 }, { header: 'action', width: 50, renderer: function (v, m, r) { var id = ext.id(); var max = 15; ext.defer(function () { ext.widget('image', { renderto: id, name: 'delete', src : 'images/book_delete.png', listeners : { afterrender: function (me) { me.getel().on('click', function() { var grid = ext.componentquery.query('bookslist')[0]; if (grid) { var sm = grid.getselectionmodel(); var rs = sm.getselection(); if (!rs.length) { ext.msg.alert('info', 'no book selected'); return; } ext.msg.confirm('remove book', 'are you sure you want to delete?', function (button) { if (button == 'yes') { grid.store.remove(rs[0]); } }); } }); } } }); }, 50); return ext.string.format('', id); } } ]; this.callparent(arguments); } }); ext.define('techzoo.view.booksform', { extend : 'ext.window.window', alias : 'widget.booksform', title : 'add book', width : 350, layout : 'fit', resizable: false, closeaction: 'hide', modal : true, config : { recordindex : 0, action : '' }, items : [{ xtype : 'form', layout: 'anchor', bodystyle: { background: 'none', padding: '10px', border: '0' }, defaults: { xtype : 'textfield', anchor: '100%' }, items : [{ name : 'title', fieldlabel: 'book title' },{ name: 'author', fieldlabel: 'author name' },{ name: 'price', fieldlabel: 'price' },{ name: 'qty', fieldlabel: 'quantity' }] }], buttons: [{ text: 'ok', action: 'add' },{ text : 'reset', handler : function () { this.up('window').down('form').getform().reset(); } },{ text : 'cancel', handler: function () { this.up('window').close(); } }] }); ext.define('techzoo.controller.books', { extend : 'ext.app.controller', stores : ['books'], views : ['bookslist', 'booksform'], refs : [{ ref : 'formwindow', xtype : 'booksform', selector: 'booksform', autocreate: true }], init: function () { this.control({ 'bookslist > toolbar > button[action=add]': { click: this.showaddform }, 'bookslist': { itemdblclick: this.onrowdblclick }, 'booksform button[action=add]': { click: this.doaddbook } }); }, onrowdblclick: function(me, record, item, index) { var win = this.getformwindow(); win.settitle('edit book'); win.setaction('edit'); win.setrecordindex(index); win.down('form').getform().setvalues(record.getdata()); win.show(); }, showaddform: function () { var win = this.getformwindow(); win.settitle('add book'); win.setaction('add'); win.down('form').getform().reset(); win.show(); }, doaddbook: function () { var win = this.getformwindow(); var store = this.getbooksstore(); var values = win.down('form').getvalues(); var action = win.getaction(); var book = ext.create('techzoo.model.book', values); if(action == 'edit') { store.removeat(win.getrecordindex()); store.insert(win.getrecordindex(), book); } else { store.add(book); } win.close(); } }); ext.application({ name : 'techzoo', controllers: ['books'], launch: function () { ext.widget('bookslist', { width : 500, height: 300, renderto: 'output' }); } } ); }); output: open the html file in any browser, the output will look similar to below. you can click to add new book record in grid. double click to any of the grid row to edit the record. view live demo

sqs, simple message queue , is a message-queue-as-a-service offering from amazon web services. it supports only a handful of messaging operations, far from the complexity of e.g. amqp , but thanks to the easy to understand interfaces, and the as-a-service nature, it is very useful in a number of situations. but how fast is sqs? how does it scale? is it useful only for low-volume messaging, or can it be used for high-load applications as well? if you know how sqs works, and want to skip the details on the testing methodology, you can jump straight to the test results . sqs semantics sqs exposes an http-based interface. to access it, you need aws credentials to sign the requests. but that’s usually done by a client library (there are libraries for most popular languages; we’ll use the official java sdk ). the basic message-related operations are: send a message, up to 256 kb in size, encoded as a string. messages can be sent in bulks of up to 10 (but the total size is capped at 256 kb). receive a message. up to 10 messages can be received in bulk, if available in the queue long-polling of messages. the request will wait up to 20 seconds for messages, if none are available initially delete a message there are also some other operations, concerning security, delaying message delivery, and changing a messages’ visibility timeout, but we won’t use them in the tests. sqs offers at-least-once delivery guarantee. if a message is received, then it is blocked for a period called “visibility timeout”. unless the message is deleted within that period, it will become available for delivery again. hence if a node processing a message crashes, it will be delivered again. however, we also run into the risk of processing a message twice (if e.g. the network connection when deleting the message dies, or if an sqs server dies), which we have to manage on the application side. sqs is a replicated message queue, so you can be sure that once a message is sent, it is safe and will be delivered; quoting from the website: amazon sqs runs within amazon’s high-availability data centers, so queues will be available whenever applications need them. to prevent messages from being lost or becoming unavailable, all messages are stored redundantly across multiple servers and data centers. testing methodology to test how fast sqs is and how it scales, we will be running various numbers of nodes, each running various number of threads either sending or receiving simple, 100-byte messages. each sending node is parametrised with the number of messages to send, and it tries to do so as fast as possible. messages are sent in bulk, with bulk sizes chosen randomly between 1 and 10. message sends are synchronous, that is we want to be sure that the request completed successfully before sending the next bulk. at the end the node reports the average number of messages per second that were sent. the receiving node receives messages in maximum bulks of 10. the amazonsqsbufferedasyncclient is used, which pre-fetches messages to speed up delivery. after receiving, each message is asynchronously deleted. the node assumes that testing is complete once it didn’t receive any messages within a minute, and reports the average number of messages per second that it received. each test sends from 10 000 to 50 000 messages per thread. so the tests are relatively short, 2-5 minutes. there are also longer tests, which last about 15 minutes. the full (but still short) code is here: sender , receiver , sqsmq . one set of nodes runs the mqsender code, the other runs the mqreceiver code. the sending and receiving nodes are m3.large ec2 servers in the eu-west region, hence with the following parameters: 2 cores intel xeon e5-2670 v2 7.5 gb rams the queue is of course also created in the eu-west region. minimal setup the minimal setup consists of 1 sending node and 1 receiving node, both running a single thread. the results are, in messages/second: average min max sender 429 365 466 receiver 427 363 463 scaling threads how do these results scale when we add more threads (still using one sender and one receiver node)? the tests were run with 1, 5, 25, 50 and 75 threads. the numbers are an average msg/second throughput. number of threads: 1 5 25 50 75 sender per thread 429,33 407,35 354,15 289,88 193,71 sender total 429,33 2 036,76 8 853,75 14 493,83 14 528,25 receiver per thread 427,86 381,55 166,38 83,92 47,46 receiver total 427,86 1 907,76 4 159,50 4 196,17 3 559,50 as you can see, on the sender side, we get near-to-linear scalability as the number of thread increases, peaking at 14k msgs/second sent (on a single node!) with 50 threads. going any further doesn’t seem to make a difference. the receiving side is slower, and that is kind of expected, as receiving a single message is in fact two operations: receive + delete, while sending is a single operation. the scalability is worse, but still we can get as much as 4k msgs/second received. scaling nodes another (more promising) method of scaling is adding nodes, which is quite easy as we are “in the cloud”. the test results when running multiple nodes, each running a single thread are: number of nodes: 1 2 4 8 sender per node 429,33 370,36 350,30 337,84 sender total 429,33 740,71 1 401,19 2 702,75 receiver per node 427,86 360,60 329,54 306,40 receiver total 427,86 721,19 1 318,15 2 451,23 in this case, both on the sending&receiving side, we get near-linear scalability, reaching 2.5k messages sent&received per second with 8 nodes. scaling nodes and threads the natural next step is, of course, to scale up both the nodes, and the threads! here are the results, when using 25 threads on each node: number of nodes: 1 2 4 8 sender per node&thread 354,15 338,52 305,03 317,33 sender total 8 853,75 16 925,83 30 503,33 63 466,00 receiver per node&thread 166,38 159,13 170,09 174,26 receiver total 4 159,50 7 956,33 17 008,67 34 851,33 again, we get great scalability results, with the number of receive operations about half the number of send operations per second. 34k msgs/second processed is a very nice number! to the extreme the highest results i managed to get are: 108k msgs/second sent when using 50 threads and 8 nodes 35k msgs/second received when using 25 threads and 8 nodes i also tried running longer “stress” tests with 200k messages/thread, 8 nodes and 25 threads, and the results were the same as with the shorter tests. running the tests – technically to run the tests, i built docker images containing the sender / receiver binaries, pushed to docker’s hub, and downloaded on the nodes by chef. to provision the servers, i used amazon opsworks. this enabled me to quickly spin up and provision a lot of nodes for testing (up to 16 in the above tests). for details on how this works, see my “cluster-wide java/scala application deployments with docker, chef and amazon opsworks” blog . the sender / receiver daemons monitored (by checking each second the last-modification date) a file on s3. if a modification was detected, the file was downloaded – it contained the test parameters – and the test started. summing up sqs has good performance and really great scalability characteristics. i wasn’t able to reach the peak of its possibilities – which would probably require more than 16 nodes in total. but once your requirements get above 35k messages per second, chances are you need custom solutions anyway; not to mention that while sqs is cheap, it may become expensive with such loads. from the results above, i think it is clear that sqs can be safely used for high-volume messaging applications, and scaled on-demand. together with its reliability guarantees, it is a great fit both for small and large applications, which do any kind of asynchronous processing; especially if your service already resides in the amazon cloud. as benchmarking isn’t easy, any remarks on the testing methodology, ideas how to improve the testing code are welcome!

When discussing transformations in regression models, I usually briefly introduce the Box-Cox transform (see e.g. an old post on that topic) and I also mention local regressions and nonparametric estimators (see e.g. another post). But while I was working on my ACT6420 course (on predictive modeling, which is a VEE for the SOA), I read something about a “Ladder of Powers Rule” also called “Tukey and Mosteller’s Bulging Rule“. To be honest, I never heard about this rule before. But that won’t be the first time I learn something while working on my notes for a course ! The point here is that, in a standard linear regression model, we have But sometimes, a linear relationship is not appropriate. One idea can be to transform the variable we would like to model, , and to consider This is what we usually do with the Box-Cox transform. Another idea can be to transform the explanatory variable, , and now, consider, For instance, this year in the course, we considered – at some point – a continuous piecewise linear functions, It is also possible to consider some polynomial regression. The ”Tukey and Mosteller’s Bulging Rule” is based on the following figure. and the idea is that it might be interesting to transform and at the same time, using some power functions. To be more specific, we will consider some linear model for some (positive) parameters and . Depending on the shape of the regression function (the four curves mentioned on the graph above, in the four quadrant) different powers will be considered. To be more specific, let us generate different models, and let us look at the associate scatterplot, > fakedataMT=function(p=1,q=1,n=99,s=.1){ + set.seed(1) + X=seq(1/(n+1),1-1/(n+1),length=n) + Y=(5+2*X^p+rnorm(n,sd=s))^(1/q) + return(data.frame(x=X,y=Y))} > par(mfrow=c(2,2)) > plot(fakedataMT(p=.5,q=2),main="(p=1/2,q=2)") > plot(fakedataMT(p=3,q=-5),main="(p=3,q=-5)") > plot(fakedataMT(p=.5,q=-1),main="(p=1/2,q=-1)") > plot(fakedataMT(p=3,q=5),main="(p=3,q=5)") If we consider the South-West part of the graph, to get such a pattern, we can consider or more generally where and are both larger than 1. And the larger and/or , the more convex the regression curve. Let us visualize that double transformation on a dataset, say the cars dataset. > base=cars > names(base)=c("x","y") > MostellerTukey=function(p=1,q=1){ + regpq=lm(I(y^q)~I(x^p),data=base) + u=seq(min(min(base$x)-2,.1),max(base$x)+2,length=501) + par(mfrow=c(1,2)) + plot(base$x,base$y,xlab="X",ylab="Y",col="white") + vic=predict(regpq,newdata=data.frame(x=u),interval="prediction") + vic[vic<=0]=.1 + polygon(c(u,rev(u)),c(vic[,2],rev(vic[,3]))^(1/q),col="light blue",density=40,border=NA) + lines(u,vic[,2]^(1/q),col="blue") + lines(u,vic[,3]^(1/q),col="blue") + v=predict(regpq,newdata=data.frame(x=u))^(1/q) + lines(u,v,col="blue") + points(base$x,base$y) + + plot(base$x^p,base$y^q,xlab=paste("X^",p,sep=""),ylab=paste("Y^",q,sep=""),col="white") + polygon(c(u,rev(u))^p,c(vic[,2],rev(vic[,3])),col="light blue",density=40,border=NA) + lines(u^p,vic[,2],col="blue") + lines(u^p,vic[,3],col="blue") + abline(regpq,col="blue") + points(base$x^p,base$y^q) + } For instance, if we call > MostellerTukey(2,1) we get the following graph, On the left, we have the original dataset, and on the right, the transformed one, , with two possible transformations. Here, we did only consider the square of the speed of the car (and only one component was transformed, here). On that transformed dataset, we run a standard linear regression. We add, here, a confidence tube. And then, we consider the inverse transformation of the prediction. This line is plotted on the left. The problem is that it should not be considered as our optimal prediction, since it is clearly biased because Note that here, it could have be possible to consider another transformation, with the same shape, but quite different > MostellerTukey(1,.5) Of course, there is no reason to consider a simple power function, and the Box-Cox transform can also be used. The interesting point is that the logarithm can be obtained as a particular case. Furthermore, it is also possible to seek optimal transformations, seen here as a pair of parameters. Consider > p=.1 > bc=boxcox(y~I(x^p),data=base,lambda=seq(.1,3,by=.1))$y > for(p in seq(.2,3,by=.1)) bc=cbind(bc,boxcox(y~I(x^p),data=base,lambda=seq(.1,3,by=.1))$y) > vp=boxcox(y~I(x^p),data=base,lambda=seq(.1,3,by=.1))$x > vq=seq(.1,3,by=.1) > library(RColorBrewer) > blues=colorRampPalette(brewer.pal(9,"Blues"))(100) > image(vp,vq,bc,col=blues) > contour(vp,vq,bc,levels=seq(-60,-40,by=1),col="white",add=TRUE) The darker, the better (here the log-likelihood is considered). The optimal pair is here > bc=function(a){p=a[1];q=a[2]; as.numeric(-boxcox(y~I(x^p),data=base,lambda=q)$y[50])} > optim(c(1,1), bc,method="L-BFGS-B",lower=c(0,0),upper=c(3,3)) $par [1] 0.5758362 0.3541601 $value [1] 47.27395 and indeed, the model we get is not bad, Fun, isn’t it?

In this post we will see how we can write Javadoc comments using Markdown instead of the typical Javadoc syntax. So what is Markdown? Markdown is a plain text formatting syntax designed so that it optionally can be converted to HTML using a tool by the same name. Markdown is popularly used to format readme files, for writing messages in online discussion forums or in text editors for the quick creation of rich text documents. (Wikipedia: Markdown) Markdown is a very easy to read formatting syntax. Different variations of Markdown can be used on Stack Overflow or GitHub to format user generated content. Setup By default the Javadoc tool uses Javadoc comments to generate API documentation in HTML form. This process can be customized used Doclets. Doclets are Java programs that specify the content and format of the output of the Javadoc tool. The markdown-doclet is a replacement for the standard Java Doclet which gives developers the option to use Markdown syntax in their Javadoc comments. We can set up this doclet in Maven using the maven-javadoc-plugin. maven-javadoc-plugin 2.9 ch.raffael.doclets.pegdown.PegdownDoclet ch.raffael.pegdown-doclet pegdown-doclet 1.1 true Writing comments in Markdown Now we can use Markdown syntax in Javadoc comments: /** * ## Large headline * ### Smaller headline * * This is a comment that contains `code` parts. * * Code blocks: * * ```java * int foo = 42; * System.out.println(foo); * ``` * * Quote blocks: * * > This is a block quote * * lists: * * - first item * - second item * - third item * * This is a text that contains an [external link][link]. * * [link]: http://external-link.com/ * * @param id the user id * @return the user object with the passed `id` or `null` if no user with this `id` is found */ public User findUser(long id) { ... } After running mvn javadoc:javadoc we can find the generated HTML API documentation in target/site/apidocs. The generated documentation for the method shown above looks like this: As we can see the Javadoc comments get nicely converted to HTML. Conclusion Markdown has the clear advantage over standard Javadoc syntax that the source it is far easier to read. Just have a look at some of the method comments of java.util.Map. Many Javadoc comments are full with formatting tags and are barely readable without any tool. But be aware that Markdown can cause problems with tools and IDEs that expect standard Javadoc syntax.

How to use the GNU linker to fill unused FLASH memory with a specific pattern for integrity and reliability.